A method for generating multi-subject customized videos

Abstract

A method for generating customized multi-subject videos according to the present invention includes five steps: S1: Data acquisition and preprocessing of the original video, video slicing to obtain a temporally coherent frame sequence, and generating a text description matching the video content through a text description model; S2: Semantic parsing of the text description to extract multiple subject nouns from the text; S3: Constructing a multi-subject feature interaction network to output enhanced features that integrate multi-subject association information; S4: In the decoding stage of the video generation model, the value matrix uses the enhanced features to integrate multi-subject association information into the video generation process; S5: Based on the multi-subject association information, noise is gradually removed through the diffusion process of the video generation model to output a customized video sequence that maintains spatiotemporal consistency of multiple subjects. The method for generating customized multi-subject videos according to the present invention can solve the problems of spatiotemporal inconsistency such as subject position drift, action logic breakage, or abrupt changes in appearance attributes.

Description

Technical Field

[0001] This invention relates to the intersection of artificial intelligence and computer vision, specifically to a method for generating multi-subject customized videos. Background Technology

[0002] With the development of deep learning technology, video generation has evolved from single-frame image synthesis to temporally coherent content generation. Early methods were based on Generative Adversarial Networks (GANs) to build inter-frame generation models, but they were limited by the mode collapse problem, making it difficult to generate high-quality long-term videos. The rise of diffusion models has significantly improved generation quality through a stepwise denoising process, but their implicit modeling characteristics result in insufficient control over multiple subjects. Existing work mainly guides generation through text prompts, but the ambiguity of text descriptions can easily lead to subject confusion, especially in multi-object scenes, where the model struggles to distinguish subject features from interaction relationships.

[0003] For the multi-agent generation problem, current mainstream solutions employ additional control signals (such as pose maps and edge detection maps) to constrain the generation process. However, these methods require manual annotation and have limited generalization capabilities. Some studies have attempted to guide generation using object detection boxes, but simple location annotations cannot encode the semantic attributes and dynamic behaviors of the agents. In terms of feature fusion, traditional methods simply concatenate text and visual features, lacking explicit modeling of the relationships between multiple agents, resulting in agent interactions in the generated results that do not conform to physical laws.

[0004] Existing datasets often focus on single-agent scenes, lacking systematic annotation of multi-agent interactions. Traditional video segmentation methods directly sample keyframes, disrupting temporal continuity and resulting in action breaks in the training data. In terms of model architecture design, mainstream video diffusion models follow image generation frameworks, failing to optimize attention mechanisms for video temporal characteristics, making it difficult to capture long-range dependencies. Furthermore, existing methods lack dynamic correction mechanisms during the inference phase; once subject feature shifts occur during generation, errors accumulate with the diffusion process, ultimately leading to video quality degradation.

[0005] The aforementioned technical bottlenecks limit the practical application of multi-subject customized videos, especially in scenarios requiring precise control of subject attributes and interaction logic (such as film and television special effects and virtual reality). This invention addresses these shortcomings by systematically reconstructing the dataset construction process, designing a multimodal feature interaction network, and optimizing the diffusion model control mechanism, thereby achieving precise control and ensuring spatiotemporal consistency in multi-subject video generation. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method for generating multi-subject customized videos to solve the problems of spatiotemporal inconsistencies such as subject position drift, action logic breaks, or abrupt changes in appearance attributes. This invention constructs a multimodal feature alignment framework and a hierarchical control mechanism to solve the above problems. First, a structured data processing flow is designed to extract multi-subject visual features and semantic descriptions from the original video. Through object detection and text parsing, a precise mapping between subject nouns and image regions is established, and an augmented dataset containing spatiotemporal correlation information is constructed. This invention provides a method for generating multi-subject customized videos, including: S1: Data acquisition and preprocessing of the original video, including extracting image regions corresponding to multiple subjects in the video through object detection algorithms, slicing the video to obtain a temporally coherent frame sequence, and generating text descriptions that match the video content through a text description model; S2: Perform semantic parsing on the text description, extract multiple subject nouns from the text, and encode the features of the image regions corresponding to each subject to obtain the visual feature vectors of each subject and the corresponding text semantic feature vectors. S3: Construct a multi-subject feature interaction network, taking the visual feature vector as the input query matrix and the text semantic feature vector as the input key matrix and value matrix. Calculate the association weights between visual features and semantic features through a multi-level attention mechanism, and output the enhanced features that fuse multi-subject association information. S4: In the decoding stage of the video generation model, enhanced features are injected into the cross-attention module. The query matrix and key matrix of this module come from the visual features of the current layer of the video generation model, and the value matrix adopts enhanced features. The attention mechanism integrates the multi-subject association information into the video generation process. S5: Based on multi-subject association information, noise is gradually removed through the diffusion process of the video generation model, and a customized video sequence that maintains the spatiotemporal consistency of multiple subjects is output.

[0007] In one embodiment of the present invention, the target detection algorithm adopts a dynamically adjusted detection strategy, automatically selects feature extraction networks of different levels according to the complexity of the video scene, establishes spatiotemporal correlation constraints between consecutive frames, corrects the deviation of single-frame detection results through cross-frame propagation mechanism, constructs a candidate region generation module based on motion trajectory prediction, optimizes bounding box accuracy by combining the mask information output by the foreground segmentation network, and uses a multi-model voting mechanism to determine the final subject location for blurry regions with detection confidence below a threshold. Furthermore, it analyzes the semantic context of video segments through a long short-term memory network, dynamically updates the subject tracking strategy, and ensures the continuity of cross-frame detection results in the temporal dimension and the integrity in the spatial dimension.

[0008] In one embodiment of the present invention, the system includes a multi-level verification feedback mechanism. During the video generation process, the spatiotemporal continuity of the subject's state is monitored in real time. By constructing a reference feature library to store key features of historical frames, a similarity measurement algorithm is used to detect subject feature drift. When an abnormal offset is detected, a feature correction module is activated to project the offset features back to a preset feature manifold space. At the same time, an interpretable analysis module is designed to visualize the multi-subject association weights, generate a diagnostic report to identify potential inconsistencies, and receive manual correction instructions through a user feedback interface. The verification results are fed back to the feature extraction network for online fine-tuning and optimization.

[0009] In one embodiment of the present invention, the text description model adopts a multimodal contrastive learning framework, jointly optimizes the feature alignment of the visual encoder and the text decoder, parses the subject-verb-object relationship by constructing a syntax tree structure of the description statement, extracts the semantic association between the action subject and the recipient by dependency parsing, establishes a hierarchical description mechanism to generate multi-granular text representations for complex scenes in videos, and introduces a semantic disambiguation module assisted by knowledge graphs, matches the detected subject nouns with concept nodes in the domain knowledge base through entity linking technology, and selects the semantic interpretation that best fits the current scene for descriptive words with polysemous meanings based on video context information.

[0010] In one embodiment of the present invention, the semantic parsing process includes a dual mechanism of noun entity classification and relation extraction. A bidirectional attention mechanism is used to construct the mapping relationship between text features and visual features. A standardized expression system of domain-specific vocabulary is established by constructing a noun ontology library. For each detected subject noun, its context modifiers are analyzed to construct attribute feature vectors. At the same time, a graph neural network is used to model the spatial positional relationship and interaction behavior pattern between multiple subjects. A semantic dependency graph is constructed to represent the action association between subjects. For referential relationships containing pronouns, a cross-sentence referential resolution mechanism is established to track the continuity of subject identity.

[0011] In one embodiment of the present invention, the visual feature encoding process adopts a multi-scale feature fusion strategy. In the spatial dimension, geometric features of the subject contour are extracted through deformable convolution. In the channel dimension, an attention reweighting mechanism is used to highlight salient regions. At the same time, a temporal feature pyramid is constructed to capture the subject's motion pattern. Independent encoding channels are established for static attribute features and dynamic behavior features respectively. The essential attributes of the subject and scene-related features are separated through a feature decoupling network. An adversarial training strategy is used to enhance the domain invariance of the feature encoder. Feature orthogonality constraints are designed to ensure the distinguishability of feature vectors of different subjects.

[0012] In one embodiment of the present invention, the multi-subject feature interaction network includes cascaded attention computing units. Each computing unit consists of a multi-head cross-attention module and a gated fusion module. During the attention weight calculation process, relative position encoding is introduced to represent the spatial topological relationship between subjects. The integrity of the original features is maintained through residual connections. At the same time, a dynamic feature selection mechanism is designed to filter noise interference. A feature memory is established in the deep layer of the network to store subject state information across time steps. An adaptive feature scaling coefficient is used to balance the contribution of different subjects to the final feature. The modeling of subject association relationships in the feature space is supervised by a contrastive loss function.

[0013] In one embodiment of the present invention, the cross-attention module adopts a dual-path information fusion architecture, wherein the main path inherits the self-attention mechanism of the original video generation model, and the auxiliary path constructs guided attention based on multi-subject features. The fusion ratio of the two-path features is dynamically adjusted through learnable weight parameters. A position bias matrix based on the spatial distribution of the subject is introduced during the attention calculation process. A feature importance evaluation module is designed to automatically identify key information injection nodes. A fine-grained feature injection strategy is adopted for high-frequency update regions. At the same time, a feature recombination mechanism is established in the channel dimension to optimize the information fusion efficiency.

[0014] In one embodiment of the present invention, the diffusion process adopts a multi-stage constraint mechanism. At each time step of the denoising process, the consistency loss function between the subjects is calculated. This function includes three components: spatial layout similarity constraint, motion trajectory smoothness constraint, and appearance feature stability constraint. The denoising direction is guided by constructing an energy function in the latent space. At the same time, a kinematic model based on physical simulation is designed to predict the rationality of the subject behavior. Verification nodes are inserted at key frame positions to check the multi-subject interaction logic. For the detected inconsistent regions, a local regeneration mechanism is initiated, and the gradient update strategy of the denoising process is adjusted through an iterative optimization algorithm.

[0015] In one embodiment of the present invention, the method includes a dynamic data augmentation module that automatically generates synthetic samples containing multi-agent interactions during the training phase. It controls the number of agents, spatial layout, and motion patterns by constructing a parameterized scene generator, enhances the robustness of the model by employing adversarial example generation technology, designs a course learning strategy to train in stages from simple to complex scenes, establishes a reward mechanism based on reinforcement learning to evaluate the multi-agent consistency of the generated results, optimizes the parameter update direction of the feature injection network through the policy gradient method, and constructs an online hard sample mining mechanism to automatically identify training samples that need to be optimized.

[0016] This invention provides a method for generating customized multi-subject videos, constructing a multimodal feature alignment framework and a hierarchical control mechanism to solve the aforementioned problems. First, a structured data processing flow is designed to extract multi-subject visual features and semantic descriptions from the original video. Accurate mappings between subject nouns and image regions are established through object detection and text parsing, constructing an augmented dataset containing spatiotemporal correlation information. Subsequently, a multi-subject feature interaction network is developed, employing an attention mechanism to fuse visual features and textual semantic features, establishing inter-subject correlation constraints in the feature space. In the generation stage, by embedding a cross-attention module in the diffusion model architecture, multi-subject correlation features are dynamically injected into the denoising process. Spatial position bias matrices and kinematic models are used to simultaneously optimize subject layout and behavioral logic. Finally, through a multi-stage consistency loss function and a dynamic verification mechanism, long-term stability of multi-subject attributes, motion trajectories, and interaction relationships in the generated video is achieved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the method for generating customized videos for multiple subjects. Detailed Implementation

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0022] Please see Figure 1 The diagram illustrates the method for generating multi-subject customized videos according to the present invention. The method comprises five steps: S1: Data acquisition and preprocessing of the original video, including extracting image regions corresponding to multiple subjects in the video using a target detection algorithm, slicing the video to obtain a temporally coherent frame sequence, and generating a text description matching the video content using a text description model; S2: Semantic parsing of the text description, extracting multiple subject nouns from the text, and simultaneously performing feature encoding on the image regions corresponding to each subject to obtain visual feature vectors and corresponding text semantic feature vectors for each subject; S3: Constructing a multi-subject feature interaction network, using the visual feature vectors as a query matrix. The input consists of text semantic feature vectors as key and value matrices. A multi-level attention mechanism is used to calculate the association weights between visual and semantic features, outputting enhanced features that fuse multi-subject association information. S4: In the decoding stage of the video generation model, the enhanced features are injected into a cross-attention module. The query and key matrices of this module come from the visual features of the current layer of the video generation model, while the value matrix uses the enhanced features. The attention mechanism integrates multi-subject association information into the video generation process. S5: Based on the multi-subject association information, noise is gradually reduced through the diffusion process of the video generation model, outputting a customized video sequence that maintains spatiotemporal consistency among the multiple subjects.

[0023] like Figure 1As shown, a dynamically adjusted detection strategy is employed in the object detection algorithm. This strategy adaptively selects different levels of feature extraction networks based on the complexity of the video scene. Lightweight networks are prioritized in simple scenes to improve processing speed, while deep networks are automatically switched to enhance feature representation capabilities in scenes with dense subjects or complex backgrounds. By establishing cross-frame spatiotemporal correlation constraints, the algorithm constructs a propagation model of the subject's motion trajectory between consecutive video frames, uses optical flow information to predict the displacement vector of the subject between adjacent frames, and combines a Kalman filter algorithm to correct the localization deviation of single-frame detection. For blurry regions with low detection confidence, a multi-model ensemble mechanism is introduced, combining the output results of multiple detectors for voting decisions, effectively reducing the false negative and false positive rates. The candidate region generation module integrates the mask information output by the motion trajectory prediction and foreground segmentation networks, optimizing the localization accuracy of bounding boxes through geometric constraints, especially maintaining detection stability in scenes where the subject is partially occluded or moving rapidly. A Long Short-Term Memory (LSTM) network is deployed to analyze the semantic context of video segments, dynamically adjusting the detection threshold and tracking strategy. For example, in dialogue scenes, continuous tracking of the main subject is prioritized, while in motion scenes, the detection sensitivity for fast-moving objects is enhanced. The detection system also includes an adaptive learning mechanism that continuously optimizes the detection model parameters through an online update module and adjusts the weight distribution of the feature extraction network using error signals fed back during the generation process, ensuring that detection accuracy and temporal consistency are maintained in long video sequences.

[0024] Furthermore, a multimodal contrastive learning framework is adopted in the design of the text description model. This framework achieves cross-modal feature alignment by jointly training a visual encoder and a text decoder. The visual encoder employs a hierarchical feature extraction structure, with the lower-level network capturing pixel-level detailed features and the higher-level network extracting semantic-level abstract features, while introducing an attention mechanism to focus on key regions. The text decoder is built based on the Transformer architecture, modeling long-range dependencies between words through a self-attention layer and establishing dynamic associations between text and visual features using a cross-attention layer. During training, a contrastive loss function is used to shorten the distance between matching image-text feature pairs and push away mismatched feature pairs, enhancing the model's ability to understand multimodal data. For complex video scenes, the model establishes a hierarchical description mechanism, first generating a general description of the global scene, and then refining it layer by layer to the detailed characterization of subject behaviors and interaction relationships. The knowledge graph-assisted semantic disambiguation module maps detected subject nouns to a domain knowledge base through entity linking technology, for example, automatically distinguishing "apple" as a fruit entity or a technology company entity based on context. For action words with multiple meanings, the model combines spatiotemporal information from the video to make semantic selections. For example, it determines whether "open" refers to opening a door or opening a box based on the subject's movement trajectory. In addition, the model constructs a grammatical dependency tree to parse the subject-verb-object structure of the sentence, extracts the semantic roles of the action performer and receiver, and establishes an event timeline to mark the order in which the actions occur, providing structured semantic guidance for subsequent feature alignment.

[0025] like Figure 1As shown, a dual mechanism of noun entity classification and relation extraction is implemented during semantic parsing. The noun entity classification module uses a sequence labeling model combining bidirectional LSTM and CRF to identify the main nouns and their attribute modifiers in the text description, such as distinguishing the color attribute and the main category in "red car". The relation extraction part constructs a graph neural network model to convert verb phrases in the text into predicate nodes, establish action association edges between subjects, and form a structured semantic dependency graph. The mapping between visual features and text features is achieved through a bidirectional attention mechanism, where the query vector on the text side and the key-value matrix on the visual side perform attention calculation, while the query vector on the visual side and the key-value matrix on the text side perform reverse attention calculation, forming a bidirectional feature interaction. The construction of the noun ontology integrates domain-specific terminology and common knowledge, and establishes a standardized concept system and synonym mapping table, such as unifying different expressions such as "dog" and "dog" as the same entity. To address the problem of pronoun referential resolution, the model uses an attention memory network to track the subjects mentioned in the previous text, and determines the referent by calculating the semantic similarity between the current pronoun and the historical noun. In terms of spatial relationship modeling, a geometric encoder is used to transform the coordinates of the detection boxes into positional feature vectors, which are then concatenated with visual features and input into a graph neural network to learn the relative positional relationships between subjects (such as left and right, containment, and adjacency). For dynamic interactive behaviors, a temporal encoder is constructed to capture the temporal evolution patterns of actions, such as analyzing the phase features of the "throwing" action: release, flight, and landing. The parsing system also includes a self-correction module. When semantic contradictions are detected (such as the same subject appearing in different locations simultaneously), a conflict resolution algorithm is activated to re-evaluate the credibility of the output results of each module, and the final parsing result is determined through a weighted voting mechanism.

[0026] In one embodiment of the present invention, a multi-scale feature fusion strategy is implemented in the visual feature encoding stage. A deformable convolutional network dynamically adjusts the sampling position of the convolutional kernels to adapt to the geometric deformation of the subject's contour, capturing the distribution of key feature points of the subject's boundary in the spatial dimension. This strategy deploys convolutional layers with different receptive fields in the feature pyramid architecture. The lower-level network extracts fine local texture features, while the higher-level network captures abstract semantic features. Multi-granularity information fusion is achieved through cross-layer feature concatenation. The attention reweighting mechanism in the channel dimension uses a channel attention module to automatically evaluate the importance of each feature channel, enhancing the response value of salient regions through a squeeze-excitation operation while suppressing irrelevant background noise. The temporal feature pyramid constructs a three-dimensional convolutional layer with multiple time steps, capturing short-term motion patterns and long-term behavioral trends respectively. Gated recurrent units integrate motion features at different time scales. For static features (such as color and shape) and dynamic behavioral features (such as motion trajectory and interactive actions) of the subject's attributes, the system establishes a dual-channel encoding network. Feature decoupling technology separates essential attributes from scene-dependent features. The static encoder eliminates interference factors such as illumination changes through adversarial training, while the dynamic encoder combines optical flow information to enhance motion feature extraction. Feature orthogonality constraints are applied by calculating the cosine similarity between feature vectors of different subjects and applying a loss function to force the clustering of similar subject features and the separation of dissimilar features. Simultaneously, a feature space projection matrix is ​​designed to eliminate the coupling effect of scene background on subject features. In the adversarial training strategy, a domain classifier is introduced to discriminate the scene from which features originate. A gradient inversion layer forces the encoder to generate domain-invariant features, improving the model's cross-scene generalization ability. The encoding system also includes a feature inpainting module. When partial feature loss or corruption is detected, a feature completion network reconstructs the complete feature representation based on information from neighboring frames.

[0027] Furthermore, the constructed cascaded multi-subject feature interaction network comprises multiple cascaded attention computation units. Each unit consists of a multi-head cross-attention module and a gated fusion module, forming a hierarchical feature processing flow. In the multi-head cross-attention module, visual features are used as the query vector and the key-value matrix of text semantic features for multi-dimensional similarity calculation. Each attention head focuses on the association patterns in different feature subspaces, capturing diverse interaction relationships through parallel computation. The gated fusion module uses learnable weight parameters to dynamically adjust the mixing ratio of original features and attention output features, retaining important information while filtering out noise interference. Relative position encoding transforms the spatial topological relationships between subjects (such as relative distance and azimuth) into positional bias vectors, embedding them into the attention weight calculation process, enabling the model to perceive geometric constraints between subjects. The residual connection structure ensures gradient flow during deep network training, preventing feature information degradation during multi-layer transmission. The dynamic feature selection mechanism evaluates the information content of each feature channel through a differentiable gated network, performs soft masking on redundant or low-confidence features, and focuses on retaining high-information-entropy features. The deep feature memory built into the network adopts a key-value pair storage structure to record the trajectory of subject state changes in historical time steps. A similarity retrieval mechanism is used to achieve feature association across time steps, effectively modeling long-range dependencies. Adaptive feature scaling coefficients are dynamically generated by a lightweight neural network, automatically adjusting the fusion weights of different subject features based on the complexity of the current scene, such as increasing the contribution of key subject features in densely interactive scenarios. The design of the contrastive loss function introduces a positive and negative sample pair supervision mechanism. Positive sample pairs come from subject features from different perspectives in the same video, while negative sample pairs are taken from similar subjects in different videos. By maximizing the mutual information of positive sample pairs while minimizing the similarity of negative sample pairs, the distinguishability of subject relationships in the feature space is enhanced.

[0028] like Figure 1As shown, the designed cross-attention module adopts a dual-path parallel architecture. The main path inherits the self-attention mechanism of the original video generation model, maintaining the model's original feature learning ability. The auxiliary path constructs a guided attention path based on multi-subject features. The fusion of the two features is dynamically weighted through learnable gating parameters. The gating coefficient is jointly determined by the noise level and scene complexity of the current generation stage. In the early stage of generation, the focus is on guiding subject features, and in the later stage, it gradually transitions to autonomous generation. The construction of the position bias matrix is ​​based on the spatial distribution information of the subject detection boxes. The center coordinates of each subject are transformed into azimuth features in polar coordinates and embedded into the attention calculation through Fourier feature mapping, so that the generation process follows spatial layout constraints. The feature importance evaluation module uses class activation mapping technology to analyze the key response regions of the generated feature map, identify the spatial locations where multi-subject information needs to be injected, and formulate differentiated feature fusion strategies accordingly. For high-frequency update regions (such as the motion trajectory of fast-moving subjects), a block attention mechanism is used to implement local fine-grained feature injection. The feature map is divided into multiple sub-regions, and attention weights are calculated independently, which improves the generation quality of local details while maintaining computational efficiency. The feature reorganization at the channel dimension breaks down inherent channel correlations through channel shuffling operations, promoting cross-channel information interaction. Simultaneously, a channel attention mechanism is designed to recalibrate the fusion weights of each channel. This module also includes a feature normalization layer combined with an activation function, employing layer normalization to stabilize the training process. Gated linear units enhance nonlinear representation capabilities, and skip connections integrate shallow detail features with deep semantic features, forming a multi-level feature fusion system. A multi-stage consistency constraint mechanism is implemented during the denoising process of the diffusion model, designing a composite loss function to simultaneously optimize stability across spatial, temporal, and appearance dimensions. Spatial layout similarity constraints penalize unreasonable displacement abrupt changes by calculating the cross-union ratio (CUI) of the subject detection boxes between adjacent frames, while an energy function is constructed to ensure that the relative positional relationships between subjects conform to physical laws. Motion trajectory smoothness constraints calculate acceleration continuity using the second derivative, predict reasonable motion paths through a Kalman filter, and apply gradient penalties to generated results deviating from the predicted trajectory. Appearance feature stability constraints ensure the consistency of subject color, texture, and other attributes along the time axis by comparing the cosine similarity of subject features between keyframes. The latent space energy function is constructed by integrating the above constraints. A differentiable optimization algorithm guides the denoising process along the energy decrease direction, enhancing physical plausibility while maintaining the generated degrees of freedom. A rigid body motion equation and collision detection module are established based on a kinematic model derived from physical simulation to predict reasonable behavior patterns in multi-agent interactions, such as verifying whether the trajectory of a thrown object conforms to parabolic motion. Keyframe verification nodes are set at important action inflection points. A pre-trained discriminant network evaluates the logical consistency of generated frames. When a phenomenon violating physical laws (such as object penetration or discontinuous motion) is detected, a local regeneration mechanism is triggered to perform targeted repair of the problem area.The iterative optimization algorithm employs an adaptive step-size strategy, increasing the number of gradient updates in regions with large errors, while also incorporating a momentum buffer mechanism to prevent the optimization process from getting trapped in local optima. This constraint system also includes a dynamic weight adjustment module, which automatically adjusts the contribution ratio of each loss term according to the current generation stage, prioritizing spatial layout stability in the early stages and enhancing the consistency of appearance details in the later stages, achieving phased and refined control.

[0029] This invention provides a method for generating customized multi-subject videos, addressing the aforementioned problems by constructing a multimodal feature alignment framework and a hierarchical control mechanism. First, a structured data processing flow is designed to extract multi-subject visual features and semantic descriptions from the original video. Precise mappings between subject nouns and image regions are established through object detection and text parsing, constructing an augmented dataset containing spatiotemporal correlation information. Subsequently, a multi-subject feature interaction network is developed, employing an attention mechanism to fuse visual and textual semantic features, establishing inter-subject correlation constraints in the feature space. During the generation stage, a cross-attention module is embedded in the diffusion model architecture to dynamically inject multi-subject correlation features into the denoising process. Spatial position bias matrices and kinematic models are used to simultaneously optimize subject layout and behavioral logic. Finally, a multi-stage consistency loss function and a dynamic verification mechanism are used to achieve long-term stability of multi-subject attributes, motion trajectories, and interaction relationships in the generated video.

[0030] Therefore, the multi-subject customized video generation method of the present invention can solve the problems of spatiotemporal inconsistency such as subject position drift, action logic breakage or appearance attribute mutation.

[0031] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for generating multi-subject customized videos, characterized in that, include: S1: Data acquisition and preprocessing of the original video, including extracting image regions corresponding to multiple subjects in the video through object detection algorithms, slicing the video to obtain a temporally coherent frame sequence, and generating text descriptions that match the video content through a text description model; S2: Perform semantic parsing on the text description, extract multiple subject nouns from the text, and simultaneously perform feature encoding on the image regions corresponding to each subject to obtain the visual feature vectors of each subject and the corresponding text semantic feature vectors. S3: Construct a multi-subject feature interaction network, using the visual feature vector as the input query matrix and the text semantic feature vector as the input key matrix and value matrix. Calculate the association weights between visual features and semantic features through a multi-level attention mechanism, and output enhanced features that fuse multi-subject association information. S4: In the decoding stage of the video generation model, the enhanced features are injected into the cross-attention module. The query matrix and key matrix of this module come from the visual features of the current layer of the video generation model, and the value matrix adopts the enhanced features. The multi-subject association information is integrated into the video generation process through the attention mechanism. S5: Based on the multi-subject association information, noise is gradually removed through the diffusion process of the video generation model to output a customized video sequence that maintains the spatiotemporal consistency of the multi-subject.

2. The method for generating a multi-subject customized video according to claim 1, characterized in that, The target detection algorithm employs a dynamically adjusted detection strategy, automatically selecting feature extraction networks of different levels based on the complexity of the video scene. It establishes spatiotemporal correlation constraints between consecutive frames, corrects deviations in single-frame detection results through a cross-frame propagation mechanism, and constructs a candidate region generation module based on motion trajectory prediction. It also optimizes bounding box accuracy by combining mask information output by the foreground segmentation network. For blurry regions with detection confidence below a threshold, a multi-model voting mechanism is used to determine the final subject location. Furthermore, it analyzes the semantic context of video segments through a long short-term memory network and dynamically updates the subject tracking strategy to ensure the temporal coherence and spatial integrity of cross-frame detection results.

3. The method for generating a multi-subject customized video according to claim 1, characterized in that, The system includes a multi-level verification feedback mechanism. During video generation, it monitors the spatiotemporal continuity of the subject's state in real time. It stores key features of historical frames by constructing a reference feature library and uses a similarity measurement algorithm to detect subject feature drift. When an abnormal offset is detected, the feature correction module is activated to project the offset features back to a preset feature manifold space. At the same time, an interpretable analysis module is designed to visualize the association weights of multiple subjects, generate a diagnostic report to identify potential inconsistencies, and receive manual correction instructions through a user feedback interface. The verification results are fed back to the feature extraction network for online fine-tuning and optimization.

4. The method for generating a multi-subject customized video according to claim 1, characterized in that, The text description model employs a multimodal contrastive learning framework, jointly optimizing feature alignment between the visual encoder and text decoder. It parses subject-verb-object relationships by constructing a syntax tree structure for the description sentences and extracts semantic associations between the action subject and the recipient using dependency parsing. For complex scenes in videos, a hierarchical description mechanism is established to generate multi-granular text representations. Simultaneously, a semantic disambiguation module assisted by a knowledge graph is introduced, matching detected subject nouns with concept nodes in the domain knowledge base through entity linking technology. For descriptive words with multiple meanings, the semantic interpretation that best fits the current scene is selected based on video context information.

5. The method for generating a multi-subject customized video according to claim 1, characterized in that, The semantic parsing process includes a dual mechanism of noun entity classification and relation extraction. A bidirectional attention mechanism is used to construct the mapping relationship between text features and visual features. A standardized expression system of domain-specific vocabulary is established by constructing a noun ontology. For each detected subject noun, its context modifiers are analyzed to construct attribute feature vectors. At the same time, graph neural networks are used to model the spatial positional relationships and interaction behavior patterns between multiple subjects. A semantic dependency graph is constructed to represent the action associations between subjects. For referential relationships containing pronouns, a cross-sentence referential resolution mechanism is established to track the continuity of subject identity.

6. The method for generating a multi-subject customized video according to claim 1, characterized in that, The visual feature encoding process employs a multi-scale feature fusion strategy. In the spatial dimension, it extracts the geometric features of the subject's contour through deformable convolution. In the channel dimension, it uses an attention reweighting mechanism to highlight salient regions. Simultaneously, it constructs a temporal feature pyramid to capture the subject's motion patterns. Independent encoding channels are established for static attribute features and dynamic behavior features, respectively. The essential attributes of the subject and scene-related features are separated through a feature decoupling network. An adversarial training strategy is adopted to enhance the domain invariance of the feature encoder, and feature orthogonality constraints are designed to ensure the distinguishability of feature vectors from different subjects.

7. The method for generating a multi-subject customized video according to claim 1, characterized in that, The multi-subject feature interaction network includes cascaded attention computing units. Each computing unit consists of a multi-head cross-attention module and a gated fusion module. In the process of attention weight calculation, relative position encoding is introduced to represent the spatial topological relationship between subjects. The integrity of the original features is maintained through residual connections. At the same time, a dynamic feature selection mechanism is designed to filter noise interference. A feature memory is established in the deep layer of the network to store subject state information across time steps. An adaptive feature scaling coefficient is used to balance the contribution of different subjects to the final feature. The modeling of subject association relationships in the feature space is supervised by a contrastive loss function.

8. The method for generating a multi-subject customized video according to claim 1, characterized in that, The cross-attention module adopts a dual-path information fusion architecture. The main path inherits the self-attention mechanism of the original video generation model, while the auxiliary path constructs guided attention based on multi-subject features. The fusion ratio of the two features is dynamically adjusted through learnable weight parameters. A position bias matrix based on the spatial distribution of the subject is introduced during the attention calculation process. A feature importance evaluation module is designed to automatically identify key information injection nodes. A fine-grained feature injection strategy is adopted for high-frequency update regions. At the same time, a feature recombination mechanism is established in the channel dimension to optimize the information fusion efficiency.

9. The method for generating a multi-subject customized video according to claim 1, characterized in that, The diffusion process employs a multi-stage constraint mechanism. At each time step of the denoising process, a consistency loss function among the subjects is calculated. This function includes three components: spatial layout similarity constraint, motion trajectory smoothness constraint, and appearance feature stability constraint. The denoising direction is guided by constructing an energy function in the latent space. At the same time, a kinematic model based on physical simulation is designed to predict the rationality of the subject's behavior. Verification nodes are inserted at key frame positions to check the multi-subject interaction logic. For detected inconsistent regions, a local regeneration mechanism is initiated, and the gradient update strategy of the denoising process is adjusted through an iterative optimization algorithm.

10. The method for generating a multi-subject customized video according to claim 1, characterized in that, The method includes a dynamic data augmentation module that automatically generates synthetic samples containing multi-agent interactions during the training phase. It controls the number of agents, spatial layout, and motion patterns by constructing a parameterized scene generator. It enhances the robustness of the model by employing adversarial example generation technology. It also designs a course learning strategy to train in stages from simple to complex scenes, establishes a reward mechanism based on reinforcement learning to evaluate the consistency of the generated results among multiple agents, optimizes the parameter update direction of the feature injection network through the policy gradient method, and constructs an online hard sample mining mechanism to automatically identify training samples that need to be optimized.

Images