Video generation model training and inference method based on multi-concept decoupling and combination

Abstract

The application discloses a video generation model training and reasoning method based on multi-concept decoupling and combination, and relates to the technical field of artificial intelligence and computer vision. The method comprises the following steps: acquiring reference image and reference video data, and constructing a basic video generation model which introduces content, style and motion low-rank matrix modules; using an iterative double-branch decoupling strategy, content features, style features and motion features are respectively decoupled from the reference data through an alternating freezing mechanism, a complementary prompt word strategy and a time perception mask strategy; in the training process, a statistical regularization loss function based on Z-score is introduced to align the weight distribution of each low-rank matrix module with the target distribution, thereby retaining the hierarchical trend while unifying the weight amplitude magnitude. The application solves the problems of feature interference and weight imbalance in multi-concept combination, realizes independent control and flexible combination of content, style and motion in video generation, and significantly improves the semantic alignment degree and dynamic consistency of the generated video.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically to a video generation technology based on a diffusion model, and more particularly to a training and inference method for a video generation model based on multi-concept decoupling and combination. Background Technology

[0002] With the rapid development of deep learning technology, text-based video models based on diffusion transformers have made significant progress in generating high-quality videos. In fields such as film production, game development, and advertising creativity, users are no longer satisfied with general video generation, but instead hope that models can achieve customized video generation based on specific reference data, that is, simultaneously retain the appearance of specific objects (content), apply specific artistic effects (style), and reproduce specific motion patterns (motion).

[0003] Current technologies face several bottlenecks in the customized generation of multi-concept videos: First, it is difficult to simultaneously and precisely control the three dimensions of content, style, and motion in a single generation. Simply linking the image's "content-style" customization model with the image-generated video model can lead to unpredictable motion generation in subsequent frames due to the uncontrollable position and posture of objects in the first frame. Second, different concept features are easily entangled during training. When learning style, the model may mistakenly include object structural features, and when learning motion, it may destroy static content features. Furthermore, joint training requires strictly paired datasets, resulting in high data collection costs and poor scalability. Sequential training can easily lead to "catastrophic forgetting" or global feature overfitting. Third, there is a serious problem of weight interference when combining multi-concept models. As an efficient fine-tuning technique, low-rank matrices show significant differences in weight amplitudes after the content, style, and motion low-rank matrices converge. The weight amplitude of the content low-rank matrix is ​​much larger than that of the other two types. Direct linear superposition can lead to content features dominating the generation result, losing the target style and motion effect, and making it impossible to achieve free combination of multiple concepts.

[0004] Therefore, there is an urgent need for a video generation model training and inference method based on multi-concept decoupling and combination that can effectively decouple content, style, and motion features, and eliminate weight interference and ensure balanced expression of each concept during combinatorial reasoning, in order to overcome the shortcomings of existing technologies. Summary of the Invention

[0005] This invention addresses the problems of conceptual entanglement, combined weight interference, and difficulty in flexibly controlling multiple dimensions in the existing technology for customized generation of multi-concept videos. It provides a training and inference method for video generation models based on multi-concept decoupling and combination, which realizes independent decoupling and flexible combination of content, style, and motion, thereby improving the semantic alignment and dynamic consistency of the generated video.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] A video generation model training method based on multi-concept decoupling and combination includes:

[0008] S1. Acquire reference image data and reference video data, wherein the reference image data is used to extract content features and style features, and the reference video data is used to extract motion features;

[0009] S2. Construct a basic video generation model based on a diffusion model, and introduce multiple low-rank matrix modules into the basic video generation model. The multiple low-rank matrix modules include at least a content low-rank matrix module, a style low-rank matrix module, and a motion low-rank matrix module.

[0010] S3. Based on the reference image data, the content low-rank matrix module and the style low-rank matrix module are trained using an iterative dual-branch decoupling strategy that integrates an alternating freeze mechanism, a complementary cue word strategy, and a time-aware masking strategy, to obtain the trained content low-rank matrix module and the trained style low-rank matrix module; the iterative dual-branch decoupling strategy includes freezing the content low-rank matrix module and updating the style low-rank matrix module in the first stage, and freezing the style low-rank matrix module and updating the content low-rank matrix module in the second stage.

[0011] S4. Based on the reference video data and the trained content low-rank matrix module, train the motion low-rank matrix module to obtain the trained motion low-rank matrix module.

[0012] S5. During the training of the content low-rank matrix module, the style low-rank matrix module, and the motion low-rank matrix module, a statistical regularization loss function based on Z-score is introduced. The statistical regularization loss function is used to constrain the weight distribution of each low-rank matrix module to align with the target distribution, so as to unify the magnitude of the weight amplitude of each low-rank matrix module while preserving the hierarchical feature trend and eliminating the magnitude difference between different low-rank matrix modules.

[0013] S6. Based on the trained content low-rank matrix module, the trained style low-rank matrix module, and the trained motion low-rank matrix module, a personalized video generation model with multi-concept combination capability is obtained by combining them.

[0014] Furthermore, in step S3, the training of the content low-rank matrix module and the style low-rank matrix module based on the reference image data using an iterative dual-branch decoupling strategy that integrates an alternating freeze mechanism, a complementary cue word strategy, and a time-aware masking strategy specifically includes:

[0015] Construct a complementary cue word set, which includes a first cue word containing only style description and a second cue word containing both content and style description;

[0016] In the first stage, based on the first prompt word and the first sampling time step range, the first diffusion loss function value is calculated, and the parameters of the style low-rank matrix module are updated based on the first diffusion loss function value;

[0017] In the second stage, based on the second prompt word and the second sampling time step range, a second diffusion loss function value is calculated, and the parameters of the content low-rank matrix module are updated based on the second diffusion loss function value; wherein, the first sampling time step range corresponds to the high-noise stage and the medium-noise stage of the diffusion process, and the characteristics of generating structure and texture in the high-noise stage of the diffusion model are used to guide the model to focus on the generation of style structure and texture; the second sampling time step range corresponds to the medium-noise stage and the low-noise stage of the diffusion process, and the characteristics of generating details in the low-noise stage of the diffusion model are used to guide the model to focus on the generation of content details.

[0018] Furthermore, in step S4, training the motion low-rank matrix module based on the reference video data and the trained content low-rank matrix module specifically includes:

[0019] The parameters of the trained low-rank content matrix module are frozen and used as a priori for static content features.

[0020] Based on the reference video data, the third cue word containing content and motion description, and the third sampling time step range, calculate the third diffusion loss function value;

[0021] The parameters of the motion low-rank matrix module are updated based on the value of the third diffusion loss function, so that the motion low-rank matrix module learns the temporal motion feature residuals relative to the static content features.

[0022] Furthermore, in step S5, the statistical regularization loss function based on Z-score includes a trend loss term and a magnitude loss term; specifically:

[0023] Obtain a pre-calculated target weight distribution vector, which contains the average absolute weight magnitude of each layer of the model;

[0024] Calculate the current weight magnitude of the low-rank matrix module being trained in each layer;

[0025] The target weight distribution vector and the current weight magnitude are respectively subjected to Z-score normalization. The Z-score normalization is calculated based on the mean and standard deviation and introduces a minimum value to avoid the denominator being 0. After processing, the normalized target vector and the normalized current vector are obtained.

[0026] The trend loss term is calculated based on the mean square error between the normalized target vector and the normalized current vector, and is used to maintain the hierarchical feature trend of the low-rank matrix module.

[0027] The magnitude loss term is calculated based on the distance between the target weight distribution vector and the current weight magnitude, and is used to constrain the absolute weight range of the low-rank matrix module.

[0028] Furthermore, the calculation formula for the statistical regularization loss function based on Z-score is as follows:

[0029]

[0030] in, The basic denoising loss for the diffusion model, For the loss term of the aforementioned magnitude, For the trend loss term, and To balance the hyperparameters.

[0031] Furthermore, the method also includes establishing a multi-concept video customization benchmark dataset, specifically including: decomposing the video generation task into three dimensions: content, style, and motion;

[0032] Style is subdivided into material style and art style, and motion is subdivided into object motion and camera motion. Based on the above dimensions, a set of test prompt words containing different combined tasks is constructed. The set of test prompt words is used for phased evaluation during model training and for comprehensive evaluation of generalization ability and decoupling combination ability after model training is completed.

[0033] To achieve the above objectives, the present invention also provides a video generation model inference method based on multi-concept decoupling and combination, comprising:

[0034] Receive target cue words, which contain multi-dimensional descriptions of target content, target style, and target motion;

[0035] From the low-rank matrix library constructed by the low-rank matrix modules trained by the above method, select the target content low-rank matrix module corresponding to the target content, the target style low-rank matrix module corresponding to the target style, and the target motion low-rank matrix module corresponding to the target motion.

[0036] The parameters of the target content low-rank matrix module, the target style low-rank matrix module, and the target motion low-rank matrix module are injected into the pre-trained diffusion video model in a linear superposition manner. Each low-rank matrix module is subjected to statistical regularization processing of Z-score, and its weight update is within the same numerical range and no single module's weight update dominates the model weight.

[0037] The target prompt words are input into a diffusion video model with low-rank matrix parameters to generate a target video in which content features, style features, and motion features are independently expressed and balancedly integrated, and the target video is free from feature interference dominated by a single concept.

[0038] Furthermore, the parameters of the target content low-rank matrix module, the target style low-rank matrix module, and the target motion low-rank matrix module are injected in a linear superposition manner, including:

[0039]

[0040] in, For the injected model weights, For frozen weights of the pre-trained model, , and These represent the weight update amounts contributed by the target content low-rank matrix module, the target style low-rank matrix module, and the target motion low-rank matrix module, respectively. Since the target content low-rank matrix module, the target style low-rank matrix module, and the target motion low-rank matrix module have undergone statistical regularization based on Z-scores during training, the magnitudes of the weight update amounts are within the same numerical range, thereby avoiding a single concept dominating the generation results.

[0041] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is a GPU, an NPU, or a CPU, and the processor executes the computer program to implement the method described above, which is capable of performing iterative training of low-rank matrix modules, calculation of Z-score statistical regularization loss, linear superposition injection of low-rank matrix parameters, and customized video generation inference.

[0042] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the method described above. The computer program includes a model building module, a decoupling training module, a regularization constraint module, a low-rank matrix combination module, and a video generation inference module. Each module works together to decouple and combine multiple concepts such as content, style, and motion to complete the customized generation of personalized videos.

[0043] Beneficial effects: (1) This invention achieves precise decoupling of multiple concepts: By using an iterative dual-branch decoupling strategy, combined with alternating freezing, complementary prompts and time-aware masking strategies, the content, style and motion features are effectively decoupled from the reference data, solving the feature entanglement problem and achieving independent control of the three dimensions;

[0044] (2) This invention eliminates the interference of combined weights: by introducing a statistical regularization loss function based on Z-scores, the weight distribution of each low-rank matrix module is aligned with the target distribution, the magnitude of the weight amplitude of different modules is unified, the problem of content features dominating the generation result is avoided, and the balanced expression is guaranteed when multiple concepts are combined.

[0045] (3) The present invention improves the quality of video generation: The present invention enables a flexible combination of content, style and motion, and the generated video is highly aligned with the target prompt words in semantics and maintains consistent dynamic features, which effectively improves the quality of customized video generation;

[0046] (4) This invention reduces data and application costs: It eliminates the need for strictly paired multi-concept datasets, reducing data collection and model training costs; during the inference phase, only the parameters of the low-rank matrix need to be linearly superimposed, eliminating the need to manually adjust the mixing ratio, simplifying the application process of multi-concept video customization, and improving the practicality and scalability of the model. Attached Figure Description

[0047] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0048] Figure 1 This is a flowchart of the main process of training a video generation model based on multi-concept decoupling and combination, as described in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of the video generation model training and inference method based on multi-concept decoupling and combination as described in an embodiment of the present invention;

[0050] Figure 3 The image shows the generated video effect of the video generation model training and inference method based on multi-concept decoupling and combination as described in the embodiment of the present invention.

[0051] Figure 4 This paper compares the video generated by the video generation model training and inference method based on multi-concept decoupling and combination according to the present invention with the video generated by various traditional methods. Detailed Implementation

[0052] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0053] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0054] The method in this embodiment is based on a diffusion converter architecture (such as the WAN2.1 model) and is implemented through a framework.

[0055] Example 1

[0056] See Figures 1-4 A video generation model training method based on multi-concept decoupling and combination includes:

[0057] S1. Acquire reference image data and reference video data, wherein the reference image data is used to extract content features and style features, and the reference video data is used to extract motion features;

[0058] In the specific implementation, reference image data is acquired to extract static visual concepts. For example, a specific "teddy bear" image is used to extract content, and a "watercolor painting" image is used to extract style. Reference video data is acquired to extract dynamic temporal concepts. For example, a "running" video is used to extract body motion, or a "zoom-in" video is used to extract camera motion.

[0059] S2. Construct a basic video generation model based on a diffusion model, and introduce multiple low-rank matrix modules into the basic video generation model. The multiple low-rank matrix modules include at least a content low-rank matrix module, a style low-rank matrix module, and a motion low-rank matrix module.

[0060] In the specific implementation, pre-trained text is loaded into a base video generation model based on a diffusion model (e.g., the WAN2.1-1.3B model). Multiple low-rank matrix modules are inserted into the base video generation model. Specifically, the weight updates contributed by the low-rank matrix modules to the target content are... The weight updates contributed by the target style low-rank matrix module () and the weight updates contributed by the target style low-rank matrix module () The rank is set to 32; the weight update amount contributed by the low-rank matrix module of the target motion ( The rank is set to 64 to accommodate more complex temporal dynamics. The model weights are represented as follows:

[0061] in For the frozen weights of the pre-trained model, The number of low-rank matrices. , This represents each distinct low-rank matrix.

[0062] S3. Based on the reference image data, the content low-rank matrix module and the style low-rank matrix module are trained using an iterative dual-branch decoupling strategy that integrates an alternating freeze mechanism, a complementary cue word strategy, and a time-aware masking strategy, to obtain the trained content low-rank matrix module and the trained style low-rank matrix module; the iterative dual-branch decoupling strategy includes freezing the content low-rank matrix module and updating the style low-rank matrix module in the first stage, and freezing the style low-rank matrix module and updating the content low-rank matrix module in the second stage.

[0063] In step S3, the training of the content low-rank matrix module and the style low-rank matrix module based on the reference image data using an iterative dual-branch decoupling strategy that integrates an alternating freeze mechanism, a complementary cue word strategy, and a time-aware masking strategy specifically includes:

[0064] Construct a complementary cue word set, which includes a first cue word containing only style description and a second cue word containing both content and style description;

[0065] In the first stage, based on the first prompt word and the first sampling time step range, the first diffusion loss function value is calculated, and the parameters of the style low-rank matrix module are updated based on the first diffusion loss function value;

[0066] In the second stage, based on the second prompt word and the second sampling time step range, a second diffusion loss function value is calculated, and the parameters of the content low-rank matrix module are updated based on the second diffusion loss function value; wherein, the first sampling time step range corresponds to the high-noise stage and the medium-noise stage of the diffusion process, and the characteristics of generating structure and texture in the high-noise stage of the diffusion model are used to guide the model to focus on the generation of style structure and texture; the second sampling time step range corresponds to the medium-noise stage and the low-noise stage of the diffusion process, and the characteristics of generating details in the low-noise stage of the diffusion model are used to guide the model to focus on the generation of content details;

[0067] In the specific implementation, based on an iterative dual-branch decoupling strategy for content-style training, this embodiment decomposes the training objective into two sub-tasks for the reference image data, using an iterative learning mechanism to separate content and style. Within this framework, a complementary cue word mechanism is applied to define style cue words. For example, "in" <style>”，定义内容提示词：例如"A<content>,in<style>”。通过在内容提示词中包含风格描述，迫使内容低秩矩阵仅学习剩余的语义对象特征。并同时运用时间感知掩码设定，通过设定总时间步，时间步低阈值Tl和时间步高阈值Th，并利用扩散模型的特性（早期生成结构，晚期生成细节），将高噪声阶段（TlTh）分配给内容学习，将低噪声阶段（0Tl）分配给风格学习。迭代双分支解耦过程如下：阶段一（更新风格）：冻结目标内容低秩矩阵模块贡献的权重更新量（），仅更新目标风格低秩矩阵模块贡献的权重更新量（）。输入提示词，在时间步0Tl内计算损失并更新梯度。此时整体有效权重为：

[0068]

[0069] 其中为梯度冻结函数，为此时的整体有效权重。

[0070] 阶段二（更新内容）：冻结目标风格低秩矩阵模块贡献的权重更新量（），仅更新目标内容低秩矩阵模块贡献的权重更新量（）。输入提示词，在时间步TlTh内计算损失并更新梯度。此时有效权重为：

[0071]

[0072] 上述两个阶段交替进行，直至模型收敛；

[0073] S4、基于所述参考视频数据和所述训练好的内容低秩矩阵模块，训练所述运动低秩矩阵模块，得到训练好的运动低秩矩阵模块；

[0074] 在步骤S4中，所述基于所述参考视频数据和所述训练好的内容低秩矩阵模块，训练所述运动低秩矩阵模块，具体包括：

[0075] 将所述训练好的内容低秩矩阵模块的参数冻结，作为静态内容特征的先验；

[0076] 基于所述参考视频数据、包含内容与运动描述的第三提示词以及第三采样时间步范围，计算第三扩散损失函数值；

[0077] 基于所述第三扩散损失函数值更新所述运动低秩矩阵模块的参数，使所述运动低秩矩阵模块学习相对于静态内容特征的时序运动特征残差；

[0078] 在具体实现中，基于迭代双分支解耦策略的内容-运动训练，针对参考视频数据，本实施例将训练目标分解为两个子任务，利用迭代学习机制分离内容与运动。在这个框架下，应用互补提示词机制，定义运动提示词例如"is<motion>”，定义内容提示词：例如"A<content>,in<style>”。通过在内容提示词中包含风格描述，迫使内容低秩矩阵仅学习剩余的语义对象特征。并同时运用时间感知掩码设定，通过设定总时间步T，时间步低阈值Tl和时间步高阈值Th，并利用扩散模型的特性，将高噪声阶段（TlTh）分配给运动学习，将低噪声阶段（0Tl）分配给内容学习。迭代双分支解耦过程如下：阶段一（更新内容）：冻结目标运动低秩矩阵模块贡献的权重更新量（），仅更新目标内容低秩矩阵模块贡献的权重更新量（）。输入提示词，在时间步TlTh内计算损失并更新梯度。此时有效权重为：

[0079]

[0080] 其中为梯度冻结函数，为此时的整体有效权重。

[0081] 阶段二（更新运动）：冻结目标内容低秩矩阵模块贡献的权重更新量（），仅更新目标运动低秩矩阵模块贡献的权重更新量（））。输入提示词，在时间步0Tl内计算损失并更新梯度。此时有效权重为：

[0082]

[0083] 上述两个阶段交替进行，直至模型收敛；

[0084] S5、在训练所述内容低秩矩阵模块、所述风格低秩矩阵模块和所述运动低秩矩阵模块的过程中，引入基于Z分数的统计正则化损失函数，通过所述统计正则化损失函数约束各低秩矩阵模块的权重分布与目标分布对齐，以在保留层级特征趋势的同时统一各低秩矩阵模块的权重幅值量级，消除不同低秩矩阵模块之间的量级差异；

[0085] 在步骤S5中，所述基于Z分数的统计正则化损失函数包括趋势损失项和量级损失项；具体为：

[0086] 获取预先计算的目标权重分布向量，所述目标权重分布向量包含模型各层的平均绝对权重幅值；

[0087] 计算当前训练的低秩矩阵模块在各层的当前权重幅值；

[0088] 对所述目标权重分布向量和所述当前权重幅值分别进行Z分数归一化处理，所述Z分数归一化基于均值和标准差计算且引入极小值避免分母为0，处理后得到归一化目标向量和归一化当前向量；

[0089] 基于所述归一化目标向量和所述归一化当前向量之间的均方误差计算所述趋势损失项，用于保持低秩矩阵模块的层级特征趋势；

[0090] 基于所述目标权重分布向量和所述当前权重幅值之间的距离计算所述量级损失项，用于约束低秩矩阵模块的绝对权重范围。

[0091] 所述基于Z分数的统计正则化损失函数的计算公式如下：

[0092]

[0093] 其中，为扩散模型的基础去噪损失，为所述量级损失项，为所述趋势损失项，和为平衡超参数；

[0094] 需要说明的是，在上述步骤S2（内容-风格训练）和步骤S3（内容-运动训练）的每一次迭代更新中，为了保证所训练出的低秩矩阵模块不仅能够准确生成目标概念，还能在后续的多概念组合推理中保持权重的统计稳定性，本实施例构建了一个包含去噪主损失与统计正则化辅助损失的复合目标函数。首先引入趋势损失，旨在约束低秩矩阵参数在模型各层级间的相对分布形态，使其符合预训练模型的固有特征响应规律。获取当前训练步数下低秩矩阵各层权重的Z分数向量以及预先统计的目标Z分数向量。利用均方误差计算两者之间的距离：其中，表示模型的总层数。扩散变换器的不同层级通常负责处理不同维度的特征。并不强制要求某层的具体数值相等，而是强制要求低秩矩阵权重的相对强弱趋势与目标分布一致。这有效防止了模型在训练过程中过度依赖某些特定层级，从而避免了在组合多个低秩矩阵时因层级响应不匹配而产生的结构崩塌。量级损失旨在严格控制低秩矩阵参数的绝对数值范围，防止因训练数据量差异导致的权重数值漂移。先计算当前各层平均权重幅值与目标幅值之间的平均绝对误差：

[0095] 在传统低秩矩阵训练中，即使两个低秩矩阵都能生成良好的图像，其权重矩阵的范数可能相差巨大。当它们在推理阶段以系数直接相加时，数值大的低秩矩阵会完全掩盖数值小的低秩矩阵。强制将所有低秩矩阵的权重幅值拉齐到同一个统计基准线上，从而确保了在推理阶段无需手动调整混合比例，即可实现多概念的平衡融合。并扩散去噪主损失:其中，为加噪后的潜空间特征，为添加的高斯噪声，为模型预测的噪声，为文本条件，为数学期望，为去噪过程中的时间步。

[0096] 最终的优化目标是上述三者的加权和：

[0097]

[0098] 通过最小化该总损失函数，本申请在利用反向传播算法更新低秩矩阵参数时，实现了生成质量（由保证）与模型解耦性及组合性（由和保证）的同步优化；

[0099] 在具体实现中，为了解决不同低秩矩阵在组合时因权重幅值差异导致的干扰，在上述步骤S3和S4的训练过程中，均引入统计正则化损失。首先离线分析一组训练良好的低秩矩阵模型，计算各层权重的平均绝对幅值，得到目标向量，其中为模型层数。在训练的每一步，计算当前低秩矩阵模块在第层的平均权重幅值:

[0100]

[0101] 其中为低秩矩阵的低秩矩阵，为该层的参数数量。然后对当前幅值向量和目标向量分别进行Z分数归一化，提取层级趋势：其中和分别为均值和标准差。

[0102] S6、基于训练好的内容低秩矩阵模块、训练好的风格低秩矩阵模块和训练好的运动低秩矩阵模块，组合得到具备多概念组合能力的个性化视频生成模型。在一具体的实例中，所述方法还包括建立多概念视频定制化基准数据集，具体包括：

[0103] 将视频生成任务分解为内容、风格和运动三个维度；

[0104] 将风格细分为材质风格和艺术风格，将运动细分为物体运动和相机运动；基于上述维度构建包含不同组合任务的测试提示词集合，所述测试提示词集合用于在模型训练过程中进行阶段性评估，以及在模型训练完成后进行泛化能力与解耦组合能力的综合评估。

[0105] 在具体实现中，首先将视频生成任务按核心概念拆解为内容、风格、运动三个独立维度，其中"内容”维度聚焦视频中的核心主体（如动物、交通工具、建筑等），"风格”维度聚焦视觉呈现效果，"运动”维度聚焦主体的时序动态变化；

[0106] 子维度细化分类：为提升评估的全面性，对风格和运动维度进一步细分：

[0107] 风格维度细分为材质风格（如金属质感、布料质感、水墨质感等）和艺术风格（如油画风格、卡通风格、写实风格等）；

[0108] 运动维度细分为物体运动（如直线运动、旋转运动、往复运动等）和视角运动（如平视推进、俯视拉远、环绕拍摄等）；

[0109] 测试提示词集合构建：基于上述"3大核心维度+4个子维度”的分类体系，构建包含不同概念组合任务的测试提示词集合，每个提示词均明确包含"内容+风格+运动”的分维度描述，示例如下：

[0110] 基础组合："白色小猫（内容）+卡通风格（艺术风格）+匀速跑跳（物体运动）”；

[0111] 复杂组合："红色跑车（内容）+金属质感（材质风格）+环绕拍摄（视角运动）”；

[0112] 泛化组合："古老城堡（内容）+水墨质感（材质风格）+旋转上升（物体运动）”；

[0113] 数据集的双向评估应用：

[0114] 训练过程阶段性评估：在模型训练每迭代200轮时，调用测试提示词集合中的基础组合任务，通过对比生成视频与参考特征的匹配度，实时验证内容 / 风格 / 运动特征的解耦稳定性，若解耦效果未达标（如风格特征干扰内容识别），则调整训练学习率或正则化权重；

[0115] 训练完成后综合评估：模型训练收敛后，依次调用测试提示词集合中的复杂组合任务和泛化组合任务-复杂组合任务用于评估模型在多概念强关联场景下的解耦与组合能力，泛化组合任务用于评估模型在未见过的概念组合场景下的适应能力，确保模型具备稳定的定制化生成性能。

[0116] 实施例2

[0117] 为了实现上述目的，本实施例还提供了一种基于多概念解耦与组合的视频生成模型推理方法，包括：

[0118] 接收目标提示词，所述目标提示词包含对目标内容、目标风格和目标运动的分维度描述；

[0119] 在具体实现中，接收用户指令,用户输入一个复杂的生成指令，例如："一只<o1>泰迪熊（内容），由<m1>玻璃材质制成（材质风格），正在<v1>骑自行车（物体运动），背景是阳光明媚的公园，<s1>油画风格（艺术风格）”。

[0120] 从由上述的方法训练得到的低秩矩阵模块构建的低秩矩阵库中，选择与所述目标内容对应的目标内容低秩矩阵模块、与所述目标风格对应的目标风格低秩矩阵模块以及与所述目标运动对应的目标运动低秩矩阵模块；

[0121] 在具体实现中，可以根据指令中的标识符，从模型库中检索对应的低秩矩阵文件：加载目标内容低秩矩阵模块贡献的权重更新量，加载目标风格低秩矩阵模块贡献的权重更新量，加载目标运动低秩矩阵模块贡献的权重更新量。

[0122] 将所述目标内容低秩矩阵模块、所述目标风格低秩矩阵模块和所述目标运动低秩矩阵模块的参数以线性叠加的方式注入到预训练的扩散视频模型中，各所述低秩矩阵模块均经Z分数的统计正则化处理，其权重更新量处于同一数值范围且无单一模块的权重更新量主导模型权重；

[0123] 将所述目标内容低秩矩阵模块、所述目标风格低秩矩阵模块和所述目标运动低秩矩阵模块的参数以线性叠加的方式注入，包括：

[0124]

[0125] 其中，为注入后的模型权重，为预训练模型的冻结权重，、和分别为目标内容低秩矩阵模块、目标风格低秩矩阵模块以及目标运动低秩矩阵模块贡献的权重更新量；由于目标内容低秩矩阵模块、目标风格低秩矩阵模块以及目标运动低秩矩阵模块在训练时经过了基于Z分数的统计正则化，所述权重更新量的量级在同一数值范围内，从而避免单一概念主导生成结果。

[0126] 将所述目标提示词输入注入了低秩矩阵参数的扩散视频模型，生成内容特征、风格特征、运动特征独立表达且均衡融合的目标视频，所述目标视频无单一概念主导的特征干扰。

[0127] 在具体实现中，本实施例将融合后的模型用于去噪推理过程，输入对应的文本提示词，生成最终的视频。该视频将同时具备泰迪熊的形状、玻璃的质感以及骑车的动作，且各属性之间界限分明，互不干扰。

[0128] 如图4为本发明与传统的几种方法对比图，以下表1为对比效果数据：

[0129] 表1 本方法与其他方法的对比结果

[0130]

[0131] 由此可以看出，本发明通过"内容 / 风格 / 运动三特征解耦+Z分数正则化统一权重”的技术方案，不仅有效解决了现有视频生成技术中多概念组合的特征纠缠、运动生成失准、权重失衡等核心痛点，还在图文匹配、风格还原、画面质量等全维度实现了技术效果的显著提升，是一种具有实质性特点和显著进步的个性化视频生成技术。

[0132] 实施例3

[0133] 为了实现上述目的，本实施例还提供了一种电子设备，包括存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，所述处理器为GPU或NPU或CPU，所述处理器运行所述计算机程序以实现如上述的方法，能够完成低秩矩阵模块的迭代训练、Z分数统计正则化损失的计算、低秩矩阵参数的线性叠加注入及定制化视频生成推理。

[0134] 实施例4

[0135] 为了实现上述目的，本实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，所述程序被处理器执行时实现如上述的方法，所述计算机程序包括模型构建模块、解耦训练模块、正则化约束模块、低秩矩阵组合模块及视频生成推理模块，各模块协同实现内容、风格、运动的多概念解耦与组合，完成个性化视频的定制化生成。

[0136] 需要说明的是，当处理器执行所述计算机程序时，实现如下功能：构建包含双分支低秩矩阵的扩散模型架构；控制迭代训练流程，交替冻结和更新不同类型的低秩矩阵模块；在反向传播过程中计算并应用包含Z-Score统计正则项的复合损失函数；在推理阶段，根据用户输入动态加载并线性组合多个低秩矩阵模块以生成视频。

[0137] 以上所述仅为本发明的较佳实施例而已，并不用以限制本发明，凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。< / style>

Claims

1. A video generation model training method based on multi-concept decoupling and combination, characterized in that, include: S1. Acquire reference image data and reference video data, wherein the reference image data is used to extract content features and style features, and the reference video data is used to extract motion features; S2. Construct a basic video generation model based on a diffusion model, and introduce multiple low-rank matrix modules into the basic video generation model. The multiple low-rank matrix modules include at least a content low-rank matrix module, a style low-rank matrix module, and a motion low-rank matrix module. S3. Based on the reference image data, the content low-rank matrix module and the style low-rank matrix module are trained using an iterative dual-branch decoupling strategy that integrates an alternating freeze mechanism, a complementary cue word strategy, and a time-aware masking strategy, to obtain the trained content low-rank matrix module and the trained style low-rank matrix module; the iterative dual-branch decoupling strategy includes freezing the content low-rank matrix module and updating the style low-rank matrix module in the first stage, and freezing the style low-rank matrix module and updating the content low-rank matrix module in the second stage. S4. Based on the reference video data and the trained content low-rank matrix module, train the motion low-rank matrix module to obtain the trained motion low-rank matrix module. S5. During the training of the content low-rank matrix module, the style low-rank matrix module, and the motion low-rank matrix module, a statistical regularization loss function based on Z-score is introduced. The statistical regularization loss function is used to constrain the weight distribution of each low-rank matrix module to align with the target distribution, so as to unify the magnitude of the weight amplitude of each low-rank matrix module while preserving the hierarchical feature trend and eliminating the magnitude difference between different low-rank matrix modules. S6. Based on the trained content low-rank matrix module, the trained style low-rank matrix module, and the trained motion low-rank matrix module, a personalized video generation model with multi-concept combination capability is obtained by combining them.

2. The video generation model training method based on multi-concept decoupling and combination according to claim 1, characterized in that, In step S3, the training of the content low-rank matrix module and the style low-rank matrix module based on the reference image data using an iterative dual-branch decoupling strategy that integrates an alternating freeze mechanism, a complementary cue word strategy, and a time-aware masking strategy specifically includes: Construct a complementary cue word set, which includes a first cue word containing only style description and a second cue word containing both content and style description; In the first stage, based on the first prompt word and the first sampling time step range, the first diffusion loss function value is calculated, and the parameters of the style low-rank matrix module are updated based on the first diffusion loss function value; In the second stage, based on the second prompt word and the second sampling time step range, a second diffusion loss function value is calculated, and the parameters of the content low-rank matrix module are updated based on the second diffusion loss function value; wherein, the first sampling time step range corresponds to the high-noise stage and the medium-noise stage of the diffusion process, and the characteristics of generating structure and texture in the high-noise stage of the diffusion model are used to guide the model to focus on the generation of style structure and texture; the second sampling time step range corresponds to the medium-noise stage and the low-noise stage of the diffusion process, and the characteristics of generating details in the low-noise stage of the diffusion model are used to guide the model to focus on the generation of content details.

3. The video generation model training method based on multi-concept decoupling and combination according to claim 1, characterized in that, In step S4, training the motion low-rank matrix module based on the reference video data and the trained content low-rank matrix module specifically includes: The parameters of the trained low-rank content matrix module are frozen and used as a priori for static content features. Based on the reference video data, the third cue word containing content and motion description, and the third sampling time step range, calculate the third diffusion loss function value; The parameters of the motion low-rank matrix module are updated based on the value of the third diffusion loss function, so that the motion low-rank matrix module learns the temporal motion feature residuals relative to the static content features.

4. The video generation model training method based on multi-concept decoupling and combination according to claim 1, characterized in that, In step S5, the statistical regularization loss function based on Z-scores includes a trend loss term and a magnitude loss term; specifically: Obtain a pre-calculated target weight distribution vector, which contains the average absolute weight magnitude of each layer of the model; Calculate the current weight magnitude of the low-rank matrix module being trained in each layer; The target weight distribution vector and the current weight magnitude are respectively subjected to Z-score normalization. The Z-score normalization is calculated based on the mean and standard deviation and introduces a minimum value to avoid the denominator being 0. After processing, the normalized target vector and the normalized current vector are obtained. The trend loss term is calculated based on the mean square error between the normalized target vector and the normalized current vector, and is used to maintain the hierarchical feature trend of the low-rank matrix module. The magnitude loss term is calculated based on the distance between the target weight distribution vector and the current weight magnitude, and is used to constrain the absolute weight range of the low-rank matrix module.

5. The video generation model training method based on multi-concept decoupling and combination according to claim 4, characterized in that, The formula for calculating the statistical regularization loss function based on Z-score is as follows: in, The basic denoising loss for the diffusion model, For the loss term of the aforementioned magnitude, For the trend loss term, and To balance the hyperparameters.

6. The video generation model training method based on multi-concept decoupling and combination according to claim 1, characterized in that, The method also includes establishing a multi-concept video-customized benchmark dataset, specifically including: The video generation task is broken down into three dimensions: content, style, and motion. Styles are further subdivided into material styles and artistic styles, and motion is further subdivided into object motion and camera motion; Based on the above dimensions, a set of test prompt words containing different combined tasks is constructed. The set of test prompt words is used for phased evaluation during model training and for comprehensive evaluation of generalization ability and decoupling combination ability after model training is completed.

7. A video generation model inference method based on multi-concept decoupling and combination, characterized in that, include: Receive target cue words, which contain multi-dimensional descriptions of target content, target style, and target motion; From the low-rank matrix library constructed by the low-rank matrix modules trained by any one of claims 1-6, select the target content low-rank matrix module corresponding to the target content, the target style low-rank matrix module corresponding to the target style, and the target motion low-rank matrix module corresponding to the target motion; The parameters of the target content low-rank matrix module, the target style low-rank matrix module, and the target motion low-rank matrix module are injected into the pre-trained diffusion video model in a linear superposition manner. Each low-rank matrix module is subjected to statistical regularization processing of Z-score, and its weight update is within the same numerical range and no single module's weight update dominates the model weight. The target prompt words are input into a diffusion video model with low-rank matrix parameters to generate a target video in which content features, style features, and motion features are independently expressed and balancedly integrated, and the target video is free from feature interference dominated by a single concept.

8. The video generation model inference method based on multi-concept decoupling and combination according to claim 7, characterized in that, The parameters of the target content low-rank matrix module, the target style low-rank matrix module, and the target motion low-rank matrix module are injected in a linear superposition manner, including: in, For the injected model weights, For frozen weights of the pre-trained model, , and These represent the weight update amounts contributed by the target content low-rank matrix module, the target style low-rank matrix module, and the target motion low-rank matrix module, respectively. Since the target content low-rank matrix module, the target style low-rank matrix module, and the target motion low-rank matrix module have undergone statistical regularization based on Z-scores during training, the magnitudes of the weight update amounts are within the same numerical range, thereby avoiding a single concept dominating the generation results.

Images