A video coloring model construction method, a video coloring method, a device, and a medium

Abstract

The application discloses a video coloring model construction method, a video coloring method, equipment and a medium, and the method comprises the following steps: sampling key frames from a grayscale video sequence; extracting scene description text and generating a color reference image; constructing a basic video generation model, introducing a main branch with parameter freezing and a control branch with parameter training; constructing a composite visual sequence and setting a time sequence mask sequence; extracting spatio-temporal control features in the control branch, and injecting the main branch with residual superposition; introducing an alignment loss function to train the control branch, extracting deep target semantic features of the color reference image, mapping the intermediate features denoised by the main branch to a semantic space; setting a loss function to jointly construct a total training target and minimize, iteratively optimizing specified parameters to obtain a video coloring model, and performing a video coloring operation. The application can effectively decouple single-frame content color editing and long-sequence time sequence color propagation, and improve the color fidelity and dynamic time sequence consistency of the generated video.

Description

Technical Field

[0001] This invention relates to the field of computer vision data processing technology, specifically to a video colorization model construction method, video colorization method, device, and storage medium that employs a diffusion model-based video processing and generation technology with text semantic guidance and spatiotemporal target decoupling. Background Technology

[0002] In the field of multimedia processing, video colorization is a key technology that plays a vital role in aesthetic enhancement, film and television production, and cultural heritage restoration. The main goal of this task is to generate reasonable and vivid color information for grayscale video sequences.

[0003] With the rapid development of deep learning technology, although text-guided single-frame image colorization technology has made significant progress and can provide fine-grained control, directly extending these capabilities to the video domain still faces huge challenges. High-quality video colorization not only requires generating high-fidelity and semantically accurate colors in the spatial dimension, but also needs to maintain strict continuity and consistency in the temporal dimension. Therefore, current technology still has many bottlenecks in video colorization tasks:

[0004] Firstly, due to the complexity of video temporal dynamics, a naive frame-by-frame colorization strategy or the direct application of existing advanced image colorization models (such as image colorization based on diffusion models) often leads to severe inter-frame inconsistencies, producing highly visually disturbing flickering artifacts in semantic regions such as the sky and clothing.

[0005] Secondly, the post-processing paradigm of "coloring first and smoothing later" relies heavily on temporal smoothing algorithms to force alignment of independently colored frames. This approach usually leads to a significant reduction in color saturation and loss of high-frequency chromaticity details, and is prone to introducing new structural artifacts in complex scenes.

[0006] Third, the joint cross-frame modeling paradigm attempts to complete coloring and cross-frame feature interaction simultaneously within a single generative model. Such methods not only lead to extremely high computational overhead, but also, when faced with dynamic videos with fast motion, occlusion, or scene switching, they are prone to content feature leakage and color drift in long video sequences due to weak spatiotemporal correspondence and insufficient object-level decoupling.

[0007] To address these issues, this application proposes a video colorization model construction method, video colorization method, device, and medium that can effectively decouple single-frame content color editing from long-sequence temporal propagation. Furthermore, by integrating advanced image color priors with the dynamic smoothing characteristics of video generation models, it eliminates long-term color drift and combined weight interference while ensuring accurate temporal propagation of high-fidelity colors, thereby solving the aforementioned technical problems. Summary of the Invention

[0008] The main objective of this invention is to provide a video coloring model construction method and a video coloring method to solve the technical problems of severe inter-frame flicker, long sequence color drift and poor generation flexibility in video coloring tasks mentioned in the background technology, so as to achieve effective decoupling between single-frame content color editing and long sequence temporal color propagation, and improve the color fidelity and dynamic temporal consistency of the generated video.

[0009] The present invention solves the above-mentioned technical problems by adopting the following technical solutions:

[0010] A video colorization model construction method and a video colorization method, executed by a computer device, include the following steps:

[0011] Step S1. Obtain the grayscale video sequence to be processed, and sample keyframes from the grayscale video sequence to provide a spatial structure basis;

[0012] Step S2. Extract scene description text of keyframes using existing visual language models, and input keyframes and scene description text into existing image coloring models for semantic-aware coloring to generate high-fidelity color reference images, which are used as global style anchors;

[0013] Step S3. Construct a basic video generation model based on a diffusion transformer, and introduce a dual-branch architecture into the basic video generation model. The dual-branch architecture includes a main branch with frozen parameters and a control branch with trainable parameters.

[0014] Step S4. Concatenate the color reference image and the grayscale video sequence along the temporal dimension to construct a composite visual sequence. The color reference image serves as a reference frame providing color conditions, and the temporal position corresponding to the grayscale video sequence serves as the target generation frame for color generation. A temporal mask sequence is set to distinguish the reference frame from the target generation frame. The composite visual sequence, the temporal mask sequence, and the scene description text are input into the control branch to extract spatiotemporal control features. The spatiotemporal control features are then injected into the main branch in a residual superposition manner to guide the main branch to perform color temporal propagation while preserving the original motion dynamics.

[0015] Step S5. Train the control branch and introduce an alignment loss function based on teacher-student feature distillation during the training process. Use the frozen teacher network to extract deep target semantic features from the color reference image. At the same time, use a trainable feature projector to map the intermediate features in the main branch denoising process to the corresponding semantic space to obtain generated features. Global semantic consistency is achieved by constraining the similarity between the generated features and the deep target semantic features to alleviate the color drift problem in the generation of long sequence videos.

[0016] Step S6. Set the stream matching loss function to jointly construct the overall training objective with the alignment loss function. The stream matching loss function is used to ensure high-fidelity video generation, and the alignment loss function is used to ensure the consistency of long program sequences. By minimizing the overall training objective, the parameters of the control branch and feature projector are iteratively optimized to finally obtain the trained video colorization model.

[0017] Preferably, step S4 specifically includes:

[0018] Step S41. Concatenate the color reference image and the grayscale video sequence along the time dimension to generate a composite visual sequence with a uniform format. The specific concatenation formula is as follows:

[0019]

[0020] in, The composite visual sequence, The color reference image, The grayscale video sequence, This indicates a splicing operation along the time dimension;

[0021] Step S42. Construct a binary mask sequence that is strictly aligned with the composite visual sequence in the time dimension as the temporal mask sequence. The specific construction formula is as follows:

[0022]

[0023] in, The time-series mask sequence, The all-zero mask assigned to the color reference image, which matches the dimensions of the color reference image, indicates that it is an inactive frame that provides invariant color and style constraints. A one-dimensional mask, matched to the dimensions of the grayscale video sequence and assigned to the grayscale video sequence, indicates that it is the target active frame for which color needs to be generated;

[0024] Step S43. The composite visual sequence With the timing mask sequence After encoding into the latent space, the data is concatenated along the channel dimension and combined with the scene description text to construct a unified video conditional unit, which is then input into the control branch. The specific formula for the unified video conditional unit is as follows:

[0025]

[0026] in, This is a unified video condition unit input to the control branch. The text prompt words corresponding to the scene description text;

[0027] Step S44. The control latent variables in the input are processed by the control branch to extract spatiotemporal control features, and the feature weights are adjusted using a zero convolutional layer connected to the output of the control branch network layer. The initial weights of the zero convolutional layer are set to zero to avoid noise interference from the control branch to the main branch in the early stage of model training. Subsequently, the spatiotemporal control features adjusted by the zero convolutional layer are added to the intermediate hidden states of the corresponding layers of the main branch to realize feature residual injection. The specific feature residual injection formula is as follows:

[0028]

[0029] in, This refers to the next level feature of the main branch after residual guidance. and These respectively represent the first branch in the main branch and the second branch in the control branch. The intermediate hidden states of each network layer This indicates the zero convolutional layer. This indicates the next level network module of the main branch.

[0030] Preferably, step S5 specifically includes:

[0031] Step S51. Use a pre-trained self-supervised visual model as a frozen teacher network to encode the features of the color reference image and extract its highly discriminative high-level semantic features as deep target semantic features.

[0032] Step S52. Perform video denoising diffusion operation. During the video denoising diffusion process, obtain the intermediate features output by the main branch at the preset shallow network layer. The shallow network layer is used to establish a strong semantic foundation so that the deep network can focus on generating high-frequency details and motion dynamics.

[0033] Step S53. A feature projector composed of lightweight neural network layers is used to perform dimensionality transformation and spatial mapping on intermediate features in the noisy latent space, and bridge them to a clean target semantic space to obtain generated features.

[0034] Step S54. Calculate the mean negative cosine similarity between each feature patch in the generated features and the corresponding feature patch in the deep target semantic features, and use it as an alignment loss function to constrain the feature distribution of the generated video to fit the semantic space of the reference image. The gradient generated by the alignment loss function is backpropagated to update the parameters of the control branch, and the control branch is forced to generate a control signal that can guide the main branch to maintain global semantic consistency, thereby effectively reducing the color drift of long-term generation.

[0035] Preferably, step S6 specifically includes:

[0036] Step S61. Based on the corrected flow method, under a specified time step, Gaussian noise and input condition sequence, use the video generation model to predict the predicted velocity field corresponding to the noisy latent variable;

[0037] Step S62. Calculate the mean square error between the predicted velocity field and the true target velocity field, and use it as the flow matching loss function to constrain the model to learn the correct denoising trajectory from the noise prior distribution to the true color video data distribution;

[0038] Step S63. Construct the overall training objective by jointly using the flow matching loss function and the alignment loss function, and apply the minimization constraint to iteratively train the parameters of the control branch and feature projector in the video colorization model.

[0039] Preferably, the formula for calculating the total training objective is as follows:

[0040]

[0041] in, For the overall training Target , For flow matching loss function, To align the loss function, To balance the hyperparameters for controlling the intensity of deep semantic guidance;

[0042] Meanwhile, during the iterative optimization of the overall training objective, the main branch, visual encoder, and text encoder of the video generation model remain frozen, and the weight parameters of the control branch and the feature projector are updated only through gradient backpropagation.

[0043] Preferably, the method further includes establishing a set of text-guided video colorization evaluation prompts, specifically including:

[0044] The evaluation dimensions for video colorization are broken down into two dimensions: spatial color fidelity and temporal dynamic consistency.

[0045] The spatial color fidelity is further subdivided into specific instance color control and global scene atmosphere rendering; the temporal dynamic consistency is further subdivided into static occlusion recovery and dynamic object color tracking.

[0046] Based on the above dimensions, a set of test prompt words containing different text descriptions and motion scenarios is constructed. The set of test prompt words is used for phased verification during model training and for comprehensive evaluation of the target color propagation accuracy and long sequence color drift suppression capability after model training is completed.

[0047] On the other hand, the present invention also discloses a video colorization method, comprising:

[0048] Receive the target grayscale video sequence to be processed and target cue words used to describe the colors of the target scene;

[0049] Sample representative keyframes from the target grayscale video sequence, and input the representative keyframes and target cue words into the image generation tool or coloring tool to generate a high-fidelity color reference image with a specified color style, which serves as a global style anchor point;

[0050] A color reference image and a target grayscale video sequence are concatenated along the temporal dimension to construct a composite visual sequence, and a temporal mask sequence that is strictly aligned with the composite visual sequence is generated, wherein an all-zero mask is assigned to the color reference image and an all-one mask is assigned to the target grayscale video sequence.

[0051] The composite visual sequence, temporal mask sequence, and target cue words are input into the video colorization model trained by any of the above-described video colorization model construction methods. The spatiotemporal control features extracted by the control branch guide the frozen main branch with residuals, and finally generate a target color video that accurately reproduces the reference color in the spatial dimension and maintains strict temporal consistency in the temporal dimension. The target color video is free from inter-frame flicker and long-sequence color drift interference.

[0052] Preferably, in the process of residual guidance of the frozen main branch by the spatiotemporal control features extracted from the control branch in the video colorization model, a feature residual injection process is also set up, wherein the specific calculation formula for feature residual injection is:

[0053]

[0054] in, and These represent the first and second branches in the main branch and the control branch, respectively. The intermediate hidden states of each network layer This represents a zero convolutional layer used to adjust the feature injection weights. This indicates the next level network module in the main branch;

[0055] By using zero convolutional layers, the spatiotemporal control features, including reference color and grayscale structure, extracted from the control branch are directly added to the data stream of the main branch, enabling the model to achieve accurate color temporal propagation without changing its basic denoising capabilities.

[0056] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0057] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0058] As can be seen from the above technical solution, the present invention provides a method for constructing a video colorization model and a video colorization method. Compared with the prior art, the present invention has the following advantages:

[0059] 1. This invention constructs a two-stage framework that operates independently, effectively decoupling single-frame high-fidelity color generation from long-sequence video color propagation. This achieves decoupling of high fidelity and high consistency, thereby overcoming the technical bottleneck of traditional methods that cannot simultaneously achieve both spatial color editing precision and temporal dynamic continuity.

[0060] 2. This invention introduces a dual-branch diffusion architecture based on main branch freezing and control branch fine-tuning, and adopts a temporal masking strategy at the input end. This enables the model to accurately propagate reference colors while strictly following the motion dynamics of the original grayscale video, thereby avoiding the color saturation reduction and structural damage caused by post-processing methods, and ultimately eliminating inter-frame flicker and structural artifacts.

[0061] 3. This invention constructs a representation alignment mechanism based on teacher-student network feature distillation, constrains the alignment of early features of the main branch with deep semantic features of the reference image, and forces the model to maintain global semantic consistency in long-term generation. This can completely solve the color leakage and drift problem in dynamic occlusion or complex motion scenes, thereby effectively suppressing the technical problem of long-sequence color drift.

[0062] 4. In the first stage, this invention allows for flexible integration with any existing visual language model and image generation tool through semantic-aware coloring. This enables users to perform highly customized local or global color control through text prompts, thereby significantly reducing the development cost of video-specific coloring models and greatly expanding the application scenarios in historical image restoration and film and television production, ultimately improving the model's scalability and generation flexibility.

[0063] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description

[0064] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0065] Figure 1 This is a schematic diagram of the video colorization model training process of the present invention;

[0066] Figure 2 This is a schematic diagram illustrating the video colorization model training method of the present invention;

[0067] Figure 3 This is a diagram showing the generated video effect of the present invention;

[0068] Figure 4 This is a comparison chart showing the effect of the video generated by this invention with the video generated by various traditional methods. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] For details in the embodiments, please refer to Figures 1 to 4 .

[0071] like Figure 1 and Figure 2 As shown, the video colorization model construction method proposed in this embodiment of the invention is based on a diffusion model (such as the Wan-T2V-1.3B model) and implemented through a two-stage video colorization framework, specifically including the following steps:

[0072] S1. Obtain the grayscale video sequence to be processed, and sample keyframes from the grayscale video sequence. The keyframes are used to provide the basis for spatial structure.

[0073] In the specific implementation, the input set of grayscale video data to be colored is obtained. To resolve the semantic ambiguity inherent in grayscale videos, the most representative frames are first extracted using a keyframe sampling module. (Usually the first frame is selected) This serves as the foundation of the spatial structure. The purpose here is to provide a high-fidelity spatial reference for subsequent color transmission.

[0074] S2. In the first stage, the scene description text of the keyframes is extracted using a visual language model, and the keyframes and scene description text are input into the image coloring model for semantic-aware coloring to generate a high-fidelity color reference image, which is used as a global style anchor.

[0075] In the first stage, semantic-aware colorization is set up to flexibly integrate any existing visual language model and image generation tool. This allows users to perform highly customized local or global color control through text prompts, thereby significantly reducing the development cost of video-specific colorization models and greatly expanding the application scenarios in historical image restoration and film and television production, ultimately improving the model's scalability and generation flexibility.

[0076] In its implementation, unlike traditional editing tasks that heavily rely on manual input prompts from external users, this invention constructs an automatic derivation mechanism of "description first, then coloring." First, a pre-trained visual language model (e.g., the Qwen2.5-VL model) is used to analyze grayscale keyframes. Automatically generate descriptive caption text that accurately reflects the semantics of the scene. This process can be represented as:

[0077]

[0078] in It describes the semantic objects and potential colors in the scene (e.g., "a yellow double-decker bus"), providing key textual constraints and color guidance for coloring.

[0079] Then, grayscale keyframes and the generated description text Both inputs are fed into a text-aware image colorization model. This colorization model generates realistic and reasonable colors while strictly preserving the grayscale input space structure. The generation process can be represented as follows:

[0080]

[0081] in This represents the final generated high-fidelity color reference image. This solves the color uncertainty of a single grayscale input and serves as a global style anchor for the subsequent temporal propagation stage of the video sequence, ensuring that the entire generated color video sequence strictly adheres to the aesthetic tone established by this keyframe.

[0082] S3. Construct a basic video generation model based on the diffusion model, and introduce a dual-branch architecture into the basic video generation model. The dual-branch architecture includes a main branch with frozen parameters and a control branch with trainable parameters.

[0083] In the specific implementation, pre-trained text is loaded into a large video generation model (e.g., the Wan-T2V model) as the main branch. To achieve accurate propagation of the target color without compromising the original high-quality video generation capabilities and temporal smoothness of the pre-trained model, this invention employs a context adapter fine-tuning strategy. A control branch corresponding to the structure is constructed in parallel, and this control branch uses low-rank adaptive techniques for parameter initialization and fine-tuning, while all parameters of the main branch remain completely frozen.

[0084] S4. In the second stage, the color reference image and the grayscale video sequence are stitched together along the temporal dimension to construct a composite visual sequence, and a temporal mask sequence is set to distinguish the reference frame from the target generated frame. The composite visual sequence, the temporal mask sequence and the text conditions are input into the control branch to extract the spatiotemporal control features, and the spatiotemporal control features are injected into the main branch in the form of residual superposition to guide the main branch to perform color temporal propagation while preserving the original motion dynamics.

[0085] Furthermore, the specific execution process of this step includes:

[0086] Color reference images and grayscale video sequences are spliced ​​together in the time dimension to generate a composite visual sequence with a unified format;

[0087] A binary mask sequence that is strictly aligned with the composite visual sequence in the temporal dimension is constructed as a temporal mask sequence; wherein, a zero mask is assigned to the color reference image to indicate that it is an inactive frame that provides invariant color and style constraints; and a one mask is assigned to the grayscale video sequence to indicate that it is the target active frame for which color needs to be generated.

[0088] The composite visual sequence and the temporal mask sequence are encoded into the latent space and then concatenated in the channel dimension, which is used as the input to the control branch as a control latent variable;

[0089] After the control branch processes the latent variables and extracts the spatiotemporal control features, it adjusts the feature weights through a zero convolutional layer and adds the adjusted spatiotemporal control features to the intermediate hidden states of the corresponding level in the main branch. This achieves residual injection of the spatiotemporal control features into the main branch, enabling the model to maintain the dynamics of the pre-trained video in the main branch while strictly adhering to the grayscale structure and propagating the reference color.

[0090] In this implementation, a unified video conditional unit is constructed to explicitly align structural priors and color references. The input condition is defined as containing text prompts. Composite visual sequences Time mask sequence tuple .make Represented by color reference image With grayscale video sequences A composite visual sequence formed by splicing along the temporal dimension, namely Timing mask sequence The instructions explicitly state that the model should utilize the appearance of the zero-mask region to guide the generation of the one-mask region. After the control branch extracts the spatiotemporal control features, the feature injection process can be represented as:

[0091]

[0092] in, and These represent the first and second branches in the main branch and the control branch, respectively. Intermediate features of each network layer This represents a zero-convolutional layer. By directly adding control features to the main data stream, the control branch effectively acts as a dynamic context adapter.

[0093] S5. During the training of the control branch, a representation alignment loss function based on teacher-student network feature distillation is introduced. The frozen teacher network is used to extract deep target semantic features from the color reference image. A trainable feature projector is used to map the intermediate features of the main branch in the denoising process to the corresponding semantic space to obtain generated features. Global semantic consistency is achieved by constraining the similarity between the generated features and the deep target semantic features, so as to alleviate the color drift problem in the generation of long sequence videos.

[0094] At this point, aligning the early features of the main branch with the deep semantic features of the reference image forces the model to maintain global semantic consistency during long-term generation. This can completely solve the color leakage and drift problem in dynamic occlusion or complex motion scenes, thereby effectively suppressing the technical problem of long-sequence color drift.

[0095] Furthermore, the specific execution flow for generating features in this step includes:

[0096] A pre-trained self-supervised visual model is used as a frozen teacher network to encode features of a color reference image and extract its highly discriminative high-level semantic features as deep target semantic features.

[0097] During the video denoising diffusion process, the intermediate features of the main branch output at the preset shallow network layers are obtained; the shallow network layers are used to establish a strong semantic foundation so that the deep network can focus on generating high-frequency details and motion dynamics.

[0098] By using a feature projector composed of lightweight neural network layers, intermediate features in the noisy latent space are transformed and spatially mapped to a clean target semantic space to obtain generated features.

[0099] The mean negative cosine similarity between each feature patch in the generated features and the corresponding feature patch in the deep target semantic features is calculated as the representation alignment loss function. The gradient generated by the representation alignment loss function is backpropagated to update the parameters of the control branch, and the control branch is forced to generate a control signal that can guide the main branch to maintain global semantic consistency, thereby effectively reducing the color drift of long-term generation.

[0100] In practical implementations, while standard denoising objectives can ensure that the model learns to reconstruct local details, they often perform poorly in maintaining global semantic consistency and color fidelity in long video sequences, easily leading to color drift or semantic shift. To address this issue, this embodiment introduces a representation alignment mechanism. The core idea is to explicitly force high-level semantic alignment between generated features and reference image features using a powerful pre-trained visual encoder. First, a frozen teacher encoder is used to extract robust semantic representations of the color reference image, and the target semantic features are represented as follows: Simultaneously, during the denoising process, the first branch of the main branch is extracted. The intermediate hidden state output by each diffusion converter module To bridge the dimensionality and semantic differences between the noisy latent space and the clean semantic space, a lightweight, trainable feature projector is introduced. The intermediate features of the noise are mapped to the target semantic space to obtain the projected generated features. .

[0101] Subsequently, the negative cosine similarity between the generated features and the reference features is calculated as the representation alignment loss function:

[0102]

[0103] in, Indicates the number of feature map patches. This indicates the calculation of cosine similarity. This embodiment specifically applies alignment constraints to the shallow network of the main branch because aligning shallow features is sufficient to establish a strong semantic foundation, thereby allowing the deep network to focus on synthesizing high-frequency details and motion dynamics.

[0104] S6. The joint flow matching loss function and the representation alignment loss function are used to construct the overall training objective. By minimizing the overall training objective, the parameters of the control branch and the feature projector are iteratively optimized to obtain the trained video colorization model.

[0105] Furthermore, the specific execution process of this step includes:

[0106] Based on the corrected flow method, under a given time step, Gaussian noise and input condition sequence, the video generation model is used to predict the predicted velocity field corresponding to the noisy latent variable.

[0107] The mean square error between the predicted velocity field and the true target velocity field is calculated and used as the flow matching loss function to constrain the model to learn the correct denoising trajectory from the noise prior distribution to the true color video data distribution.

[0108] Obtain deep target semantic features from a color reference image extracted by a pre-trained teacher network, as well as generated features obtained by feature projector mapping;

[0109] Calculate the cosine similarity between the corresponding patches of the generated features and the deep target semantic features on all feature patches, and use the negative mean as the representation alignment loss function to constrain the feature distribution of the generated video to fit the semantic space of the reference image.

[0110] In this implementation, the overall training objective is to ensure high-fidelity video generation and strict semantic adherence to reference keyframes. Given input conditions... The flow matching loss is defined as the difference between the predicted velocity field and the actual velocity field. Mean square error between:

[0111]

[0112] in This indicates the freeze parameter for the main branch. This represents the trainable parameters of the control branch.

[0113] Combining the flow matching loss with the representation alignment loss, the final optimization objective formula is:

[0114]

[0115] in This is a balancing hyperparameter that controls the semantic guidance strength of the reference image. In the optimization strategy, the pre-trained main branch, variational autoencoder, text encoder, and teacher encoder are all kept frozen. The optimization process is only performed on the parameters of the control branch and the feature projector. By minimizing the overall training objective, the control branch can learn to inject precise spatiotemporal features, guiding the main branch to synthesize a target video that is structurally consistent with the grayscale input and color-aligned with the reference image.

[0116] The above-mentioned dual-branch diffusion architecture based on main branch freezing and control branch fine-tuning, and the use of a temporal masking strategy at the input end, enables the model to accurately propagate reference colors while strictly following the motion dynamics of the original grayscale video, thereby avoiding the color saturation reduction and structural damage caused by post-processing methods, and ultimately eliminating inter-frame flicker and structural artifacts.

[0117] In a specific example, the video colorization model construction method also includes: establishing a set of text-guided video colorization evaluation prompts, the specific process of which includes:

[0118] The evaluation dimensions for video colorization are broken down into two dimensions: spatial color fidelity and temporal dynamic consistency.

[0119] Spatial color fidelity is further subdivided into specific instance color control and global scene atmosphere rendering, and temporal dynamic consistency is further subdivided into static occlusion recovery and dynamic object color tracking.

[0120] Based on the above dimensions, a set of test prompt words containing different text descriptions and motion scenarios is constructed. The set of test prompt words is used for phased verification during model training, and for comprehensive evaluation of the target color propagation accuracy and long sequence color drift suppression ability after model training is completed.

[0121] In the specific implementation, the evaluation task is first decomposed into two independent dimensions based on core capabilities: spatial and temporal. The spatial dimension focuses on the accurate coloring of specific objects and the rendering of overall ambient lighting; the temporal dimension focuses on the color retention capability when an object reappears after being occluded, and the color attachment stability of objects during rapid movement. Based on this classification system, test prompt word sets of different complexities are constructed for bidirectional evaluation. During training, the basic combined task is periodically called to compare the feature distribution of the generated video with the reference features, verifying the decoupling stability in real time. After training, test videos of complex motion scenes are called to evaluate the generalization ability, ensuring that the model has stable long-sequence consistent generation performance.

[0122] On the other hand, the present invention also discloses a video colorization method, comprising:

[0123] L1. Receives the target grayscale video sequence to be processed, as well as target prompts describing the colors of the target scene.

[0124] In the specific implementation, it receives a target grayscale video sequence to be colored uploaded by the user (such as a black and white historical video of "an old steam train running"), and receives target prompts automatically generated by the visual language model or user-defined, such as: "A red vintage passenger train is running on a tree-lined track under a clear blue sky."

[0125] L2. Sample representative keyframes from the target grayscale video sequence, and input the representative keyframes and target cue words into the image generation or coloring tool to generate a high-fidelity color reference image with a specified color style, which serves as a global style anchor point.

[0126] In its implementation, the system automatically extracts the first frame of the video as a representative keyframe. This keyframe, along with the target cues, is then input into an image coloring tool based on a pre-trained diffusion model. Using semantic cues such as "red," "green trees," and "azure," the coloring tool accurately generates a high-fidelity color reference image while maintaining the original image structure, thus establishing the overall color tone and atmosphere of the video.

[0127] L3. The color reference image and the target grayscale video sequence are concatenated along the temporal dimension to construct a composite visual sequence, and a temporal mask sequence that is strictly aligned with the composite visual sequence is generated; wherein an all-zero mask is assigned to the color reference image and an all-one mask is assigned to the target grayscale video sequence.

[0128] In the specific implementation, the generated single color reference image is aligned and stitched with subsequent grayscale video frames in the time dimension to form a unified composite input stream. At the same time, a mask sequence is constructed, assigning a value of 0 to the reference image to indicate that its color is a known static condition; and assigning a value of 1 to the grayscale video frame to indicate that it is a dynamic target for which color needs to be generated.

[0129] L4. Input the composite visual sequence, temporal mask sequence, and target cue words into the video colorization model trained by the above method. Use the spatiotemporal control features extracted from the control branch in the video colorization model to guide the frozen main branch with residuals, and generate a target color video that accurately reproduces the reference color in the spatial dimension and maintains strict temporal consistency in the temporal dimension. The target color video is free from inter-frame flicker and long-sequence color drift interference.

[0130] Furthermore, the frozen main branch is guided by residuals through the spatiotemporal control features extracted from the control branch in the video colorization model, including a feature residual injection process. The specific calculation formula is as follows:

[0131]

[0132] in, and These represent the first and second branches in the main branch and the control branch, respectively. The intermediate hidden states of each network layer This represents a zero convolutional layer used to adjust the feature injection weights. This represents the next level network module of the main branch; by using zero convolutional layers, the spatiotemporal control features extracted from the control branch, including reference color and grayscale structure, are directly added to the data stream of the main branch, enabling the model to achieve accurate color temporal propagation without changing its basic denoising capabilities.

[0133] In this implementation, the constructed composite sequence and mask are fed into a dual-branch model for the denoising inference process. The final output video will strictly maintain the original train's dynamic movement, and the colors of the red paint on the train body and the green trees on the tracks will smoothly transition with the movement of the viewpoint and changes in lighting, avoiding the feature interference problems of the train body color suddenly turning blue or flickering as seen in traditional methods. The final generated video effect is as follows. Figure 3 As shown;

[0134] In summary, this colorization method effectively decouples single-frame high-fidelity color generation from long-sequence video color propagation by constructing a two-stage framework that operates independently. This decoupling achieves high fidelity and high consistency, thus overcoming the technical bottleneck of traditional methods that cannot simultaneously balance spatial color editing precision and temporal dynamic continuity.

[0135] Furthermore, based on the methods of the above embodiments, the results are compared with those of existing technologies. Table 1 below shows various evaluations of the video colorization results of the methods used in the above embodiments and traditional methods, and the final display results are compared as follows: Figure 4 As shown.

[0136] Table 1: Comparison of Results between This Method and Traditional Methods

[0137]

[0138] It can be seen that this invention, through the combination of "divide and conquer two-stage architecture, two-branch residual guidance, and teacher-student network representation alignment", not only effectively solves the core pain points of existing video colorization technology, such as long sequence color drift, severe inter-frame flicker and structural damage, but also achieves significant improvement in technical effects in dimensions such as perceptual quality (PSNR, SSIM, LPIPS), subject consistency and picture aesthetic quality. It is a video colorization generation technology with substantial characteristics and significant progress.

[0139] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is a GPU, an NPU, or a CPU. When the computer program is executed by the processor, the processor performs the steps of the above method to complete the construction and iterative training of a dual-branch video diffusion model, the joint calculation of the stream matching and representation alignment loss function, the residual injection of spatiotemporal control features, and the text semantic-guided video colorization inference.

[0140] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor performs the steps of the method described above. The computer program includes a data preprocessing and coloring module, a dual-branch model construction module, a representation alignment and joint optimization training module, and a video temporal color propagation inference module. Each module works together to achieve spatiotemporal decoupling of content semantic editing and temporal dynamic propagation, thereby completing high-fidelity video coloring generation without inter-frame flicker.

[0141] Furthermore, it should be noted that when the processor executes the computer program, it performs the following functions: constructs a diffusion transformer architecture containing a main branch and a control branch; controls the training process, freezes the main branch, and fine-tunes the control branch using flow matching and representation alignment loss; forces the generated features to align with the high-level semantic features of the reference image extracted by the teacher network during backpropagation; and in the inference phase, performs highly consistent long-sequence video colorization inference based on the user-input text and reference image, using a dual-branch residual injection mechanism and a masking strategy.

[0142] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the video coloring model construction methods and video coloring methods in the above embodiments.

[0143] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0144] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus.

[0145] Memory, used to store computer programs;

[0146] The processor, when executing the program stored in memory, implements the above-mentioned video colorization model construction method and video colorization method.

[0147] The communication bus mentioned in the above-mentioned electronic devices can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0148] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0149] The memory may include random access memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0150] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0151] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0153] Furthermore, it should be noted that if any directional indication (such as up, down, left, right, front, back, etc.) is involved in the embodiments of the present invention, the directional indication is only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0154] Furthermore, those skilled in the art should understand that in the actual use of the embodiments of this application, there may be preset thresholds used as the basis for judging the corresponding technical solutions. These thresholds are conventional technical means commonly used in the field to implement functions such as state judgment, condition recognition, and control logic switching. The specific values, setting basis, value selection methods, determination methods, and adjustment rules of the thresholds involved in this technical solution are all conventional technical choices that can be reasonably determined by those skilled in the art based on conventional technical factors such as actual application scenarios, system working states, characteristics of the detection object, hardware performance parameters, and functional requirements, through conventional experiments, calibrations, and debugging. The specific setting and adjustment of the aforementioned thresholds will not cause this technical solution to be unimplementable as a whole, nor will it affect the realization of the core concept and the achievement of the technical effects of this technical solution.

[0155] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, in the embodiments of this invention, "multiple" refers to two or more. Moreover, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A method for constructing a video colorization model, characterized in that, include: Step S1. Obtain the grayscale video sequence to be processed and sample keyframes; Step S2. Extract the scene description text of the keyframes, and perform semantic-aware colorization based on the keyframes and scene description text to generate a high-fidelity color reference image; Step S3. Construct a basic video generation model based on a diffusion transformer, and introduce a main branch with frozen parameters and a control branch with trainable parameters into the basic video generation model; Step S4. The color reference image and the grayscale video sequence are spliced ​​along the temporal dimension to construct a composite visual sequence. The color reference image is used as a reference frame, and the temporal position corresponding to the grayscale video sequence is used as the target generated frame. A temporal mask sequence is set to distinguish the reference frame from the target generated frame. The composite visual sequence, the temporal mask sequence, and the scene description text are input into the control branch to extract the spatiotemporal control features and inject them into the main branch in a residual superposition manner to guide the main branch to perform color temporal propagation while preserving the original motion dynamics. Step S5. Train the control branch and introduce an alignment loss function based on teacher-student feature distillation during the training process. Use the frozen teacher network to extract deep target semantic features from the color reference image. At the same time, use a trainable feature projector to map the intermediate features in the main branch denoising process to the corresponding semantic space to obtain generated features. Step S6. Set the stream matching loss function to jointly align the loss function to construct the overall training objective. By minimizing the overall training objective, iteratively optimize the parameters of the control branch and the feature projector to finally obtain the trained video colorization model.

2. The video colorization model construction method as described in claim 1, characterized in that, Step S4 specifically includes: Step S41. The color reference image and grayscale video sequence are spliced ​​together in the time dimension to generate a composite visual sequence with a uniform format; Step S41. Concatenate the color reference image and the grayscale video sequence along the time dimension to generate a composite visual sequence with a uniform format. The specific concatenation formula is as follows: in, The composite visual sequence, The color reference image, The grayscale video sequence, This indicates a splicing operation along the time dimension; Step S42. Construct a binary mask sequence that is strictly aligned with the composite visual sequence in the time dimension as the temporal mask sequence. The specific construction formula is as follows: in, The time-series mask sequence, The all-zero mask assigned to the color reference image, which matches the dimensions of the color reference image, indicates that it is an inactive frame that provides invariant color and style constraints. A one-dimensional mask, matched to the dimensions of the grayscale video sequence and assigned to the grayscale video sequence, indicates that it is the target active frame for which color needs to be generated; Step S43. The composite visual sequence With the timing mask sequence After encoding into the latent space, the data is concatenated along the channel dimension and combined with the scene description text to construct a unified video conditional unit, which is then input into the control branch. The specific formula for the unified video conditional unit is as follows: in, This is a unified video condition unit input to the control branch. The text prompt words corresponding to the scene description text; Step S44. The latent control variables in the input are processed through the control branch to extract spatiotemporal control features, and the feature weights are adjusted using a zero-convolutional layer connected to the output of the control branch network layer, wherein the initial weights of the zero-convolutional layer are set to zero; subsequently, the spatiotemporal control features adjusted by the zero-convolutional layer are added to the intermediate hidden states of the corresponding level of the main branch to achieve feature residual injection. The specific feature residual injection formula is as follows: in, This refers to the next level feature of the main branch after residual guidance. and These respectively represent the first branch in the main branch and the second branch in the control branch. The intermediate hidden states of each network layer This indicates the zero convolutional layer. This indicates the next level network module of the main branch.

3. The video colorization model construction method as described in claim 2, characterized in that, Step S5 specifically includes: Step S51. Use a pre-trained self-supervised visual model as a frozen teacher network to encode the features of the color reference image and extract its highly discriminative high-level semantic features as deep target semantic features. Step S52. Perform video denoising and diffusion operation. During the video denoising and diffusion process, obtain the intermediate features output by the main branch at the preset shallow network layer. Step S53. A feature projector composed of lightweight neural network layers is used to perform dimensionality transformation and spatial mapping on intermediate features in the noisy latent space, and bridge them to a clean target semantic space to obtain generated features. Step S54. Calculate the mean negative cosine similarity between each feature patch in the generated features and the corresponding feature patch in the deep target semantic features, and use it as the alignment loss function.

4. The video colorization model construction method as described in claim 3, characterized in that, Step S6 specifically includes: Step S61. Based on the corrected flow method, under a specified time step, Gaussian noise and input condition sequence, use the video generation model to predict the predicted velocity field corresponding to the noisy latent variable; Step S62. Calculate the mean square error between the predicted velocity field and the actual target velocity field, and use it as the flow matching loss function; Step S63. Construct the overall training objective by jointly using the flow matching loss function and the alignment loss function, and apply the minimization constraint to iteratively train the parameters of the control branch and feature projector in the video colorization model.

5. The video colorization model construction method as described in claim 4, characterized in that, The formula for calculating the overall training objective is as follows: in, For the overall training Target , For flow matching loss function, To align the loss function, To control the balance hyperparameters for the intensity of deep semantic guidance.

6. The video colorization model construction method as described in claim 1, characterized in that, The method also includes establishing a set of text-guided video colorization evaluation prompts, specifically including: The evaluation dimensions for video colorization are broken down into two dimensions: spatial color fidelity and temporal dynamic consistency. The spatial color fidelity is further subdivided into preset instance color control and global scene atmosphere rendering; the temporal dynamic consistency is further subdivided into static occlusion recovery and dynamic object color tracking. Based on the above dimensions, a set of test prompt words containing different text descriptions and motion scenarios is constructed. The set of test prompt words is used for phased verification during model training and for comprehensive evaluation of the target color propagation accuracy and long sequence color drift suppression capability after model training is completed.

7. A video colorization method, characterized in that, Applied to a video colorization model constructed by any one of claims 1 to 6, comprising: Receive the target grayscale video sequence to be processed and target cue words used to describe the colors of the target scene; Sample representative keyframes from the target grayscale video sequence, and input the representative keyframes and target cue words into the image generation tool or coloring tool to generate a high-fidelity color reference image with a specified color style, which serves as a global style anchor point; A color reference image and a target grayscale video sequence are concatenated along the temporal dimension to construct a composite visual sequence, and a temporal mask sequence that is strictly aligned with the composite visual sequence is generated, wherein an all-zero mask is assigned to the color reference image and an all-one mask is assigned to the target grayscale video sequence. The composite visual sequence, temporal mask sequence, and target cue words are input into the video colorization model. The spatiotemporal control features extracted by the control branch guide the frozen main branch with residuals, and finally generate a target color video that accurately reproduces the reference color in the spatial dimension and maintains strict temporal consistency in the temporal dimension.

8. The video colorization method as described in claim 7, characterized in that, In the video colorization model, during the process of residual guidance of the frozen main branch by the spatiotemporal control features extracted from the control branch, a feature residual injection process is also set up, wherein the specific calculation formula for feature residual injection is as follows: in, and These represent the first and second branches in the main branch and the control branch, respectively. The intermediate hidden states of each network layer This represents a zero convolutional layer used to adjust the feature injection weights. This indicates the next level network module in the main branch.

9. A computer-readable storage medium, characterized in that, The device stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

Images