Text-guided creative and time consistency-oriented video coloring method and system

Abstract

The invention relates to the technical field of computer vision, in particular to a text-guided creativity and time consistency-oriented video coloring method and system. By introducing a time deformable attention block, according to the position and shape change dynamic state of an object in a video frame, dynamic features of an instance are captured in the time dimension, so that feature representation consistency is kept, and color flickering and shifting in the coloring process are effectively prevented; semantic representation of a text noun concept is adjusted through a cross-modal pre-fusion module, mask cross attention in the module is used for enhancing understanding of the model on the noun concept and instance perception, and color distribution accuracy is improved; the global structure of the colored video is maintained through gray level video information introduced in the video decoding process; a cross-fragment fusion mechanism is introduced during reasoning, so that the long-term consistency of video coloring is maintained; through text guidance, the user can color the video according to own demands and creativity.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a text-guided video colorization method and system that is creatively and temporally consistent. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, especially diffusion models in the field of computer vision, the automated processing capabilities of video content have been significantly improved. Diffusion models initially achieved great success in image generation, enabling the creation of high-quality, diverse image content. With technological advancements, the application of these models has expanded to the video field, making video generation and editing more intelligent and automated.

[0003] Against this backdrop, video colorization, which converts monochrome or grayscale video into color video, has become an important research direction. Video colorization can not only breathe new life into historical images but also enhance the visual appeal of modern video content and improve the viewing experience.

[0004] Despite its significant application value, video coloring technology faces several key challenges. First, video coloring models need to assign appropriate colors to each object in the video while maintaining the realism of the content. This requires the model to not only understand the semantic information of the video content but also to be able to reasonably infer colors based on the context. Second, video coloring also needs to address the temporal consistency issue between frames, ensuring that colors remain consistent throughout the video playback and avoiding flickering or color shifts.

[0005] To address these challenges, some researchers have employed automatic coloring to infer appropriate colors from monochromatic semantic cues; others have treated the coloring task as video translation, combining language-based image coloring methods and post-processing algorithms to resolve video flickering or color shifts.

[0006] like Figure 1 As shown, one of the existing technologies, "Temporally Consistent Video Colorization Based on Deep Feature Propagation and Self-Regularization Learning", proposes the TCVC framework, which realizes automatic colorization with temporal consistency by constructing a deep feature network.

[0007] First, the input monochrome video sequence is divided into several intervals, each containing several consecutive frames. The start and end frames are selected as anchor frames, and the remaining frames are used as internal frames. The depth features of the anchor frames are extracted using a pre-trained image coloring model. Then, the feature information of the anchor frames is propagated to all other frames in the video sequence through a bidirectional feature propagation mechanism. Furthermore, a self-regularization learning scheme is introduced to enhance temporal consistency by minimizing the differences in prediction results at different time steps, thus avoiding reliance on color video data. Finally, a feature fusion module is used to fuse the features and convert the fused features into predicted color channels to achieve automatic colorization.

[0008] like Figure 2 As shown, in the second existing technology, "Realistic Video Colorization Achieved Through Gated Color-Guided Image Diffusion Model", GCG and CDGA modules are proposed. By using a pre-trained text-based image model, gated color guidance and sliding reference grid strategies are combined with two-stage training to complete language-guided video colorization.

[0009] First, a gated color guidance module is introduced. Through a gating mechanism, color information matching the structure is extracted when the reference frame is valid, and mismatched color information is filtered out, thereby achieving colorization of grayscale frames. At the same time, text prompts are used as global semantic guidance to help the model understand the video content and generate semantically consistent colorization results. Then, in the first training stage, selected keyframes are jointly colorized, and an extended attention mechanism is used to enable keyframes to share global information to ensure color consistency. Finally, in the second training stage, a sliding reference grid strategy is used to combine keyframes and colorized frames into a grid to provide color information for the remaining grayscale frames, improving colorization coherence.

[0010] Regarding existing technologies such as "time-consistent video colorization based on deep feature / propagation and self-regularization learning" and "realistic video colorization achieved through a gated color-guided image diffusion model":

[0011] 1. Automatic coloring requires no external guidance or control, and uses universal colors to ensure reasonable coloring results;

[0012] 2. A sliding reference grid is used to maintain color consistency, but long video instances are not considered.

[0013] The aforementioned technical problems result in the following drawbacks in the existing technology:

[0014] 1. Unable to personalize colors according to user needs;

[0015] 2. Instances in long videos still struggle to maintain color consistency over long periods. Summary of the Invention

[0016] To address the aforementioned technical problems, this invention provides a text-guided video coloring method and system that prioritizes creativity and temporal consistency. It improves upon existing technologies, enhances the accuracy of color allocation, and maintains the global structure of the colorized video.

[0017] To achieve the above objectives, the technical solution adopted by this invention is as follows: First, a text-guided video colorization method oriented towards creativity and temporal consistency is provided, comprising the following steps:

[0018] S1 Preprocessing:

[0019] Segment the text T in each text-to-color video pair (T,V) into words, and represent the video V of each text-to-color video pair (T,V) as color video frames [V1,V2,...,V]. n Simultaneously extract the corresponding black and white video frames. Where n is the number of frames;

[0020] S2 feature extraction:

[0021] The segmented text is encoded into a text feature tensor using a text encoder. The color video frames and black-and-white video frames are encoded separately to generate the color video frame feature z0 and the grayscale video frame feature tensor z. g Initialize time step t;

[0022] S3 generates a noisy latent representation:

[0023] At the current time step t, initialize a random Gaussian noise with the same shape as the color video frame feature z0 as the actual Gaussian noise ∈ t Based on the color video frame features z0 and the actual Gaussian noise ∈ t The corresponding noisy latent representation z is generated by weighted summation. t ;

[0024] S4 Denoising Network Training:

[0025] Noisy latent representation z t and grayscale video frame feature tensor z g After concatenation, the video feature representation h is obtained by inputting it into the denoising network U-Net, and then trained through spatial blocks, temporally deformable attention blocks, and cross attention blocks.

[0026] S4.1 Training Phase 1: Training the denoising network U-Net:

[0027] S4.1.1 In each spatial block, compute the self-attention of the video feature representation h;

[0028] S4.1.2 In each temporally deformable attention block, compute the updated video feature tensor.

[0029] S4.1.3 In each cross-attention block, compute the updated video feature tensor. With text feature tensor Cross attention;

[0030] In the second phase of training S4.2, only the cross-modal pre-fusion module is trained, and the weights of the denoising network U-Net are frozen.

[0031] Text feature tensor and grayscale video frame feature tensor z g Inputting a cross-modal prefusion module consisting of L cross-modal prefusion layers yields a new text feature tensor y. tex ;

[0032] S5 denoising network output:

[0033] The U-Net denoising network outputs predicted noise. Calculate prediction noise and actual Gaussian noise ∈ t The loss function is then optimized.

[0034] S6 fusion features:

[0035] Use a skip window to capture long-term color dependencies, selecting N with different resolutions. f Frame video segments, time interval d∈{1,2,4,…,N} d}, where N d It is the largest power of 2 result that does not exceed the total number of video frames. The distance-based weighted fusion operation performs cross-segment feature fusion on each frame that is close to the center of the segment.

[0036] S7 generates the denoised latent representation:

[0037] At time step t, the noisy latent representation z is used. t Subtract prediction noise Get z t-1 Repeat this step until time step t becomes 0 to obtain the denoised latent representation.

[0038] S8 video shading:

[0039] For the denoised latent representation Decode the video to obtain the colored video V.

[0040] Preferably, the text encoder encoding process described in step S2 is as follows:

[0041]

[0042] Where T is the text, Tokenizer(x) is the tokenizer, and TextEncoder(x) is the text encoder. It is the encoded text feature tensor;

[0043] The process of encoding the color video frame and the black and white video frame separately in step S2 is as follows:

[0044] z i =ε(V i ),

[0045]

[0046] Among them, V i It is a color video frame. These are black and white video frames, ε(x) is the encoder of the variational autoencoder, ε lum (x) is the brightness encoder, z i These are encoded color video frames. These are encoded black and white video frames.

[0047] Preferably, the noisy latent representation z in step S3 t The calculation is as follows:

[0048]

[0049] in, These are predefined time-step related parameters in the diffusion model.

[0050] Preferably, the updated video feature tensor described in step S4.1.2 The calculation process is as follows:

[0051] Based on manually preset time sampling rate and resolution sampling rate, the video feature representation h is uniformly sampled in both time and resolution dimensions, and the reference point sequence p is calculated:

[0052] p = UniformSample(h),

[0053] Where UniformSample(x) represents uniform sampling;

[0054] Through linear projection W o After projecting the video feature representation h, it is input into a lightweight 3D convolutional network δ. offset Predict the offset of each reference point on (x), and limit the result to the range [-1,1] using the tanh(x) function to calculate the estimated offset Δp:

[0055] Δp=tanh(δ offset (hW o )),

[0056] Among them, W o It is the projection matrix, δ offset (x) is a lightweight 3D convolutional network, and tanh(x) is the hyperbolic tangent function;

[0057] Based on the reference point sequence p and the estimated offset Δp, the estimated deformation points of the video feature representation h are weighted using the estimated deformation point weighting function φ(h,p), and the intermediate transformed result is calculated.

[0058]

[0059] Where α is a parameter, and φ(h,p) is the weighting function for estimating deformation points;

[0060] The weighted function φ(h,p) for estimating deformation points is calculated as follows:

[0061] g(a,b) = max(0,1-|ab|),

[0062]

[0063] Among them, (r x ,r y ,r t ) is the location index of the video feature representation h, (p x ,p y ,p t ) is the position index of the reference point sequence p;

[0064] Using projection matrix and Transformed intermediate results Projection as key Sum Using projection matrix Project the video feature representation h as the query q. i Use query q i ,key Sum Calculate multi-head attention and compute the updated video feature tensor.

[0065]

[0066] in, It is the projection matrix, q i , It is the projected attention query, key and value, i is the head number of multi-head attention, and MHA(x) is multi-head attention.

[0067] Preferably, the new text feature tensor y in step S4.2 tex The calculation is as follows:

[0068] First, the output of each layer is calculated. After L layers of calculation, the instance-aware text feature tensor is obtained. Output of layer l The calculation process is as follows:

[0069] right Perform multi-head self-attention calculation:

[0070]

[0071] in, It is the output of the previous layer, for When l is 1, it is the text feature tensor. When l is [2,3,...L], it represents the output of the previous cross-modal pre-fusion layer; LN(x) is layer normalization, and MSA(x) is multi-head self-attention;

[0072] calculate and grayscale video frame feature tensor z g Mask cross attention:

[0073]

[0074] Where M is a mask used to mask color-related words in the text, and MCA(x,y,M) is mask cross-attention;

[0075] Output of the current layer The calculation process is as follows:

[0076]

[0077] Where FFN(x) is a feedforward neural network and LN(x) is a layer normalization;

[0078] Combining text feature tensors and instance-aware text feature tensors Generate a new text feature tensor y tex :

[0079]

[0080] in, It is a text feature tensor. It is an instance-aware text feature tensor, W e It is a linear layer.

[0081] Preferably, the calculation process of the loss function in step S5 is as follows:

[0082]

[0083] Where, ∈ θ It is prediction noise, ∈ t It is actual Gaussian noise. It is the mean squared error loss function.

[0084] Preferably, the distance-based weighted fusion operation in step S6 is calculated as follows:

[0085]

[0086] Where, N f It is the number of frames contained in each segment, c i,j It contains the index of the center frame of the j-th video segment within the i-th frame of the video, f i,j It is the feature of the i-th frame extracted from the temporally deformable attention block in the j-th video segment. It is the result of fusing features in the i-th frame, and k represents the number of skip windows.

[0087] Preferably, the decoding process in step S8 is as follows:

[0088] Latent representation after denoising Add grayscale video frame feature tensor z g Obtain video features Decode it:

[0089]

[0090]

[0091] Decoding yields a color frame sequence [V1, V2, ..., V n The color frame sequence is combined to obtain the colored color video V;

[0092] Where D(x) is the decoder of the variational autoencoder.

[0093] A text-guided video colorization system oriented towards creativity and temporal consistency, characterized by comprising:

[0094] Preprocessing module:

[0095] This is used to segment the text T in each text-to-color video pair (T,V) into words, and to represent the video V of each text-to-color video pair (T,V) as a color video frame [V1,V2,...,V]. nSimultaneously extract the corresponding black and white video frames. Where n is the number of frames;

[0096] Feature extraction module:

[0097] This is used to encode the segmented text into a text feature tensor using a text encoder. The color video frames and black-and-white video frames are encoded separately to generate the color video frame feature z0 and the grayscale video frame feature tensor z. g Initialize time step t;

[0098] Noisy latent representation generation module:

[0099] At the current time step t, initialize a random Gaussian noise with the same shape as the color video frame feature z0 as the actual Gaussian noise ∈ t Based on the color video frame features z0 and the actual Gaussian noise ∈ t The corresponding noisy latent representation z is generated by weighted summation. t ;

[0100] Denoising network training module:

[0101] Used to transform the noisy latent representation z t and grayscale video frame feature tensor z g After concatenation, the video feature representation h is obtained by inputting it into the denoising network U-Net, and then trained through spatial blocks, temporally deformable attention blocks, and cross attention blocks.

[0102] Denoising network output module:

[0103] Used to predict noise output in the denoising network U-Net Calculate prediction noise and actual Gaussian noise ∈ t The loss function is then optimized.

[0104] Feature fusion module:

[0105] Used to capture long-term color dependencies using a skip window, selecting N with different resolutions. f Frame video segments, time interval d∈{1,2,4,…,N} d}, where N d It is the largest power of 2 result that does not exceed the total number of video frames. The distance-based weighted fusion operation performs cross-segment feature fusion on each frame that is close to the center of the segment.

[0106] Denoising latent representation generation module:

[0107] For time step t, use the noisy latent representation z t Subtract prediction noise Get z t-1 Repeat this step until time step t becomes 0 to obtain the denoised latent representation.

[0108] Video coloring module:

[0109] Used for the latent representation after denoising Decode the video to obtain the colored video V.

[0110] The above technical solution has the following advantages or beneficial effects:

[0111] This invention is based on a denoising diffusion model. By introducing a time-deformable attention block, it captures the dynamic features of instances in the time dimension based on the dynamic changes in the position and shape of objects in video frames, thereby maintaining the consistency of feature representation and effectively preventing color flickering and offset during the coloring process. A cross-modal pre-fusion module adjusts the semantic representation of textual noun concepts, and the use of masked cross-attention in the module enhances the model's understanding of noun concepts and instance perception, improving the accuracy of color allocation. Grayscale video information introduced during video decoding maintains the global structure of the colorized video. A cross-segment fusion mechanism is introduced during inference to maintain long-term consistency in video coloring. With text guidance, users can colorize videos according to their own needs and creativity.

[0112] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the invention will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0113] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in the invention and should not be construed as limiting the scope of the invention.

[0114] Figure 1 A schematic diagram of the TCVC framework in time-consistent video colorization based on deep feature propagation and self-regularization learning in existing technologies;

[0115] Figure 2 A schematic diagram of the GCC+CDGA framework in existing technologies for achieving realistic video colorization through a gated color-guided image diffusion model;

[0116] Figure 3 A flowchart of a text-guided video colorization method oriented towards creativity and temporal consistency provided by the present invention;

[0117] Figure 4 A structural diagram of a text-guided video colorization system oriented towards creativity and temporal consistency is provided for this invention;

[0118] Figure 5 This invention provides a structural diagram of a time-deformable attention block in text-guided creative and temporally consistent video colorization. Detailed Implementation

[0119] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0120] This invention first provides a text-guided video colorization method oriented towards creativity and temporal consistency, the process of which is as follows: Figure 3 As shown, it includes the following steps:

[0121] S1 Preprocessing:

[0122] Segment the text T in each text-color video pair (T,V) into words, and represent the video V of each text-color video pair (T,V) as color video frames [V1,V2,...,V]. n Simultaneously, the color video frames are converted to LAB color mode, and the L channel is extracted to obtain the corresponding black and white video frames. Where n is the number of frames;

[0123] S2 feature extraction:

[0124] The segmented text is encoded into a text feature tensor using the CLIP text encoder. The color video frames and black-and-white video frames are encoded separately to generate the color video frame feature z0 and the grayscale video frame feature tensor z. g Initialize time step t; where color video frames are encoded using a variational autoencoder ε, and black and white video frames are encoded using a luma encoder ε. lum Encode it.

[0125] The text encoder encoding process is as follows:

[0126]

[0127] Where T is the text, Tokenizer(x) is the tokenizer, and TextEncoder(x) is the text encoder. It is a text feature tensor.

[0128] The process of encoding color video frames and black-and-white video frames separately is as follows:

[0129] z i =ε(V i ),

[0130]

[0131] Among them, V i It is a color video frame. These are black and white video frames, ε(x) is the encoder of the variational autoencoder, ε lum (x) is the brightness encoder, z i These are encoded color video frames. These are encoded black and white video frames.

[0132] S3 noise-enhanced latent representation generation:

[0133] At the current time step t, initialize a random Gaussian noise with the same shape as the color video frame feature z0 as the actual Gaussian noise ∈ t Based on the color video frame features z0 and the actual Gaussian noise ∈ t The corresponding noisy latent representation z is generated by weighted summation. t ;

[0134] The noiseification latent representation z t The calculation is as follows:

[0135]

[0136] in, These are predefined time-step related parameters in the diffusion model.

[0137] S4 Denoising Network Training:

[0138] Noisy latent representation z t and grayscale video frame feature tensor z g After concatenation, the input is a denoising network U-Net based on SD (StableDiffusion, a base model based on a latent diffusion model). U-Net consists of multiple large blocks, each of which includes three parts: a spatial block, a temporally deformable attention block, and a cross attention block. The video feature representation h obtained by the denoising network U-Net is trained through the spatial block, the temporally deformable attention block, and the cross attention block.

[0139] S4.1 Training Phase 1: Training the denoising network U-Net:

[0140] S4.1.1 In each spatial block, compute the self-attention of the video feature representation h:

[0141] h = SA(h),

[0142] Where h represents the video feature representation, and SA(x) represents self-attention;

[0143] S4.1.2 as Figure 5 As shown, in each time-deformable attention block, the updated video feature tensor is computed. Capture the dynamic characteristics of instances;

[0144] The updated video feature tensor The calculation process is as follows:

[0145] Based on manually preset time sampling rate and resolution sampling rate, the video feature representation h is uniformly sampled in the time and resolution (width, height) dimensions, and the reference point sequence p is calculated:

[0146] p = UniformSample(h),

[0147] UniformSample(x) represents uniform sampling.

[0148] Through linear projection W o After projecting the video feature representation h, it is input into a lightweight 3D convolutional network δ. offset Predict the offset of each reference point on (x), and limit the result to the range [-1,1] using the tanh(x) function to calculate the estimated offset Δp:

[0149] Δp=tanh(δ offset (hW o )),

[0150] Among them, W o It is the projection matrix, δ offset (x) is a lightweight 3D convolutional network, and tanh(x) is the hyperbolic tangent function;

[0151] Based on the reference point sequence p and the estimated offset Δp, the estimated deformation points of the video feature representation h are weighted using the estimated deformation point weighting function φ(h,p), and the intermediate transformed result is calculated.

[0152]

[0153] Where α is a parameter, and φ(h,p) is the weighting function for estimating deformation points;

[0154] The weighted function φ(h,p) for estimating deformation points is calculated as follows:

[0155] g(a,b) = max(0,1-|ab|),

[0156]

[0157] Among them, (r x ,r y ,r t ) is the location index of the video feature representation h, (p x ,p y ,p t ) is the position index of the deformation point tensor p;

[0158] Using projection matrix and Transformed intermediate results Projection as key Sum Using projection matrix Project the video feature representation h as the query q. i Use query q i ,key Sum Calculate multi-head attention and compute the updated video feature tensor.

[0159]

[0160] in, It is the projection matrix, q i , It is the projected attention query, key and value, i is the head number of multi-head attention, and MHA(x) is multi-head attention.

[0161] S4.1.3 In each cross-attention block, compute the updated video feature tensor. With text feature tensor Cross-attention captures text semantics, ensuring consistency between video content and text.

[0162]

[0163] Where CA(x,y) is the cross attention of x and y;

[0164] In the second phase of training S4.2, only the cross-modal pre-fusion module is trained, and the weights of the denoising network U-Net are frozen.

[0165] In text feature tensor Before entering the cross-attention block, convert the text feature tensor and grayscale video frame feature tensor z g The input consists of a cross-modal pre-fusion module composed of L cross-modal pre-fusion layers, used to adjust the semantic representation of noun concepts in the text, and to compute the new text feature tensor y.tex ;

[0166] The new text feature tensor y tex The calculation is as follows:

[0167] First, the output of each layer is calculated. After L layers of calculation, the instance-aware text feature tensor is obtained. Output of layer l The calculation process is as follows:

[0168] right Perform multi-head self-attention calculation:

[0169]

[0170] in, It is the output of the previous layer, for When l is 1, it is the text feature tensor. When l is [2,3,...L], it represents the output of the previous cross-modal pre-fusion layer; LN(x) is layer normalization, and MSA(x) is multi-head self-attention;

[0171] calculate and grayscale video frame feature tensor z g Mask cross attention:

[0172]

[0173] Where M is a mask used to mask color-related words in the text, and MCA(x,y,M) is mask cross-attention;

[0174] Output of the current layer The calculation process is as follows:

[0175]

[0176] Here, FFN(x) is a feedforward neural network, and LN(x) is a layer normalization.

[0177] Combining text feature tensors and instance-aware text feature tensors Generate a new text feature tensor y tex :

[0178]

[0179] in, It is a text feature tensor. It is an instance-aware text feature tensor, W e It is a linear layer.

[0180] S5 denoising network output:

[0181] The U-Net denoising network outputs predicted noise. Calculate prediction noise and actual Gaussian noise ∈ t The loss function uses AdamW (Adam with Weight Decay, a common optimizer used to control model parameter updates); the optimizer updates parameters during backpropagation, and the learning rate LR (Learning Rate, a hyperparameter used to control the step size of model parameter updates in each iteration) is set to 1e-5.

[0182] The calculation process of the loss function is as follows:

[0183]

[0184] Where, ∈ θ It is prediction noise, ∈ t It is actual Gaussian noise. It is the mean squared error loss function.

[0185] S6 fusion features:

[0186] Use a skip window to capture long-term color dependencies. First, select N with different resolutions. f Frame video segments, time interval d∈{1,2,4,…,N} d}, where N d It is the largest power of 2 of the total number of video frames. Secondly, because frames closer to the center of a segment contain more representative content, the distance-weighted fusion operation performs cross-segment feature fusion on each frame close to the segment center, further mitigating the inconsistency of segment boundaries.

[0187] The distance-based weighted fusion operation is calculated as follows:

[0188]

[0189] Where, N f It is the number of frames contained in each segment, c i,j It contains the index of the center frame of the j-th video segment within the i-th frame of the video, f i,j It is the feature of the i-th frame extracted from the temporally deformable attention block in the j-th video segment. It is the result of fusing features in the i-th frame, and k represents the number of skip windows.

[0190] S7 generates the denoised latent representation:

[0191] At time step t, the noisy latent representation z is used. t Subtract prediction noise Get z t-1 Repeat this step until time step t becomes 0 to obtain the denoised latent representation.

[0192] S8 video shading:

[0193] For the denoised latent representation Decode the video to obtain the colored video V.

[0194] The decoding process is as follows:

[0195] Latent representation after denoising Add grayscale video frame feature tensor z g Obtain video features Decode it:

[0196]

[0197] Decoding yields a color frame sequence [V1, V2, ..., V n The color frame sequence is combined to obtain the colored color video V;

[0198] Where D(x) is the decoder of the variational autoencoder.

[0199] A text-guided video colorization system oriented towards creativity and temporal consistency, structured as follows: Figure 4 As shown, the feature is that it includes:

[0200] Preprocessing module:

[0201] This is used to segment the text T in each text-to-color video pair (T,V) into words, and to represent the video V of each text-to-color video pair (T,V) as a color video frame [V1,V2,...,V]. n Simultaneously extract the corresponding black and white video frames. Where n is the number of frames;

[0202] Feature extraction module:

[0203] This is used to encode the segmented text into a text feature tensor using a text encoder. The color video frames and black-and-white video frames are encoded separately to generate the color video frame feature z0 and the grayscale video frame feature tensor z. g Initialize time step t;

[0204] Noisy latent representation generation module:

[0205] At the current time step t, initialize a random Gaussian noise with the same shape as the color video frame feature z0 as the actual Gaussian noise ∈ tBased on the color video frame features z0 and the actual Gaussian noise ∈ t The corresponding noisy latent representation z is generated by weighted summation. t ;

[0206] Denoising network training module:

[0207] Used to transform the noisy latent representation z t and grayscale video frame feature tensor z g After concatenation, the video feature representation h is obtained by inputting it into the denoising network U-Net, and then trained through spatial blocks, temporally deformable attention blocks, and cross attention blocks.

[0208] Denoising network output module:

[0209] Used to predict noise output in the denoising network U-Net Calculate prediction noise and actual Gaussian noise ∈ t The loss function is then optimized.

[0210] Feature fusion module:

[0211] Used to capture long-term color dependencies using a skip window, selecting N with different resolutions. f Frame video segments, time interval d∈{1,2,4,…,N} d}, where N d It is the largest power of 2 result that does not exceed the total number of video frames. The distance-based weighted fusion operation performs cross-segment feature fusion on each frame that is close to the center of the segment.

[0212] Denoising latent representation generation module:

[0213] For time step t, use the noisy latent representation z t Subtract prediction noise Get z t-1 Repeat this step until time step t becomes 0 to obtain the denoised latent representation.

[0214] Video coloring module:

[0215] Used for the latent representation after denoising Decode the video to obtain the colored video V.

[0216] The above technical solution has the following advantages or beneficial effects: Based on the denoising diffusion model, this invention introduces a time-deformable attention block to capture the dynamic features of instances in the time dimension according to the dynamic changes in the position and shape of objects in video frames, thereby maintaining the consistency of feature representation and effectively preventing color flickering and offset during the coloring process; by adjusting the semantic representation of text noun concepts through a cross-modal pre-fusion module, the model's understanding of noun concepts and instance perception is enhanced by using mask cross-attention in the module, improving the accuracy of color allocation; by introducing grayscale video information during video decoding, the global structure of the colorized video is maintained; by introducing a cross-segment fusion mechanism during inference, the long-term consistency of video coloring is maintained; and through text guidance, users can colorize videos according to their own needs and creativity.

[0217] In the description of this specification, references to terms such as "an embodiment," "a preferred embodiment," "an example," "a specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0218] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0219] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0220] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0221] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments. Since the device implementation corresponds to the method implementation, the description of the device implementation is relatively brief, and the relevant description can be found by referring to the description of the method implementation.

[0222] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0223] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A text-guided video colorization method oriented towards creativity and temporal consistency, characterized in that, Includes the following steps: S1 Preprocessing: Segment the text T in each text-to-color video pair (T,V) into words, and represent the video V of each text-to-color video pair (T,V) as color video frames [V1,V2,...,V]. n Simultaneously extract the corresponding black and white video frames. Where n is the number of frames; S2 feature extraction: The segmented text is encoded into a text feature tensor using a text encoder. The color video frames and black-and-white video frames are encoded separately to generate the color video frame feature z0 and the grayscale video frame feature tensor z. g Initialize time step t; S3 generates a noisy latent representation: At the current time step t, initialize a random Gaussian noise with the same shape as the color video frame feature z0 as the actual Gaussian noise ∈ t Based on the color video frame features z0 and the actual Gaussian noise ∈ t The corresponding noisy latent representation z is generated by weighted summation. t ; S4 Denoising Network Training: Noisy latent representation z t and grayscale video frame feature tensor z g After concatenation, the video feature representation h is obtained by inputting it into the denoising network U-Net, and then trained through spatial blocks, temporally deformable attention blocks, and cross attention blocks. S4.1 Training Phase 1: Training the denoising network U-Net: S4.1.1 In each spatial block, compute the self-attention of the video feature representation h; S4.1.2 In each temporally deformable attention block, compute the updated video feature tensor. S4.1.3 In each cross-attention block, compute the updated video feature tensor. With text feature tensor Cross attention; In the second phase of training S4.2, only the cross-modal pre-fusion module is trained, and the weights of the denoising network U-Net are frozen. Text feature tensor and grayscale video frame feature tensor z g Inputting a cross-modal prefusion module consisting of L cross-modal prefusion layers yields a new text feature tensor y. tex ; S5 denoising network output: The U-Net denoising network outputs predicted noise. Calculate prediction noise and actual Gaussian noise ∈ t The loss function is then optimized. S6 fusion features: Use a skip window to capture long-term color dependencies, selecting N with different resolutions. f Frame video segments, time interval d∈{1,2,4,…,N} d }, where N d It is the largest power of 2 result that does not exceed the total number of video frames. The distance-based weighted fusion operation performs cross-segment feature fusion on each frame that is close to the center of the segment. S7 generates the denoised latent representation: At time step t, the noisy latent representation z is used. t Subtract prediction noise Get z t-1 Repeat this step until time step t becomes 0 to obtain the denoised latent representation. S8 video shading: For the denoised latent representation Decode the video to obtain the colored video V.

2. The text-guided video colorization method for creative and temporal consistency as described in claim 1, characterized in that: The text encoder encoding process described in step S2 is as follows: Where T is the text, Tokenizer(x) is the tokenizer, and TextEncoder(x) is the text encoder. It is the encoded text feature tensor; The process of encoding the color video frame and the black and white video frame separately in step S2 is as follows: z i =ε(V i ), Among them, V i It is a color video frame. These are black and white video frames, ε(x) is the encoder of the variational autoencoder, ε lum (x) is the brightness encoder, z i These are encoded color video frames. These are encoded black and white video frames.

3. The text-guided video colorization method for creative and temporal consistency as described in claim 1, characterized in that: The noisy latent representation z described in step S3 t The calculation is as follows: in, These are predefined time-step related parameters in the diffusion model.

4. The text-guided video colorization method for creative and temporal consistency as described in claim 1, characterized in that: The updated video feature tensor described in step S4.1.2 The calculation process is as follows: Based on manually preset time sampling rate and resolution sampling rate, the video feature representation h is uniformly sampled in both time and resolution dimensions, and the reference point sequence p is calculated: p = UniformSample(h), Where UniformSample(x) represents uniform sampling; Through linear projection W o After projecting the video feature representation h, it is input into a lightweight 3D convolutional network δ. offset Predict the offset of each reference point on (x), and limit the result to the range [-1,1] using the tanh(x) function to calculate the estimated offset Δp: Δp=tanh(δ offset (hW o )), Among them, W o It is the projection matrix, δ offset (x) is a lightweight 3D convolutional network, and tanh(x) is the hyperbolic tangent function; Based on the reference point sequence p and the estimated offset Δp, the estimated deformation points of the video feature representation h are weighted using the estimated deformation point weighting function φ(h,p), and the intermediate transformed result is calculated. Where α is a parameter, and φ(h,p) is the weighting function for estimating deformation points; The weighted function φ(h,p) for estimating deformation points is calculated as follows: g(a,b) = max(0,1-|ab|), Among them, (r x ,r y ,r t ) is the location index of the video feature representation h, (p x ,p y ,p t ) is the position index of the reference point sequence p; Using projection matrix and Transformed intermediate results Projection as key Sum Using projection matrix Project the video feature representation h as the query q. i Use query q i ,key Sum Calculate multi-head attention and compute the updated video feature tensor. in, It is the projection matrix, q i , It is the projected attention query, key and value, i is the head number of multi-head attention, and MHA(x) is multi-head attention.

5. The text-guided video colorization method for creative and temporal consistency as described in claim 1, characterized in that: The new text feature tensor y mentioned in step S4.2 tex The calculation is as follows: First, the output of each layer is calculated. After L layers of calculation, the instance-aware text feature tensor is obtained. Output of layer l The calculation process is as follows: right Perform multi-head self-attention calculation: in, It is the output of the previous layer, for When l is 1, it is the text feature tensor. When l is [2,3,...L], it represents the output of the previous cross-modal pre-fusion layer; LN(x) is layer normalization, and MSA(x) is multi-head self-attention; calculate and grayscale video frame feature tensor z g Mask cross attention: Where M is a mask used to mask color-related words in the text, and MCA(x,y,M) is mask cross-attention; Output of the current layer The calculation process is as follows: Where FFN(x) is a feedforward neural network and LN(x) is a layer normalization; Combining text feature tensors and instance-aware text feature tensors Generate a new text feature tensor y tex : in, It is a text feature tensor. It is an instance-aware text feature tensor, W e It is a linear layer.

6. The text-guided video colorization method for creative and temporal consistency as described in claim 1, characterized in that: The calculation process of the loss function in step S5 is as follows: Where, ∈ θ It is prediction noise, ∈ t It is actual Gaussian noise. It is the mean squared error loss function.

7. The text-guided video colorization method for creative and temporal consistency as described in claim 1, characterized in that: The distance-based weighted fusion operation described in step S6 is calculated as follows: Where, N f It is the number of frames contained in each segment, c i,j It contains the index of the center frame of the j-th video segment within the i-th frame of the video, f i,j It is the feature of the i-th frame extracted from the temporally deformable attention block in the j-th video segment. It is the result of fusing features in the i-th frame, and k represents the number of skip windows.

8. A text-guided video colorization method for creative and temporal consistency as described in claim 1, characterized in that: The decoding process described in step S8 is as follows: Latent representation after denoising Add grayscale video frame feature tensor z g Obtain video features Decode it: Decoding yields a color frame sequence [V1, V2, ..., V n The color frame sequence is combined to obtain the colored color video V; Where D(x) is the decoder of the variational autoencoder.

9. A text-guided video colorization system oriented towards creativity and temporal consistency, characterized in that, include: Preprocessing module: This is used to segment the text T in each text-to-color video pair (T,V) into words, and to represent the video V of each text-to-color video pair (T,V) as a color video frame [V1,V2,...,V]. n Simultaneously extract the corresponding black and white video frames. Where n is the number of frames; Feature extraction module: This is used to encode the segmented text into a text feature tensor using a text encoder. The color video frames and black-and-white video frames are encoded separately to generate the color video frame feature z0 and the grayscale video frame feature tensor z. g Initialize time step t; Noisy latent representation generation module: At the current time step t, initialize a random Gaussian noise with the same shape as the color video frame feature z0 as the actual Gaussian noise ∈ t Based on the color video frame features z0 and the actual Gaussian noise ∈ t The corresponding noisy latent representation z is generated by weighted summation. t ; Denoising network training module: Used to transform the noisy latent representation z t and grayscale video frame feature tensor z g After concatenation, the video feature representation h is obtained by inputting it into the denoising network U-Net, and then trained through spatial blocks, temporally deformable attention blocks, and cross attention blocks. Denoising network output module: Used to predict noise output in the denoising network U-Net Calculate prediction noise and actual Gaussian noise ∈ t The loss function is then optimized. Feature fusion module: Used to capture long-term color dependencies using a skip window, selecting N with different resolutions. f Frame video segments, time interval d∈{1,2,4,…,N} d }, where N d It is the largest power of 2 result that does not exceed the total number of video frames. The distance-based weighted fusion operation performs cross-segment feature fusion on each frame that is close to the center of the segment. Denoising latent representation generation module: For time step t, use the noisy latent representation z t Subtract prediction noise Get z t-1 Repeat this step until time step t becomes 0 to obtain the denoised latent representation. Video coloring module: Used for the latent representation after denoising Decode the video to obtain the colored video V.

Images