A video colorization method based on semantic correspondence and alignment

Abstract

A video colorization method based on semantic correspondence and alignment is proposed, constructing a video colorization model SCAColor. This model includes a semantic correspondence pre-colorization subnet, a bidirectional optical flow-guided semantic alignment subnet, and a color refinement subnet. The semantic correspondence pre-colorization subnet learns the semantic association between grayscale frames and reference frames to generate preliminary colorization results and a semantic similarity map. The bidirectional optical flow-guided semantic alignment subnet incorporates the colorization result of the previous frame and the true color value of the current frame, correcting feature biases while enhancing temporal consistency. The color refinement subnet further optimizes local colors and outputs the final colorization result. Extensive experimental results on public benchmark datasets demonstrate that the video colorization method based on the video colorization model can achieve reasonable colorization of videos with natural and rich colors and smooth inter-frame transitions, effectively avoiding color bleeding and exhibiting good network robustness.

Description

Technical Field

[0001] This invention belongs to the field of video image processing technology, specifically relating to a video coloring method based on semantic correspondence and alignment. Background Technology

[0002] Images and videos, as crucial carriers of visual information, hold an irreplaceable position in communication and technological development. Color, a key visual element in images, significantly impacts human perception. Compared to grayscale images, the human eye is far more sensitive to color images—while it can only distinguish a limited range of grayscale levels, it can recognize millions of different colors. In the fields of visual science and computer graphics, colorization techniques are a core research area. Their goal is to imbue grayscale images or video sequences with appropriate and vivid colors, thereby enhancing their realism and expressiveness. Colorization techniques not only improve image quality but also enhance visual appeal, effectively improving the efficiency of information transmission and the viewer's comprehension.

[0003] Images record information through spatial pixels, while videos consist of a series of consecutive image frames, containing not only spatial information but also dynamic content in the temporal dimension. Compared to image colorization, research on video colorization is relatively limited. Applying image colorization methods directly to video can easily lead to inconsistent color propagation between adjacent frames, resulting in visual defects such as flickering, color jumps, and even artifacts. Therefore, video colorization technology urgently requires more in-depth and specialized research to address its unique challenges in temporal consistency and visual coherence. Summary of the Invention

[0004] To address the aforementioned problems, the purpose of this invention is to provide a video coloring method based on semantic correspondence and alignment, which solves the common problems of inter-frame color transitions and inaccurate local coloring in existing video coloring technologies.

[0005] To achieve the above objectives, the technical solution adopted by the present invention includes: A method for constructing a video colorization model based on semantic correspondence and alignment includes the following steps: S1. Obtain the training dataset. The training dataset includes multiple training samples. Each training sample includes a continuous sequence of color video frames, a corresponding color reference frame, and a corresponding sequence of grayscale video frames. The corresponding color reference frame includes the first and last frames of the continuous color video frame sequence. The corresponding grayscale video frame sequence is obtained by extracting the luminance channels of each frame from the continuous color video frame sequence. S2. Construct a video colorization network, which includes a semantic correspondence pre-colorization subnetwork, a bidirectional optical flow guided semantic alignment subnetwork, and a color refinement subnetwork connected in sequence. The semantic correspondence pre-colorization subnetwork is used to generate preliminary colorization results and a semantic similarity map based on the input current grayscale video frame and the corresponding color reference frame. The bidirectional optical flow guided semantic alignment subnetwork is used to perform cross-frame color alignment and feature fusion on the preliminary colorization results, semantic similarity map, colorization results of the previous frame, and the true color value of the current frame, and output the associated features and the color alignment features of the current frame. The color refinement subnetwork is used to concatenate and fuse the semantic similarity map, associated features, and color alignment features of the current frame, and output the final colorization result of the current frame. S3 takes the training dataset as input to the video colorization network and passes it through the total loss function. The training process is performed to obtain a well-trained video colorization model;

[0006] in, In order to perceive loss, For context loss, To smooth out the loss, Indicating resistance to loss, express loss, These are the weighting coefficients for each loss term.

[0007] Preferably, the semantic correspondence pre-coloring subnet includes: an initial feature extraction module, used to extract multi-scale features of the input current grayscale video frame and the corresponding color reference frame; a feature refinement module, used to optimize the multi-scale features and output enhanced grayscale frame features and reference frame features; and a non-local semantic correspondence module, used to calculate the semantic similarity matrix between the enhanced grayscale frame features and the reference frame features, and generate preliminary coloring results and a semantic similarity map based on the semantic similarity matrix and the color reference frame.

[0008] Preferably, the feature refinement module includes a feature purification residual layer, which consists of multiple cascaded residual blocks for progressively optimizing the multi-scale features of the input; at least one residual block is a shallow residual block for extracting and separating local details and noise, and at least one residual block is a deep residual block for integrating global context and enhancing semantic structure.

[0009] Preferably, the bidirectional optical flow-guided semantic alignment subnet includes: an optical flow estimation module for calculating the bidirectional optical flow between the ground truth of the current video grayscale frame and the colorization result of the previous frame; a deformable alignment module for spatially aligning the colorization result of the previous frame and the ground truth of the current video grayscale frame using the bidirectional optical flow, generating color alignment features of the previous frame and color alignment features of the current frame; and a dynamic fusion module for fusing the color alignment features of the previous frame, the preliminary colorization result, and the semantic similarity map, specifically including: first, extracting a first luminance channel and a second luminance channel from the color alignment features of the previous frame and the preliminary colorization result, respectively; second, calculating the element-wise absolute difference between the first luminance channel and the second luminance channel; third, generating a first fusion weight for the color alignment features of the previous frame and a second fusion weight for the preliminary colorization result based on the element-wise absolute difference and the semantic similarity map, respectively; and finally, performing a weighted summation of the color alignment features of the previous frame and the preliminary colorization result according to the first fusion weight and the second fusion weight, and outputting the associated features.

[0010] Preferably, the optical flow estimation module is a SpyNet network, which includes an image pyramid architecture consisting of multiple cascaded identical processing units. Each processing unit corresponds to a level of the image pyramid, and optical flow is estimated by progressively refining from low resolution to high resolution.

[0011] Preferably, the color alignment feature of the previous frame in the deformable alignment module is aligned through the following steps: First, the coloring result of the previous frame is concatenated with the input forward optical flow to obtain the first concatenated feature; second, the first concatenated feature is input into an offset estimation network composed of multiple concatenated convolutions, and the last convolution of the offset estimation network outputs the first residual optical flow; third, the first residual optical flow is added to the forward optical flow to obtain the first sampling offset; simultaneously, the first concatenated feature is input into a mask estimation network composed of multiple concatenated convolutions, and the last convolution of the mask estimation network outputs the normalized first attention mask; finally, the first sampling offset and the first attention mask are used to adaptively spatially sample the coloring result of the previous frame to output the spatially aligned color alignment feature of the previous frame. The current frame color alignment feature in the deformable alignment module achieves feature alignment through the following steps: First, the ground truth value of the current video grayscale frame is concatenated with the input reverse optical flow to obtain the second concatenated feature; second, the second concatenated feature is input into an offset estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the offset estimation network outputs the second residual optical flow; third, the second residual optical flow is added to the reverse optical flow to obtain the second sampling offset; simultaneously, the second concatenated feature is input into a mask estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the mask estimation network outputs the normalized second attention mask; finally, the second sampling offset and the second attention mask are used to adaptively spatially sample the ground truth value of the input current video grayscale frame to output the spatially aligned current frame color alignment feature.

[0012] Preferably, the color thinning subnet includes an encoder, which includes a shallow semantic feature representation layer, a downsampling layer, a hole enhancement layer and a deep semantic feature representation layer connected in series; the color thinning subnet also includes a decoder, which includes a first upsampling layer, a second upsampling layer and an upsampling attention layer connected in series.

[0013] Preferred, perceived loss weights Context loss weights Smoothing loss weights Counteracting loss weights , Loss weights .

[0014] A video colorization method based on semantic correspondence and alignment is provided, which inputs a continuous sequence of grayscale video frames to be colored and the corresponding color reference frames into a video colorization model to obtain a colored video sequence; the video colorization model is constructed by the video colorization model construction method based on semantic correspondence and alignment disclosed in this application.

[0015] A video colorization system based on semantic correspondence and alignment includes: a data input unit for receiving a grayscale video sequence to be colored and a corresponding color reference frame input by a user; a model loading and execution unit for loading a video colorization model and calling the model to perform frame-by-frame colorization processing on the grayscale video sequence, wherein the video colorization model is constructed by the video colorization model construction method based on semantic correspondence and alignment disclosed in this application; and a result output unit for outputting the colored color video sequence.

[0016] Compared with the prior art, the advantages of the present invention are: (1) The video coloring method based on semantic correspondence and alignment of the present invention constructs a video coloring model SCAColor, which includes a semantic correspondence pre-coloring subnet, a bidirectional optical flow guided semantic alignment subnet, and a color refinement subnet. The semantic correspondence pre-coloring subnet generates preliminary coloring results and semantic similarity maps by learning the semantic association between grayscale frames and reference frames. The bidirectional optical flow guided semantic alignment subnet introduces the coloring results of the previous frame and the true color values ​​of the current frame, which improves temporal consistency while correcting feature deviations. The color refinement subnet further optimizes local colors and outputs the final coloring results. Extensive experimental results on public benchmark datasets show that the video coloring method based on the video coloring model can achieve reasonable coloring of videos with natural and rich colors and smooth inter-frame continuity, effectively avoid color bleeding, and has good network robustness.

[0017] (2) The video coloring method based on semantic correspondence and alignment of the present invention, through the bidirectional optical flow-guided semantic alignment subnet, realizes the accurate transmission and utilization of cross-frame color information by optical flow estimation, deformable alignment and feature fusion, which not only ensures the continuity of color between frames, but also improves the coloring accuracy by using real frames.

[0018] (3) The video colorization method based on semantic correspondence and alignment of the present invention includes a parallel attention layer containing pixel attention branches and channel attention branches. With the help of the parallel attention mechanism, it adaptively recalibrates the feature map, allowing the model to focus on more informative regions while effectively suppressing irrelevant information.

[0019] (3) The video coloring method based on semantic correspondence and alignment of the present invention uses a feature purification residual layer to progressively optimize features through multi-level residual blocks: shallow blocks extract local details and filter out noise, and deep blocks integrate global context and enhance semantic structure, providing high-quality feature input for the semantic correspondence module and enhancing the alignment accuracy between grayscale frames and reference frames. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof. In the drawings: Figure 1It is a network structure diagram based on semantic correspondence and alignment video coloring; Figure 2 It is a semantically corresponding precolored subnet structure diagram; Figure 3 This is a structural diagram of VGG19; Figure 4 This is a structural diagram of the feature refinement module; Figure 5 This is a diagram of the feature extraction layer structure; Figure 6 This is a diagram of the residual layer structure after feature purification. Figure 7 This is a structural diagram of the non-local semantic correspondence module; Figure 8 This is a diagram of the bidirectional optical flow-guided semantic alignment subnet structure; Figure 9 This is a diagram of the SpyNet architecture; Figure 10 This is a structural diagram of the deformable alignment module; Figure 11 This is a diagram of the dynamic fusion module structure; Figure 12 It is a color-refined subnet structure diagram; Figure 13 This is a diagram of the encoder's internal structure; Figure 14 This is a diagram of the internal structure of the downsampling block; Figure 15 This is a diagram of the internal structure of the void reinforcement layer; Figure 16 This is a diagram of the decoder's internal structure; Figure 17 This is a diagram of the internal structure of the upsampling layer; Figure 18 This is a diagram of the internal structure of the upsampling attention layer; Figure 19 This is a diagram of the parallel attention layer structure; Figure 20 This is a comparison diagram of the present invention and other video coloring methods in Example 1; Figure 21 This is a comparison diagram of the present invention and other video coloring methods in Example 2; Figure 22 This is a comparison diagram of the present invention and other video coloring methods in Example 3; Figure 23 This is a comparison diagram of the present invention and other video coloring methods in Example 4; Figure 24 This is a comparison diagram of the present invention and other video coloring methods in Example 5; Figure 25 This is a visualization of the ablation experiment of the module of this invention. Detailed Implementation

[0021] The invention is not limited to the specific embodiments described below. All equivalent modifications made based on the technical solutions of this application fall within the protection scope of this invention. Unless otherwise specified, all components and devices in this invention utilize components and devices known in the prior art.

[0022] Example 1 This embodiment discloses a method for constructing a video colorization model based on semantic correspondence and alignment, characterized by the following steps: S1, Obtain the training dataset, which includes multiple sets of training samples. Each set of training samples includes a continuous sequence of color video frames, a corresponding color reference frame, and a corresponding sequence of grayscale video frames. The corresponding color reference frame includes the first and last frames of a continuous color video frame sequence, and the corresponding video grayscale frame sequence is obtained by extracting the luminance channels of each frame from the continuous color video frame sequence.

[0023] S2, construct the video coloring network (SCAColor), such as Figure 1 As shown, the video coloring network includes a semantic correspondence pre-coloring subnet (SCPC), a bidirectional optical flow-guided semantic alignment subnet (BOFSA), and a color refinement subnet (CR) connected in sequence. The semantic correspondence pre-coloring subnet is used to generate preliminary coloring results and semantic similarity maps based on the input current grayscale video frame and the corresponding color reference frame. The bidirectional optical flow-guided semantic alignment subnet is used to perform cross-frame color alignment and feature fusion on the preliminary coloring result, semantic similarity map, coloring result of the previous frame and real color value of the current frame, and output associated features and color alignment features of the current frame. The color refinement subnet is used to concatenate and fuse the semantic similarity map, association features, and the current frame color alignment features to output the final coloring result of the current frame. S3, the training dataset is used as input to the video colorization network, and the total loss function is applied. The training process yields a well-trained video colorization model.

[0024] In this embodiment, the semantic correspondence pre-coloring subnet first learns the semantic association between the current grayscale frame and the reference frame, and generates the preliminary coloring result of the current frame. With semantic similarity graph This lays the foundation for subsequent color optimization. Secondly, the bidirectional optical flow-guided semantic alignment subnets are calculated separately. Frame pointing Frame and Frame pointing The optical flow of the frame is used to compensate for the current frame and adjacent frames to generate Frame pointing Previous frame color alignment feature and Frame pointing Frame association features This subnet utilizes, on the one hand, The coloring result of a frame constrains the coloring results of subsequent adjacent frames, overcoming inter-frame color jumps; on the other hand, it utilizes the training set... The color ground truth values ​​of the frames are used to enhance the semantic correspondence of colors. Finally, the color alignment features of the previous frame are concatenated. , Frame pointing Frame association features With semantic similarity graph The color refinement subnet, which includes upsampling parallel attention blocks, is fed into the network to achieve precise optimization of local colors (such as edges and texture areas), ensuring overall color harmony and effectively avoiding color bleeding and leakage.

[0025] This embodiment grayscale video frames (in and These represent the height and width of a video frame, respectively; 1 indicates a grayscale frame containing only the luminance channel. The overall training process for SCAColor is as follows: (1) Semantic correspondence learning: the grayscale video sequence With color reference frame Input a semantic correspondence pre-coloring subnet, learn the semantic correspondence between grayscale frames and reference frames, and obtain preliminary coloring results. (including brightness) With chromaticity (channels), and semantic similarity graphs .

[0026] (2) Bidirectional alignment and feature fusion: In the bidirectional optical flow-guided semantic alignment subnet: forward alignment will The frame's coloring result is passed to Frame generation Frame alignment features, and with As a weighting signal, for Dynamically weighted fusion with alignment features yields... Frame pointing Frame association features Back alignment utilizes a sequence of real color frames. generate Frame pointing Frame color alignment features .

[0027] (3) Generation of final coloring results: color refinement subnet reception , and Output the final color frame sequence. .

[0028] Next, let's start with the number... Taking the frame coloring process as an example, this paper details the processing flow of the SCAColor network and the detailed structure of each module.

[0029] The semantically corresponding precolored subnet in this embodiment is as follows: Figure 2 As shown, it consists of an initial feature extraction module (VGG19), a feature refinement module (FRM), and a non-local semantic correspondence module (SCM). It learns the semantic correspondence between the current grayscale frame and the reference frame and outputs the preliminary colorization result of the current grayscale frame. (Including basic luminance and chrominance information) and semantic similarity map of quantized inter-frame semantic matching strength .

[0030] The initial feature extraction module VGG19 is used to extract multi-scale features from the input current grayscale video frame and the corresponding color reference frame; such as... Figure 3 As shown, the "19" in VGG19 refers to the number of layers including parameters, consisting of 16 convolutional layers and 3 fully connected layers. All convolutional layers have the same configuration, with a kernel size of 3. 3. The stride is 1, the padding is 1, and both closely follow the ReLU nonlinear activation function, using repeated small convolutional kernels (3... 3) Feature extraction is performed stepwise with pooling layers. The network contains five max-pooling layers (2×2 size, stride 2), all placed after each convolutional block to reduce the feature map size, decrease computational cost, and retain key features. The first two fully connected layers have an output dimension of 4096, and the third fully connected layer has an output dimension of 1000, corresponding to 1000 classification categories. The classification task is completed through the final softmax layer.

[0031] Fully connected layers and softmax layers are originally used for classification, but in video colorization, the core requirement is feature extraction rather than direct classification. Therefore, this invention uses a VGG19 model pre-trained on the large-scale image dataset ImageNet, but only uses its convolutional and pooling layers to extract multi-scale features from video frames. (The last sentence appears to be incomplete and possibly refers to a different approach.) (Where H and W represent the height and width of the video frame, respectively) and color reference frame. As input, the VGG19 network outputs six intermediate feature maps, denoted as follows: ( For convolutional block index, For the corresponding feature map channel number, (cumulative downsampling factor), and Since each max-pooling layer achieves a 2x downsampling, the cumulative downsampling factors for the six feature maps are 1, 2, 4, 8, 16, and 32, respectively, corresponding to sizes of... , , , , , .like Figure 3 As shown, the video frames have been uniformly adjusted to [the specified format] during the data preprocessing stage. Size (i.e.) ).

[0032] For video colorization, this invention selects feature maps from the middle 2-5 layers using key-value control. On the one hand, deeper feature maps (such as those sampled 32 times) may lose the local details required for the task due to excessive compression of spatial information; on the other hand, these features already cover a sufficient scale span, while shallower features (such as...) , It retains rich detail and texture, deep features ( , The key semantic structure is encoded, and the combination of the two can take into account both the local texture details and global semantic consistency required for video coloring, providing multi-scale feature support for the reasonable transfer of color between grayscale frames and reference frames.

[0033] The feature refinement module is used to optimize the multi-scale features and output enhanced grayscale frame features and reference frame features; for example Figure 4 As shown, the Feature Refinement Module (FRM) consists of a Feature Extraction Layer (FEL), an Upsample Layer, and a Feature Purification Residual Layer (FPRL).

[0034] (1) Feature extraction layer The feature extraction layer (FEL), as the core component of the FRM module, is responsible for extracting multi-scale inputs. , , , Its characteristics, its internal structure as follows Figure 5 As shown.

[0035] Conv This method is used to extract local features from the input feature map, effectively capturing spatial relationships and semantic information within local regions. It controls the output size by setting different strides: when stride=1, the feature map size remains unchanged; when stride=2, the feature map size is reduced to half of the input size. For example... Figure 4 As shown, this mechanism enables feature preservation or downsampling at different levels. IN (Instance normalization) normalizes each channel of each sample separately, which helps accelerate model training and improve model stability. PReLU is the activation function. RefPad (Reflection padding) employs a reflection padding strategy to alleviate the problem of edge information loss during convolution.

[0036] (2) Feature purification residual layer In FRM, feature depth optimization is achieved by the Feature Purification Residual Layer (FPRL). For example... Figure 6 As shown, this layer consists of three residual blocks with the same structure connected in series. Each residual block adopts a dual-unit design of convolution + IN + PReLU and introduces cross-layer residual connections to effectively alleviate gradient vanishing and ensure stable optimization of deep networks.

[0037] FPRL progressively optimizes feature maps through multi-layer convolution and non-linear cascading. Its core lies in the fact that residual blocks of different depths process information at different levels. Shallow residual blocks have smaller receptive fields, primarily focusing on local details (such as edges and color blocks), and initially separating these basic patterns from background noise. Deep residual blocks have larger receptive fields, capable of integrating broad-area contextual information. Building upon the shallow output, they synthesize and enhance complex textures and semantic structures while filtering out residual interference unrelated to the overall content. This process provides clean, high-quality feature input for subsequent nonlocal semantic correspondence modules.

[0038] The nonlocal semantic correspondence module (SCM) is used to calculate the semantic similarity matrix between the enhanced grayscale frame features and the reference frame features, and based on the semantic similarity matrix and the color reference frame, to generate preliminary colorization results and a semantic similarity map; for example... Figure 7As shown. Its core is to realize the reference frame by calculating the pairwise semantic similarity between two types of features. The precise transmission of color involves the following process: First, for and Perform feature transformations separately, and subtract the mean, Norm normalization eliminates feature differences, ensuring the comparability of similarities between different features and enhancing the stability of similarity calculation. Subsequently, matrix multiplication is used to calculate the similarity matrix for all spatial locations of the two types of features. ,express In position Features of the place and In position The semantic association strength between features at a given location. The calculation formula for this process is as follows:

[0039] in, Represents the similarity matrix. express In position The characteristics of the place, express The mean eigenvector, express In position The characteristics of the place, express The mean eigenvector, express Norm.

[0040] pass Reference frame Towards Mapping yields preliminary coloring results. The calculation formula is as follows:

[0041] in, This indicates the preliminary coloring results. Represents the normalization function. Represents the similarity matrix. This chapter defines the temperature coefficient. A value of 0.01 causes the weight distribution to focus on the most similar reference frame position. Indicates the reference frame.

[0042] To quantify matching reliability, this invention calculates the maximum similarity score for each location as a matching confidence map. This is used by subsequent modules to correct unreliable color mapping areas. Each position on ,exist The In the middle, use Iterate through all positions in the reference frame and find the position with the highest similarity value. This maximum value is then used as the position. The confidence score of the matching reference color is used to generate a semantic similarity map. The calculation process is as follows:

[0043] in, Represents a semantic similarity graph. Indicates index Perform the maximum value operation. This represents the similarity matrix.

[0044] The bidirectional optical flow-guided semantic alignment subnet of this embodiment is as follows: Figure 8 As shown, this includes channel extraction operations. and The system consists of an optical flow estimation module (SpyNet), a deformable correspondence alignment (DCA) module, and a dynamic fusion (DF) module. Through optical flow estimation, deformable alignment, and feature fusion, it achieves accurate transmission and utilization of color information across frames, ensuring the continuity of colors between frames and improving color accuracy by leveraging real frames.

[0045] in and Used to extract the luminance channel respectively With color channels ; The optical flow estimation module is used to calculate the bidirectional optical flow between the ground truth of the current video grayscale frame and the coloring result of the previous frame; the deformable alignment module uses the bidirectional optical flow to spatially align the coloring result of the previous frame and the ground truth of the current video grayscale frame respectively, generating color alignment features of the previous frame and color alignment features of the current frame; the dynamic fusion module is used to fuse the color alignment features of the previous frame, the preliminary coloring result and the semantic similarity map, and output the association features.

[0046] The optical flow estimation module is a SpyNet network, consisting of an image pyramid architecture composed of multiple cascaded identical processing units. Each processing unit corresponds to a level of the image pyramid, estimating optical flow through progressive refinement from low to high resolution. point to Forward optical flow (Depicting the first) Frame to the (frame pixel motion), and from point to Reverse optical flow (Depicting the first) Frame to the (Frame pixel motion). Bidirectional optical flow provides guidance for inter-frame pixel displacement in subsequent deformable alignment.

[0047] The SpyNet network employs a pyramid architecture consisting of six identical processing modules, estimating optical flow through progressive refinement from low to high resolution. Figure 9 This demonstrates the structure of one of the modules. Input Frame and The images were downsampled and constructed into a 6-level image pyramid. Each level of frame is denoted as and The corresponding optical flow is denoted as The processing flow of each module is as follows: For the low-resolution optical flow from the previous level... Upsampling is performed to obtain optical flow. .use For the current level Perform optical flow warping to obtain the warped frame. ,Will and The features are concatenated along the channel dimension, and then sequentially passed through five convolutional layers for feature extraction, outputting the optical flow correction for the current layer. Finally, the correction amount is added to the upsampled optical flow through residual connection to output the optical flow of the current level. The optical flow output of the last layer. That is, the final forward optical flow Through this pyramidal processing, the network can improve upon the initial estimate from the low-resolution stage at each higher-resolution stage, resulting in more accurate optical flow. Forward optical flow The reverse optical flow is calculated through the above process. The calculation process is exactly the same, only requiring swapping and The input order is sufficient.

[0048] The deformable alignment module (DCA) utilizes bidirectional optical flow to respectively align the first... The coloring results are the same as the first Spatial alignment is performed on the truth values ​​of the frames. Figure 8 In the middle, the DCA input of the upper branch and Output aligned to the first Frames Frame color alignment features DCA input of the lower branch and Output aligned to The Frame color alignment features Finally, extract of Color channels are used in subsequent coloring processes. Through the deformable alignment module, cross-frame features are matched in spatial location, providing an aligned feature basis for subsequent fusion.

[0049] The structure of the deformable alignment module is as follows: Figure 10 As shown, its core lies in using deformable convolution to align features between frames, enhancing the temporal consistency of video colorization. In video colorization tasks, complex non-rigid motions often exist between adjacent frames, such as clothing wrinkles and water ripples. Traditional convolution operations are limited by a fixed sampling grid, making it difficult to adapt to such deformations, resulting in blurred or distorted features after alignment, affecting colorization quality. Based on this, this invention designs a specific deformable alignment process. First, a SpyNet network is used to obtain... point to Forward optical flow to Frame coloring results Perform concatenation, then apply Conv... Estimate the residual optical flow between the two The process is shown in formula (4):

[0050] in, This indicates a channel separation operation. This indicates a splicing operation. This represents the offset estimation function.

[0051] Adding the original optical flow to the residual optical flow yields the final offsets DCN of the deformable convolution, as shown in Equation (5):

[0052] in, This represents the offset of the deformable convolution.

[0053] Will and After splicing, it is processed through a convolutional layer Conv After applying the Softmax normalization operation, the deformable convolutional mask DCN is estimated, as shown in Equation (6):

[0054] in, A mask representing deformable convolution. This indicates a splicing operation. This represents the mask estimation function.

[0055] As shown in Table 1, and They all have a 5-layer structure and share the first 4 layers. The first 4 layers consist of 2 pairs of... It consists of convolution and LeakyReLU concatenation, with the 5th layer being a single... The convolution, but and The number of output channels is different.

[0056] Table 1 and Structure

[0057] Finally, using the obtained DCN offsets and DCN masks, deformable convolution pairs Perform alignment to generate forward alignment features. As shown in formula (7):

[0058] in, This indicates the color alignment feature of the previous frame. Represents a deformable convolution function. express The coloring result of the frame. , These represent the mask and offset of deformable convolution, respectively.

[0059] This alignment feature will be used in the subsequent coloring process, effectively improving the performance of video coloring in complex motion scenes.

[0060] In this embodiment, the color alignment feature of the previous frame in the deformable alignment module is aligned through the following steps: First, the coloring result of the previous frame is concatenated with the input forward optical flow to obtain the first concatenated feature; second, the first concatenated feature is input into an offset estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the offset estimation network outputs the first residual optical flow; third, the first residual optical flow is added to the forward optical flow to obtain the first sampling offset; simultaneously, the first concatenated feature is input into a mask estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the mask estimation network outputs the normalized first attention mask; finally, the first sampling offset and the first attention mask are used to adaptively spatially sample the coloring result of the previous frame to output the spatially aligned color alignment feature of the previous frame. In this embodiment, the current frame color alignment feature in the deformable alignment module achieves feature alignment through the following steps: First, the ground truth value of the current video grayscale frame is concatenated with the input reverse optical flow to obtain a second concatenated feature; second, the second concatenated feature is input into an offset estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the offset estimation network outputs a second residual optical flow; third, the second residual optical flow is added to the reverse optical flow to obtain a second sampling offset; simultaneously, the second concatenated feature is input into a mask estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the mask estimation network outputs a normalized second attention mask; finally, the second sampling offset and the second attention mask are used to adaptively spatially sample the ground truth value of the input current video grayscale frame to output the spatially aligned current frame color alignment feature.

[0061] Dynamic fusion module DF Figure 11 As shown, it specifically includes: First, the first luminance channel and the second luminance channel are extracted from the color alignment features and preliminary coloring results of the previous frame, respectively. Second, the element-wise absolute difference between the first luminance channel and the second luminance channel is calculated. Third, based on the element-wise absolute difference and the semantic similarity map, a first fusion weight for the color alignment features of the previous frame and a second fusion weight for the preliminary coloring results are generated, respectively. Finally, the color alignment features of the previous frame and the preliminary coloring results are weighted and summed according to the first fusion weight and the second fusion weight to output the association features.

[0062] The dynamic fusion module DF first passes through the channel extractor From respectively and Extracting the luminance component and Subsequently, the element-wise absolute difference between the two brightness features was calculated. This is used to quantify the degree of content change caused by object movement or scene transitions. This difference... The inverted value is then input into the Sigmoid function to generate attention weights for the alignment path. Simultaneously, the semantic similarity graph S also generates attention weights for non-local paths using the Sigmoid function. .in, A larger value indicates higher inter-frame brightness consistency, and the scene is relatively static, making it more dependent on... Frame to Frame color alignment features To strengthen timing constraints; and The larger the value, the more significant the semantic change in the current region (such as the movement of dynamic objects or camera transitions). In this case, more reference should be made to the [previous text]. Initial coloring result of the frame This enhances semantic matching. This inter-frame brightness difference perception mechanism enables the model to dynamically and autonomously adjust the fusion strategy according to the scene, overcoming the limitations of traditional static weighting or single feature dependence, and achieving a dynamic balance between temporal constraints and semantic reference.

[0063] Based on weight and The DF module performs weighted fusion of the two features, as shown in formula (8):

[0064] in, for Frame pointing Frame association characteristics.

[0065] DF combines the temporal alignment information provided by the alignment subnet with the semantic-based coloring information generated by the semantic correspondence module, thus compensating for the shortcomings of a single module in terms of temporal continuity or content accuracy. (Association Features) At the same time, it retained Frame timing information and The semantic information of the frames is extracted, thus ensuring both temporal consistency and content matching in subsequent processing. Furthermore, the entire fusion structure is very lightweight, relying solely on feature difference calculation and weighted fusion to meet the real-time requirements of video colorization tasks while maintaining high performance.

[0066] In this embodiment, the color refinement sub-mesh (CR) is the core module for the model to generate the final refined color image. Its function is to map the fused features after inter-frame alignment to... Color space Channel, and then the input brightness channel. Combined, they form a complete color image. For example... Figure 12 As shown, this subnet adopts an encoder-decoder structure, which includes an encoder and a decoder containing an upsampled parallel attention layer (UPAL). Through multi-level feature extraction, residual fusion and feature enhancement, it achieves accurate prediction from structural features to precise color.

[0067] The encoder comprises a shallow semantic feature representation layer, three downsampling layers, a hole enhancement layer, and a deep semantic feature representation layer, which are sequentially connected in series; denoted as follows: , , , , , Its detailed structure is as follows: Figure 13As shown.

[0068] (1) Shallow semantic feature representation layer This section consists of 3 consecutive The system consists of convolutional layers, each followed by a ReLU activation function, and finally an instance-normalized IN function. The first two convolutional layers take the input features... The number of channels increased from 6 to 64, and the third convolutional layer was used for initial feature refinement. The instance normalization layer was used to stabilize the feature distribution during training and avoid gradient oscillations.

[0069] (2) Downsampling layer This section consists of three cascaded downsampling blocks, with the following structure: Figure 14 As shown. The core of the downsampling layer is a convolutional kernel. Grouped convolution with a specific number of groups, while setting... This achieves a 2x downsampling, reducing computational load while maintaining channel specificity. , and The number of groups in the grouped convolutions are 64, 128, and 256, respectively. Three groups are then appended to the grouped convolutions. Ordinary convolution, ReLU activation, and instance normalization are used to increase channel dimensions and refine features while compressing spatial dimensions.

[0070] (3) Void reinforcement layer This section consists of two stacked void reinforcement layers, the structure of which is as follows: Figure 15 As shown. Each void reinforcement layer uses 3 layers with a dilation rate of 2. Dilated convolution significantly expands the receptive field without increasing the number of parameters, thereby capturing the global color context and semantic relationships of the image. A subsequent instance normalization layer stabilizes the feature distribution.

[0071] (4) Deep semantic feature representation layer This section consists of 3 layers. It consists of ordinary convolution, ReLU activation function and instance normalization, and aims to further refine local detailed features based on the obtained broad receptive field, so as to provide high-dimensional semantic features (512 channels, size H / 8×W / 8) for the decoder.

[0072] The changes in the number of output channels and feature map size at each stage of the encoder are shown in Table 2.

[0073] Table 2 Encoder Parameter Settings

[0074] The decoder is responsible for progressively upsampling the high-dimensional semantic features output by the encoder to the original resolution and predicting the final... Color channels, their structure is as follows Figure 16 As shown, it includes a first upsampling layer, a second upsampling layer, and an upsampling attention layer connected in series, denoted as follows: , and .

[0075] (1) Upsampling layer The structure of the upsampling layer is as follows Figure 17 As shown, taking the first upsampling layer as an example, the input... For the feature map of a certain size, first, the feature map size is upsampled by a factor of 2 using Nearest Neighbor Interpolation (NNI), followed by a... Convolutional layers are used to integrate contextual semantic information and reduce the number of channels from 512 to 256. The size of the encoder output is... The middle layer features are processed in layer 1 Convolution yields residual features to unify their distribution, facilitating fusion. Next, the upsampled features and residual features are added element-wise, activated by ReLU, and then two consecutive layers are applied. The convolutional layer performs feature optimization, and the final output is normalized by instance.

[0076] (2) Upsampling attention layer The upsampling attention layer UPAL is key to generating the final color; its internal structure is as follows: Figure 18 As shown. Input The feature map of this size, similar to the upsampling layer, first performs a 2x upsampling and convolution on the input, then concatenates it with the encoder output of size [size missing]. Shallow features (via After convolutional adjustment, the features are fused. The fused features are then fed into a parallel attention layer (PAL) to enhance key features related to color generation and suppress irrelevant information. Finally, through one... Convolution and LeakyReLU activation functions further refine the features, and then use... Convolution compresses the number of channels to 2, corresponding to Color space The Tanh activation function constrains the output value to the interval [-1, 1], then multiplies it by 128 to match the space. The typical numerical range of the channel is [-128, 127].

[0077] Parallel Attention Layer (PAL) better preserves detail information in image output. The high-resolution features generated by PAL contain rich spatial details without requiring downsampling. PAL consists of four multi-layer attention modules (MLAM) connected in series, and each MLAM module is composed of eight parallel attention blocks (PABs) connected in series. Each PAB module consists of pixel attention blocks and channel attention blocks connected in parallel. The PAB module structure is as follows: Figure 19 As shown, it contains two parallel sub-branches: the pixel attention (PA) branch and the channel attention (CA) branch.

[0078] Pixel attention focuses more on the edges and textures of objects in an image, paying better attention to pixel features. Therefore, incorporating pixel attention can improve the detail recovery performance of colored video frames. The input feature map first passes through two consecutive... The system consists of convolutional layers, followed by ReLU activation. These convolutional operations extract local spatial features. Then, sigmoid activation is used to generate pixel attention weights, which are then multiplied element-wise with the original input feature map. The purpose of this process is to highlight pixel regions carrying important semantic information, enabling the model to focus on relevant object locations and details.

[0079] In the channel attention branch, the input feature map is first subjected to a global pooling layer to aggregate spatial dimensional information. Then, two layers with ReLU activation functions sandwiched in between are applied. Convolutional layers. The first convolutional layer reduces the channel dimension to improve computational efficiency, while the second restores the number of channels. Next, a sigmoid activation generates a channel attention map, which is then multiplied element-wise with the original input feature map. This process allows the model to selectively emphasize or suppress different channels, effectively capturing the interdependencies between channels and highlighting channels more relevant to color mapping. Finally, the outputs of the PA and CA branches are concatenated along the channel dimension. Then, through a single... The convolutional layers fuse and concatenate the features, then add them element-wise to the original input feature map. This design ensures that the enhanced features not only include information from the recalibration by the attention mechanism but also retain the original feature information.

[0080] PAL utilizes a parallel attention mechanism to adaptively recalibrate feature maps, allowing the model to focus on more informative regions while effectively suppressing irrelevant information. This is crucial for improving the overall performance of the network.

[0081] The output parameters of each stage of the decoder are shown in Table 3. Table 3 Decoder Parameter Settings

[0082] The total loss function in this embodiment It consists of the following parts: (1) Perceived loss

[0083] To enhance the perceptual plausibility of the output image, this invention introduces perceptual loss to measure the generated video frames. With real video frames Differences at the semantic level:

[0084] in, and These represent the output video frames extracted from the VGG19 network and the actual video frames at the [number]th ... Feature maps of each convolutional block.

[0085] In this invention, Indicates the first in the VGG19 network A convolutional block. For example... Figure 3 As shown, this invention selects the 5th convolutional block of VGG19 (i.e. The features are from the second convolutional layer in the video frame. These deep features, after multiple convolutions and abstractions, capture high-level semantic information of the video frame, rather than low-level details such as edges and textures. Therefore, they can effectively measure the perceptual semantic consistency between the output video frame and the real video frame, providing a reliable feature basis for perceptual loss. Perceptual Loss It helps the network select appropriate colors from the color space and spread them effectively.

[0086] (2) Context loss

[0087] This invention employs contextual loss to ensure that the colors of the output video frames are consistent with those of the reference image. This loss effectively achieves color transfer in semantically relevant regions by measuring local feature similarity and combining it with overall image context information.

[0088] First, calculate the values ​​from the output video frames. Feature points in Compared with reference image Feature points in Cosine distance between And perform normalization:

[0089] in, This represents the normalized relative distance. express and The original cosine distance between them This indicates the feature points in the reference image. The minimum value of cosine distance, It represents a very small constant.

[0090] Furthermore, the pairwise similarity between feature points is defined as:

[0091] in, Indicates the first Feature points of video frames output in the layer Feature points of the reference image Similarity between them Indicates along the dimension Normalize, Represents the normalized feature distance. This represents the bandwidth parameter, which is set to 0.1 in this invention.

[0092] The range is [0,1], used to measure features. and The degree of similarity. Considering Some objects in It may not exist in the present invention for each feature. Search for it The most similar features .

[0093] Context loss is defined as maximizing the overall similarity between the output video frame and the reference image. This invention uses multi-layer feature maps. And it employs a coarse-to-fine search strategy.

[0094]

[0095] in, This represents the summation of multiple features of VGG19. Indicates hierarchical weight. Indicates the first The number of feature points in the layer Indicates to Each feature point Calculate its relationship with All feature points The maximum similarity, and then for all Sum.

[0096] (3) Smoothing loss

[0097] To enhance the spatial smoothness of the colorization results and avoid color abrupt changes or artifacts, this invention introduces a smoothing loss. This loss is based on the following assumption: if the real video frame If adjacent pixels are similar in chromaticity, then the video frame is output. The adjacent pixels at the corresponding positions should also be similar. Defined as the difference between the current pixel color and the weighted average color of its 8 neighbors:

[0098] in, This represents the total number of pixels involved in the calculation. Indicates the color channel and Summing them separately Iterate through and generate video frames All pixels , Indicates the first Pixels in frame coloring results The value on color channel c, Represents pixels 8 neighboring pixels, Represents pixels and The weights between them express The weighted average of neighboring pixels on color channel c.

[0099] (4) Combating losses

[0100] This invention employs adversarial loss to enhance the realism of generated video frames. Unlike image discriminators, this invention uses a video discriminator to evaluate consecutive frames. In adversarial training, a temporal discriminator identifies frame sequences with flickering or structural anomalies and assigns them lower scores. To minimize this discriminative loss, the colorization network is thus driven to generate more temporally coherent and visually natural video results. However, adversarial training often struggles to stabilize on large-scale datasets such as ImageNet. Therefore, this invention employs a relativistic discriminative strategy, aiming to determine the authenticity of pairs of frames (denoted as ). and Is it compared to the generated frame pair (denoted as...)? and It has higher visual realism. The corresponding adversarial loss function is defined as follows:

[0101] in, This indicates the generation of video frame pairs. (Taken from distribution) ) Expectation Indicates a pair of real video frames (From distribution) ) Expectation This indicates the output of the discriminator.

[0102] The definition of relative discriminator loss is similar.

[0103] (5) loss

[0104] Although the aforementioned loss function enables the network to generate high-quality colorization results given a reference image, this invention still aims to ensure good degradation performance when the reference image and video frame belong to the same scene. In this case, due to the presence of real labels, a method can be introduced... Loss to measure output With real video frames Color differences between them:

[0105] in, and They represent the first The output of the frame is different from the truth value. The values ​​on the chroma channel, express Norm.

[0106] Considering all the above-mentioned losses, the optimization objective of this invention is:

[0107] in, The weighting coefficients for each loss term, and the perceived loss weights. Context loss weights Smoothing loss weights Counteracting loss weights , Loss weights Guided by these combined losses, this invention successfully unifies feature correspondence and color propagation within a single network, enabling the network to generate visually plausible and color-consistent video colorization results based on example images.

[0108] Example 2 This embodiment discloses a video coloring method based on semantic correspondence and alignment. The continuous grayscale frame sequence of the video to be colored and the corresponding color reference frame are input into the video coloring model to obtain the colored color video sequence. The video coloring model is constructed by the video coloring model construction method based on semantic correspondence and alignment disclosed in Embodiment 1.

[0109] The following are the details of the experiments and analysis conducted on this embodiment: 1. Experimental environment and parameter settings All experiments in this invention were conducted on a single NVIDIA L20 GPU. The network was designed and implemented using the PyTorch deep learning platform. The network employed the Adam optimizer, with an initial learning rate set to 2e-4. The StepLR scheduler was used to control the learning rate decay, which was reduced to 0.1 times the original rate every 10 epochs.

[0110] To balance the impact of different loss terms on model optimization, this invention sets the following weighting coefficients for each loss function: perceptual loss weight. Context loss weights Smoothing loss weights Counteracting loss weights , Loss weights .

[0111] Regarding the training strategy, to reduce the model's dependence on a single reference image, this invention uses the first and last frames of each video segment as input reference images. This method enhances the model's robustness while also effectively expanding the diversity of training samples.

[0112] 2. Comparative Experiment and Analysis To verify the performance of SCAColor, this invention compares and demonstrates the coloring results of SCAColor with other methods on the DAVIS and VIDEVO datasets from two dimensions: quantitative indicators and qualitative analysis.

[0113] DDColor and CT 2 DDColor, a typical automatic image colorization method, proposes a dual-decoder structure aimed at recovering reasonable color information; CT 2 An end-to-end Transformer coloring model based on color tagging was constructed, which models the coloring task as a classification problem and introduces a brightness selection module to limit the range of candidate colors, thereby effectively alleviating the problems of semantic color error and insufficient saturation.

[0114] VCGAN and TCVC are typical automatic video colorization methods: VCGAN designs a hybrid generative adversarial network to enhance colorization quality and spatiotemporal consistency through a global feature extractor and a placeholder feature extractor; TCVC proposes a cross-frame bidirectional propagation mechanism to achieve video colorization by generating consecutive adjacent features.

[0115] DeepExemplar and SwinTExCo belong to the category of reference-based video colorization methods: DeepExemplar first proposed an end-to-end example-driven colorization network, which integrates semantic correspondence and color propagation through a unified recurrent framework and guides the colorization process frame by frame with the help of reference images; SwinTExCo introduces the Swin Transformer as the backbone network into the example-based video colorization task, making full use of its global and local feature extraction capabilities to improve the colorization effect.

[0116] (1) Quantitative results and analysis Tables 4 and 5 present the quantitative evaluation results of SCAColor and all the comparison methods after completing the video colorization task on the DAVIS and VIDEVO datasets, respectively, covering PSNR, SSIM, CDC, FID and LPIPS metrics; the publication year is indicated in square brackets after the name of each comparison method.

[0117] Table 4. Video colorization results on the DAVIS dataset (bold indicates optimal, italic indicates suboptimal).

[0118] Table 5. Video colorization results on the VIDEVO dataset (bold indicates optimal, italic indicates suboptimal).

[0119] SCAColor demonstrates robust performance in terms of color fidelity and structural integrity. On the PSNR metric, which measures pixel-level accuracy, SCAColor achieves a best performance of 29.12 dB on the VIDEVO dataset and a near-optimal 27.78 dB on the DAVIS dataset, comparable to state-of-the-art methods such as TCVC and SwinTExCo. This indicates that the method can effectively perform accurate color mapping. On the SSIM metric, which evaluates structural similarity, SCAColor achieves the highest score of 0.9523 on the DAVIS dataset, demonstrating its ability to reconstruct image structural information.

[0120] SCAColor achieved comprehensive leadership across metrics measuring generated quality and visual perception. In the FID (Film ID) metric, reflecting visual realism, SCAColor ranked first on both datasets, significantly lower than other comparative methods, indicating that its colorization results most closely resemble the distribution of real-world color images in the high-level feature space. In the LPIPS (Perceptual Similarity Perception) metric, SCAColor also achieved the best results on both datasets, demonstrating that its colorization results show the smallest difference from real videos in the human visual perception system. These two performance indicators demonstrate SCAColor's superior performance in terms of the visual naturalness of its output.

[0121] SCAColor also performs exceptionally well in temporal consistency, a key challenge in video colorization. SCAColor achieved the lowest CDC value on both datasets, indicating that it can effectively and smoothly propagate the color information of the reference frame to the entire video sequence over time, effectively avoiding flickering and color jumps.

[0122] While different methods have their strengths and weaknesses across various metrics, the SCAColor method achieves comprehensive leadership in the three key metrics of FID, LPIPS, and CDC, while also maintaining a leading position in PSNR and SSIM. This result fully validates the comprehensive advantages of the SCAColor method in generating visually more natural, perceptually more realistic, and temporally more consistent video colorization results.

[0123] (2) Qualitative results and analysis Figure 20 and Figure 21 This embodiment demonstrates the coloring results of SCAColor, a video coloring method based on semantic correspondence and alignment, and five representative video coloring methods—Ddcolor, VCGAN, TCVC, DeepExemplar, and SwinTExCo—in the same video clip at frames 1, 10, 20, 30, and 40.

[0124] In terms of color stability and temporal consistency, SCAColor demonstrated excellent stability throughout the sequence, with smooth and consistent coloring effects and no perceptible flickering or color abrupt changes. In contrast, the SwinTExCo method exhibited significant instability, a deficiency particularly pronounced in the temporal dimension. Specifically, unnatural colored fluorescent noise intermittently appeared on the cow's back; similarly, aberrant color patches that changed over time appeared on the goat's body surface. This inconsistency severely disrupted the visual coherence of the video, indicating a deficiency in its temporal color propagation mechanism.

[0125] In terms of color naturalness and artifact control, SCAColor also outperforms other comparison methods. As shown in the figure, SCAColor renders the most realistic, highly saturated, and semantically consistent colors for animal fur and natural environments. DDColor and VCGAN, on the other hand, tend to produce flat colors or color casts. While the TCVC method is generally stable, its coloring effect appears slightly dull in some frames, indicating insufficient dynamic range. The Deep Exemplar method exhibits color overflow or blurred edges in certain frames.

[0126] In terms of preserving and propagating the colors of the reference frame, the reference-based method is generally superior to the fully automatic method. However, SCAColor not only accurately inherits the reference color in the initial frame, but also effectively propagates this accurate and stable color information to subsequent frames, avoiding unstable artifacts such as those produced by SwinTExCo, and overcoming the problem of gradual color fading or distortion that may occur with Deep Exemplar.

[0127] To visually evaluate the generalization ability of different coloring methods in diverse scenes, this invention qualitatively compares DDColor, TCVC, Deep Exemplar, SwinTExCo, and the method described in this chapter, SCAColor, in three scenes with different semantic features: people, architecture, and natural scenery. The results are as follows: Figure 22 , 23 As shown in Figure 24.

[0128] SCAColor performs excellently in all scenarios in terms of color accuracy and semantic consistency. For example... Figure 22 As shown in the scene depicting people, (e) SCAColor successfully and accurately transferred the clothing colors of the person in the reference image to the target person, and the skin tone was reproduced naturally, making its effect closest to the real image (f). In contrast, while (a) DDColor and (b) TCVC had overall harmonious colors, their saturation was insufficient or there were color casts, failing to make full use of the reference information; (c) Deep Exemplar showed obvious errors in clothing and skin color.

[0129] In terms of maintaining details and structural integrity Figure 23 The contrast is particularly evident in architectural scenes. (e)SCAColor renders the detailed structure and texture of the building facade clearly and with distinct layers. Other methods, such as (a)DDColor, exhibit obvious problems with blurred details and color overflow, resulting in the loss of wall textures.

[0130] (d)SwinTExCo exhibits significant limitations in terms of shading stability and artifact control. The method produces noticeable unstable shading artifacts in all three scene types, manifesting as aberrant colored fluorescent dots scattered across the resulting images. In scenes with people, these fluorescent dots appear on skin and dark clothing; in architectural scenes, they disrupt the clarity of walls; and in landscape scenes, these noise dots are mixed into the grass. This systematic instability severely impairs the visual realism and usability of the generated results, indicating a flaw in its internal color propagation mechanism.

[0131] In conclusion, the cross-scene qualitative comparison fully validates the robustness and superiority of the SCAColor method. It not only accurately understands and transfers the color semantics of the reference image, but also achieves precise coloring on complex structural details, effectively avoiding instability phenomena such as fluorescence noise generated by SwinTExCo. SCAColor generates visually realistic and perceptually natural coloring results in diverse scenarios, demonstrating its powerful potential as a reliable video coloring solution.

[0132] 7.3 Ablation Experiment and Analysis To comprehensively evaluate the contributions of the Bidirectional Optical Flow Guided Semantic Alignment Subnetwork (BOFSA), Feature Purification Residual Layer (FPRL), and Parallel Attention Layer (PAL) in the proposed SCAColor network, this invention designs an ablation experiment scheme as shown in Table 6 and provides detailed experimental results. The ablation experiments were conducted on the DAVIS dataset.

[0133] Table 6 Ablation Experimental Protocol Setup

[0134] The experiments began with a general baseline model (Deep Exemplar) based on a reference colorization task. After introducing a Bidirectional Optical Flow Guided Semantic Alignment Subnet (BOFSA), all metrics were significantly improved, with PSNR increasing from 25.35 dB to 26.88 dB and FID decreasing dramatically from 111.33 to 86.00. This demonstrates that the BOWSA module effectively improves the accuracy of color propagation and enhances the visual quality of the generated images through precise temporal alignment.

[0135] Building upon this foundation, the addition of a Feature Extraction Residual Layer (FPRL) further improves the model's Structure Preservation Simulation (SSIM) and Perceptual Similarity Perception (LPIPS). SSIM increases from 0.9215 to 0.9322, and LPIPS decreases from 0.2011 to 0.1695, demonstrating that this module helps enhance feature representation and optimize detail generation.

[0136] Finally, a parallel attention layer (PAL) was introduced to form the complete SCAColor model, achieving optimal performance across all metrics. In particular, SSIM improved to 0.9523, LPIPS dropped significantly to 0.1217, and FID further decreased to 64.53, indicating that the PAL module played a crucial role in maintaining structural integrity and enhancing visual realism by effectively fusing global and local features.

[0137] like Figure 25As shown, this invention verifies the contribution of each module to the video colorization effect through ablation experiments. The baseline model has significant shortcomings in color consistency and region coherence. After introducing the Bidirectional Optical Flow Semantic Alignment (BOFSA) module, the color continuity between adjacent frames is significantly improved. Further addition of the Feature Purification Residual Layer (FPRL) further enhances the color stability of detail regions and reduces artifacts. Finally, in the complete model incorporating Perceptual Alignment Loss (PAL), the colorization result achieves optimal temporal consistency and visual realism, effectively alleviating color flicker and region blurring issues.

[0138] Ablation experiments fully demonstrate the complementarity and necessity of the three proposed modules, and their gradual integration contributes to the superior performance of the SCAColor model. Example 3 This embodiment discloses a video colorization system based on semantic correspondence and alignment, including: The data input unit is used to receive the grayscale video sequence to be colored and the corresponding color reference frame input by the user; The model loading and execution unit is used to load the video coloring model and call the model to perform frame-by-frame coloring processing on the grayscale video sequence. The video coloring model is constructed by the video coloring model construction method based on semantic correspondence and alignment disclosed in Embodiment 1. The result output unit is used to output the colored video sequence.

[0139] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0140] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0141] Furthermore, the various implementation methods disclosed in this solution can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content invented by this disclosure.

Claims

1. A method for constructing a video colorization model based on semantic correspondence and alignment, characterized in that, Includes the following steps: S1, Obtain the training dataset, which includes multiple sets of training samples. Each set of training samples includes a continuous sequence of color video frames, a corresponding color reference frame, and a corresponding sequence of grayscale video frames. The corresponding color reference frame includes the first and last frames of a continuous color video frame sequence, and the corresponding video grayscale frame sequence is obtained by extracting the luminance channel of each frame from the continuous color video frame sequence. S2, Construct a video coloring network, the video coloring network comprising a semantic correspondence pre-coloring subnet, a bidirectional optical flow guided semantic alignment subnet, and a color refinement subnet connected in sequence; The semantic correspondence pre-coloring subnet is used to generate preliminary coloring results and semantic similarity maps based on the input current grayscale video frame and the corresponding color reference frame. The bidirectional optical flow-guided semantic alignment subnet is used to perform cross-frame color alignment and feature fusion on the preliminary coloring result, semantic similarity map, coloring result of the previous frame and real color value of the current frame, and output associated features and color alignment features of the current frame. The color refinement subnet is used to concatenate and fuse the semantic similarity map, association features, and the current frame color alignment features to output the final coloring result of the current frame. S3, the training dataset is used as input to the video colorization network, and the total loss function is applied. The training process is performed to obtain a well-trained video colorization model; in, In order to perceive loss, For context loss, To smooth out the loss, Indicating resistance to loss, express loss, These are the weighting coefficients for each loss term.

2. The video colorization model construction method based on semantic correspondence and alignment as described in claim 1, characterized in that, The semantically corresponding precolored subnet includes: The initial feature extraction module is used to extract multi-scale features of the input current grayscale video frame and the corresponding color reference frame; The feature refinement module is used to optimize the multi-scale features and output the enhanced grayscale frame features and reference frame features. The nonlocal semantic correspondence module is used to calculate the semantic similarity matrix between the enhanced grayscale frame features and the reference frame features, and to generate preliminary coloring results and semantic similarity maps based on the semantic similarity matrix and the color reference frame.

3. The video colorization model construction method based on semantic correspondence and alignment as described in claim 2, characterized in that, The feature refinement module includes a feature purification residual layer, which is composed of multiple cascaded residual blocks and is used to progressively optimize the input multi-scale features. At least one residual block is a shallow residual block used to extract and separate local details from noise, and at least one residual block is a deep residual block used to integrate global context and enhance semantic structure.

4. The video colorization model construction method based on semantic correspondence and alignment as described in claim 1, characterized in that, The bidirectional optical flow-guided semantic alignment subnet includes: The optical flow estimation module is used to calculate the bidirectional optical flow between the ground truth of the current video grayscale frame and the coloring result of the previous frame. The deformable alignment module uses the bidirectional optical flow to spatially align the coloring result of the previous frame with the ground truth value of the current video grayscale frame, generating color alignment features of the previous frame and color alignment features of the current frame. The dynamic fusion module is used to fuse the color alignment features of the previous frame, the preliminary coloring result, and the semantic similarity map, specifically including: First, the first luminance channel and the second luminance channel are extracted from the color alignment features and preliminary coloring results of the previous frame, respectively. Next, calculate the element-wise absolute difference between the first brightness channel and the second brightness channel; Next, based on the element-wise absolute difference and semantic similarity map, a first fusion weight for the color alignment feature of the previous frame and a second fusion weight for the preliminary coloring result are generated respectively. Finally, based on the first fusion weight and the second fusion weight, the color alignment features of the previous frame and the preliminary coloring result are weighted and summed to output the associated features.

5. The video colorization model construction method based on semantic correspondence and alignment as described in claim 4, characterized in that, The optical flow estimation module is a SpyNet network, which includes an image pyramid architecture consisting of multiple cascaded identical processing units. Each processing unit corresponds to a level of the image pyramid, and optical flow is estimated by progressively refining from low resolution to high resolution.

6. The video colorization model construction method based on semantic correspondence and alignment as described in claim 4, characterized in that, The color alignment feature of the previous frame in the deformable alignment module is aligned through the following steps: First, the coloring result of the previous frame is concatenated with the input forward optical flow to obtain the first concatenated feature. Secondly, the first concatenated features are input into an offset estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the offset estimation network outputs the first residual optical flow. Next, the first residual optical flow is added to the forward optical flow to obtain the first sampling offset; at the same time, the first concatenated features are input into a mask estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the mask estimation network outputs a normalized first attention mask. Finally, the first sampling offset and the first attention mask are used to adaptively sample the coloring result of the previous frame input, and the spatially aligned color alignment feature of the previous frame is output. The current frame color alignment feature in the deformable alignment module achieves feature alignment through the following steps: First, the ground truth value of the current video grayscale frame is concatenated with the input reverse optical flow to obtain the second concatenated feature. Secondly, the second concatenated features are input into an offset estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the offset estimation network outputs the second residual optical flow. Next, the second residual optical flow is added to the reverse optical flow to obtain the second sampling offset; at the same time, the second concatenated features are input into a mask estimation network composed of multiple concatenated convolutions, and the last convolutional layer of the mask estimation network outputs a normalized second attention mask; Finally, the second sampling offset and the second attention mask are used to adaptively spatially sample the ground truth value of the current video grayscale frame, and the spatially aligned color alignment feature of the current frame is output.

7. The video colorization model construction method based on semantic correspondence and alignment as described in claim 1, characterized in that, The color refinement subnet includes an encoder, which includes a shallow semantic feature representation layer, a downsampling layer, a hole enhancement layer and a deep semantic feature representation layer connected in series. The color refinement subnet also includes a decoder, which comprises a first upsampling layer, a second upsampling layer, and an upsampling attention layer connected in series.

8. The video colorization model construction method based on semantic correspondence and alignment as described in any one of claims 1 to 7, characterized in that, Perceived loss weights Context loss weights Smoothing loss weights Counteracting loss weights , Loss weights .

9. A video colorization method based on semantic correspondence and alignment, characterized in that, The continuous grayscale video frame sequence to be colored and the corresponding color reference frame are input into the video coloring model to obtain the colored video sequence. The video coloring model is constructed by the video coloring model construction method based on semantic correspondence and alignment as described in any one of claims 1-8.

10. A video coloring system based on semantic correspondence and alignment, characterized in that, include: The data input unit is used to receive the grayscale video sequence to be colored and the corresponding color reference frame input by the user; The model loading and execution unit is used to load the video coloring model and call the model to perform frame-by-frame coloring processing on the grayscale video sequence. The video coloring model is constructed by the video coloring model construction method based on semantic correspondence and alignment as described in any one of claims 1-8. The result output unit is used to output the colored video sequence.

Images