A video recoloring method based on similar frame assistance

Abstract

A video coloring method based on similar frame assistance aims to significantly improve video quality. First, through a preprocessing step, key frames are extracted from correctly colored videos and necessary image processing is performed, while generating reference frames similar to the frames to be colored. Next, these frames (black and white, correctly colored and similarly colored) are jointly input into a deep learning network for training, which can learn and utilize the information between these frames to improve the coloring effect. Finally, using a confidence ranking mechanism, the prediction results are evaluated and corrected by comparing the similarity of the model output with the correctly colored frames and the similarly colored frames. This method not only improves the visual effect of the video, but also provides an innovative solution for the video processing field.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video colorization method based on similar frame assistance. Background Technology

[0002] Video colorization, which converts black-and-white videos into color videos, is an important research topic in the field of digital image processing. The development and implementation of this technology is not only of great significance for the restoration of historical video materials, but also provides a broad application space for modern film production, advertising production, and personal video editing. The origin of video colorization technology can be traced back to the early 20th century, when most films were black and white. With the development of technology, people began to try to colorize these films manually, such as by coloring the film or using coloring templates. However, this method was time-consuming and labor-intensive, and the effect was limited by the technological level at the time. By the mid-20th century, with the emergence of color film technology, people's interest in colorizing early black and white films gradually increased. With the advancement of computer technology, video colorization began to shift to digital processing. The initial digital colorization technology relied on manual operation. Artists needed to color each frame of the image individually, which was still very time-consuming. However, over time, automated and semi-automated methods began to emerge, greatly improving the efficiency and quality of colorization.

[0003] The basic principle of video colorization is to analyze black and white images using algorithms and predict the color value of each pixel. This process involves multiple aspects, including color theory, image segmentation, feature extraction, and machine learning. Understanding how colors combine and interact in nature is crucial for predicting the correct color. Through image segmentation, an image can be divided into multiple regions or objects, each of which can be colorized independently. Sufficient information, such as texture, shape, and edges, can be extracted from black and white images to help the algorithm understand the image content. Machine learning algorithms, especially deep learning, are used to analyze large amounts of color image data to train models for color prediction.

[0004] Modern video colorization techniques increasingly rely on deep learning and artificial intelligence. By training on large amounts of color image data, deep learning models can learn complex color patterns and automatically colorize black and white images. These methods not only improve the accuracy of colorization but also significantly increase processing speed.

[0005] Video colorization technology has a wide range of applications. In the restoration of historical images, it restores the colors of historical films, documentaries and old photos, providing richer visual materials for historical research and education. In film and advertising production, video colorization can be used to create specific visual styles or emotional effects. For personal entertainment, colorizing personal videos can enhance their visual appeal and emotional expression.

[0006] In the future, with the improvement of computing power and the continuous progress of deep learning technology, video colorization technology will become more efficient and accurate. In addition, with the development of artificial intelligence, we may see more innovative applications, such as real-time video colorization and personalized color selection. The development of video colorization technology not only represents the progress of technology, but also provides us with a new way to view and understand history, while creating more possibilities for modern visual arts.

[0007] However, existing video colorization techniques are not natural and accurate enough when used, and the overall coherence and naturalness are insufficient. To solve the above problems, a video colorization method based on similar frame assistance is proposed. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a video colorization method based on similar frame assistance. The core of this method lies in utilizing a "similar frame assistance" strategy to improve the video colorization process. This method uses color information from similar frames to guide and optimize the colorization process. By combining color information extracted from the original frame and similar frames, this method can generate more natural and accurate colorization results. This step also includes fine-tuning the colors to ensure overall consistency and naturalness. Specifically, it comprises the following steps:

[0009] S1. Video Preprocessing Stage: Image similarity processing is an image editing technique designed to enhance the visual effect of an original image or adapt it to a specific visual style through a series of fine adjustments. These processes include:

[0010] First, color temperature adjustment, by varying the range by 5% above and below the correct coloring frame, adjusts the warm or cool tone of the image to create or improve a specific atmosphere, making the image appear warmer or calmer. Next is HSV adjustment, which adjusts hue, saturation, and brightness, also within a range of 7% above and below the correct coloring frame. Hue adjustment changes the main color tone of the image, saturation adjustment is used to enhance or weaken the vividness of colors, brightness adjustment affects the overall brightness, contrast adjustment has a range of up to 15% for the darkest and brightest areas, and 10% for other areas, which helps to enhance the depth and dimension of the image, increase vividness and dynamism, or create a softer, smoother effect. Finally, brightness adjustment changes the overall brightness level of the image, which can be used to correct exposure problems or create a specific visual atmosphere. Through these comprehensive adjustment methods, we can significantly improve the naturalness and realism of the coloring effect.

[0011] S2. Coloring Network: First, all input frames—including similarly colored frames, black and white frames, and correctly colored frames—are scaled proportionally to a uniform size of 340×340 pixels. This step ensures that all input data is consistent in size, which helps reduce the computational burden when the model is processed, while also maintaining the integrity of the image content.

[0012] Subsequently, these resized frames are normalized, specifically according to the mean and variance of the ImageNet dataset. Normalization is a standard step in deep learning preprocessing, which helps the model converge faster and improves the stability of the training process. By using the mean and variance of ImageNet, it is ensured that the input data is consistent with a large number of pre-trained models in terms of color distribution, thereby leveraging the advanced capabilities of these models in color recognition and processing.

[0013] Finally, these carefully preprocessed frames are fed into the coloring network for training.

[0014] The structure of the coloring network in step S2 is as follows:

[0015] Block 1:

[0016] Three 3×3 convolutional layers, each with 32 kernels, followed by a GeLU activation function, and a 3×3 max-pooling layer with a stride of 3.

[0017] The second block (Block 2) consists of three 5×5 convolutional layers, each with 64 kernels. Each convolutional layer is followed by a GeLU activation function and a 5×5 max-pooling layer with a stride of 5.

[0018] The third block:

[0019] Two 7×7 convolutional layers, each with 128 convolutional kernels, followed by a GeLU activation function after each convolutional layer, a 7×7 max pooling layer with a stride of 7, and blocks 1-3 are the pre-feature extraction blocks;

[0020] Block 4:

[0021] Two 7×7 convolutional layers, each with 128 kernels, followed by a GeLU activation function, and a 5×5 max-pooling layer with a stride of 5.

[0022] Block 5:

[0023] Two 5×5 convolutional layers, each with 128 convolutional kernels, followed by a GeLU activation function after each convolutional layer, a 5×5 max pooling layer with a stride of 5, and the 4th-5th blocks are post-feature extraction blocks;

[0024] First, three independent pre-feature extraction blocks are introduced, which process black and white frames, correctly colored frames, and similarly colored frames respectively. The aim is to extract key feature information from each type of frame. This step is crucial because it provides the foundation for subsequent feature fusion and optimization.

[0025] The output features of each preceding feature extraction block are then integrated and fused using a weighted summation method. In this process, each type of feature is assigned a corresponding weight based on its importance to the final coloring result. The weights can be determined based on experience or through data-driven methods. This weighted summation process can be represented as:

[0026] F combined =w1·F bw +w2·F correct +w3·F similar ,

[0027] Where F combined F represents the characteristics after fusion. bw F correct and F similar These represent the features of black and white frames, correctly colored frames, and similarly colored frames, respectively, while w1, w2, and w3 are the corresponding weights.

[0028] Next, these fused features are fed into the post-feature extraction block for further processing and refinement to form more discriminative feature representations. The post-feature extraction block optimizes and refines the features through further deep learning techniques, such as convolutional layers or fully connected layers.

[0029] These optimized features were then used to calculate the cross-entropy loss to evaluate the difference between the model's predictions and the actual values. Cross-entropy loss is a loss function commonly used in supervised learning to measure the difference between predicted and true values, and its calculation formula is as follows:

[0030] L cross-entropy =-∑(ylog(p)+(1-y)log(1-p)),

[0031] Where y is the true label and p is the probability predicted by the model.

[0032] Finally, based on these losses, the data enters the scoring and ranking module, where the final evaluation and sorting will be performed to complete the entire image processing workflow. The scoring and ranking module may include a series of scoring criteria, such as color accuracy and naturalness, to ensure the quality of the coloring results. Through this comprehensive processing workflow, we can achieve high-quality and natural video coloring effects.

[0033] S3. Result Ranking:

[0034] Scoring and ranking module:

[0035] Cosine similarity calculation layer

[0036] Fully connected layer 1:256 dimensions

[0037] Fully connected layer 2: 128-dimensional

[0038] Fully connected layer 3:2D;

[0039] The output of the post-feature extraction block first enters the scoring and ranking module. In this module, cosine similarity is calculated to determine the similarity between black and white frames, correctly colored frames, and similarly colored frames. This step is based on the principle of cosine similarity, evaluating the similarity of feature vector directions between different frames. The similarity calculation formula is as follows:

[0040]

[0041] A·B represents the dot product of two vectors A and B.

[0042]

[0043] ||A|| and ||B|| are the Euclidean norms of vectors A and B, respectively.

[0044]

[0045] Therefore, cosine similarity actually calculates the cosine of the angle between two vectors in n-dimensional space. This value ranges from -1 to 1, where 1 indicates that the two vectors are in the same direction, 0 indicates that they are orthogonal, and -1 indicates that they are in opposite directions. In many applications, especially in the case of positive data (such as image pixels), the value of cosine similarity is usually between 0 and 1.

[0046] Next, the features of correctly colored frames and similarly colored frames are fed into a fully connected layer for further scoring. In this stage, the L2 loss is calculated, a method for measuring the difference between predicted and actual values, commonly used in regression tasks and for optimizing model accuracy.

[0047]

[0048] Where yi is the true value of the i-th sample. It is the model prediction value of the i-th sample, and n is the total number of samples;

[0049] S4. Joint Loss Optimization:

[0050] Subsequently, this L2-norm loss is combined with the previously calculated cross-entropy loss. These two losses are weighted in a 1.1:0.8 ratio. This weighting combines two different types of losses to simultaneously optimize the model's performance in different aspects—the cross-entropy loss focuses on classification accuracy, while the L2-norm loss focuses on regression accuracy. Finally, the combined total loss is used in the model's backpropagation process.

[0051] Loss total =1.1·Loss CE +0.8 Loss L2 ,

[0052] Stochastic Gradient Descent (SGD) is used as the optimizer to fine-tune the model parameters. The momentum parameter of the SGD optimizer is set to 0.8, which helps to accelerate the training process of the model while maintaining sufficient stability. The momentum mechanism plays a role in smoothing the optimization path and effectively avoids excessive oscillation of the model during training.

[0053] The initial learning rate is set to 2e-4, which is a relatively small value designed to ensure a smooth optimization process and avoid overfitting, thus preventing potential overfitting problems. In machine learning, the appropriate choice of learning rate is crucial because it directly affects the speed and effectiveness of model learning.

[0054] A batch size of 16 samples is set. A smaller batch size is beneficial to increasing the generalization ability of the model during training, while maintaining reasonable requirements for hardware resources. In each iteration, the model parameters are updated according to the calculated gradient of the loss function. In this process, a weighted loss function is designed, which combines cross-entropy loss and L2 loss, targeting classification accuracy and regression precision respectively. Through this method, the model can be optimized in multiple aspects, thereby improving the accuracy of the regression task while maintaining classification accuracy.

[0055] The beneficial effects of this invention are:

[0056] By combining these strategies, this invention ensures that the model is optimized in an efficient and robust manner during training, thereby maximizing its overall performance. This meticulous training strategy enables the model to achieve higher accuracy and better results when dealing with complex video colorization tasks. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall process of a video colorization method based on similar frame assistance according to the present invention;

[0058] Figure 2 This is a schematic diagram of the coloring network and scoring network structure of the present invention;

[0059] Figure 3 This is a schematic diagram of the loss function of the present invention. Detailed Implementation

[0060] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Example 1

[0062] This invention provides a video colorization method based on similar frame assistance. The core of this method lies in using a "similar frame assistance" strategy to improve the video colorization process. This method utilizes color information from similar frames to guide and optimize the colorization process. By combining color information extracted from the original frame and similar frames, this method can generate more natural and accurate colorization results. This step also includes fine-tuning the colors to ensure overall consistency and naturalness. Specifically, it consists of the following steps:

[0063] S1. Video Preprocessing Stage: Image similarity processing is an image editing technique designed to enhance the visual effect of an original image or adapt it to a specific visual style through a series of fine adjustments. These processes include:

[0064] First, color temperature adjustment, by varying the range by 5% above and below the correct coloring frame, adjusts the warm or cool tone of the image to create or improve a specific atmosphere, making the image appear warmer or calmer. Next is HSV adjustment, which adjusts hue, saturation, and brightness, also within a range of 7% above and below the correct coloring frame. Hue adjustment changes the main color tone of the image, saturation adjustment is used to enhance or weaken the vividness of colors, brightness adjustment affects the overall brightness, contrast adjustment has a range of up to 15% for the darkest and brightest areas, and 10% for other areas, which helps to enhance the depth and dimension of the image, increase vividness and dynamism, or create a softer, smoother effect. Finally, brightness adjustment changes the overall brightness level of the image, which can be used to correct exposure problems or create a specific visual atmosphere. Through these comprehensive adjustment methods, we can significantly improve the naturalness and realism of the coloring effect.

[0065] S2. Coloring Network: First, all input frames—including similarly colored frames, black and white frames, and correctly colored frames—are scaled proportionally to a uniform size of 340×340 pixels. This step ensures that all input data is consistent in size, which helps reduce the computational burden when the model is processed, while also maintaining the integrity of the image content.

[0066] Subsequently, these resized frames are normalized, specifically according to the mean and variance of the ImageNet dataset. Normalization is a standard step in deep learning preprocessing, which helps the model converge faster and improves the stability of the training process. By using the mean and variance of ImageNet, it is ensured that the input data is consistent with a large number of pre-trained models in terms of color distribution, thereby leveraging the advanced capabilities of these models in color recognition and processing.

[0067] Finally, these carefully preprocessed frames are fed into the coloring network for training.

[0068] The structure of the coloring network in step S2 is as follows:

[0069] Block 1:

[0070] Three 3×3 convolutional layers, each with 32 kernels, followed by a GeLU activation function, and a 3×3 max-pooling layer with a stride of 3.

[0071] The second block (Block 2) consists of three 5×5 convolutional layers, each with 64 kernels. Each convolutional layer is followed by a GeLU activation function and a 5×5 max-pooling layer with a stride of 5.

[0072] The third block:

[0073] Two 7×7 convolutional layers, each with 128 convolutional kernels, followed by a GeLU activation function after each convolutional layer, a 7×7 max pooling layer with a stride of 7, and blocks 1-3 are the pre-feature extraction blocks;

[0074] Block 4:

[0075] Two 7×7 convolutional layers, each with 128 kernels, followed by a GeLU activation function, and a 5×5 max-pooling layer with a stride of 5.

[0076] Block 5:

[0077] Two 5×5 convolutional layers, each with 128 convolutional kernels, followed by a GeLU activation function after each convolutional layer, a 5×5 max pooling layer with a stride of 5, and the 4th-5th blocks are post-feature extraction blocks;

[0078] First, three independent pre-feature extraction blocks are introduced, which process black and white frames, correctly colored frames, and similarly colored frames respectively. The aim is to extract key feature information from each type of frame. This step is crucial because it provides the foundation for subsequent feature fusion and optimization.

[0079] The output features of each preceding feature extraction block are then integrated and fused using a weighted summation method. In this process, each type of feature is assigned a corresponding weight based on its importance to the final coloring result. The weights can be determined based on experience or through data-driven methods. This weighted summation process can be represented as:

[0080] F combined =w1·F bw +w2·F correct +w3·F similar ,

[0081] Where F combined F represents the characteristics after fusion. bw F correct and F similar These represent the features of black and white frames, correctly colored frames, and similarly colored frames, respectively, while w1, w2, and w3 are the corresponding weights.

[0082] Next, these fused features are fed into the post-feature extraction block for further processing and refinement to form more discriminative feature representations. The post-feature extraction block optimizes and refines the features through further deep learning techniques, such as convolutional layers or fully connected layers.

[0083] These optimized features were then used to calculate the cross-entropy loss to evaluate the difference between the model's predictions and the actual values. Cross-entropy loss is a loss function commonly used in supervised learning to measure the difference between predicted and true values, and its calculation formula is as follows:

[0084] L cross-entropy =-∑(ylog(p)+(1-y)log(1-p)),

[0085] Where y is the true label and p is the probability predicted by the model.

[0086] Finally, based on these losses, the data enters the scoring and ranking module, where the final evaluation and sorting will be performed to complete the entire image processing workflow. The scoring and ranking module may include a series of scoring criteria, such as color accuracy and naturalness, to ensure the quality of the coloring results. Through this comprehensive processing workflow, we can achieve high-quality and natural video coloring effects.

[0087] S3. Result Ranking:

[0088] Scoring and ranking module:

[0089] Cosine similarity calculation layer

[0090] Fully connected layer 1:256 dimensions

[0091] Fully connected layer 2: 128-dimensional

[0092] Fully connected layer 3:2D;

[0093] The output of the post-feature extraction block first enters the scoring and ranking module. In this module, cosine similarity is calculated to determine the similarity between black and white frames, correctly colored frames, and similarly colored frames. This step is based on the principle of cosine similarity, evaluating the similarity of feature vector directions between different frames. The similarity calculation formula is as follows:

[0094]

[0095] A·B represents the dot product of two vectors A and B.

[0096]

[0097] ||A|| and ||B|| are the Euclidean norms of vectors A and B, respectively.

[0098]

[0099] Therefore, cosine similarity actually calculates the cosine of the angle between two vectors in n-dimensional space. This value ranges from -1 to 1, where 1 indicates that the two vectors are in the same direction, 0 indicates that they are orthogonal, and -1 indicates that they are in opposite directions. In many applications, especially in the case of positive data (such as image pixels), the value of cosine similarity is usually between 0 and 1.

[0100] Next, the features of correctly colored frames and similarly colored frames are fed into a fully connected layer for further scoring. In this stage, the L2 loss is calculated, a method for measuring the difference between predicted and actual values, commonly used in regression tasks and for optimizing model accuracy.

[0101]

[0102] Where yi is the true value of the i-th sample. It is the model prediction value of the i-th sample, and n is the total number of samples;

[0103] S4. Joint Loss Optimization:

[0104] Subsequently, this L2-norm loss is combined with the previously calculated cross-entropy loss. These two losses are weighted in a 1.1:0.8 ratio. This weighting combines two different types of losses to simultaneously optimize the model's performance in different aspects—the cross-entropy loss focuses on classification accuracy, while the L2-norm loss focuses on regression accuracy. Finally, the combined total loss is used in the model's backpropagation process.

[0105] Loss total =1.1·Loss CE +0.8 Loss L2 ,

[0106] Stochastic Gradient Descent (SGD) is used as the optimizer to fine-tune the model parameters. The momentum parameter of the SGD optimizer is set to 0.8, which helps to accelerate the training process of the model while maintaining sufficient stability. The momentum mechanism plays a role in smoothing the optimization path and effectively avoids excessive oscillation of the model during training.

[0107] The initial learning rate is set to 2e-4, which is a relatively small value designed to ensure a smooth optimization process and avoid overfitting, thus preventing potential overfitting problems. In machine learning, the appropriate choice of learning rate is crucial because it directly affects the speed and effectiveness of model learning.

[0108] A batch size of 16 samples is set. A smaller batch size is beneficial to increasing the generalization ability of the model during training, while maintaining reasonable requirements for hardware resources. In each iteration, the model parameters are updated according to the calculated gradient of the loss function. In this process, a weighted loss function is designed, which combines cross-entropy loss and L2 loss, targeting classification accuracy and regression precision respectively. Through this method, the model can be optimized in multiple aspects, thereby improving the accuracy of the regression task while maintaining classification accuracy.

[0109] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. The various components mentioned in this invention are common technologies in the existing field. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A video colorization method based on similar frame assistance, characterized in that, Includes the following steps: S1. Video preprocessing: First, obtain the correctly colored video and perform similarity processing on each frame to obtain similar frames; S2. Training with three types of frames: Black and white frames, correctly colored frames, and similarly colored frames are all fed into the coloring network for training; S3. Result Confidence Ranking: The model output will be compared with the correct coloring frame and similar coloring frame to calculate the similarity and rank the two with the prediction result. Incorrect rankings will be corrected. The structure of the colored network in step S2 is as follows: Block 1: Three 3×3 convolutional layers, each with 32 kernels, followed by a GeLU activation function, and a 3×3 max-pooling layer with a stride of 3. Second Block 2: Three 5×5 convolutional layers, each with 64 kernels, followed by a GeLU activation function, and a 5×5 max-pooling layer with a stride of 5. Block 3: Two 7×7 convolutional layers, each with 128 kernels, followed by a GeLU activation function, and a 7×7 max-pooling layer with a stride of 7. Blocks 1-3 are the preliminary feature extraction blocks; Block 4: Two 7×7 convolutional layers, each with 128 kernels, followed by a GeLU activation function, and a 5×5 max-pooling layer with a stride of 5. Block 5: Two 5×5 convolutional layers, each with 128 kernels, followed by a GeLU activation function, and a 5×5 max-pooling layer with a stride of 5. Blocks 4-5 are the post-feature extraction blocks; Three independent pre-processing feature extraction blocks process black-and-white frames, correctly colored frames, and similarly colored frames respectively to extract their key features. These extracted features are then combined using a weighted stacking method, where each feature receives a different weight based on its importance. The combined features are then fed into a post-processing feature extraction block for further processing and refinement to form a more discernible feature representation. These features are then used to calculate the cross-entropy loss to evaluate the difference between the prediction and the actual value. Finally, based on these losses, the data enters the scoring and ranking module for final evaluation and sorting, completing the entire image processing workflow.

2. The video colorization method based on similar frame assistance according to claim 1, characterized in that, Similarity processing in S1 is an image editing technique that enhances visual effects and adapts to visual styles by making a series of fine adjustments to the original image. These adjustments include: changing the color temperature: This involves adjusting the color temperature of the image to give it different hues. The adjustment of color temperature is used to create a sense of atmosphere. HSV adjustment: Changes the color characteristics of an image by adjusting hue, saturation, and brightness. Hue adjustment changes the colors in the image, saturation adjustment changes the intensity of the colors, and brightness adjustment changes the overall brightness of the image. Contrast adjustment: Used to change the difference between the darkest and brightest areas of an image, thereby enhancing the depth and dimension of the image; Brightness Adjustment: Changes the overall brightness level of the image. Brightness adjustment corrects underexposed or overexposed photos and creates visual atmosphere.

3. The video colorization method based on similar frame assistance according to claim 2, characterized in that, The structure of step S3 is as follows: Scoring and ranking module: Cosine similarity calculation layer Fully connected layer 1:256 dimensions Fully connected layer 2: 128-dimensional Fully connected layer 3:2D The output of the post-feature extraction block first enters the scoring and ranking module. In this module, cosine similarity is calculated to determine the similarity between black and white frames and correctly colored frames as well as similar colored frames. This step is based on the principle of cosine similarity and evaluates the similarity of feature vector directions between different frames. Next, the features of correctly colored frames and similar colored frames are fed into a fully connected layer for further scoring. In this stage, the L2 loss is calculated.

4. The video colorization method based on similar frame assistance according to claim 3, characterized in that, The L2 loss is combined with the previously calculated cross-entropy loss and weighted. Finally, the combined total loss is used in the backpropagation process of the model. In backpropagation, the model parameters are updated according to the gradient of the loss function to optimize the model's performance.

Images