Video coloring method based on similar frame assistance

Abstract

A video coloring method based on similar frame assistance aims at remarkably improving video quality and comprises the following steps: firstly, through a preprocessing step, extracting a key frame from a correctly colored video, carrying out necessary image processing, and generating a reference frame similar to a frame to be colored at the same time; the frames (black and white, correct coloring and similar coloring) are jointly input into a deep learning network for training, the network can learn and utilize information among the frames to improve the coloring effect, and finally, by utilizing a confidence ranking mechanism, the similarity between the correct coloring frame and the similar coloring frame is output through a comparison model, so that the coloring effect is improved. According to the method, the visual effect of the video is improved, and an innovative solution is provided for the field of video processing.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a video colorization method based on similar frame assistance. Background Art

[0002] Video colorization technology, that is, converting the original black-and-white video into color video, is an important research topic in the field of digital image processing. The development and implementation of this technology is not only of great significance to the restoration of historical image materials, but also provides a broad application space for modern film production, advertising production and personal video editing. The origin of video colorization technology can be traced back to the early 20th century, when most movies were black-and-white. With the development of technology, people began to try to color these movies manually, such as by coloring the film or using coloring templates. However, this method is time-consuming and labor-intensive, and the effect is limited by the technical level at the time. In the middle of the 20th century, with the emergence of color film technology, people's interest in coloring early black-and-white movies gradually increased. With the advancement of computer technology, video colorization began to turn to digital processing. The initial digital colorization technology relied on manual operation, and artists needed to color each frame of the image separately. This process was still very time-consuming. However, with the passage of time, automated and semi-automated methods began to appear, greatly improving the efficiency and quality of colorization.

[0003] The basic principle of video colorization is to analyze black and white images through algorithms and predict the color value of each pixel. This process involves many aspects, including color theory, image segmentation, feature extraction, and machine learning. Understanding how colors combine and interact in nature is crucial to predicting the correct color. Through image segmentation, the image can be divided into multiple regions or objects, each of which can be colored independently. Sufficient information such as texture, shape, and edges can be extracted from black and white images to help the algorithm understand the image content. Machine learning algorithms, especially deep learning, are used to analyze large amounts of color image data to train the model to predict colors.

[0004] Modern video colorization technology increasingly relies on deep learning and artificial intelligence. By training a large amount of color image data, deep learning models can learn complex color patterns and automatically colorize black and white images. These methods not only improve the accuracy of colorization, but also significantly increase the processing speed.

[0005] The application field of video colorization technology is very wide. In the restoration of historical images, it can restore the colors of historical films, documentaries and old photos, providing richer visual materials for historical research and education. In film and advertising production, video colorization can be used to create a specific visual style or emotional effect. For personal entertainment, coloring personal videos can enhance the visual appeal and emotional expression of the video.

[0006] In the future, with the improvement of computing power and the continuous advancement of deep learning technology, video colorization technology will become more efficient and accurate. In addition, with the development of artificial intelligence, we may see more innovative applications, such as real-time video colorization, personalized color selection, etc. The development of video colorization technology not only represents the advancement of technology, but also provides us with a new way to view and understand history, while creating more possibilities for modern visual art.

[0007] However, the existing video colorization technology is not natural and accurate enough when used, and the overall coherence and naturalness are not enough. To solve the above problems, a video colorization method based on similar frame assistance is proposed. Summary of the invention

[0008] In view of the above technical problems, the present invention provides a video colorization method based on similar frame assistance. The core of the method is to use the "similar frame assistance" strategy to improve the video colorization process. This method can use the color information in these similar frames to guide and optimize the colorization process. By combining the color information extracted from the original frame and the similar frame, the method can generate a more natural and accurate colorization result. This step also includes subtle adjustments to the color to ensure overall coherence and naturalness. Specifically, it is divided into the following steps:

[0009] S1. Video preprocessing stage: Image similarity processing is an image editing technique that aims to enhance the visual effect or adapt it to a specific visual style by making a series of fine adjustments to the original image. These processes include:

[0010] First is the color temperature adjustment, which adjusts the warm and cool tones of the image by changing the range to 5% above and below the correct color frame to create or improve a specific atmosphere, making the image appear warmer or calmer. Next is the HSV adjustment, which is the adjustment of hue, saturation and brightness. The change range is also 7% above and below the correct color frame. Hue adjustment can change the main color of the image, saturation adjustment is used to strengthen or weaken the vividness of the color, and brightness adjustment affects the overall brightness. Contrast adjustment has a fluctuation of up to 15% for the darkest and brightest areas, and 10% for other areas. This helps to enhance the depth and dimension of the image, increase vividness and movement, or create a softer and smoother effect. Finally, brightness adjustment changes the overall brightness level of the image, which can be used to correct exposure problems or create a specific visual atmosphere. Through these comprehensive adjustment methods, we can greatly improve the naturalness and realism of the coloring effect.

[0011] S2. Colorization network: First, all input frames, including similar color frames, black and white frames, and correctly colored frames, are scaled to a uniform size of 340×340 pixels. This step ensures the consistency of the size of all input data, which helps reduce the computational burden of model processing while maintaining the integrity of the image content.

[0012] These resized frames are then normalized to the mean and variance of the ImageNet dataset. Normalization is a standard step in deep learning preprocessing, which helps the model converge faster and improves the stability of the training process. By using the mean and variance of ImageNet, the input data is ensured to be consistent in color distribution with a large number of pre-trained models, thereby taking advantage of the advanced capabilities of these models in color recognition and processing.

[0013] Finally, these carefully pre-processed frames are fed together into the colorization network for training;

[0014] The coloring network of step S2 has the following structure:

[0015] Block 1:

[0016] Three 3×3 convolutional layers, each with 32 kernels, each followed by a GeLU activation function, a 3×3 max pooling layer with a stride of 3,

[0017] The second block (Block 2): 3 5×5 convolutional layers, each with 64 convolution kernels, each followed by a GeLU activation function, a 5×5 maximum pooling layer with a stride of 5,

[0018] Block 3:

[0019] Two 7×7 convolutional layers, each with 128 kernels, each followed by a GeLU activation function, a 7×7 maximum pooling layer with a stride of 7, and blocks 1-3 are pre-feature extraction blocks;

[0020] Block 4:

[0021] Two 7×7 convolutional layers, each with 128 kernels, each followed by a GeLU activation function, a 5×5 max pooling layer with a stride of 5,

[0022] Block 5:

[0023] Two 5×5 convolutional layers, each with 128 kernels, each followed by a GeLU activation function, a 5×5 maximum pooling layer with a stride of 5, and blocks 4-5 are post-feature extraction blocks;

[0024] First, three independent pre-feature extraction blocks are introduced, which process black and white frames, correctly colored frames, and similarly colored frames respectively, aiming to extract key feature information from each type of frame. This step is crucial because it provides the basis for the subsequent feature fusion and optimization.

[0025] The output features of each front feature extraction block are then integrated and fused by weighted superposition. In this process, each type of feature is given a corresponding weight according to its importance to the final colorization result. The weight can be determined based on experience or through data-driven methods. This weighted superposition process can be expressed as:

[0026] F combined =w1·F bw +w2·F correct +w3·F similar ,

[0027] where F combined Represents the fused features, F bw , F correct and F similar They represent the features of black and white frames, correctly colored frames, and similarly colored frames, respectively, while w1, w2, and w3 are the corresponding weights;

[0028] Then, these fused features are fed into the post-feature extraction block for further processing and refinement to form a more discriminative feature representation. The post-feature extraction block optimizes and refines the features through further deep learning techniques, convolutional layers or fully connected layers;

[0029] These optimized features are then used to calculate the cross entropy loss to evaluate the difference between the model's predicted values ​​and the actual values. The cross entropy loss is a loss function commonly used in supervised learning to measure the difference between the predicted values ​​and the actual values. The calculation formula is:

[0030] L cross-entropy =-∑(ylog(p)+(1-y)log(1-p)),

[0031] Where y is the true label and p is the probability predicted by the model.

[0032] Finally, based on these losses, the data enters the scoring and ranking module, where the final evaluation and sorting will be performed to complete the entire image processing process. The scoring and ranking module may include a series of scoring criteria, such as color accuracy, naturalness, etc., to ensure the quality of the colorization results. Through this comprehensive processing flow, we can achieve high-quality and natural video colorization effects;

[0033] S3. Result ranking:

[0034] Scoring and ranking module:

[0035] Cosine similarity calculation layer,

[0036] Fully connected layer 1: 256 dimensions,

[0037] Fully connected layer 2: 128 dimensions,

[0038] Fully connected layer 3: 2D;

[0039] The output of the post-feature extraction block first enters the scoring and ranking module, in which the cosine similarity calculation is performed to determine the similarity between the black and white frame and the correctly colored frame and the similar colored frame. This step is based on the principle of cosine similarity to evaluate the similarity of the feature vector direction between different frames. The similarity calculation formula is:

[0040]

[0041] A·B represents the dot product of two vectors A and B.

[0042]

[0043] ||A|| and ||B|| are the Euclidean norms of vectors A and B, respectively.

[0044]

[0045] Therefore, cosine similarity actually calculates the cosine value of the angle between two vectors in n-dimensional space. This value ranges from -1 to 1, where 1 means that the two vectors are in exactly the same direction, 0 means that they are orthogonal, and -1 means that the two directions are completely opposite. In many applications, especially in the case of positive data (such as image pixels), the value of cosine similarity is usually between 0 and 1;

[0046] Next, the features of the correctly colored frame and the similar colored frame are fed into a fully connected layer for further scoring. In this step, the L2 Loss is calculated, which is a method to measure the difference between the predicted value and the actual value. It is usually used for regression tasks and to optimize the accuracy of the model.

[0047]

[0048] Among them, yi is the true value of the i-th sample, is the model prediction value of the i-th sample, and n is the total number of samples;

[0049] S4. Joint loss optimization:

[0050] Subsequently, this bi-norm loss is combined with the previously calculated cross entropy loss. The two losses are weighted in a ratio of 1.1:0.8. This weighted combination of two different types of losses aims to optimize the performance of the model in different aspects at the same time - the cross entropy loss focuses on classification accuracy, while the bi-norm loss focuses on regression accuracy. Finally, the combined total loss is used in the back-propagation process of the model.

[0051] Loss total =1.1·Loss CE +0.8 Loss L2 ,

[0052] Stochastic Gradient Descent (SGD) is used as the optimizer to fine-tune the model parameters. The momentum parameter of the SGD optimizer is set to 0.8, which helps to accelerate the training process of the model while maintaining sufficient stability. The momentum mechanism plays the role of smoothing the optimization path and effectively avoids excessive oscillation of the model during training.

[0053] The initial learning rate is set to 2e-4, which is a relatively small value designed to ensure the smoothness of the optimization process and avoid over-adjustment, thereby preventing possible overfitting problems. In machine learning, appropriate learning rate selection is crucial because it directly affects the speed and effect of model learning;

[0054] A batch size of 16 samples is set. A smaller batch size is conducive to increasing the generalization ability of the model training process while maintaining a reasonable demand for hardware resources. In each iteration, the model parameters are updated according to the calculated loss function gradient. In this process, a weighted loss function is designed, which combines the cross entropy loss and L2 loss, targeting the accuracy of classification and the precision of regression respectively. Through this method, the model can be optimized in multiple aspects, thereby maintaining the accuracy of classification while also improving the accuracy of regression tasks.

[0055] Beneficial effects of the present invention:

[0056] Through the comprehensive application of these strategies, the present invention can ensure that the model is optimized in an efficient and robust manner during the training process, thereby maximizing its overall performance. This meticulous training strategy enables the model to achieve higher accuracy and better results when processing complex video colorization tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the overall process of a video colorization method based on similar frame assistance of the present invention;

[0058] Figure 2 It is a schematic diagram of the coloring network and scoring network structure of the present invention;

[0059] Figure 3 It is a Loss schematic diagram of the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Example 1

[0062] The present invention provides a video colorization method based on similar frame assistance. The core of the method is to use the "similar frame assistance" strategy to improve the video colorization process. This method can use the color information in these similar frames to guide and optimize the colorization process. By combining the color information extracted from the original frame and the similar frame, the method can generate a more natural and accurate colorization result. This step also includes subtle adjustments to the color to ensure overall coherence and naturalness. Specifically, it is divided into the following steps:

[0063] S1. Video preprocessing stage: Image similarity processing is an image editing technique that aims to enhance the visual effect or adapt it to a specific visual style by making a series of fine adjustments to the original image. These processes include:

[0064] First is the color temperature adjustment, which adjusts the warm and cool tones of the image by changing the range to 5% above and below the correct color frame to create or improve a specific atmosphere, making the image appear warmer or calmer. Next is the HSV adjustment, which is the adjustment of hue, saturation and brightness. The change range is also 7% above and below the correct color frame. Hue adjustment can change the main color of the image, saturation adjustment is used to strengthen or weaken the vividness of the color, and brightness adjustment affects the overall brightness. Contrast adjustment has a fluctuation of up to 15% for the darkest and brightest areas, and 10% for other areas. This helps to enhance the depth and dimension of the image, increase vividness and movement, or create a softer and smoother effect. Finally, brightness adjustment changes the overall brightness level of the image, which can be used to correct exposure problems or create a specific visual atmosphere. Through these comprehensive adjustment methods, we can greatly improve the naturalness and realism of the coloring effect.

[0065] S2. Colorization network: First, all input frames, including similar color frames, black and white frames, and correctly colored frames, are scaled to a uniform size of 340×340 pixels. This step ensures the consistency of the size of all input data, which helps reduce the computational burden of model processing while maintaining the integrity of the image content.

[0066] These resized frames are then normalized to the mean and variance of the ImageNet dataset. Normalization is a standard step in deep learning preprocessing, which helps the model converge faster and improves the stability of the training process. By using the mean and variance of ImageNet, the input data is ensured to be consistent in color distribution with a large number of pre-trained models, thereby taking advantage of the advanced capabilities of these models in color recognition and processing.

[0067] Finally, these carefully pre-processed frames are fed together into the colorization network for training;

[0068] The coloring network of step S2 has the following structure:

[0069] Block 1:

[0070] Three 3×3 convolutional layers, each with 32 kernels, each followed by a GeLU activation function, a 3×3 max pooling layer with a stride of 3,

[0071] The second block (Block 2): 3 5×5 convolutional layers, each with 64 convolution kernels, each followed by a GeLU activation function, a 5×5 maximum pooling layer with a stride of 5,

[0072] Block 3:

[0073] Two 7×7 convolutional layers, each with 128 kernels, each followed by a GeLU activation function, a 7×7 maximum pooling layer with a stride of 7, and blocks 1-3 are pre-feature extraction blocks;

[0074] Block 4:

[0075] Two 7×7 convolutional layers, each with 128 kernels, each followed by a GeLU activation function, a 5×5 max pooling layer with a stride of 5,

[0076] Block 5:

[0077] Two 5×5 convolutional layers, each with 128 kernels, each followed by a GeLU activation function, a 5×5 maximum pooling layer with a stride of 5, and blocks 4-5 are post-feature extraction blocks;

[0078] First, three independent pre-feature extraction blocks are introduced, which process black and white frames, correctly colored frames, and similarly colored frames respectively, aiming to extract key feature information from each type of frame. This step is crucial because it provides the basis for the subsequent feature fusion and optimization.

[0079] The output features of each front feature extraction block are then integrated and fused by weighted superposition. In this process, each type of feature is given a corresponding weight according to its importance to the final colorization result. The weight can be determined based on experience or through data-driven methods. This weighted superposition process can be expressed as:

[0080] F combined =w1·F bw +w2·F correct +w3·F similar ,

[0081] where F combined Represents the fused features, F bw , F correct and F similar They represent the features of black and white frames, correctly colored frames, and similarly colored frames, respectively, while w1, w2, and w3 are the corresponding weights;

[0082] Then, these fused features are fed into the post-feature extraction block for further processing and refinement to form a more discriminative feature representation. The post-feature extraction block optimizes and refines the features through further deep learning techniques, convolutional layers or fully connected layers;

[0083] These optimized features are then used to calculate the cross entropy loss to evaluate the difference between the model's predicted values ​​and the actual values. The cross entropy loss is a loss function commonly used in supervised learning to measure the difference between the predicted values ​​and the actual values. The calculation formula is:

[0084] L cross-entropy =-∑(ylog(p)+(1-y)log(1-p)),

[0085] Where y is the true label and p is the probability predicted by the model.

[0086] Finally, based on these losses, the data enters the scoring and ranking module, where the final evaluation and sorting will be performed to complete the entire image processing process. The scoring and ranking module may include a series of scoring criteria, such as color accuracy, naturalness, etc., to ensure the quality of the colorization results. Through this comprehensive processing flow, we can achieve high-quality and natural video colorization effects;

[0087] S3. Result ranking:

[0088] Scoring and ranking module:

[0089] Cosine similarity calculation layer,

[0090] Fully connected layer 1: 256 dimensions,

[0091] Fully connected layer 2: 128 dimensions,

[0092] Fully connected layer 3: 2D;

[0093] The output of the post-feature extraction block first enters the scoring and ranking module, in which the cosine similarity calculation is performed to determine the similarity between the black and white frame and the correctly colored frame and the similar colored frame. This step is based on the principle of cosine similarity to evaluate the similarity of the feature vector direction between different frames. The similarity calculation formula is:

[0094]

[0095] A·B represents the dot product of two vectors A and B.

[0096]

[0097] ||A|| and ||B|| are the Euclidean norms of vectors A and B, respectively.

[0098]

[0099] Therefore, cosine similarity actually calculates the cosine value of the angle between two vectors in n-dimensional space. This value ranges from -1 to 1, where 1 means that the two vectors are in exactly the same direction, 0 means that they are orthogonal, and -1 means that the two directions are completely opposite. In many applications, especially in the case of positive data (such as image pixels), the value of cosine similarity is usually between 0 and 1;

[0100] Next, the features of the correctly colored frame and the similar colored frame are fed into a fully connected layer for further scoring. In this step, the L2 Loss is calculated, which is a method to measure the difference between the predicted value and the actual value. It is usually used for regression tasks and to optimize the accuracy of the model.

[0101]

[0102] Among them, yi is the true value of the i-th sample, is the model prediction value of the i-th sample, and n is the total number of samples;

[0103] S4. Joint loss optimization:

[0104] Subsequently, this bi-norm loss is combined with the previously calculated cross entropy loss. The two losses are weighted in a ratio of 1.1:0.8. This weighted combination of two different types of losses aims to optimize the performance of the model in different aspects at the same time - the cross entropy loss focuses on classification accuracy, while the bi-norm loss focuses on regression accuracy. Finally, the combined total loss is used in the back-propagation process of the model.

[0105] Loss total =1.1·Loss CE +0.8 Loss L2 ,

[0106] Stochastic Gradient Descent (SGD) is used as the optimizer to fine-tune the model parameters. The momentum parameter of the SGD optimizer is set to 0.8, which helps to accelerate the training process of the model while maintaining sufficient stability. The momentum mechanism plays the role of smoothing the optimization path and effectively avoids excessive oscillation of the model during training.

[0107] The initial learning rate is set to 2e-4, which is a relatively small value designed to ensure the smoothness of the optimization process and avoid over-adjustment, thereby preventing possible overfitting problems. In machine learning, appropriate learning rate selection is crucial because it directly affects the speed and effect of model learning;

[0108] A batch size of 16 samples is set. A smaller batch size is conducive to increasing the generalization ability of the model training process while maintaining a reasonable demand for hardware resources. In each iteration, the model parameters are updated according to the calculated loss function gradient. In this process, a weighted loss function is designed, which combines the cross entropy loss and L2 loss, targeting the accuracy of classification and the precision of regression respectively. Through this method, the model can be optimized in multiple aspects, thereby maintaining the accuracy of classification while also improving the accuracy of regression tasks.

[0109] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. The various components mentioned in the present invention are common technologies in the existing field. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A video colorization method based on similar frame assistance, characterized in that: The following steps are involved: S1. Video preprocessing: First, obtain the correctly colored video, and perform similarity processing on each frame image to obtain similar frames; S2. Training three types of frames together: black and white frames, correctly colored frames, and similarly colored frames are fed into the colorization network for training; S3. Result confidence ranking: The results output by the model will be similarly calculated with the correctly colored frame and the similarly colored frame to rank the two with the predicted results, and correct the incorrect rankings.

2. The video colorization method based on similar frame assistance according to claim 1, characterized in that: Similarity processing in S1 is an image editing technique that enhances the original image and adapts it to a specific visual style by making a series of subtle adjustments to the original image, including: Change color temperature: involves adjusting the color temperature of the image to give the image a different hue. The adjustment of color temperature is used to create a specific atmosphere. HSV adjustment: Change the color characteristics of an image by adjusting hue, saturation, and brightness. Hue adjustment changes the main color in the image, saturation adjustment changes the intensity of the color, and brightness adjustment changes the overall brightness of the image. Contrast adjustment: used to change the difference between the darkest and lightest areas in an image, thereby enhancing the depth and dimension of the image; Brightness Adjustment: Changing the overall brightness level of an image, brightness adjustment corrects under- or over-exposed photos and creates a specific visual atmosphere.

3. The video colorization method based on similar frame assistance according to claim 2, characterized in that: The structure of the coloring network in step S2 is: Block 1: 3 3×3 convolutional layers, each with 32 convolution kernels, Each convolutional layer is followed by a GeLU activation function. A 3×3 max pooling layer with a stride of 3, Block 2: 3 5×5 convolutional layers, each with 64 convolution kernels, Each convolutional layer is followed by a GeLU activation function. A 5×5 max pooling layer with a stride of 5, Block 3: Two 7×7 convolutional layers, each with 128 convolution kernels, Each convolutional layer is followed by a GeLU activation function. A 7×7 max pooling layer with a stride of 7, Blocks 1-3 are pre-feature extraction blocks; Block 4: Two 7×7 convolutional layers, each with 128 convolution kernels, Each convolutional layer is followed by a GeLU activation function. A 5×5 max pooling layer with a stride of 5, Block 5: Two 5×5 convolutional layers, each with 128 convolution kernels, Each convolutional layer is followed by a GeLU activation function. A 5×5 max pooling layer with a stride of 5, Blocks 4-5 are post-feature extraction blocks; Three independent pre-feature extraction blocks process black and white frames, correctly colored frames, and similarly colored frames respectively to extract their respective key features. These extracted features are then combined together by weighted superposition, where each feature obtains different weights according to its importance. The combined features are sent to the post-feature extraction block for further processing and refinement to form a more discernible feature representation; then, these features are used to calculate the cross entropy loss to evaluate the difference between the prediction and the actual value; finally, based on these losses, the data enters the scoring and ranking module for final evaluation and sorting to complete the entire image processing flow.

4. The video colorization method based on similar frame assistance according to claim 3, characterized in that: The structure of step S3 is: Scoring and ranking module: Cosine similarity calculation layer, Fully connected layer 1: 256 dimensions, Fully connected layer 2: 128 dimensions, Fully connected layer 3: 2 dimensions, The output of the post-feature extraction block first enters the scoring and ranking module, in which cosine similarity calculation is performed to determine the similarity between the black and white frame and the correctly colored frame and the similarly colored frame. This step is based on the principle of cosine similarity and evaluates the similarity of the feature vector directions between different frames. Next, the features of the correctly colored frame and the similarly colored frame are sent to a fully connected layer for further scoring. In this link, the bi-norm loss is calculated.

5. The video colorization method based on similar frame assistance according to claim 4, characterized in that: The binary loss is combined with the previously calculated cross entropy loss for weighted processing; finally, the combined total loss is used in the back-propagation process of the model. In the back-propagation process, the parameters of the model are updated according to the gradient of the loss function to optimize the performance of the model.

Images