Video coloring method, compression method, system and electronic equipment

Abstract

The present invention provides a video colorization method, comprising: using a neural network to extract a multi-layer feature map from the grayscale information of a reference frame and a target frame; performing optical flow prediction and optical flow optimization based on the multi-layer feature map; and using the result obtained by the optical flow optimization to propagate the color in the reference frame to the target frame. The present invention improves the accuracy of long-term optical flow prediction, and the generated color video frame has better fidelity and more accurate color, and has extremely strong practical application value. When applied to video encoding tasks, the bit rate can be reduced while maintaining the reconstruction quality.

Description

Technical Field

[0001] The invention relates to a video coloring scheme based on optical flow optimization, in particular to a video coloring method, a compression method, a system and an electronic device. Background Art

[0002] Video colorization is the process of coloring video frames with only grayscale information using reference color information. The reference information for video colorization can be manually specified by the user or extracted from a reference image. Video colorization methods based on training and learning have also emerged with the development of deep learning technology. After searching the literature on the prior art, Cheng et al. published "Deep Colorization" at the International Conference on Computer Vision (ICCV) in 2015, proposing to use deep learning technology to solve the colorization problem. This method generates color information of grayscale images through a deep neural network and further improves the color information using joint bilateral filtering. Zhang et al. published "Deep exemplar-based video colorization" at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2019, using VGG-net to extract local features of color video frames and grayscale video frames, using correlation matrices to calculate the global correlation of local features at different positions, and using neural networks to color grayscale video frames. The colorization method based on optical flow uses a manually designed method to achieve optical flow prediction and color propagation. Xia et al. published "Robust and automatic video colorization via multiframe reordering refinement" at the 2016 IEEE International Conference on Image Processing (ICIP) conference, using dense optical flow estimation technology to predict optical flow and using confidence propagation to achieve color propagation.

[0003] However, among the above methods: the colorization method based on deep neural network is limited by the dataset used for training. If it is applied to a video with a color distribution that is significantly different from the training dataset, obvious color difference will occur during the colorization process. In addition, the colorization accuracy of the method based on optical flow drops significantly in the case of long-term prediction. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a video colorization method, a compression method, a system and an electronic device.

[0005] According to one aspect of the present invention, there is provided a video colorization method, comprising:

[0006] Use a neural network to extract multi-layer feature maps from the grayscale information of the reference frame and the target frame;

[0007] Performing optical flow prediction and optical flow optimization based on the multi-layer feature map;

[0008] The color in the reference frame is propagated to the target frame using the result obtained by optical flow optimization.

[0009] Preferably, the optical flow prediction includes:

[0010] Sort the multi-layer feature maps in order of size from small to large;

[0011] In adjacent feature maps, the optical flow information of the smaller-sized feature maps is used layer by layer to predict the optical flow information of the larger-sized feature maps;

[0012] Until the optical flow information of the feature map of the largest size is obtained, it is used as the first optical flow information.

[0013] Preferably, the optical flow optimization includes: iteratively performing grayscale propagation and optical flow updating on the grayscale information of the reference frame and the first optical flow information to obtain second optical flow information.

[0014] Preferably, the second optical flow information is recorded as Δ * (z)=(u * (z),v * (z)), where z = (x, y) is the position of the feature map coefficient on the two-dimensional coordinate axis, u * (z) represents the motion information of the pixel in the x-axis direction, v * (z) represents the motion information of the pixel in the y-axis direction, and C1(z) is the reference frame color information;

[0015] The color propagation is realized by a function, which is:

[0016]

[0017] in and and

[0018] Preferably, the optical flow prediction includes:

[0019] Feature deformation, the feature deformation includes:

[0020] The feature layer corresponding to the minimum size directly outputs the feature map corresponding to the reference frame;

[0021] Other layers implement feature deformation through deformation functions, including:

[0022] When predicting the optical flow information of a larger size, the feature map corresponding to the reference frame of the smaller size is deformed using the optical flow information of the smaller size to obtain the deformed feature map;

[0023] The reference frame feature map at the lth scale is recorded as The optical flow information predicted by the l-1th layer is recorded as w (l-1) , then the deformed feature map is And U(w (l-1) ) to achieve w (l-1) Upsample it to make it similar in scale to Alignment;

[0024] Loss calculation:

[0025] The error between the feature map obtained after the output feature deformation and the corresponding feature map of the target frame of the same size is calculated, and the corresponding feature map of the target frame of the same size is recorded as The error calculation output is

[0026] Optical flow estimation:

[0027] Through the neural network, the error ∈ calculated by inputting the loss under a certain size (l) , feature maps corresponding to the reference frame and the target frame Output the optical flow obtained under this layer.

[0028] Preferably, the grayscale propagation outputs propagated grayscale information of the target frame by inputting the grayscale of the reference frame and predicting the optical flow information, including:

[0029] In the first iteration, the first optical flow information is used as the predicted optical flow;

[0030] The i-1th predicted optical flow information is recorded as Δ (i-1) (z)=(u (i-1) (z),v (i-1) (z)), the grayscale information of the reference frame is recorded as I1(z), then the i-th grayscale propagation is realized by the following function:

[0031]

[0032] in

[0033] and and

[0034] The optical flow update comprises:

[0035] Measure the propagation grayscale information of the target frame The gap between the real grayscale information I2(z);

[0036] Update the i-1th predicted optical flow information through back propagation and record it as Δ (i-1) (z), the i-th predicted optical flow information is recorded as Δ (i) (z);

[0037] The gap is calculated as remember is D about Δ (i-1) The gradient of (z), λ is the update step size, and the back-propagation update is

[0038] The optical flow information obtained after iterative convergence or the last optical flow update is the second optical flow information Δ * (z).

[0039] According to a second aspect of the present invention, there is provided a video colorization system, comprising:

[0040] A feature extraction module, wherein the feature extraction module uses a pyramid-structured neural network to extract a multi-layer feature map from grayscale information of a reference frame and a target frame;

[0041] An optical flow prediction module, wherein the optical flow prediction module predicts optical flow information from the multi-layer feature map, wherein the prediction process is implemented by iteration, firstly extracting optical flow information of smaller size from the feature map of smaller size, and then predicting optical flow information of larger size step by step;

[0042] An optical flow optimization module, wherein the optical flow optimization module uses a pseudo loss function and grayscale propagation for iterative optimization;

[0043] Color Propagation Module: Uses optimized optical flow to propagate the reference frame color to the target frame.

[0044] According to a third aspect of the present invention, there is provided a video compression method, comprising:

[0045] Chroma information temporal downsampling: remove the chroma information of some video frames in the color video to form a mixed frame sequence containing color frames and grayscale frames;

[0046] Color and grayscale frame mixed video coding: Intra-frame coding is performed on color frames, and grayscale frames are encoded using color frames as reference frames to form a binary code stream of a mixed frame sequence;

[0047] Color and grayscale mixed video decoding: Intra-frame decoding of color frames from binary code streams, and using the decoded color frames as reference frames, decoding grayscale frames, and restoring mixed frame sequences;

[0048] Chroma information reconstruction: At the decoding end, the decoded color frame is used as a reference frame and the decoded grayscale frame is used as a target frame, and the chroma information of the grayscale frame is reconstructed according to the video colorization method.

[0049] According to a fourth aspect of the present invention, there is provided a video compression system, comprising:

[0050] Chroma information temporal downsampling module: removes the chroma information of some video frames in the color video to form a mixed frame sequence containing color frames and grayscale frames;

[0051] Color and grayscale frame mixed video encoding module: performs intra-frame encoding on color frames, and uses color frames as reference frames to encode grayscale frames to form a binary code stream of mixed frame sequences;

[0052] Color and grayscale mixed video decoding module: It performs intra-frame decoding on color frames from binary code streams, and uses the decoded color frames as reference frames to decode grayscale frames and restore the mixed frame sequence;

[0053] Chroma information reconstruction module: At the decoding end, the decoded color frame is used as the reference frame and the decoded gray frame is used as the target frame, and the chroma information of the gray frame is reconstructed according to the video colorization method described in the right.

[0054] According to a fifth aspect of the present invention, an electronic device is provided, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement any one of the following methods:

[0055] The video coloring method, or

[0056] The video coloring system, or

[0057] The video compression method, or

[0058] The video compression system.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] A video colorization method in an embodiment of the present invention improves the accuracy of optical flow long-term prediction, and the generated color video frame has better fidelity and more accurate color, which has great practical application value. When applied to video encoding tasks, the bit rate can be reduced while maintaining the reconstruction quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0062] Figure 1 is a flow chart of a video coloring method in one embodiment of the present invention;

[0063] Figure 2 is a flow chart of a video coloring system in one embodiment of the present invention;

[0064] Figure 3 is a flow chart of a video compression method in one embodiment of the present invention;

[0065] Figure 4 The flowchart of a video compression system in one embodiment of the present invention is shown in FIG. DETAILED DESCRIPTION

[0066] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0067] The present invention provides an embodiment, a video colorization method, comprising:

[0068] S01, uses a neural network to extract multi-layer feature maps from the grayscale information of the reference frame and the target frame;

[0069] S02, performing optical flow prediction and optical flow optimization based on the multi-layer feature map;

[0070] S03, using the result obtained by optical flow optimization to propagate the color in the reference frame to the target frame.

[0071] like Figure 1 The above is a video coloring method further optimized based on the above embodiment of the present invention, and its process includes:

[0072] S1, feature extraction: extract multiple layers of feature maps from the grayscale information of the frame, and provide a series of paired feature maps corresponding to the reference frame and the target frame from large to small sizes;

[0073] S2, optical flow prediction: predict the optical flow information from the multi-layer feature map obtained in S1, extract the optical flow information of smaller size from the feature map of smaller size layer by layer, use it to predict the optical flow information of larger size, and output the final optical flow information of the largest size as the first optical flow information;

[0074] S3, optical flow optimization: grayscale propagation and optical flow update are iteratively performed using the grayscale information of the reference frame and the first optical flow information obtained in S2 to obtain the second optical flow information;

[0075] S4, color propagation: Use the second optical flow information obtained in S3 to propagate the reference frame color to the target frame.

[0076] Furthermore, in a preferred embodiment of the present invention, several video clips are combined for video colorization. In this embodiment, each video clip has 20 frames, wherein the first frame is a color frame and the remaining 19 frames are grayscale frames.

[0077] As a preferred embodiment, the above S1 is implemented, including: recording the grayscale information of the first frame as I1, and recording the remaining 19 grayscale frames as I n , pyramid-shaped feature extraction is achieved through an 18-layer convolutional neural network. The convolution kernel size of each layer of the neural network is 3x3, and the number of output channels of every 3 layers from front to back is 16, 32, 64, 96, 128, 192. The convolution step size of the 1st, 4th, 7th, 10th, 13th, and 16th layers is 2, and the convolution step size of the remaining layers is 1. Every three layers output a feature map. n , then we get 6 sizes The feature map of 20 grayscale information I n (1≤n≤20) are input respectively, and 120 feature maps are obtained, which are recorded as The feature map corresponding to the lth scale of the nth frame;

[0078] As a preferred embodiment, the above S2 is implemented, including: obtaining 19 pieces of initial optical flow information (first optical flow information) are the optical flow information between the 1st frame and the nth frame (2≤n≤20). The optical flow prediction is performed by the 6-layer feature (i.e. The smaller the size of l, the smaller the size of the corresponding feature map) is, the feature deformation, loss calculation and optical flow estimation are iterated in the order of gradually decreasing l;

[0079] S210, feature deformation: For the first layer of features, directly output the feature map corresponding to the reference frame, that is, For other layers, the output Where z = (x, y) is the position of the pixel point on the two-dimensional coordinate axis. is the optical flow information predicted by the l-1 layer, and For Upsample twice spatially;

[0080] S220, loss calculation: output N is The number of elements in ;

[0081] S230, Optical Flow Estimation: Through 7-layer Convolutional Neural Network The convolution kernel size of each layer is 3×3, the step size is 1, and the corresponding number of output channels is 128, 128, 128, 96, 64, 32, and 2 respectively. The first layer input of the network is ∈ (l) The composite feature obtained by connecting forward and backward on the channel. In addition, the last layer of the network contains an additional addition term The residual structure is formed. The network output is the predicted optical flow and

[0082] As a preferred embodiment, the above S3 optical flow optimization includes: using pseudo loss function and grayscale propagation to iteratively optimize and improve the accuracy of optical flow information. Specifically, the i-th iteration implements the optical flow obtained in the i-1th iteration. The above includes the following two sub-steps:

[0083] S310, Grayscale Propagation: Optical flow contains components in two directions, using Indicates that u(z) represents the motion information of the pixel in the x-axis direction, and v(z) represents the motion information of the pixel in the y-axis direction. Grayscale propagation is achieved through the following function:

[0084]

[0085] in and and

[0086] S320, pseudo loss function: used to measure the gap between the pseudo grayscale information and the real grayscale information of the target frame, and optimize the optical flow through back propagation. In this embodiment, the pseudo loss function is the predicted and the true gray value I n With the goal of minimizing the pseudo loss function, back propagation is used to update the optical flow, and we get

[0087] As a preferred embodiment, the above S4, color propagation, includes using the optimized optical flow to propagate the reference frame color to the target frame. The color of the first frame is represented as C1, and the optical flow obtained by iterating S3 is Color propagation is implemented by the following function:

[0088]

[0089] in and and The color propagation function is consistent with the grayscale propagation function of S310.

[0090] For ease of understanding, the optical flow symbols in this application are explained: w is the optical flow involved in the optical flow prediction item in S2, and the final result is the first optical flow information; Δ represents the optical flow in the optical flow optimization of S3, Represents the result of the i-1th optimization, that is, the input of the i-th optimization. * It is the final result of iterative optimization, that is, the second optical flow information.

[0091] In this embodiment, the average reconstruction quality of the dyed video after 0, 10, and 100 times of post-training optimization described in step 2 is 42.17dB, 42.49dB, and 42.56dB respectively. On the same test video, two recent video dyeing methods were used, and the results of 33.54dB and 36.67dB were obtained respectively.

[0092] Based on the same inventive concept, another embodiment of the present invention provides a video coloring system, whose structure is as follows: Figure 2 As shown, including:

[0093] A feature extraction module S100, wherein the feature extraction module uses a pyramid-structured neural network to extract a multi-layer feature map from grayscale information of a reference frame and a target frame;

[0094] An optical flow prediction module S200, wherein the optical flow prediction module predicts optical flow information from the multi-layer feature map, wherein the prediction process is implemented by iteration, firstly extracting optical flow information of a smaller size from the feature map of a smaller size, and then predicting optical flow information of a larger size step by step;

[0095] Optical flow optimization module S300, the optical flow optimization module uses a pseudo loss function and grayscale propagation iterative optimization;

[0096] Color propagation module S400: uses the optimized optical flow to propagate the reference frame color to the target frame.

[0097] Based on the same inventive concept, another embodiment of the present invention further provides a video compression method, whose flow chart is as follows: Figure 3 As shown, including:

[0098] S5, chroma information temporal downsampling: removing chroma information of some video frames in the color video to form a mixed frame sequence including color frames and grayscale frames;

[0099] S6, color and grayscale frame mixed video coding: perform intra-frame coding on the color frame, and use the color frame as a reference frame to encode the grayscale frame to form a binary code stream of the mixed frame sequence;

[0100] S7, color and grayscale mixed video decoding: perform intra-frame decoding on the color frame from the binary code stream, and use the decoded color frame as a reference frame to decode the grayscale frame and restore the mixed frame sequence;

[0101] S8, chromaticity information reconstruction: at the decoding end, the decoded color frame is used as a reference frame and the decoded grayscale frame is used as a target frame, and the chromaticity information of the grayscale frame is reconstructed according to the video colorization method in the above embodiment.

[0102] Based on the same inventive concept, another embodiment of the present invention further provides a video compression system based on a video colorization method, and its structure diagram is as follows: Figure 4 As shown, including:

[0103] Chroma information temporal downsampling module S500: removes chroma information of some video frames in the color video to form a mixed frame sequence including color frames and grayscale frames;

[0104] Color and grayscale frame mixed video encoding module S600: performs intra-frame encoding on color frames, and uses color frames as reference frames to encode grayscale frames to form a binary code stream of a mixed frame sequence;

[0105] Color and grayscale frame mixed video decoding module S700: performs intra-frame decoding on color frames from binary code streams, and uses the decoded color frames as reference frames to decode grayscale frames and restore the mixed frame sequence;

[0106] Chroma information reconstruction module S800: At the decoding end, the decoded color frame is used as a reference frame and the decoded gray frame is used as a target frame, and the chroma information of the gray frame is reconstructed according to the video colorization method in the above embodiment.

[0107] Based on the same inventive concept, another embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement any one of the following methods:

[0108] - the video colorization method described above, or,

[0109] - the video colorization system described above, or,

[0110] - the video compression method as described above, or,

[0111] -The video compression system as described above.

[0112] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A video coloring method, characterized in that: include: Use a neural network to extract multi-layer feature maps from the grayscale information of the reference frame and the target frame; Performing optical flow prediction and optical flow optimization based on the multi-layer feature map; Propagate the color in the reference frame to the target frame using the result obtained by optical flow optimization; The optical flow prediction includes: Sort the multi-layer feature maps in ascending order of size and predict layer by layer; The prediction is: in feature maps of adjacent sizes, optical flow information of feature maps of smaller sizes is used to predict optical flow information of feature maps of larger sizes; Until the optical flow information of the feature map with the largest size is obtained, it is used as the first optical flow information; The optical flow optimization comprises: Iteratively perform grayscale propagation and optical flow update on the grayscale information of the reference frame and the first optical flow information to obtain second optical flow information; The second optical flow information is recorded as Δ * (z)=(u * (z),v * (z)), where z = (x, y) is the position of the feature map coefficient on the two-dimensional coordinate axis, u * (z) represents the motion information of the pixel in the x-axis direction, v * (z) represents the motion information of the pixel in the y-axis direction, and C1(z) is the reference frame color information; The color propagation is realized by a function, which is: in and and 2. A video coloring method according to claim 1, characterized in that: The optical flow prediction includes: Feature deformation, the feature deformation includes: The feature layer corresponding to the minimum size directly outputs the feature map corresponding to the reference frame; Other layers implement feature deformation through deformation functions, including: When predicting the optical flow information of a larger size, the feature map corresponding to the reference frame of the smaller size is deformed using the optical flow information of the smaller size to obtain the deformed feature map; The reference frame feature map at the lth scale is recorded as The optical flow information predicted by the l-1th layer is recorded as w (l-1) , then the deformed feature map is And U(w (l-1) ) to achieve w (l-1) Upsample it to make it similar in scale to Alignment; Loss calculation, the loss calculation includes: The error between the feature map obtained after the output feature deformation and the corresponding feature map of the target frame of the same size is calculated, and the corresponding feature map of the target frame of the same size is recorded as The error calculation output is Optical flow estimation, the optical flow estimation includes: Through neural network Implementation, input the error ∈ calculated by loss under the l-th layer size (l) , feature maps corresponding to the reference frame and the target frame Output the optical flow obtained under this layer 3. A video coloring method according to claim 1, characterized in that: The grayscale propagation outputs the propagated grayscale information of the target frame by inputting the grayscale of the reference frame and predicting the optical flow information, including: In the first iteration, the first optical flow information is used as the predicted optical flow; The i-1th predicted optical flow information is recorded as Δ (i-1) (z)=(u (i-1) (z),v (i-1) (z)), the grayscale information of the reference frame is recorded as I1(z), then the i-th grayscale propagation is realized by the following function: in and and The optical flow update comprises: Measure the propagation grayscale information of the target frame The gap between the real grayscale information I2(z); Update the i-1th predicted optical flow information through back propagation and record it as Δ (i-1) (z), the i-th predicted optical flow information is recorded as Δ (i) (z); The gap is calculated as remember is D about Δ (i-1) The gradient of (z), λ is the update step size, and the back-propagation update is The optical flow information obtained after iterative convergence or the last optical flow update is the second optical flow information Δ * (z).

4. A video coloring system, characterized in that: include: A feature extraction module, wherein the feature extraction module uses a pyramid-structured neural network to extract a multi-layer feature map from grayscale information of a reference frame and a target frame; An optical flow prediction module, wherein the optical flow prediction module predicts optical flow information from the multi-layer feature map, wherein the prediction process is implemented by iteration, firstly extracting optical flow information of smaller size from the feature map of smaller size, and then predicting optical flow information of larger size step by step; An optical flow optimization module, wherein the optical flow optimization module uses a pseudo loss function and grayscale propagation for iterative optimization; Color propagation module: Uses optimized optical flow to propagate the reference frame color to the target frame; The optical flow prediction module comprises: Sort the multi-layer feature maps in ascending order of size and predict layer by layer; The prediction is: in feature maps of adjacent sizes, optical flow information of feature maps of smaller sizes is used to predict optical flow information of feature maps of larger sizes; Until the optical flow information of the feature map with the largest size is obtained, it is used as the first optical flow information; The optical flow optimization module comprises: Iteratively perform grayscale propagation and optical flow update on the grayscale information of the reference frame and the first optical flow information to obtain second optical flow information; The second optical flow information is recorded as Δ * (z)=(u * (z),v * (z)), where z = (x, y) is the position of the feature map coefficient on the two-dimensional coordinate axis, u * (z) represents the motion information of the pixel in the x-axis direction, v * (z) represents the motion information of the pixel in the y-axis direction, and C1(z) is the reference frame color information; The color propagation is realized by a function, which is: in and and 5. A video compression method, characterized in that: include: Chroma information temporal downsampling: remove the chroma information of some video frames in the color video to form a mixed frame sequence containing color frames and grayscale frames; Color and grayscale frame mixed video coding: intra-frame coding is performed on the color frame, and the grayscale frame is encoded using the color frame as a reference frame to form a binary code stream of the mixed frame sequence; Color and grayscale mixed video decoding: intra-frame decoding of the color frame from the binary code stream, and decoding of the grayscale frame using the decoded color frame as a reference frame to restore a mixed frame sequence; Chroma information reconstruction: at the decoding end, the decoded color frame is used as a reference frame and the decoded grayscale frame is used as a target frame, and the chroma information of the grayscale frame is reconstructed according to the video colorization method described in any one of claims 1 to 3.

6. A video compression system, characterized in that: include: Chroma information temporal downsampling module: removes the chroma information of some video frames in the color video to form a mixed frame sequence containing color frames and grayscale frames; Color and grayscale frame mixed video encoding module: performs intra-frame encoding on color frames, and uses color frames as reference frames to encode grayscale frames to form a binary code stream of mixed frame sequences; Color and grayscale mixed video decoding module: It performs intra-frame decoding on color frames from binary code streams, and uses the decoded color frames as reference frames to decode grayscale frames and restore the mixed frame sequence; Chroma information reconstruction module: at the decoding end, the decoded color frame is used as a reference frame and the decoded grayscale frame is used as a target frame, and the chroma information of the grayscale frame is reconstructed according to the video colorization method described in any one of claims 1 to 3.

7. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement any one of the following methods: - the video coloring method according to claims 1 to 3, or, - the video coloring system of claim 4, or, - The video compression method according to claim 5, or, -The video compression system as claimed in claim 6.

Images