Video re-coloring system based on time sequence auxiliary palette

Abstract

The invention discloses a video re-coloring system based on a time sequence auxiliary palette, and the system comprises a data importing module which is used for loading a group of video frames; the data processing module is used for processing the loaded group of video frames, calculating to obtain a color palette corresponding to each video frame, performing optimization calculation on the color palettes to obtain a time sequence auxiliary color palette, performing weight decomposition on each video frame according to the time sequence auxiliary color palette, and performing weight decomposition on each video frame according to the time sequence auxiliary color palette; a convex hull deformation migration algorithm is utilized to transmit the change of the color of a certain frame by the user to other frames so as to realize video re-coloring; the result visualization module is used for visualizing the processing result of each step, so that a user can conveniently operate video re-coloring and re-coloring result comparison; and the result exporting module is used for exporting the recoloring video result. According to the method, the limitation that the change of the palette is difficult to transmit to the whole video is broken through, the change of the palette is reasonably transmitted to the video by using the convex hull deformation migration algorithm, and a reasonable recoloring video result is quickly obtained.

Description

Technical Field

[0001] The present invention relates to the technical fields of geometric processing and video processing, and in particular to a video recoloring system based on a temporal auxiliary palette. Background Art

[0002] In recent years, people's demand for video editing has been increasing. However, the high complexity and high time consumption of video editing make it difficult for people to get started. Even though there are video editing software under Adobe and DaVinci Resolve video color grading software on the market currently, their high entry threshold results in a small number of users. And the image editing method based on the palette is becoming more and more mature. This method is widely used in image editing, art creation, short video editing, etc. due to its simple operation, high computing efficiency, excellent editing results, etc. This method calculates the representative colors in the image, and the representative colors form a palette. Users only need to modify the colors of the palette to complete the recoloring of the image. Therefore, video color editing based on the palette has become a popular research topic.

[0003] Currently, video color editing methods are mainly divided into three categories: example-based methods, stroke-based methods, and palette-based methods. The first two methods both require deep learning on the video selected by the user. The example-based method mainly uses the style transfer of example pictures to the entire video; the stroke-based method is that the user provides a color for a certain area of a certain frame of the video, and then transfers the color change to the entire video; the palette-based method extracts a palette from each video frame in the video, and then performs algorithm processing on the palette sequence to obtain a video palette. Users can quickly achieve video recoloring by operating the video palette. In the example-based method, it is difficult for users to replace the colors in the video with ideal colors. In the stroke-based method, it is difficult to find a large number of works recolored by artists in the training data required, and at the same time, a certain training time is required for each edited video, which increases the difficulty of using the first two methods. The palette-based method has the characteristics of convenience and good editing effect, but currently the palette-based method still has the disadvantages of long palette calculation time and difficult transfer of palette changes to the entire video. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and propose a video recoloring system based on a temporal auxiliary palette, which provides a temporal auxiliary palette for users, breaks through the limitation that it is difficult to transfer palette changes to the entire video, and uses the method of convex hull deformation migration to reasonably transfer the palette changes to the video. Compared with the previous video recoloring methods based on the palette, it can obtain a reasonable recolored video result more quickly.

[0005] To achieve the above object, the technical solution provided by the present invention is: A video recoloring system based on a temporal-assisted color palette, comprising:

[0006] A data import module for loading a set of video frames;

[0007] A data processing module for processing the loaded set of video frames, calculating the color palette corresponding to each video frame, performing optimization calculations on the color palette to obtain a temporal-assisted color palette, and performing weight decomposition on each video frame according to the temporal-assisted color palette, and using the convex hull deformation migration algorithm to transfer the color change of a certain frame by the user to other frames to achieve video recoloring;

[0008] A result visualization module for visualizing the processing results of each step to facilitate the user to operate video recoloring and compare the recoloring results;

[0009] A result export module for exporting the recolored video result.

[0010] The data import module loads a set of video frames that the user needs to recolor, where:

[0011] The data loading module reads a set of video frames from the local, and its format is PNG format. This set of video frames is used to represent a video.

[0012] The data processing module is used for preprocessing the color palette of the loaded set of video frames, optimizing the color palette corresponding to each video frame by using an optimization algorithm, performing weight decomposition on each video frame according to the temporal-assisted color palette result, and using the convex hull deformation migration algorithm for video recoloring, including the following steps:

[0013] 1) Extract the color palette from the loaded set of video frames:

[0014] First, each pixel can be represented as a point in the RGB three-dimensional color space, and all the pixels in each video frame can be represented as a point cloud in the RGB three-dimensional color space. For the loaded set of video frames, the pixel point clouds of each video frame are placed in the same three-dimensional color space, so as to obtain a large point cloud with video temporal characteristics and color characteristics. The color palette extraction algorithm based on the auxiliary color palette is used for this large point cloud to extract the global color palette P V ={p i ∈[0,1] 3 ,i = 1,2,...,n P ,n P +1,...,n P +n A}, where p i is the i-th color in the global color palette, and nP is the number of main colors of the color palette, n A is the number of auxiliary colors of the color palette;

[0015] Then, the pixel point cloud of each video frame is mapped according to the global color palette P V to obtain the corresponding color palette, that is, the t-th video frame I t is mapped according to the global color palette P V to obtain the corresponding color palette where is the i-th color in the color palette P corresponding to the t-th video frame, and the number of colors in the color palette corresponding to each video frame is the same as that of the global color palette P t* and due to the mapping relationship, the i-th color in the color palette P corresponding to the t-th video frame V is similar to the i-th color p in the global color palette i is similar.

[0016] 2) Optimize the color palette corresponding to each video frame using an optimization algorithm:

[0017]

[0018]

[0019]

[0020] In the formula, P t is the optimized color palette of the t-th video frame, is the color palette loss term of the t-th video frame, which is used to constrain the difference between the colors in t and the colors in , P t-1 is the optimized color palette of the (t - 1)-th video frame, P t+1 is the optimized color palette of the (t + 1)-th video frame, is the i-th color in P t , is the i-th color in P t-1 , is the i-th color in P t+1 , is the optimized color palette P of the t-th video frame t and the smooth loss terms of the optimized color palettes P t-1 and P t+1 of the previous and next frames, which is used to constrain the color change amplitude between frames, λ data is the weight of the color palette loss term , λ reg is the smooth loss term ​The weight, T is the total number of video frames, ε is the temporal auxiliary palette loss term, comprehensively considering the palette loss term and the smoothing loss term corresponding to each video frame; after the optimization is completed, the optimized palette of each video frame is obtained, and all the optimized palettes are integrated into a temporal auxiliary palette P in the time dimension. The temporal auxiliary palette describes the color characteristics of the entire group of video frames, and because the temporal auxiliary palette has auxiliary vertices, the video reconstruction accuracy is higher;

[0021] 3) Decompose the weight of each video frame according to the temporal auxiliary palette:

[0022] After optimization, each video frame has its optimized palette P t , since the palette is a geometric convex hull in the RGB three-dimensional color space, all the pixel point clouds within the convex hull can be obtained by the linear addition of the palette:

[0023]

[0024] In the formula, I t (x, y) is the pixel color with abscissa x and ordinate y of the t-th video frame, is the i-th color of the palette For I t (x, y) weight;

[0025] 4) Use the convex hull deformation migration algorithm for video recoloring:

[0026]

[0027] In the formula, is the i-th color in the palette after the change of the t-th video frame, is the pixel color with abscissa x and ordinate y of the t-th video frame after recoloring;

[0028] After the user changes the color of the optimized palette of a certain video frame, the system calculates the change range of the optimized palettes of the remaining video frames according to the convex hull deformation migration algorithm, and transfers the change to the optimized palettes of other video frames, and then reconstructs the image according to the decomposed weights, so as to realize video recoloring.

[0029] Furthermore, the result visualization module includes a temporal auxiliary palette visualization module, a three-dimensional convex hull visualization module, and a video visualization module before and after recoloring, where:

[0030] The time-series assisted palette visualization module visualizes the time-series assisted palette. Users can observe the changes in the colors of each optimized palette frame on this module to correspond to the color changes of the entire video over time. Users can view the colors contained in the optimized palette of a video frame by sliding the time axis. Additionally, users can re-color the entire video by changing the optimized palette of a certain video frame in the time-series assisted palette visualization module;

[0031] The three-dimensional convex hull visualization module visualizes the convex hull topological structure of the optimized palette of each frame in the RGB three-dimensional color space, facilitating users to observe the color changes of the video from the perspective of the convex hull;

[0032] The video visualization module before and after re-coloring visualizes the original video and the re-colored video. Users can adjust how to perform more ideal re-coloring by observing the color changes of the video before and after re-coloring.

[0033] Furthermore, the result export module is used to export the re-colored video to the local for saving. The saving methods are divided into two types: image and video. The format of the exported image is PNG, and the format of the exported video is MP4. Additionally, users can choose to merge the original video and the re-colored video and then save it as a comparison result before and after re-coloring.

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] 1. The present invention provides an optimization algorithm to obtain a time-series assisted palette through optimization calculation. The time-series assisted palette has the advantages of high reconstruction accuracy and good color representativeness. It can accurately reflect the color changes of the entire video, providing a convenient and intuitive video re-coloring control platform for users.

[0036] 2. The present invention first applies the convex hull deformation migration algorithm in the field of video re-coloring, transferring the geometric convex hull structure changes of the palette in the RGB three-dimensional space to the optimized palettes of the remaining frames of the video, thereby achieving fast and efficient video re-coloring.

[0037] 3. The present invention provides a video re-coloring system. Users can perform functions such as video frame loading, video palette optimization and calculation, video re-coloring, and result export on the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the relationship between the various modules of the system of the present invention.

[0039] Figure 2 It is a flowchart of the video re-coloring of the present invention.

[0040] Figure 3 It is the design visualization interface of the system of the present invention.

[0041] Figure 4 They are the various visualization modules of the system of the present invention.

[0042] Figure 5 It is Example 1 of the result of video recoloring of the present invention.

[0043] Figure 6 It is Example 2 of the result of video recoloring of the present invention. Detailed implementation manners

[0044] The present invention will be further described in detail below in conjunction with embodiments and the accompanying drawings.

[0045] This embodiment discloses a video recoloring system based on a temporal auxiliary color palette, which is a video recoloring system developed using the Python language and can run on Windows devices. The system visualization interface is as Figure 3 shown. It includes:

[0046] A data import module for loading a set of video frames;

[0047] A data processing module for processing the loaded set of video frames, calculating the color palette corresponding to each video frame, performing optimization calculations on the color palette to obtain a temporal auxiliary color palette, and performing weight decomposition on each video frame according to the temporal auxiliary color palette, and using the convex hull deformation migration algorithm to transfer the change of the color of a certain frame by the user to other frames to achieve video recoloring. The flow chart of the system for video recoloring is as Figure 2 shown;

[0048] A result visualization module for visualizing the processing results of each step, facilitating the user to operate video recoloring and compare the recoloring results. Its composition is as Figure 4 shown;

[0049] A result export module for exporting the recolored video result.

[0050] The relationship between the system modules is as Figure 1 shown. The relationship between the data import module, the data processing module and the result export module conforms to the flow chart of the system for video recoloring. The results generated by the three can all be output through the result visualization module. The result of the data import module can be output as a set of loaded video frames through the visualization module, and the data processing module can be output as the visualization of the temporal auxiliary color palette and the convex hull visualization of the temporal auxiliary color palette in the RGB three-dimensional space through the visualization module. The result export module can be output as a recolored video through the visualization module.

[0051] Specifically, the data import module loads a set of video frames that the user needs to recolour, where:

[0052] The data loading module reads a set of video frames locally, and the format is PNG format. This set of video frames is used to represent a video.

[0053] Specifically, the data processing module is used to perform palette preprocessing on the loaded set of video frames, optimize the palette corresponding to each video frame using an optimization algorithm, decompose the weights of each video frame according to the timing-assisted palette result, and perform video recolouring using the convex hull deformation migration algorithm, including the following steps:

[0054] 1) Extract the palette from the loaded set of video frames:

[0055] First, each pixel can be represented as a point in the RGB three-dimensional color space, and all the pixels in each video frame can be represented as a point cloud in the RGB three-dimensional color space. For the loaded set of video frames, the pixel point clouds of each video frame are placed in the same three-dimensional color space, so as to obtain a large point cloud with video timing characteristics and color characteristics. The global palette P with video color characteristics is extracted from this large point cloud using the palette extraction algorithm based on the auxiliary palette. V ={p i ∈[0,1] 3 ,i=1,2,...,n P ,n P +1,...,n P +n A}, where p i is the i-th color in the global palette, n P is the number of colors in the main palette, and n A is the number of colors in the auxiliary palette;

[0056] Then, the pixel point cloud of each video frame is mapped according to the global palette P V to obtain the corresponding palette, that is, the t-th video frame I t is mapped according to the global palette P V to obtain the corresponding palette where is the palette corresponding to the t-th video frame The i-th color in, and the number of palette colors corresponding to each video frame is the same as the number of colors in the global palette P V Due to the mapping relationship, the i-th color in the palette P corresponding to the t-th video frame is similar to the i-th color p i in the global palette.

[0057] 2) Optimize the color palette corresponding to each video frame using an optimization algorithm:

[0058]

[0059] In the formula, P t is the optimized color palette of the t-th video frame, is the color palette loss term of the t-th video frame, used to constrain the difference between the colors in P t and the colors in ; P t-1 is the optimized color palette of the (t - 1)-th video frame, and P t+1 is the optimized color palette of the (t + 1)-th video frame, is the i-th color in P t ; is the i-th color in P t-1 ; is the i-th color in P t+1 ; is the optimized color palette P of the t-th video frame t and the smooth loss terms of the optimized color palettes P t-1 and P t+1 of the previous and next frames, used to constrain the color change amplitude between frames. λ data is the weight of the color palette loss term , and λ reg is the weight of the smooth loss term . T is the total number of video frames, and ε is the temporal auxiliary color palette loss term, comprehensively considering the color palette loss term and the smooth loss term corresponding to each video frame. After the optimization, the optimized color palette of each video frame is obtained, and all the optimized color palettes are integrated into a temporal auxiliary color palette P in the time dimension. The temporal auxiliary color palette describes the color characteristics of the entire group of video frames, and due to the temporal auxiliary color palette having auxiliary vertices, the video reconstruction accuracy is higher.

[0060] 3) Decompose the weights of each video frame according to the temporal auxiliary color palette:

[0061] After optimization, each video frame has its optimized color palette P t . Since the color palette is a geometric convex hull in the RGB three-dimensional color space, all the pixel point clouds within the convex hull can be obtained by linearly adding the color palette:

[0062]

[0063] In the formula, I t (x, y) is the pixel color with abscissa x and ordinate y in the t-th video frame, is the i-th color of the color palette For I t The weight of (x, y).

[0064] 4) Use the convex hull deformation migration algorithm for video recoloring:

[0065]

[0066] In the formula, is the i-th color in the palette after the change of the t-th video frame, is the pixel color with abscissa x and ordinate y in the t-th video frame after recoloring.

[0067] After the user changes the color of the optimized palette of a certain video frame, the system calculates the change range of the optimized palettes of the remaining video frames according to the convex hull deformation migration algorithm, and transfers the change to the optimized palettes of other video frames, and then reconstructs the image according to the decomposed weights, so as to realize video recoloring.

[0068] Specifically, the result visualization module includes a temporal auxiliary palette visualization module, a three-dimensional convex hull visualization module, and a video visualization module before and after recoloring. Examples of the three visualization modules are shown as Figure 4 shown, where:

[0069] The temporal auxiliary palette visualization module visualizes the temporal auxiliary palette. Users can observe the changes in the colors of the optimized palette of each frame on this module to correspond to the temporal changes in the colors of the entire video. Users can view the colors contained in the optimized palette of this video frame by sliding the time axis. In addition, users can realize the recoloring of the entire video by changing the optimized palette of a certain video frame in the temporal auxiliary palette visualization module;

[0070] The three-dimensional convex hull visualization module visualizes the convex hull topological structure of the optimized palette of each frame in the RGB three-dimensional color space, which is convenient for users to observe the changes in video colors from the perspective of the convex hull;

[0071] The video visualization module before and after recoloring visualizes the original video and the recolored video. Users can adjust how to perform more ideal recoloring by observing the color changes in the video before and after recoloring.

[0072] Specifically, the result export module is used to export the recolored video to the local for saving. The saving methods are divided into two types: image and video. The format of the exported image is PNG, and the format of the exported video is MP4. In addition, users can choose to merge the original video and the recolored video and then save them as the comparison result before and after recoloring. Examples of the comparison between the two recolored video results and the original video are shown as Figure 5 、 Figure 6 shown.

[0073] The above-described embodiments are only the preferred embodiments of the present invention, and do not limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A video recoloring system based on a timing-assisted color palette, characterized in that, Including: A data import module for loading a set of video frames; A data processing module for processing the loaded set of video frames, calculating the color palette corresponding to each video frame, performing optimization calculations on the color palette to obtain a timing-assisted color palette, and performing weight decomposition on each video frame according to the timing-assisted color palette, and using the convex hull deformation migration algorithm to transfer the color change of a certain frame by the user to other frames to achieve video recoloring; A result visualization module for visualizing the processing results of each step to facilitate the user to operate video recoloring and compare the recoloring results; A result export module for exporting the recolored video result.

2. The video recoloring system based on a timing-assisted color palette according to claim 1, wherein: The data import module loads a set of video frames that the user needs to recolor, specifically as follows: The data loading module reads a set of video frames from the local, and its format is PNG format. This set of video frames is used to represent a video.

3. The video recoloring system based on a timing-assisted palette according to claim 1, characterized in that: The data processing module performs the following operations: 1) Extract the color palette from the loaded set of video frames: First, each pixel is represented as a point in the RGB three-dimensional color space, and all the pixels in each video frame are represented as a point cloud in the RGB three-dimensional color space. For a set of loaded video frames, the pixel point clouds of each video frame are placed in the same three-dimensional color space, so as to obtain a large point cloud with video temporal characteristics and color characteristics. The palette extraction algorithm based on the auxiliary palette is used for this large point cloud to extract the global palette P with video color characteristics V ={p i ∈[0,1] 3 , i = 1, 2,..., n P , n P +1,..., n P +n A}, where p i is the i-th color in the global palette P V , n P is the number of main colors of the palette, n A is the number of auxiliary colors of the palette; Then, the pixel point cloud of each video frame is mapped according to the global color palette P V to obtain the corresponding color palette, that is, the t-th video frame I t is mapped according to the global color palette P V to obtain the corresponding color palette where is the i-th color in the color palette corresponding to the t-th video frame The number of color palette colors corresponding to each video frame is the same as that of the global color palette P V And due to the mapping relationship, the i-th color in the color palette corresponding to the t-th video frame is similar to the i-th color p in the global color palette i ; 2) Optimize the color palette corresponding to each video frame using an optimization algorithm: Wherein, P t is the optimized color palette of the t-th video frame, is the color palette loss term of the t-th video frame, which is used to constrain the difference between the colors in P t and the colors in , P t-1 is the optimized color palette of the (t - 1)-th video frame, P t+1 is the optimized color palette of the (t + 1)-th video frame, is the i-th color in P t , is the i-th color in P t-1 , is the i-th color in P t+1 , is the smooth loss term between the optimized color palette P t of the t-th video frame and the optimized color palettes P t-1 and P t+1 of the previous and next frames, which is used to constrain the color change amplitude between frames, λ data is the weight of the color palette loss term , λ reg is the weight of the smooth loss term , T is the total number of video frames, ε is the temporal auxiliary color palette loss term, comprehensively considering the color palette loss term and the smooth loss term corresponding to each video frame; after the optimization is completed, the optimized color palette of each video frame is obtained, and all the optimized color palettes are integrated into a temporal auxiliary color palette P in the time dimension. The temporal auxiliary color palette describes the color characteristics of the entire group of video frames, and due to the temporal auxiliary color palette having auxiliary vertices, the video reconstruction accuracy is higher; 3) Perform weight decomposition on each video frame according to the timing-assisted color palette: After optimization, each video frame has its optimized color palette P t , since the color palette is a geometric convex hull in the RGB three-dimensional color space, all the pixel point clouds within the convex hull can be obtained by linearly adding the color palette: Where, I t (x, y0 is the pixel color with the abscissa x and the ordinate y in the t-th video frame, is the i-th color of the color palette For I t (x, y0's weight; 4) Use the convex hull deformation migration algorithm for video recoloring: wherein, is the i-th color in the optimized color palette after the change of the t-th video frame, is the pixel color with abscissa x and ordinate y in the t-th video frame after recoloring; After the user changes the color of the optimized color palette of a certain video frame, the system calculates the change range of the optimized color palette of the remaining frames of the video according to the convex hull deformation migration algorithm, and transfers the change to the optimized color palettes of other video frames, and then reconstructs the image according to the decomposed weights, so as to achieve video recoloring.

4. The video recoloring system based on the timing-assisted color palette according to claim 1, wherein: The result visualization module includes a timing-assisted color palette visualization module, a three-dimensional convex hull visualization module, and a pre- and post-recoloring video visualization module, where: The timing-assisted color palette visualization module visualizes the timing-assisted color palette. The user can observe the color changes of each color in the optimized color palette of each frame on this module to correspond to the color changes of the entire video in terms of timing. The user can view the colors contained in the optimized color palette of the video frame by sliding the time axis. In addition, the user can achieve the recoloring of the entire video by changing the optimized color palette of a certain video frame in the timing-assisted color palette visualization module; The three-dimensional convex hull visualization module visualizes the convex hull topological structure of the optimized color palette of each frame in the RGB three-dimensional color space, facilitating the user to observe the color changes of the video from the perspective of the convex hull; The pre- and post-recoloring video visualization module visualizes the original video and the recolored video. The user can adjust how to perform more ideal recoloring by observing the color changes of the video before and after recoloring.

5. The video recoloring system based on a timing-assisted palette according to claim 1, characterized in that: The result export module is used to export the recolored video to the local for saving. The saving methods are divided into two types: image and video. The format of the exported image is PNG, and the format of the exported video is MP4. In addition, the user can choose to merge the original video and the recolored video and then save them as the comparison result before and after recoloring.

Images