Video re-coloring method and system based on Bezier curve

Abstract

The invention provides a video re-coloring method and system based on a Bezier curve, and relates to the technical field of video processing. According to the method, the Bezier curve fitting technology is adopted, a large amount of iterative operation is avoided, and the calculation efficiency is greatly improved. Therefore, the extraction of the video palette can be quickly completed within several seconds to dozens of seconds. In addition, the method endows the user with higher flexibility, allows the overall color of the video to be modified by adjusting the palette of the local frame, and has more advantages in the aspect of local color control compared with the limitation that the video color can only be edited through a global palette in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a video recoloring method and system based on Bezier curves. Background Art

[0002] As a popular color editing technology, image recoloring has attracted widespread attention in academia and industry. Related technologies have been widely used in many fields such as photo editing, advertising design, and film production, showing strong practical value and broad development prospects. At present, a representative technology in the field of video recoloring is to form a video palette by constructing a 4D geometric polyhedron. This technology successfully extends the image recoloring principle based on the convex hull to video scenes, achieving an intuitive and natural video color editing effect.

[0003] However, existing 4D geometric polyhedron-based video recoloring techniques have the following defects: the efficiency of extracting the video palette (i.e., the 4D geometric polyhedron) is not high, and this process involves a large number of iterative optimizations of the initial geometric polyhedron to simplify its topological structure, which causes significant time consumption and seriously affects the user experience. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the deficiencies of the prior art, the present invention provides a video recoloring method and system based on Bezier curves, which solves the technical problem of low efficiency of the existing video recoloring technology based on 4D geometric polyhedrons.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] In a first aspect, the present invention provides a video recoloring method based on Bezier curves, comprising:

[0009] Obtain the video to be processed, perform frame sampling on the video to be processed, and construct a sampled frame set;

[0010] A clustering algorithm is used to extract the representative color of each frame in the sampled frame set;

[0011] Connect similar colors between adjacent frames in the sampling frame set in sequence to form a number of color chains, and fit each of the color chains into an initial Bezier curve;

[0012] Modify the representative colors of any one or several frames in the sampled frame set, refit the initial Bezier curve according to the local color editing, and obtain a set of new Bezier curves;

[0013] The edited color palette of each frame in the video to be processed is obtained by intercepting the Bezier curve; the color palette change of each frame is mapped to all pixels in the video to be processed using the color transfer function, so as to realize the recoloring processing of the entire input video.

[0014] Preferably, the step of performing frame sampling on the video to be processed and constructing a set of sampled frames includes:

[0015] A sampling frame set is constructed by selecting one frame every x frames in the video to be processed, where x is a positive integer greater than or equal to 2.

[0016] Preferably, when a clustering algorithm is used to extract the representative color of each frame in the sampling frame set, during the extraction process, the representative color of the previous frame is used as the initial cluster center of the next frame.

[0017] Preferably, refitting the initial Bezier curve according to the local color editing comprises:

[0018] Refit the Bezier curve according to the local color editing. During the fitting process, the following energy function is used for optimization:

[0019] E refit =E edit +λE trend

[0020] Among them, E refit is the energy loss. During the fitting process, the goal is to minimize the energy loss; λ is the weight; E edit is the editing loss term, that is, the difference between the editing effect expected by the user and the actual effect, defined as:

[0021]

[0022] Where, F represents the index set of the user edited frame; m represents the number of Bezier curves; represents the frame palette of the i-th frame intercepted in the fitted j-th Bezier curve; represents the i-th frame palette intercepted from the j-th Bezier curve adjusted by the user;

[0023] E trend The color editing trend item is the difference between the color change trend of the video itself and the actual effect, which is defined as:

[0024]

[0025] Wherein, n represents the number of video frames in the video to be processed; and Respectively represent the gradient changes of the Bezier curve before and after fitting.

[0026] Preferably, the method of obtaining the edited color palette of each frame in the video to be processed by intercepting the Bezier curve includes:

[0027] The color palette corresponding to each video frame is obtained by intercepting the Bezier curve through a set of hyperplanes parallel to the time axis.

[0028] Preferably, the color migration function includes:

[0029]

[0030] Among them, I i,j and I i ′ ,j represent the colors of the jth pixel in the i-th frame before and after editing, m represents the number of Bezier curves, P i,k and P i ′ ,k Respectively represent the value of the jth color in the palette of the i-th frame before and after editing, I i,j About P i,k Similarity weight; d(I i,j ,P i,k ) indicates two colors I i,j With P i,k The Euclidean distance.

[0031] In a second aspect, the present invention provides a video recoloring system based on Bezier curves, comprising:

[0032] The sampling module is used to obtain the video to be processed, perform frame sampling on the video to be processed, and construct a sampling frame set;

[0033] A clustering module is used to extract the representative color of each frame in the sample frame set by using a clustering algorithm; during the extraction process, the representative color of the previous frame is used as the initial clustering center of the next frame;

[0034] A fitting module, used for sequentially connecting similar colors between adjacent frames in the sampling frame set to form a number of color chains, and fitting each of the number of color chains into an initial Bezier curve;

[0035] A local modification module is used to modify the representative colors of any one or several frames in the sampled frame set, and refit the initial Bezier curve according to the local color editing to obtain a set of new Bezier curves;

[0036] The recoloring module is used to obtain the edited palette of each frame in the video to be processed by intercepting the Bezier curve; the color migration function is used to map the palette changes of each frame to all pixels in the video to be processed, so as to realize the recoloring processing of the entire input video.

[0037] In a third aspect, the present invention provides a computer-readable storage medium, characterized in that it stores a computer program for video recoloring based on Bezier curves, wherein the computer program enables a computer to execute the video recoloring method based on Bezier curves as described above.

[0038] In a fourth aspect, the present invention provides an electronic device, comprising:

[0039] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include instructions for executing the Bezier curve-based video recoloring method as described above.

[0040] (III) Beneficial effects

[0041] The present invention provides a video recoloring method and system based on Bezier curves. Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention adopts Bezier curve fitting technology to avoid a large number of iterative operations, thereby greatly improving the calculation efficiency. Therefore, it only takes a few seconds to tens of seconds to quickly complete the extraction of the video palette. In addition, the present invention gives users higher flexibility, allowing the overall color of the video to be modified by adjusting the palette of the local frame. Compared with the limitation of the prior art that the video color can only be edited through the global palette, the present invention has more advantages in local color control. Finally, the present invention uses Bezier curves to construct a palette, which not only accurately captures the main color changes in the video, but also ensures a smooth transition of colors, making it more convenient for users to edit color effects and achieving a natural and smooth video recoloring effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A block diagram of a video recoloring method based on Bezier curves according to an embodiment of the present invention;

[0045] Figure 2 This is a specific flow chart of a video recoloring method based on Bezier curves according to an embodiment of the present invention;

[0046] Figure 3FIG. 4 is a schematic diagram of a video recoloring effect according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The embodiments of the present application provide a video recoloring method and system based on Bezier curves, thereby solving the technical problem of low efficiency of existing video recoloring technology based on 4D geometric polyhedrons, and using a set of 4D Bezier curves to accurately represent and construct the color palette of the video, thereby supporting efficient color editing.

[0049] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0050] Existing 4D geometric polyhedron-based video recoloring technologies still face two major challenges. First, the efficiency of extracting the video palette (i.e., the 4D geometric polyhedron) is not high. This process involves a large number of iterative optimizations of the initial geometric polyhedron to simplify its topological structure, which causes significant time consumption and seriously affects the user experience. Second, in the process of simplifying the 4D geometric polyhedron, the color representation of the video palette is gradually weakened, which makes local color adjustments likely to cause global color changes.

[0051] To solve the above problems, an embodiment of the present invention proposes a novel video recoloring method. The method uses a set of 4D Bezier curves to accurately represent and construct the color palette of the video. This method not only significantly improves the solution efficiency, but also ensures that the extracted color palette is more representative. Secondly, a color transfer function is designed, which can efficiently map the color changes of the local frame to the entire video, thereby achieving seamless color transition. The proposed method not only supports flexible local editing, but also can output high-quality recoloring effects in real time, significantly improving the practicality and convenience of video color editing.

[0052] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The embodiment of the present invention provides a video recoloring method based on Bezier curve, such as Figure 1 As shown, the method includes:

[0054] S1. Obtain the video to be processed, perform frame sampling on the video to be processed, and construct a sampling frame set;

[0055] S2, using a clustering algorithm to extract the representative color of each frame in the sampled frame set;

[0056] S3, connecting similar colors between adjacent frames in the sampling frame set in sequence to form a number of color chains, and fitting each of the number of color chains into an initial Bezier curve;

[0057] S4, modifying the representative colors of any one or several frames in the sampling frame set, refitting the initial Bezier curve according to the local color editing, and obtaining a set of new Bezier curves;

[0058] S5. Obtain the edited color palette of each frame in the video to be processed by intercepting the Bezier curve; use the color transfer function to map the color palette change of each frame to all pixels in the video to be processed, so as to realize the recoloring processing of the entire input video.

[0059] The embodiments of the present invention significantly improve the computational efficiency of palette extraction. Taking a video containing 500 frames as an example, the existing method may take several hours to complete the palette extraction work, while the present invention adopts Bezier curve fitting technology to avoid a large number of iterative operations, so that the computational efficiency is greatly improved. Therefore, it only takes a few seconds to tens of seconds to quickly complete the extraction of the video palette. In addition, the present invention gives users higher flexibility and allows the overall color of the video to be modified by adjusting the palette of the local frame. Compared with the limitation of the prior art that the video color can only be edited through the global palette, the present invention has more advantages in local color control. Finally, the present invention uses Bezier curves to construct a palette, which not only accurately captures the main color changes in the video, but also ensures a smooth transition of colors, making it more convenient for users to edit color effects and achieve a natural and smooth video recoloring effect.

[0060] Combine the following Figure 2 The embodiments of the present invention are described in detail. It should be noted that Figure 2 The dominant color of the video frame refers to the representative color of each frame in the frame set obtained by clustering. The video palette and the palette refitting correspond to the initial Bezier curve and the refitted Bezier curve after local modification.

[0061] In one embodiment, S1, obtaining a video to be processed, sampling frames of the video to be processed, and constructing a sampling frame set. Specifically, it includes:

[0062] A sampling frame set is constructed by selecting one frame every x (x is a positive integer greater than or equal to 2, and in the embodiment of the present invention, x=5) frames in the video to be processed.

[0063] In one embodiment, S2, a clustering algorithm is used to extract the representative color of each frame in the sample frame set; it should be noted that, in the extraction process, the representative color of the previous frame is used as the initial cluster center of the next frame. Specifically, it includes:

[0064] The improved k-means algorithm is used to extract the representative color of each frame. Specifically, the k-means algorithm is first used to extract the representative color of the first frame, and then the representative color of the first frame is used as the initial clustering center of the second frame, and the k-means algorithm is executed to extract the representative color. And so on, until the representative colors of all frames in the sampled frame set are extracted. It should be noted that in the specific implementation process, other clustering algorithms, such as k-mean++, DBSCAN and other clustering algorithms, can also be used.

[0065] In one embodiment, S3, similar colors between adjacent frames in the sampling frame set are sequentially connected to form a plurality of color chains, and the plurality of color chains are respectively fitted into an initial Bezier curve.

[0066] Specifically include:

[0067] In the above process of extracting representative colors, the representative colors of the previous frame are used to initialize the clustering algorithm of the next frame, so that the colors between adjacent frames can correspond naturally. Therefore, after extracting representative colors frame by frame, the same or similar colors between adjacent frames are connected to naturally form a series of color chains. Subsequently, it is only necessary to fit these color chains to Bezier curves to complete the construction of the color palette. The fitting energy function of the Bezier curve is:

[0068]

[0069] Where n represents the number of input video frames, t represents the index of the video frame, represents the i-th representative color of the t-th frame corresponding to the fitted Bezier curve, Represents the i-th representative color of the extracted t-th frame, and the function expects the two to be as close as possible.

[0070] In one embodiment, S4, the representative colors of any one or several frames in the sampled frame set are modified, and the initial Bezier curve is refitted according to the local color editing to obtain a set of new Bezier curves. Specifically, it includes:

[0071] The user can modify the representativeness of any one or several frames in the sampled frame set and refit the Bezier curve based on these local color edits. Specifically, the following energy function is used for optimization to achieve accurate fitting of the Bezier curve:

[0072] Erefit =E edit +λE trend

[0073] Among them, E refit is the energy loss. During the fitting process, the goal is to minimize the energy loss. λ is the weight, which is used to balance the relative contributions of the two items. E edit is the editing loss term, that is, the difference between the editing effect expected by the user and the actual effect, defined as:

[0074]

[0075] Where, F represents the index set of the user edited frame; m represents the number of Bezier curves; represents the frame palette of the i-th frame intercepted in the fitted j-th Bezier curve; Indicates the i-th frame palette captured from the j-th Bezier curve adjusted by the user.

[0076] E trend The color editing trend item is the difference between the color change trend of the video itself and the actual effect, which is defined as:

[0077]

[0078] Wherein, n represents the number of video frames in the video to be processed; and They represent the gradient changes of the Bezier curve before and after fitting, respectively, and are used to measure the changing trend of the Bezier curve before and after editing.

[0079] In one embodiment, S5, by intercepting the Bezier curve to obtain the edited palette of each frame in the video to be processed; using the color migration function to map the palette change of each frame to all pixels in the video to be processed, to achieve the recoloring of the entire input video. Specifically including:

[0080] The color palette corresponding to each video frame is obtained by intercepting the re-fitted set of Bezier curves through a set of hyperplanes (t=0, t=1 / n, ... t=1) parallel to the time axis.

[0081] Using the color migration function, the palette changes of each frame are mapped to all pixels in the frame, thus adjusting the color of the entire video. The specific color migration function is defined as follows:

[0082]

[0083] Among them, I i,j and I i ′ ,jrepresent the colors of the jth pixel in the i-th frame before and after editing, m represents the number of Bezier curves, P i,k and P i ′ ,k Respectively represent the value of the jth color in the palette of the i-th frame before and after editing, I i,j About P i,k Specifically, Defined as:

[0084]

[0085] Among them, d(I i,j ,P i,k ) indicates two colors I i,j With P i,k This operation actually converts d(I i,j ,P i,k ) is normalized to [0,1].

[0086] The effect of video recoloring is as follows Figure 3 shown.

[0087] An embodiment of the present invention provides a video recoloring system based on Bezier curves, the system comprising:

[0088] The sampling module is used to obtain the video to be processed, perform frame sampling on the video to be processed, and construct a sampling frame set;

[0089] A clustering module, used for extracting a representative color of each frame in a set of sampled frames by using a clustering algorithm;

[0090] A fitting module, used for sequentially connecting similar colors between adjacent frames in the sampling frame set to form a number of color chains, and fitting each of the number of color chains into an initial Bezier curve;

[0091] A local modification module is used to modify the representative colors of any one or several frames in the sampled frame set, and refit the initial Bezier curve according to the local color editing to obtain a set of new Bezier curves;

[0092] The recoloring module is used to obtain the edited palette of each frame in the video to be processed by intercepting the Bezier curve; the color migration function is used to map the palette changes of each frame to all pixels in the video to be processed, so as to realize the recoloring processing of the entire input video.

[0093] It can be understood that the Bezier curve-based video recoloring system provided in an embodiment of the present invention corresponds to the above-mentioned Bezier curve-based video recoloring method, and the explanations, examples, beneficial effects and other parts of the relevant contents can refer to the corresponding contents in the Bezier curve-based video recoloring method, which will not be repeated here.

[0094] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program for video recoloring based on Bezier curves, wherein the computer program enables a computer to execute the video recoloring method based on Bezier curves as described above.

[0095] An embodiment of the present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the Bezier curve-based video recoloring method as described above.

[0096] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0097] The embodiments of the present invention significantly improve the computational efficiency of palette extraction. Taking a video containing 500 frames as an example, the existing method may take several hours to complete the palette extraction work, while the present invention adopts Bezier curve fitting technology to avoid a large number of iterative operations, so that the computational efficiency is greatly improved. Therefore, it only takes a few seconds to tens of seconds to quickly complete the extraction of the video palette. In addition, the present invention gives users higher flexibility and allows the overall color of the video to be modified by adjusting the palette of the local frame. Compared with the limitation of the prior art that the video color can only be edited through the global palette, the present invention has more advantages in local color control. Finally, the present invention uses Bezier curves to construct a palette, which not only accurately captures the main color changes in the video, but also ensures a smooth transition of colors, making it more convenient for users to edit color effects and achieve a natural and smooth video recoloring effect.

[0098] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0099] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video recoloring method based on Bezier curve, characterized in that: include: Obtain the video to be processed, perform frame sampling on the video to be processed, and construct a sampled frame set; A clustering algorithm is used to extract the representative color of each frame in the sampled frame set; Connect similar colors between adjacent frames in the sampling frame set in sequence to form a number of color chains, and fit each of the color chains into an initial Bezier curve; Modify the representative colors of any one or several frames in the sampled frame set, refit the initial Bezier curve according to the local color editing, and obtain a set of new Bezier curves; Obtain the edited color palette of each frame in the video to be processed by intercepting the Bezier curve; The color transfer function is used to map the palette changes of each frame to all pixels in the video to be processed, thereby realizing the recoloring of the entire input video.

2. The video recoloring method based on Bezier curve according to claim 1, characterized in that: The step of sampling frames of the video to be processed and constructing a set of sampled frames includes: A sampling frame set is constructed by selecting one frame every x frames in the video to be processed, where x is a positive integer greater than or equal to 2.

3. The video recoloring method based on Bezier curve according to claim 1, characterized in that: When a clustering algorithm is used to extract the representative color of each frame in a set of sampled frames, during the extraction process, the representative color of the previous frame is used as the initial clustering center of the next frame.

4. The video recoloring method based on Bezier curve according to claim 1, characterized in that: The re-fitting of the initial Bezier curve according to the local color editing comprises: Refit the Bezier curve according to the local color editing. During the fitting process, the following energy function is used for optimization: AND refit =And edit +λE trend Among them, E refit is the energy loss. During the fitting process, the goal is to minimize the energy loss; λ is the weight; E edit is the editing loss term, that is, the difference between the editing effect expected by the user and the actual effect, defined as: Where, F represents the index set of the user edited frame; m represents the number of Bezier curves; represents the frame palette of the i-th frame intercepted in the fitted j-th Bezier curve; represents the i-th frame palette intercepted from the j-th Bezier curve adjusted by the user; E trend The color editing trend item is the difference between the color change trend of the video itself and the actual effect, which is defined as: Wherein, n represents the number of video frames in the video to be processed; and Respectively represent the gradient changes of the Bezier curve before and after fitting.

5. The video recoloring method based on Bezier curve according to claim 1, characterized in that: The method of obtaining the edited color palette of each frame in the video to be processed by intercepting the Bezier curve includes: The color palette corresponding to each video frame is obtained by intercepting the Bezier curve through a set of hyperplanes parallel to the time axis.

6. The video recoloring method based on Bezier curve according to any one of claims 1 to 5, characterized in that: The color migration function includes: Among them, I i,j and I i ′ ,j Respectively represent the color of the jth pixel in the i-th frame before and after editing, m represents the number of Bezier curves, P i,k and P′ i,k Respectively represent the value of the jth color in the palette of the i-th frame before and after editing, I i,j About P i,k Similarity weight; d(I i,j ,P i,k ) indicates two colors I i,j With P i,k The Euclidean distance.

7. A video recoloring system based on Bezier curve, characterized in that: include: The sampling module is used to obtain the video to be processed, perform frame sampling on the video to be processed, and construct a sampling frame set; A clustering module is used to extract the representative color of each frame in the sample frame set by using a clustering algorithm; during the extraction process, the representative color of the previous frame is used as the initial clustering center of the next frame; A fitting module, used for sequentially connecting similar colors between adjacent frames in the sampling frame set to form a number of color chains, and fitting each of the number of color chains into an initial Bezier curve; A local modification module is used to modify the representative colors of any one or several frames in the sampled frame set, and refit the initial Bezier curve according to the local color editing to obtain a set of new Bezier curves; A recoloring module, used to obtain the edited color palette of each frame in the video to be processed by intercepting the Bezier curve; The color transfer function is used to map the palette changes of each frame to all pixels in the video to be processed, thereby realizing the recoloring of the entire input video.

8. A computer-readable storage medium, characterized in that: The computer program for video recoloring based on Bezier curves is stored therein, wherein the computer program enables a computer to execute the video recoloring method based on Bezier curves according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include instructions for executing the Bezier curve-based video recoloring method according to any one of claims 1 to 6.

Images