A video coloring method, device, equipment and storage medium

Abstract

The present invention relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for video colorization. Continuous frames are used to receive a colorization requirement video sent by a video end to be colored, and when it is determined that the continuous frames do not store a video colorization strategy that matches the colorization requirement video, the colorization requirement video is divided into multiple colorization video frame sets according to the video scene of the colorization requirement video, so as to match each colorization video frame set step by step in a multi-source colorization sample set, so that when it is determined that the colorization sample set results corresponding to all the colorization video frame sets are matched, the results of each colorization sample set are merged to obtain a first colorization strategy, and then the first colorization strategy is fed back to the video end to be colored using continuous frames, which solves the problem of low video colorization efficiency of existing video colorization methods, can reduce the amount of video colorization calculation, and improve the video colorization response speed, thereby improving data matching efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method, apparatus, device and storage medium for video colorization. Background Art

[0002] Video colorization is to color each frame of a video. Compared with image colorization, it not only needs to consider spatial consistency, but also faces the challenge of temporal consistency. At present, there are three types of video colorization algorithms: fully automatic video colorization algorithm, reference frame-based video colorization algorithm, and graffiti-based video colorization algorithm. Among them, the reference frame-based video colorization algorithm uses a reference frame to color the grayscale frame, which is mainly divided into two categories. One is to use an unrelated example image as a reference frame to color the grayscale frame, with the goal of coloring it reasonably and maintaining temporal consistency; the other is to use a highly related example image as a reference frame to color the grayscale frame, with the goal of faithfully propagating the color of the reference frame to subsequent grayscale frames and maintaining temporal consistency.

[0003] When commercially colorizing black-and-white movies, in order to achieve satisfactory visual effects, the movie is generally divided into many video clips, and a reference frame is provided for colorization for each clip. This paper notes that when coloring the current clip, the reference frame of the adjacent clip is closer to the current clip, and the reference information provided by it can also be used. In this way, the total number of reference frames used is only one more than that of the algorithm based on a single-frame reference frame, but the reference information provided by the previous and next reference frames can be considered when coloring each frame, thereby achieving better colorization effects and improving temporal consistency. Since most of the existing reference-based video colorization algorithms use single-frame reference frames for video colorization, they cannot make good use of double-frame reference frames. Even if there are video colorization algorithms that can use multiple-frame reference frames, their colorization effects are not good. Summary of the invention

[0004] The embodiments of the present invention provide a method, apparatus, device and storage medium for video colorization, which can reduce the amount of video colorization calculations, improve the video colorization response speed, and thus improve data matching efficiency.

[0005] According to one aspect of the present invention, a method for video colorization is provided, which is performed by a video colorization palette, and includes:

[0006] Using continuous frames to receive the coloring requirement video sent by the video end to be colored;

[0007] When it is determined that the continuous frames do not store a video colorization strategy matching the colorization demand video, segmenting the colorization demand video into a plurality of colorization video frame sets according to a video scene of the colorization demand video;

[0008] In the multi-source colored sample set, matching each of the colored video frame sets step by step;

[0009] When it is determined that the coloring sample set results corresponding to all the coloring video frame sets are matched, the coloring sample set results are combined to obtain a first coloring strategy, and the first coloring strategy is fed back to the video end to be colored using continuous frames.

[0010] In addition, the video coloring method according to the above embodiment of the present invention may also have the following additional technical features:

[0011] Optionally, according to a video scene of the coloring demand video, the coloring demand video is divided into a plurality of coloring video frame sets, including:

[0012] Obtaining a set of previous and next frames in sequence according to the order of scene conversion levels of the previous and next frames of the previous and next frame sets;

[0013] Detect whether the video scene contains at least one set of preceding and following frames;

[0014] If yes, then in the coloring demand video, a coloring video frame set corresponding to each preceding and following frame set in the video scene is divided;

[0015] After denoising the video scene according to the front and back frame sets, the operation of sequentially acquiring a front and back frame set according to the front and back frame scene conversion level order of the front and back frame sets is returned to be executed until the video scene is segmented within the front and back frame sets of all the front and back frame scene conversion levels.

[0016] An embodiment of the above invention has the following advantages or beneficial effects: when the preceding and following frame sets are determined and included in the video scene, the shading requirement video is divided into multiple shading video frame sets according to the preceding and following frame sets, so that when each shading video frame set is matched in the shading sample set, the hit rate of the shading sample set can be improved, and the utilization rate of the shading sample set can be improved.

[0017] Optionally, the multi-source shading sample set includes at least two of: a dual-frame optical flow propagation shading sample set, a dual-frame semantic matching shading sample set, and a fusion shading sample set.

[0018] An embodiment of the above invention has the following advantages or beneficial effects: a specific coloring sample set setting method of a multi-source coloring sample set is proposed to further improve the video coloring response speed and improve the data matching efficiency.

[0019] Optionally, in the multi-source colored sample set, each colored video frame set is matched step by step, including:

[0020] Matching each set of colored video frames in the double-frame optical flow propagation coloring sample set;

[0021] When it is determined that no matching results of the coloring sample sets respectively corresponding to all the coloring video frame sets are found, the scene jump frame coloring video frame set not obtained is obtained, and the scene jump frame coloring video frame set not obtained is matched in the double-frame semantic matching coloring sample set;

[0022] When it is determined that no matching result of the coloring sample sets respectively corresponding to the unacquired coloring video frame sets of all scene jump frames is found, a distributed unacquired coloring video frame set is acquired, and the distributed unacquired coloring video frame set is matched in the fused coloring sample set.

[0023] An embodiment of the above invention has the following advantages or beneficial effects: when using a dual-frame optical flow propagation coloring sample set, a dual-frame semantic matching coloring sample set, and a fused coloring sample set to match each coloring video frame set step by step, and determining the coloring sample set results that are not matched to the coloring video frame set in the current coloring sample set, the next level coloring sample set is used to match the unmatched coloring video frame set, thereby reducing the amount of matching calculations in the video coloring process.

[0024] Optionally, after matching each set of colored video frames step by step in the multi-source colored sample set, the method further includes:

[0025] When it is determined that no rendering sample set results corresponding to all rendering video frame sets are matched, an unobtained rendering video frame set is obtained;

[0026] Matching the unobtained colored video frame set in each original data table of the persistent storage layer to obtain a persistent coloring strategy;

[0027] The persistent coloring strategy is combined with the coloring sample set result obtained by matching the used coloring sample set to obtain a second coloring strategy, and the second coloring strategy is fed back to the video end to be colored using continuous frames.

[0028] An embodiment of the above invention has the following advantages or beneficial effects: a supplementary processing strategy is added when a colored sample set result corresponding to a colored video frame set is not matched in a multi-source colored sample set, that is, when a colored sample set result corresponding to a colored video frame set is not matched, the unmatched colored video frame set is matched in the original data table of the persistent storage layer, which can ensure the matching success rate of the colored required video and the accuracy of the colored strategy.

[0029] Optionally, the method further includes at least one of the following:

[0030] According to the previous and next frame data of each original data table of the persistent storage layer, each original data table is segmented to obtain at least one data segmentation table, and the fused coloring sample set is updated according to each data segmentation table;

[0031] When a coloring sample set result corresponding to the coloring video frame set is matched in the fused coloring sample set, the dual-frame semantic matching coloring sample set is updated according to the matched coloring sample set result;

[0032] When a coloring sample set result corresponding to the coloring video frame set is matched in the dual-frame semantic matching coloring sample set, a historical hit count of the coloring sample set corresponding to the matched coloring sample set result is determined, and according to the coloring sample set result whose historical hit count of the coloring sample set is greater than a hit count threshold, the dual-frame optical flow propagation coloring sample set is updated;

[0033] The continuous frame rendering sample set is updated according to the first rendering strategy, or the continuous frame rendering sample set is updated according to the second rendering strategy.

[0034] An embodiment of the above invention has the following advantages or beneficial effects: an update strategy for coloring sample sets at each level is added, and by updating the coloring sample sets at each level, each coloring video frame set can be matched in the coloring sample sets at each level, thereby improving data matching efficiency.

[0035] Optionally, the method further includes at least one of the following:

[0036] Determine a continuous frame invalid coloring sample set according to the inertia threshold and / or the first preceding and succeeding frame threshold, and update the continuous frame coloring sample set according to the continuous frame invalid coloring sample set;

[0037] Determine a scene jump frame invalid shading sample set according to the LUT lookup table and / or the second previous and next frame thresholds, and update the dual-frame optical flow propagation shading sample set according to the scene jump frame invalid shading sample set;

[0038] Determine the first and last frame invalid coloring sample sets according to the LUT lookup table and / or the third preceding and following frame thresholds, and update the double-frame semantic matching coloring sample sets according to the first and last frame invalid coloring sample sets;

[0039] The persistent layer invalid shading sample set is determined according to the data elimination cycle and / or the previous and next frames of the shading sample set data, and the fused shading sample set is updated according to the persistent layer invalid shading sample set.

[0040] An embodiment of the above invention has the following advantages or beneficial effects: a coloring sample set failure strategy is added to each level of coloring sample sets, and the accuracy of video coloring can be ensured by determining the failed coloring sample sets in each level of coloring sample sets.

[0041] According to another aspect of the present invention, there is provided a video coloring device, which is configured on a video coloring palette and includes:

[0042] A coloring requirement video receiving module is used to receive the coloring requirement video sent by the video end to be colored using continuous frames;

[0043] A colorization demand video segmentation module, configured to segment the colorization demand video into a plurality of colorization video frame sets according to a video scene of the colorization demand video when it is determined that the continuous frames do not store a video colorization strategy matching the colorization demand video;

[0044] A rendering video frame set matching module, used for matching each rendering video frame set in a multi-source rendering sample set step by step;

[0045] The coloring strategy feedback module is used to combine the coloring sample set results to obtain a first coloring strategy when it is determined that the coloring sample set results corresponding to all the coloring video frame sets are matched, and use continuous frames to feed back the first coloring strategy to the video end to be colored.

[0046] According to another aspect of the present invention, there is provided a device, comprising:

[0047] at least one processor; and

[0048] a memory communicatively connected to the at least one processor; wherein,

[0049] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video coloring method described in any embodiment of the present invention.

[0050] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video colorization method described in any embodiment of the present invention when executed.

[0051] The technical solution of the embodiment of the present invention uses continuous frames to receive the coloring requirement video sent by the video end to be colored, and when it is determined that the continuous frames do not store the video coloring strategy that matches the coloring requirement video, the coloring requirement video is divided into multiple coloring video frame sets according to the video scene of the coloring requirement video, so as to match each coloring video frame set step by step in the multi-source coloring sample set, so that when it is determined that the coloring sample set results corresponding to all the coloring video frame sets are matched, the results of each coloring sample set are merged to obtain the first coloring strategy, and then the first coloring strategy is fed back to the video end to be colored using continuous frames, which solves the problem of low video coloring efficiency of the existing video coloring method, can reduce the amount of video coloring calculation, improve the video coloring response speed, and thus improve the data matching efficiency.

[0052] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easily understood using the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 is a flow chart of a video coloring method provided by Embodiment 1 of the present invention;

[0055] Figure 2 is a flow chart of a video coloring method provided by Embodiment 2 of the present invention;

[0056] Figure 3 is an example flow chart of a method for video colorization provided by Embodiment 3 of the present invention;

[0057] Figure 4 is a schematic diagram of a video coloring device provided by a fourth embodiment of the present invention;

[0058] Figure 5 It is a schematic diagram of the structure of a device for implementing the video colorization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0060] Embodiment 1

[0061] Figure 1 This is a flowchart of a method for video colorization provided by the first embodiment of the present invention. This embodiment can be applied to reduce the amount of video colorization calculation and improve the response speed of video colorization. The method can be executed by a video colorization device, which can be implemented in software and / or hardware, and can generally be directly integrated in a device that executes the method. The device can be a server device, such as a video colorization drawing board that implements the function of video colorization. Specifically, Figure 1As shown, the video coloring method may specifically include the following steps:

[0062] S110, using continuous frames to receive the coloring-required video sent by the video end to be colored.

[0063] The coloring requirement video may be any request for matching data. It is understandable that the coloring requirement video may be any coloring requirement video initiated by any user at the end of the video to be colored.

[0064] In the embodiment of the present invention, the video coloring drawing board uses continuous frames to receive the coloring requirement video sent by the video end to be colored. It can be understood that after the user initiates the coloring requirement video at the video end to be colored, the video coloring drawing board uses continuous frames to receive the coloring requirement video, so that the video coloring drawing board performs video coloring according to the coloring requirement video.

[0065] S120: When it is determined that the continuous frames do not store a video colorization strategy that matches the colorization-required video, segment the colorization-required video into a plurality of colorization video frame sets according to a video scene of the colorization-required video.

[0066] Among them, the video colorization strategy can be the result obtained by matching the colorization requirement video in the continuous frame colorization sample set. The video scene can be the frame segments before and after the data to be matched in the colorization requirement video. It can be understood that when matching the data, the data in any frame segments before and after can be matched. Exemplarily, when matching the data, the data from 6 am to 4 pm is matched, and the video scene can be from 6 am to 4 pm. The colorization video frame set can be a task obtained by segmenting the colorization requirement video. Exemplarily, assuming that the colorization requirement video is to match the data from 6 am to 4 pm, the colorization requirement video can be segmented into a colorization video frame set that matches the data from 6 am to 12 noon, and a colorization video frame set that matches the data from 12 noon to 4 pm.

[0067] In an embodiment of the present invention, after the video coloring drawing board uses continuous frames to receive the coloring requirement video sent by the video end to be colored, the video coloring drawing board can further determine whether the continuous frames store a video coloring strategy that matches the coloring requirement video, and when it is determined that the continuous frames do not store a video coloring strategy that matches the coloring requirement video, the video coloring drawing board divides the coloring requirement video into a plurality of coloring video frame sets according to the video scene of the coloring requirement video. It is understandable that when it is determined that the continuous frames store a video coloring strategy that matches the coloring requirement video, the video coloring strategy can be fed back to the video end to be colored using continuous frames.

[0068] S130 , matching the rendered video frame sets in a multi-source rendered sample set step by step.

[0069] Among them, the multi-source coloring sample set can be a plurality of coloring sample sets in the video coloring palette. Optionally, the multi-source coloring sample set may include: at least two of: a dual-frame optical flow propagation coloring sample set, a dual-frame semantic matching coloring sample set, and a fusion coloring sample set. Among them, the dual-frame optical flow propagation coloring sample set can be a coloring sample set in the scene jump frame memory of the video coloring palette. The dual-frame semantic matching coloring sample set can be a coloring sample set in the dual-frame semantic matching coloring sample set layer of the video coloring palette. The fusion coloring sample set can be a coloring sample set in the persistent storage layer of the video coloring palette. The above technical solution uses a specific coloring sample set setting method for proposing a multi-source coloring sample set to further improve the video coloring response speed and improve the data matching efficiency.

[0070] In the prior art, after receiving the colorization request video sent by the video end to be colored, the backend server usually goes through the four-layer structure of continuous frames, scene jump frames, double-frame semantic matching colorization sample set layer and persistent storage layer in the backend server. However, for each backward matching layer in the backend server, the previous and next frames of the video responding to the colorization request will be longer.

[0071] Different from the video colorization method in the prior art, the technical solution of this embodiment uses the establishment of a multi-source colorization sample set, which can match each colorization video frame set step by step in the multi-source colorization sample set, thereby improving the video colorization response speed and improving the data matching efficiency.

[0072] In an embodiment of the present invention, after the video rendering drawing board divides the rendering requirement video into multiple rendering video frame sets according to the video scene of the rendering requirement video, the video rendering drawing board can further match each rendering video frame set step by step in the multi-source rendering sample set.

[0073] It should be noted that, in a multi-source coloring sample set, when matching each coloring video frame set step by step, if the coloring sample set result corresponding to the coloring video frame set is matched in the current coloring sample set, the coloring video frame set matched to the coloring sample set result will no longer be matched in the next-level coloring sample set. Exemplarily, assuming that the coloring requirement video is divided into coloring video frame set 1, coloring video frame set 2 and coloring video frame set 3, and the multi-source coloring sample set includes a dual-frame optical flow propagation coloring sample set and a dual-frame semantic matching coloring sample set, then the coloring video frame set 1, the coloring video frame set 2 and the coloring video frame set 3 can be matched in the dual-frame optical flow propagation coloring sample set. If the coloring sample set result of the coloring video frame set 1 is not matched in the dual-frame optical flow propagation coloring sample set, the coloring video frame set 1 is further matched in the dual-frame semantic matching coloring sample set.

[0074] S140: When it is determined that the coloring sample set results corresponding to all the coloring video frame sets are matched, the coloring sample set results are combined to obtain a first coloring strategy, and the first coloring strategy is fed back to the video end to be colored using continuous frames.

[0075] The coloring sample set result may be a result obtained by matching any coloring sample set in the video coloring palette with the coloring video frame set. The first coloring strategy may be a coloring strategy obtained by coloring the video according to the coloring requirement video.

[0076] In an embodiment of the present invention, after the video rendering sketchpad matches each rendered video frame set step by step in the multi-source rendering sample set, the video rendering sketchpad can also merge the results of each rendered sample set to obtain a first rendering strategy when it determines that the rendered sample set results corresponding to all the rendered video frame sets are matched, and use continuous frames to feed back the first rendering strategy to the video end to be rendered.

[0077] The technical solution of this embodiment uses continuous frames to receive the coloring requirement video sent by the video end to be colored, and when it is determined that the continuous frames do not store the video coloring strategy that matches the coloring requirement video, the coloring requirement video is divided into multiple coloring video frame sets according to the video scene of the coloring requirement video, so as to match each coloring video frame set step by step in the multi-source coloring sample set, so that when it is determined that the coloring sample set results corresponding to all the coloring video frame sets are matched, the results of each coloring sample set are merged to obtain the first coloring strategy, and then the first coloring strategy is fed back to the video end to be colored using continuous frames, which solves the problem of low video coloring efficiency of existing video coloring methods, can reduce the amount of video coloring calculation, improve the video coloring response speed, and thus improve data matching efficiency.

[0078] Embodiment 2

[0079] Figure 2 This is a flowchart of a video colorization method provided by the second embodiment of the present invention. This embodiment is a further refinement of the above-mentioned technical solutions, and provides a plurality of specific optional implementation methods for segmenting the colorization demand video into multiple colorization video frame sets according to the video scene of the colorization demand video, and matching each colorization video frame set step by step in the multi-source colorization sample set. The technical solution in this embodiment can be combined with each optional solution in one or more of the above-mentioned embodiments. Figure 2 As shown, the method may include the following steps:

[0080] S210, using continuous frames to receive the coloring-required video sent by the video end to be colored.

[0081] S220: When it is determined that the continuous frames do not store a video colorization strategy that matches the colorization requirement video, sequentially obtain a set of previous and next frames according to the previous and next frame scene conversion level order of the previous and next frame sets.

[0082] Among them, the front and back frame set can be a module that characterizes the front and back frames. The front and back frame scene conversion level order can be the level order of the front and back frame set determined according to the front and back frame scene conversion. Exemplarily, when the front and back frame set is a whole day, the front and back frame scene conversion level order can be the first order or other orders, and the embodiment of the present invention does not limit this. It can be understood that for the same front and back frame set, the front and back frame scene conversion level order is also the same due to the same front and back frame scene conversion. Different front and back frame sets have different front and back frame scene conversion level orders due to different front and back frame scene conversions. The front and back frame set can be a front and back frame set currently determined according to the front and back frame scene conversion level order.

[0083] In an embodiment of the present invention, after the video coloring drawing board uses continuous frames to receive the coloring requirement video sent by the video end to be colored, the video coloring drawing board can further obtain a set of previous and next frames in sequence according to the previous and next frame scene conversion level order of the previous and next frame sets when it is determined that the continuous frames do not store a video coloring strategy that matches the coloring requirement video.

[0084] S230, detecting whether the video scene contains at least one set of preceding and following frames; if so, executing S240; if not, returning to executing S220.

[0085] In an embodiment of the present invention, after the video coloring drawing board sequentially obtains a set of previous and next frames in the order of the scene conversion levels of the previous and next frames of the previous and next frames, it can further detect whether the video scene contains at least one set of previous and next frames. Exemplarily, assuming that the previous and next frames are a whole day, it is detected whether the video scene includes at least one whole day. It is understandable that a video scene may contain multiple sets of previous and next frames. If it is detected that the video scene contains at least one set of previous and next frames, then in the coloring requirement video, a set of colored video frames corresponding to each set of previous and next frames in the video scene can be divided. If it is detected that the video scene does not contain a set of previous and next frames, it can return to execute the order of the scene conversion levels of the previous and next frames, and sequentially obtain a set of previous and next frames to obtain a set of previous and next frames of the next scene conversion level of the previous and next frames.

[0086] S240: Divide the coloring-required video into coloring video frame sets corresponding to the preceding and following frame sets in the video scene.

[0087] In an embodiment of the present invention, after detecting that a video scene contains a set of front and back frames, the video coloring drawing board can further divide the coloring video frame sets corresponding to each set of front and back frames in the video scene in the coloring demand video. It can be understood that there can be multiple coloring video frame sets corresponding to the set of front and back frames in the video scene. Exemplarily, if the set of front and back frames is a whole hour, and the video scene is from 7 am to 11:10 am, then the coloring video frame sets corresponding to the set of front and back frames in the video scene may include 2, namely, the coloring video frame set from 7 am to 8 am, and the coloring video frame set from 8 am to 11 am.

[0088] It can be understood that after obtaining a set of front and back frames according to the level order of the front and back frame scene conversion of the front and back frame sets, it is possible to further detect whether the video scene contains the front and back frame set, and when it is detected that the video scene includes the front and back frame set, in the coloring requirement video, divide the coloring video frame set corresponding to the front and back frame set in the video scene.

[0089] Optionally, the video rendering palette may use scene jump frames to divide the rendering demand video into rendering video frame sets corresponding to each of the previous and next frame sets in the video scene.

[0090] S250, after denoising the video scene according to the front and back frame sets, return to execute the operation of sequentially acquiring a front and back frame set according to the front and back frame scene conversion level order of the front and back frame sets, until the video scene is segmented within the front and back frame sets of all the front and back frame scene conversion levels.

[0091] In an embodiment of the present invention, after the video coloring drawing board divides the coloring video frame sets corresponding to the front and back frame sets in the video scene in the coloring requirement video, it can further denoise the video scene according to the front and back frame sets, and then return to execute the operation of obtaining a front and back frame set in sequence according to the front and back frame scene conversion level order of the front and back frame sets, until the video scene is segmented within the front and back frame sets of all front and back frame scene conversion levels.

[0092] Exemplarily, assuming that the video scene is from 6:00 a.m. to 7:10 a.m., and the front and back frame set is a whole hour, after the video scene is denoised according to the front and back frame set, the denoised video scene is from 7:00 a.m. to 7:10 a.m., and the front and back frame set in the next order of front and back frame scene conversion levels is returned as the front and back frame set, and the denoised video scene is denoised according to the front and back frame set obtained again, until the video scene is segmented within the front and back frame sets of all front and back frame scene conversion levels.

[0093] The above technical solution uses a determined set of preceding and following frames, and when the preceding and following frame sets are included in a video scene, the coloring requirement video is divided into multiple coloring video frame sets according to the preceding and following frame sets, so that when each coloring video frame set is matched in the coloring sample set, the hit rate of the coloring sample set can be improved, and the utilization rate of the coloring sample set can be improved.

[0094] S260: Match each of the rendered video frame sets in the dual-frame optical flow propagation rendered sample set.

[0095] In the embodiment of the present invention, after segmenting the video scene into the previous and next frame sets of all previous and next frame scene transition levels, the video shading palette can further match each shading video frame set in the double-frame optical flow propagation shading sample set.

[0096] S270, determining whether the coloring sample set results corresponding to all the coloring video frame sets are matched; if not, executing S280; if so, executing S290.

[0097] In an embodiment of the present invention, after the video coloring drawing board matches each coloring video frame set in the dual-frame optical flow propagation coloring sample set, it can further determine whether the coloring sample set results corresponding to all the coloring video frame sets are matched. If the coloring sample set results corresponding to all the coloring video frame sets are not matched, the scene jump frame unobtained coloring video frame set can be obtained, and the scene jump frame unobtained coloring video frame set can be matched in the dual-frame semantic matching coloring sample set. If the coloring sample set results corresponding to all the coloring video frame sets are matched, the coloring sample set results can be further merged to obtain the first coloring strategy, and the first coloring strategy can be fed back to the video end to be colored using continuous frames.

[0098] S280 , obtaining a set of scene jump frames that are not rendered, and matching the set of scene jump frames that are not rendered in a double-frame semantic matching rendering sample set.

[0099] The scene jump frame that does not obtain the rendering video frame set may be a rendering video frame set that does not match the rendering sample set result in the dual-frame optical flow propagation rendering sample set.

[0100] In an embodiment of the present invention, after determining that no matching has been found for the coloring sample set results corresponding to all the coloring video frame sets, the video coloring drawing board can further obtain the scene jump frame unacquired coloring video frame set, so as to match the scene jump frame unacquired coloring video frame set in the double-frame semantic matching coloring sample set.

[0101] S290, determining whether the coloring sample set results corresponding to all the coloring video frame sets are matched; if not, executing S2100; if so, executing S2120.

[0102] In an embodiment of the present invention, after obtaining a set of scene jump frames that have not been colored, and matching the set of scene jump frames that have not been colored in a dual-frame semantic matching coloring sample set, the video coloring drawing board can further determine whether the coloring sample set results corresponding to all the scene jump frames that have not been colored are matched. If the coloring sample set results corresponding to all the coloring video frame sets are not matched, a distributed set of unobtained coloring video frames can be obtained, and the distributed set of unobtained coloring video frames can be matched in the fused coloring sample set. If the coloring sample set results corresponding to all the coloring video frame sets are matched, the results of each coloring sample set can be further merged to obtain a first coloring strategy, and the first coloring strategy can be fed back to the video end to be colored using continuous frames.

[0103] S2100 , obtaining a distributed unobtained rendering video frame set, and matching the distributed unobtained rendering video frame set in a fused rendering sample set.

[0104] The distributed unobtained rendering video frame set may be a rendering video frame set that is not matched to a rendering sample set result in the dual-frame semantic matching rendering sample set.

[0105] In an embodiment of the present invention, after determining that no matching has been found for the coloring sample set results corresponding to the coloring video frame sets that have not been acquired for all scene jump frames, the video coloring drawing board can further acquire a distributed coloring video frame set that has not been acquired, so as to match the distributed coloring video frame set in the fused coloring sample set.

[0106] The above technical scheme uses the dual-frame optical flow propagation coloring sample set, the dual-frame semantic matching coloring sample set, and the fused coloring sample set to match each coloring video frame set step by step, and when determining the coloring sample set result that is not matched to the coloring video frame set in the current coloring sample set, the next level coloring sample set is used to match the unmatched coloring video frame set, thereby reducing the amount of matching calculations in the video coloring process.

[0107] In an embodiment of the present invention, the video coloring drawing board matches the unobtained coloring video frame set in each original data table of the persistent storage layer. After obtaining the persistent coloring strategy, the persistent coloring strategy can be further merged with the coloring sample set result obtained by matching the used coloring sample set to obtain a second coloring strategy, and the second coloring strategy is fed back to the video end to be colored using continuous frames.

[0108] The above technical solution adds a supplementary processing strategy when the coloring sample set results corresponding to the coloring video frame set are not matched in the multi-source coloring sample set. That is, when the coloring sample set results corresponding to the coloring video frame set are not matched, the unmatched coloring video frame set is matched in the original data table of the persistent storage layer, which can ensure the matching success rate of the coloring requirement video and the accuracy of the coloring strategy.

[0109] Optionally, the method may also include at least one of the following: performing data table segmentation on each original data table of the persistent storage layer according to the previous and next frame data of each original data table to obtain at least one data segmentation table, and updating the fused coloring sample set according to each data segmentation table; when a coloring sample set result corresponding to the colored video frame set is matched in the fused coloring sample set, updating the dual-frame semantic matching coloring sample set according to the matched coloring sample set result; when a coloring sample set result corresponding to the colored video frame set is matched in the dual-frame semantic matching coloring sample set, determining the number of historical hits of the coloring sample set corresponding to the matched coloring sample set result, and updating the dual-frame optical flow propagation coloring sample set according to the coloring sample set result whose historical hit number of the coloring sample set is greater than a hit number threshold; updating the continuous frame coloring sample set according to the first coloring strategy, or updating the continuous frame coloring sample set according to the second coloring strategy.

[0110] The data segmentation table may be a table obtained by segmenting the original data table.

[0111] Specifically, the video coloring drawing board can perform data table segmentation on each original data table of the persistent storage layer according to the front and back frame data of each original data table, obtain at least one data segmentation table, and update the fused coloring sample set according to each data segmentation table. Optionally, the data table segmentation can be performed on each original data table according to the front and back frame sets of each original data table of the persistent storage layer, to obtain at least one data segmentation table. Exemplarily, if the front and back frame sets include a whole day and a whole hour, the data segmentation table may include a day-level data segmentation table and an hour-level data segmentation table.

[0112] Specifically, when a coloring sample set result corresponding to the coloring video frame set is matched in the fused coloring sample set, the video coloring palette can update the dual-frame semantic matching coloring sample set according to the matched coloring sample set result.

[0113] The historical hit count of the coloring sample set may be the number of times the coloring sample set result found in the double-frame semantic matching coloring sample set is matched during the historical matching process. Exemplarily, the coloring sample set result found in the double-frame semantic matching coloring sample set is coloring sample set result A. During the historical matching process, the number of times the coloring sample set result A is matched is 10, and the historical hit count of the coloring sample set is 10. The hit count threshold may be a pre-set threshold of the number of matches.

[0114] Specifically, when the video colorization sketchpad matches a coloring sample set result corresponding to a colored video frame set in a dual-frame semantic matching coloring sample set, it can further determine the historical hit count of the coloring sample set corresponding to the matched coloring sample set result, and update the dual-frame optical flow propagation coloring sample set based on the coloring sample set result whose historical hit count of the coloring sample set is greater than a hit count threshold.

[0115] Specifically, the video rendering palette may update the continuous frame rendering sample set according to the first rendering strategy, or update the continuous frame rendering sample set according to the second rendering strategy.

[0116] The above technical solution adds an update strategy for coloring sample sets at each level. By updating the coloring sample sets at each level, each coloring video frame set can be matched in the coloring sample sets at each level, thereby improving data matching efficiency.

[0117] Optionally, the method may further include at least one of the following: determining a continuous frame invalid coloring sample set according to an inertia threshold and / or a first before and after frame threshold, and updating the continuous frame coloring sample set according to the continuous frame invalid coloring sample set; determining a scene jump frame invalid coloring sample set according to a LUT lookup table and / or a second before and after frame threshold, and updating a dual-frame optical flow propagation coloring sample set according to the scene jump frame invalid coloring sample set; determining the first and last frame invalid coloring sample sets according to the LUT lookup table and / or a third before and after frame threshold, and updating a dual-frame semantic matching coloring sample set according to the first and last frame invalid coloring sample sets; determining a persistent layer invalid coloring sample set according to a data elimination cycle and / or the before and after frames of the coloring sample set data, and updating a fused coloring sample set according to the persistent layer invalid coloring sample set.

[0118] The inertia threshold may be a threshold of frames before and after the coloring strategy stored in the continuous frame coloring sample set. The first threshold of frames before and after may be a threshold of frames before and after the failure of a continuous frame coloring sample set, for example, a few days or a few hours, etc., which is not limited in the embodiment of the present invention. The continuous frame failure coloring sample set may be a failure coloring sample set in the continuous frame coloring sample set.

[0119] The second preceding and following frame threshold may be a preset preceding and following frame threshold at which a dual-frame optical flow propagation coloring sample set fails. The scene jump frame failure coloring sample set may be a failure coloring sample set in the dual-frame optical flow propagation coloring sample set.

[0120] Specifically, the video coloring palette can determine the scene jump frame invalid coloring sample set according to the LUT lookup table or the second before and after frame threshold, or according to the LUT lookup table and the second before and after frame threshold, so as to update the dual-frame optical flow propagation coloring sample set according to the scene jump frame invalid coloring sample set. Exemplarily, the scene jump frame invalid coloring sample set is determined according to the LUT lookup table and the second before and after frame threshold, and the second before and after frame threshold is used as a period, and the coloring sample set that is least frequently used in the period is determined as the scene jump frame invalid coloring sample set according to the LUT lookup table.

[0121] The third preceding and following frame threshold may be a preset preceding and following frame threshold at which a double-frame semantic matching coloring sample set fails. The first and last frame failed coloring sample set may be a failed coloring sample set in the double-frame semantic matching coloring sample set.

[0122] Specifically, the video coloring palette can determine the first and last frame invalid coloring sample sets according to the LUT lookup table or the third before and after frame threshold, or according to the LUT lookup table and the third before and after frame threshold, so as to update the double-frame semantic matching coloring sample set according to the first and last frame invalid coloring sample sets. Exemplarily, the first and last frame invalid coloring sample sets are determined according to the LUT lookup table and the third before and after frame threshold, and the third before and after frame threshold is used as a period, and the least recently used coloring sample set in the period is determined as the double-frame semantic matching coloring sample set according to the LUT lookup table.

[0123] Specifically, the video rendering palette can determine the invalid rendering sample set of the persistence layer according to the data elimination cycle or the frames before and after the rendering sample set data, or according to the data elimination cycle and the frames before and after the rendering sample set data, so as to update the fused rendering sample set according to the invalid rendering sample set of the persistence layer.

[0124] The above technical solution adds a coloring sample set invalidation strategy for each level of coloring sample sets, and uses the invalid coloring sample sets in each level of coloring sample sets to ensure the accuracy of video coloring.

[0125] The technical solution of this embodiment uses continuous frames to receive the coloring requirement video sent by the video end to be colored, and when the video coloring strategy matching the coloring requirement video is not stored in the continuous frames, the coloring requirement video is divided into multiple coloring video frame sets according to the previous and next frame sets, and each coloring video frame set is matched step by step in the double-frame optical flow propagation coloring sample set, the double-frame semantic matching coloring sample set and the fusion coloring sample set. If the coloring sample set results corresponding to all the coloring video frame sets are matched, the results of each coloring sample set are merged and fed back to the video end to be colored using continuous frames. If the coloring sample set results corresponding to all the coloring video frame sets are not matched, the unobtained coloring video frame set is matched in the original data table, and the coloring strategy and the results of each coloring sample set are merged and fed back to the video end to be colored using continuous frames, thereby reducing the amount of calculation in the video colorization process and improving the response speed of video colorization, thereby improving the data matching efficiency.

[0126] Embodiment 3

[0127] In order to enable those skilled in the art to better understand the method of video colorization, this embodiment uses a specific example for illustration. Figure 3 is an example flow chart of a method for video colorization provided by Embodiment 3 of the present invention. Figure 3 As shown, the specific process includes:

[0128] Step 301: The video coloring drawing board uses continuous frames to receive the coloring requirement video sent by the video end to be colored. The video coloring strategy that matches the coloring requirement video is matched in the continuous frame coloring sample set, and when the video coloring strategy that matches the coloring requirement video is matched, the video coloring strategy is fed back to the video end to be colored. If the video coloring strategy that matches the coloring requirement video is matched, the coloring requirement video is sent to the scene jump frame of the video coloring drawing board.

[0129] Step 302: The video coloring drawing board scene jump frame receives the coloring requirement video, and uses the scene jump frame to segment the coloring requirement video according to the front and back frame sets to obtain multiple coloring video frame sets. Specifically, when segmenting the coloring requirement video, the matching start and end front and back frames in the coloring requirement video can be decomposed according to the front and back frame sets.

[0130] Step 303: In the multi-source coloring sample set of the video coloring palette, each coloring video frame set is matched step by step. Each coloring video frame set is matched in turn in the dual-frame optical flow propagation coloring sample set. If the coloring sample set result corresponding to the coloring video frame set is not matched in the dual-frame optical flow propagation coloring sample set, enter the dual-frame semantic matching coloring sample set layer, and match the unmatched coloring video frame set in the dual-frame semantic matching coloring sample set.

[0131] Step 304: merging the coloring sample set results. The coloring sample set results obtained from each layer are merged according to the pre-set data aggregation requirements. After assembling the merged coloring sample set results into standard format data, continuous frames are used to feed back to the video end to be colored.

[0132] Specifically, the updating of the hierarchical coloring sample set of the video coloring palette may include: the updating of the fused coloring sample set may include: using the offline analysis layer of the video coloring palette to segment the original data table in the persistent storage layer according to the offline analysis task, and updating the fused coloring sample set according to the segmented data table. Among them, the offline analysis task may be a task that uses the offline analysis layer to segment the original data table into a whole day, a whole hour, a whole thirty minutes, a whole ten minutes or a whole minute. The updating of the double-frame semantic matching coloring sample set may include: obtaining the coloring sample set result that matches the coloring video frame set in the fused coloring sample set, and updating the double-frame semantic matching coloring sample set according to the matched coloring sample set result. The updating of the double-frame optical flow propagation coloring sample set may include: determining the coloring sample set result that matches the coloring video frame set in the double-frame semantic matching coloring sample set, the number of hits in the preset front and back frame segments, and when the number of hits of the coloring sample set result exceeds the preset number threshold, updating the double-frame optical flow propagation coloring sample set according to the coloring sample set result. The updating of the continuous frame rendering sample set may include: obtaining a merged rendering sample set result fed back to the to-be-rendered video end using continuous frames, and updating the continuous frame rendering sample set according to the merged rendering sample set result.

[0133] Specifically, the failure of the hierarchical coloring sample set of the video coloring palette may include: the failure of the fusion coloring sample set may include: determining the persistent layer invalid coloring sample set according to the data elimination cycle and the frames before and after the coloring sample set data. Specifically, when entering the data elimination cycle, determine the coloring sample set whose frames before and after the coloring sample set data exceed the preset frames before and after frames, and determine the coloring sample set as the persistent layer invalid coloring sample set. The failure of the double-frame semantic matching coloring sample set may include: determining the first and last frame invalid coloring sample set according to the LUT lookup table and the timing failure threshold. Specifically, determine the least recently used coloring sample set according to the LUT lookup table within the preset frame period, and determine the coloring sample set as the first and last frame invalid coloring sample set. The failure of the double-frame optical flow propagation coloring sample set may include: determining the scene jump frame invalid coloring sample set according to the LUT lookup table and the timing failure threshold. Specifically, determine the least recently used coloring sample set according to the LUT lookup table within the preset frame period, and determine the coloring sample set as the scene jump frame invalid coloring sample set. The failure of a continuous frame coloring sample set may include: determining a continuous frame failure coloring sample set according to an inertia failure threshold and a timing failure threshold. Among them, inertia failure may be that after receiving the coloring requirement video, if the video coloring strategy of the coloring requirement video is matched in the continuous frames, it is further determined whether the storage front and back frames of the video coloring strategy have exceeded the inertia failure threshold. If it exceeds the inertia failure threshold, the video coloring strategy is determined as a continuous frame failure coloring sample set. Timing failure may be that within a preset front and back frame period, the coloring sample set of the continuous frame is determined to store the front and back frames, and the coloring sample set whose storage front and back frames of the coloring sample set exceed the timing failure threshold is determined as a continuous frame failure coloring sample set.

[0134] The above technical solution uses the grading of the coloring sample set of the video coloring palette to improve the response speed of the coloring requirement video and reduce the load of the data cluster; and uses the task segmentation of the coloring requirement video to improve the hit rate of the coloring sample set and the utilization rate of the coloring sample set.

[0135] Embodiment 4

[0136] Figure 4 is a schematic diagram of a video coloring device provided by Embodiment 4 of the present invention, such as Figure 4 As shown, the device includes: a coloring requirement video receiving module 510, a coloring requirement video segmentation module 520, a coloring video frame set matching module 530 and a coloring strategy feedback module 540, wherein:

[0137] The coloring requirement video receiving module 510 is used to receive the coloring requirement video sent by the video end to be colored using continuous frames;

[0138] A colorization demand video segmentation module 520 is used to segment the colorization demand video into a plurality of colorization video frame sets according to a video scene of the colorization demand video when it is determined that the continuous frames do not store a video colorization strategy matching the colorization demand video;

[0139] A rendered video frame set matching module 530, configured to match each rendered video frame set in a multi-source rendered sample set step by step;

[0140] The coloring strategy feedback module 540 is used to combine the coloring sample set results to obtain a first coloring strategy when it is determined that the coloring sample set results corresponding to all the coloring video frame sets are matched, and to feed back the first coloring strategy to the video end to be colored using continuous frames.

[0141] The technical solution of this embodiment uses continuous frames to receive the coloring requirement video sent by the video end to be colored, and when it is determined that the continuous frames do not store the video coloring strategy that matches the coloring requirement video, the coloring requirement video is divided into multiple coloring video frame sets according to the video scene of the coloring requirement video, so as to match each coloring video frame set step by step in the multi-source coloring sample set, so that when it is determined that the coloring sample set results corresponding to all the coloring video frame sets are matched, the results of each coloring sample set are merged to obtain the first coloring strategy, and then the first coloring strategy is fed back to the video end to be colored using continuous frames, which solves the problem of low video coloring efficiency of existing video coloring methods, can reduce the amount of video coloring calculation, improve the video coloring response speed, and thus improve data matching efficiency.

[0142] Optionally, the colorization-required video segmentation module 520 can be specifically used to: obtain a set of front and back frames in sequence according to the order of the front and back frame scene conversion levels of the front and back frame sets; detect whether the video scene contains at least one front and back frame set; if so, divide the colorization-required video frame sets corresponding to each front and back frame set in the video scene in the colorization-required video; after denoising the video scene according to the front and back frame sets, return to execute the operation of obtaining a set of front and back frames in sequence according to the order of the front and back frame scene conversion levels of the front and back frame sets, until the video scene is segmented within the front and back frame sets of all front and back frame scene conversion levels.

[0143] Optionally, the multi-source shading sample set may include at least two of: a dual-frame optical flow propagation shading sample set, a dual-frame semantic matching shading sample set, and a fusion shading sample set.

[0144] Optionally, the colored video frame set matching module 530 can be specifically used to: match each colored video frame set in the dual-frame optical flow propagation colored sample set; when it is determined that the colored sample set results corresponding to all the colored video frame sets are not matched, obtain the scene jump frame unacquired colored video frame set, and match the scene jump frame unacquired colored video frame set in the dual-frame semantic matching colored sample set; when it is determined that the colored sample set results corresponding to all the scene jump frame unacquired colored video frame sets are not matched, obtain the distributed unacquired colored video frame set, and match the distributed unacquired colored video frame set in the fused colored sample set.

[0145] Optionally, the colored video frame set matching module 530 can also be specifically used to: obtain the unobtained colored video frame set when it is determined that the colored sample set results corresponding to all the colored video frame sets are not matched; match the unobtained colored video frame set in each original data table of the persistent storage layer to obtain a persistent coloring strategy; merge the persistent coloring strategy with the colored sample set results obtained by matching the used colored sample set to obtain a second coloring strategy, and use continuous frames to feed back the second coloring strategy to the video end to be colored.

[0146] Optionally, the device for video colorization can be specifically used to: perform data table segmentation on each original data table of the persistent storage layer according to the previous and next frame data of each original data table to obtain at least one data segmentation table, and update the fused coloring sample set according to each data segmentation table; when a coloring sample set result corresponding to the colored video frame set is matched in the fused coloring sample set, update the dual-frame semantic matching coloring sample set according to the matched coloring sample set result; when a coloring sample set result corresponding to the colored video frame set is matched in the dual-frame semantic matching coloring sample set, determine the historical hit count of the coloring sample set corresponding to the matched coloring sample set result, and update the dual-frame optical flow propagation coloring sample set according to the coloring sample set result whose historical hit count of the coloring sample set is greater than a hit count threshold; update the continuous frame coloring sample set according to the first coloring strategy, or update the continuous frame coloring sample set according to the second coloring strategy.

[0147] Optionally, the video coloring device can also be specifically used to: determine a continuous frame invalid coloring sample set based on an inertia threshold and / or a first before and after frame threshold, and update the continuous frame coloring sample set based on the continuous frame invalid coloring sample set; determine a scene jump frame invalid coloring sample set based on a LUT lookup table and / or a second before and after frame threshold, and update a dual-frame optical flow propagation coloring sample set based on the scene jump frame invalid coloring sample set; determine the first and last frame invalid coloring sample sets based on the LUT lookup table and / or a third before and after frame threshold, and update a dual-frame semantic matching coloring sample set based on the first and last frame invalid coloring sample sets; determine a persistent layer invalid coloring sample set based on a data elimination cycle and / or the before and after frames of the coloring sample set data, and update a fused coloring sample set based on the persistent layer invalid coloring sample set.

[0148] The video colorization device provided in the embodiment of the present invention can execute the video colorization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0149] Embodiment 5

[0150] Figure 5 A schematic diagram of a device 10 that can be used to implement an embodiment of the present invention is shown. The device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0151] like Figure 5 As shown, the device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage module 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other using a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0152] A number of components in the device 10 are connected to the I / O interface 15, including: an input module 16, such as a keyboard, a mouse, etc.; an output module 17, such as various types of displays, speakers, etc.; a storage module 18, such as a disk, an optical disk, etc.; and a communication module 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication module 19 allows the device 10 to exchange information / data with other devices using a computer network such as the Internet and / or various telecommunication networks.

[0153] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing module (CPU), a graphics processing module (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a method for video colorization.

[0154] In some embodiments, the method of video colorization may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as storage module 18. In some embodiments, part or all of the computer program may be loaded and / or installed on device 10 via ROM 12 and / or communication module 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method of video colorization described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the method of video colorization using any other suitable means (e.g., by means of firmware).

[0155] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0156] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0157] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) that the user can use to provide input to the device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0159] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser that a user can use to interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. Any form or medium of digital data communication (e.g., a communication network) can be used to interconnect the components of the system. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0160] The computing system may include a video terminal to be colored and a server. The video terminal to be colored and the server are generally far away from each other and usually interact using a communication network. The relationship between the video terminal to be colored and the server is generated using a computer program running on a corresponding computer and having a video terminal to be colored-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0162] Those skilled in the art will appreciate that the contents disclosed in this disclosure may be subject to various variations and improvements. For example, the various devices or components described above may be implemented using hardware, or may be implemented using software, firmware, or a combination of some or all of the three.

[0163] Flowcharts are used in this disclosure to illustrate the steps of the method according to the embodiments of the present disclosure. It should be understood that the preceding or following steps are not necessarily performed precisely in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes.

[0164] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed using a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / module in the above embodiment can be implemented in the form of hardware or in the form of a software functional module. The present disclosure is not limited to any particular form of combination of hardware and software.

[0165] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present disclosure belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or highly formal sense, unless explicitly defined as such herein.

[0166] The above is an explanation of the present disclosure and should not be considered as a limitation thereof. Although several exemplary embodiments of the present disclosure are described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined in the claims. It should be understood that the above is an explanation of the present disclosure and should not be considered to be limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.

[0167] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0168] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for video colorization, performed by a video colorization palette, characterized in that: include: Using continuous frames to receive the coloring requirement video sent by the video end to be colored; When it is determined that the continuous frames do not store a video colorization strategy matching the colorization demand video, segmenting the colorization demand video into a plurality of colorization video frame sets according to a video scene of the colorization demand video; In the multi-source colored sample set, matching each of the colored video frame sets step by step; When it is determined that the coloring sample set results corresponding to all the coloring video frame sets are matched, the coloring sample set results are combined to obtain a first coloring strategy, and the first coloring strategy is fed back to the video end to be colored using continuous frames; According to the video scene of the coloring requirement video, the coloring requirement video is divided into a plurality of coloring video frame sets, including: Obtaining a set of previous and next frames in sequence according to the order of scene conversion levels of the previous and next frames of the previous and next frame sets; Detecting whether the video scene includes at least one set of preceding and following frames; If yes, dividing the coloring required video into coloring video frame sets corresponding to the respective preceding and following frame sets in the video scene; After denoising the video scene according to the front and back frame sets, returning to perform the operation of sequentially acquiring a front and back frame set according to the front and back frame scene conversion level order of the front and back frame sets until the video scene is segmented within the front and back frame sets of all the front and back frame scene conversion levels; The multi-source coloring sample set includes: at least two items of a dual-frame optical flow propagation coloring sample set, a dual-frame semantic matching coloring sample set, and a fusion coloring sample set; In the multi-source colored sample set, each of the colored video frame sets is matched step by step, including: Matching each of the colored video frame sets in the dual-frame optical flow propagation colored sample set; When it is determined that no coloring sample set results corresponding to all the coloring video frame sets are matched, obtaining a scene jump frame unobtained coloring video frame set, and matching the scene jump frame unobtained coloring video frame set in a double-frame semantic matching coloring sample set; When it is determined that no matching result of the coloring sample sets respectively corresponding to the unobtained coloring video frame sets of all scene jump frames is found, a distributed unobtained coloring video frame set is obtained, and the distributed unobtained coloring video frame set is matched in the fused coloring sample set; After matching each of the rendered video frame sets in the multi-source rendered sample sets step by step, the method further includes: when it is determined that no rendered sample set results corresponding to all rendered video frame sets are matched, obtaining an unobtained rendered video frame set; Matching the unobtained colored video frame set in each original data table of the persistent storage layer to obtain a persistent coloring strategy; The persistent coloring strategy is combined with the coloring sample set result obtained by matching the used coloring sample set to obtain a second coloring strategy, and the second coloring strategy is fed back to the video end to be colored using continuous frames.

2. A method for video colorization according to claim 1, characterized in that: The method further includes at least one of the following: performing data table segmentation on each original data table according to the previous and next frame data of each original data table of the persistent storage layer to obtain at least one data segmentation table, and updating the fused shading sample set according to each of the data segmentation tables; When a coloring sample set result corresponding to the coloring video frame set is matched in the fused coloring sample set, updating the dual-frame semantic matching coloring sample set according to the matched coloring sample set result; When a coloring sample set result corresponding to the coloring video frame set is matched in the dual-frame semantic matching coloring sample set, determining a historical hit count of the coloring sample set corresponding to the matched coloring sample set result, and updating the dual-frame optical flow propagation coloring sample set according to a coloring sample set result whose historical hit count of the coloring sample set is greater than a hit count threshold; The continuous frame rendering sample set is updated according to the first rendering strategy, or the continuous frame rendering sample set is updated according to the second rendering strategy.

3. A video colorization method according to claim 2, characterized in that: The method further includes at least one of the following: determining a continuous frame invalid rendering sample set according to an inertia threshold and / or a first preceding and succeeding frame threshold, and updating a continuous frame rendering sample set according to the continuous frame invalid rendering sample set; Determine a scene jump frame invalid shading sample set according to the LUT lookup table and / or the second previous and next frame thresholds, and update the dual-frame optical flow propagation shading sample set according to the scene jump frame invalid shading sample set; Determine the first and last frame invalid coloring sample sets according to the LUT lookup table and / or the third preceding and following frame thresholds, and update the double-frame semantic matching coloring sample set according to the first and last frame invalid coloring sample sets; The persistent layer invalid shading sample set is determined according to the data elimination cycle and / or the previous and next frames of the shading sample set data, and the fused shading sample set is updated according to the persistent layer invalid shading sample set.

4. A video coloring device, configured on a video coloring drawing board, characterized in that: include: A coloring requirement video receiving module is used to receive the coloring requirement video sent by the video end to be colored using continuous frames; A colorization demand video segmentation module, configured to segment the colorization demand video into a plurality of colorization video frame sets according to a video scene of the colorization demand video when it is determined that the continuous frames do not store a video colorization strategy matching the colorization demand video; A rendering video frame set matching module, used for matching each rendering video frame set in a multi-source rendering sample set step by step; A coloring strategy feedback module, configured to combine the coloring sample set results to obtain a first coloring strategy when it is determined that the coloring sample set results corresponding to all the coloring video frame sets are matched, and to feed back the first coloring strategy to the video end to be colored using continuous frames; According to the video scene of the coloring requirement video, the coloring requirement video is divided into a plurality of coloring video frame sets, including: Obtaining a set of previous and next frames in sequence according to the order of scene conversion levels of the previous and next frames of the previous and next frame sets; Detecting whether the video scene includes at least one set of preceding and following frames; If yes, dividing the coloring required video into coloring video frame sets corresponding to the respective preceding and following frame sets in the video scene; After denoising the video scene according to the front and back frame sets, returning to perform the operation of sequentially acquiring a front and back frame set according to the front and back frame scene conversion level order of the front and back frame sets until the video scene is segmented within the front and back frame sets of all the front and back frame scene conversion levels; The multi-source coloring sample set includes: at least two items of a dual-frame optical flow propagation coloring sample set, a dual-frame semantic matching coloring sample set, and a fusion coloring sample set; In the multi-source colored sample set, each of the colored video frame sets is matched step by step, including: Matching each of the colored video frame sets in the dual-frame optical flow propagation colored sample set; When it is determined that no coloring sample set results corresponding to all the coloring video frame sets are matched, obtaining a scene jump frame unobtained coloring video frame set, and matching the scene jump frame unobtained coloring video frame set in a double-frame semantic matching coloring sample set; When it is determined that no matching result of the coloring sample sets respectively corresponding to the unobtained coloring video frame sets of all scene jump frames is found, a distributed unobtained coloring video frame set is obtained, and the distributed unobtained coloring video frame set is matched in the fused coloring sample set; After matching each of the rendered video frame sets step by step in the multi-source rendered sample sets, the method further includes: When it is determined that no rendering sample set results corresponding to all rendering video frame sets are matched, an unobtained rendering video frame set is obtained; Matching the unobtained colored video frame set in each original data table of the persistent storage layer to obtain a persistent coloring strategy; The persistent coloring strategy is combined with the coloring sample set result obtained by matching the used coloring sample set to obtain a second coloring strategy, and the second coloring strategy is fed back to the video end to be colored using continuous frames.

5. A device, characterized in that: The device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, wherein the computer program is executed by the at least one processor so as to enable the at least one processor to perform the video colorization method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video colorization method according to any one of claims 1 to 3 when the processor executes the computer instructions.

Images