Interactive video coloring method, system, equipment and medium

Abstract

The invention provides an interactive video coloring method, system and device and a medium, the interactive video coloring method achieves the purpose of improving the video coloring quality by adjusting a feature map of an image coloring network before a color channel is reconstructed, and the structure of the image coloring network is not changed in the coloring process. And only one layer of feature map is correspondingly adjusted. The memory unit and the correction unit are used for respectively recording the semantic and color information of the historical coloring frame, and the intermediate structure of the current frame is modified by using the coloring result of the latest frame, so that the visual quality of the output video is remarkably improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to an interactive video colorization method, system, device and medium. Background Art

[0002] In recent years, with the development of hardware equipment and 5G networks, social applications based on short videos have sprung up like mushrooms after rain. Traditional social software and e-commerce software have also had to make adjustments and have added pages dedicated to displaying short videos. The emergence of such emerging things has profoundly changed people's production and life. Software applications that are limited to text and images as information carriers have gradually declined and cannot fully meet the social needs of the majority of users. At the same time, ordinary users, especially young people, are increasingly inclined to show personalized life fragments. However, the video software on the market can only provide limited video color adjustment or deformation editing functions, such as beauty effects, etc. Such fixed functions greatly limit the flexibility of video processing and reduce the user's imagination of customized video processing. Further, for the film and television industry, editing recorded videos will cost a lot of manpower and material resources, and often requires professionals to perform complicated human-computer interactions to achieve the desired effect. Therefore, developing a simple video editing method that can provide a higher degree of freedom can bring great convenience to both ordinary users and people engaged in the film and television industry.

[0003] At present, many interactive image colorization methods have achieved good results, such as color point-based methods and methods that support multimodal input, but there are few studies on interactive video colorization. If the image colorization model is directly applied to video colorization without change, the color reconstruction effect will be greatly reduced. The reason is that the video adds a time dimension compared to the image data. The final output result must not only have excellent performance on a single video frame, but also ensure the coherence between video frames. Since users cannot tolerate the interactive burden of inputting before coloring each frame, the migration of user-guided information over time also hinders the development of interactive video colorization. One migration method is to provide a path for the flow of interactive information between frames through optical flow. Unfortunately, all current optical flow estimation methods cannot obtain accurate results, and inaccurate optical flow leads to the accumulation of errors as the video length increases. At this time, the color strokes input by the user are inevitably mapped to the position after the offset. Therefore, the video colorization model must also adapt to the situation of inaccurate guidance information position. Summary of the invention

[0004] The present disclosure provides a method, system, device and storage medium for an interactive video colorization method.

[0005] According to one aspect of the present disclosure, there is provided an interactive video colorization method, comprising:

[0006] Obtaining a target grayscale video clip to be colored and a color point and a mask input by a user; wherein the target grayscale video clip includes a plurality of continuous grayscale frames;

[0007] Translating the current grayscale frame, the color point and the mask to a pre-trained interactive colorization model;

[0008] Wherein, the interactive coloring model includes a backbone network unit, a memory unit and a correction unit;

[0009] The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask;

[0010] Acquire a historical color state of a historical coloring frame from the memory unit; the historical coloring frame is a frame before the current grayscale frame;

[0011] The correction unit evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, and performs a correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map;

[0012] Reconstructing the color channel of the current grayscale frame using the modified second feature map to generate a colored frame, wherein the memory unit stores the color state of the current frame after the coloring is completed;

[0013] Repeat the above steps to colorize the target grayscale video clip frame by frame to generate a color video.

[0014] In the method according to some embodiments of the present disclosure, the backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask, including:

[0015] Using a pre-trained grayscale image-based backbone network unit model, the grayscale frame, the color point and the mask are combined to generate a first feature map to be corrected.

[0016] In the method according to some embodiments of the present disclosure, the backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask, including:

[0017] A U-Net-based backbone network unit model is used, wherein the backbone network unit model comprises ten convolution modules; wherein the first to fourth convolution modules increase the number of channels and reduce the size of feature maps by downsampling, the fifth to sixth convolution modules are hole convolutions, the seventh to tenth convolution modules reduce the number of channels and restore the size of feature maps by upsampling, the first to fourth convolution modules are connected to the sixth to tenth convolution modules by jump connections,

[0018] The grayscale frame, the color point and the mask are concatenated and input into the backbone network unit model to generate the first feature map to be corrected.

[0019] In the method according to some embodiments of the present disclosure, the method for obtaining the historical color state of the historical coloring frame includes:

[0020] Divide each frame of image into rectangular blocks, calculate the visual code and add the position code, input it into the Transformer-based visual encoder, and obtain the visual embedding tensor;

[0021] Iteratively updating the memory unit by using the visual embedding tensor one by one, and generating a candidate memory unit based on the current memory state and the visual embedding tensor;

[0022] Obtaining a gating vector for controlling the update degree of each memory unit through the current memory state and the candidate memory unit, and obtaining the memory unit after the iteration update through weighted summation;

[0023] The memory state outputted by the last iteration is used as the historical color state of the historical coloring frame to participate in subsequent feature correction.

[0024] In the method according to some embodiments of the present disclosure, evaluating, by the correction unit, the difference between the historical color state of the historical colorization frame and the feature map of the current grayscale frame includes:

[0025] The correction unit obtains the historical color state of the historical coloring frame from the memory unit, and evaluates the difference between the historical color state of the historical coloring frame and the feature map of the current grayscale frame to obtain the difference between the current feature and the historical state, that is, the difference is the offset caused by time.

[0026] In the method according to some embodiments of the present disclosure, the step of performing a correction process on the first characteristic map to be corrected according to the difference to obtain a corrected second characteristic map includes:

[0027] Based on the attention mechanism, the offset correction tensor is calculated through the channel scaling weight and the spatial scaling weight;

[0028] The first feature map is corrected based on the offset correction tensor to obtain a corrected second feature map.

[0029] In the method according to some embodiments of the present disclosure, it also includes:

[0030] The interactive video colorization model is pre-trained based on preset optimization objectives and loss functions to obtain a pre-trained interactive colorization model.

[0031] According to another aspect of the present disclosure, there is provided an interactive video colorization system, comprising:

[0032] An acquisition unit, which acquires a target grayscale video clip to be colored and a color point and a mask input by a user; wherein the target grayscale video clip includes a plurality of continuous grayscale frames;

[0033] Inputting the current grayscale frame, the color point and the mask into a pre-trained interactive colorization model;

[0034] Wherein, the interactive coloring model includes a backbone network unit, a memory unit and a correction unit;

[0035] A backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask;

[0036] A memory unit for storing a historical color state of a historical coloring frame; the historical coloring frame is a frame before the current grayscale frame;

[0037] A correction unit, which evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, and performs a correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map;

[0038] Reconstructing the color channel of the current grayscale frame using the modified second feature map to generate a colored frame, wherein the memory unit stores the color state of the current frame after the coloring is completed;

[0039] The above process is repeated to colorize the target grayscale video clip frame by frame to generate a color video.

[0040] According to another aspect of the present disclosure, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0041] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0042] The present invention achieves the purpose of improving the video colorization quality by adjusting the feature map of the image colorization network before reconstructing the color channel. During the colorization process, the structure of the image colorization network is not changed, and only one layer of feature map is adjusted accordingly. The memory unit and the correction unit are used to record the semantics and color information of the historical colorization frames respectively, and the intermediate structure of the current frame is modified using the colorization result of the latest frame, thereby significantly improving the visual quality of the output video.

[0043] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0045] Figure 1 is a flowchart according to an exemplary embodiment of the present disclosure;

[0046] Figure 2 is a schematic diagram of a framework overview according to an exemplary embodiment of the present disclosure;

[0047] Figure 3 is a schematic diagram of a process of generating a user input according to an exemplary embodiment of the present disclosure;

[0048] Figure 4 is a schematic diagram of a memory unit according to an exemplary embodiment of the present disclosure;

[0049] Figure 5 is a schematic diagram of a correction unit according to an exemplary embodiment of the present disclosure;

[0050] Figure 6 is a block diagram of an interactive video colorization system according to an exemplary embodiment of the present disclosure;

[0051] Figure 7 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0053] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0054] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0055] Existing methods for solving the instability problem of video colorization can be summarized into three aspects: network structure, loss function, and post-processing. First, some methods use feature matching of optical flow or latent space to cascade multiple networks and align features of adjacent frames. Other methods improve the network structure by proposing multiple sub-networks that extract features with learning priors at different scales. Second, in addition to many loss functions directly applied to video colorization such as reconstruction loss, perceptual loss, and segmentation loss in image processing, new loss functions such as short-term loss and long-term loss have been proposed and widely used. These loss functions can improve the color similarity of adjacent frames by matching the pixel-level non-occluded areas of the warped frame with the current output, as well as modeling each long-range connection between two generated frames to enforce the temporal consistency of the correctly warped area. Finally, researchers proposed methods for refining color video frames through post-processing, including spatiotemporal smoothing and two-stage processing combined with super-resolution models.

[0056] However, most of these methods are too dependent on optical flow data, while existing optical flow estimation algorithms based on color videos cannot provide accurate estimations and their performance on grayscale videos will be greatly degraded. In addition, post-processing methods cannot fundamentally solve the temporal consistency problem. Based on this, Figure 1 and Figure 2 As shown, this embodiment provides an interactive video coloring method, including:

[0057] S101, obtaining a target grayscale video clip to be colored and a color point and a mask input by a user; wherein the target grayscale video clip includes a plurality of continuous grayscale frames;

[0058] In one embodiment, obtaining sufficient video data and diverse user input data for training will cost a huge amount of manpower and material resources. Therefore, the user's interactive input can be simulated by using artificially synthesized data, which is also a common method in the current interactive colorization literature. The generation process is as follows: Figure 3As shown in the figure (color images are used because it is necessary to distinguish between grayscale frames and original frames). First, the optical flow estimation method is used to calculate the optical flow between two adjacent frames of the grayscale video clip. Then, pixels are selected at any position of the first frame of the true value video clip to simulate the user's color strokes, and the mask of the pixel points is saved. The number of pixels used in training is evenly distributed in the interval [0,50] to cover the real user interaction in a wide range of situations as much as possible. Finally, the pre-calculated optical flow is used to map the simulated user input and its mask on the first frame to all frames of the entire grayscale video clip.

[0059] Some definitions:

[0060] A grayscale frame refers to an image frame that contains only brightness information. It consists of grays with different brightness levels and has no color information. A grayscale frame is usually a black and white version of a video, and the brightness value usually varies between 0 (black) and 255 (white). In the video colorization process, the grayscale frame is the basis of the original video, and color information needs to be added to each frame through an algorithm. Each frame in the video can be regarded as a separate grayscale image.

[0061] User-entered color points: User-entered color points refer to the color information manually annotated by the user on the grayscale frame. These points contain the color specified by the user (such as red, blue, green, etc.) and are usually applied to a specific object or area. Through selective annotation, the user can provide initial color clues for the grayscale frame. These color points provide necessary hints for the automatic colorization algorithm, and the algorithm will use these points to decide how to color the same or similar areas in other frames.

[0062] Mask: It is used to mark the position of user input. 1 indicates that there is a color point input by the user at this pixel position, and 0 indicates that there is no user input at this pixel position.

[0063] S102, the current grayscale frame, color point and mask are converted into a pre-trained interactive coloring model; wherein the interactive coloring model includes a backbone network unit, a memory unit and a correction unit;

[0064] S103, the backbone network unit generates a first feature map to be corrected based on the current grayscale frame, color points and mask;

[0065] The grayscale image information of the current frame is extracted from the video clip as the basic input of the model. The user interactively inputs the specific pixels selected to specify the colors that these points should present when generating color images. The mask corresponding to the color point is used to identify which areas should apply the color input by the user, thereby ensuring accuracy in the subsequent feature map generation.

[0066] The backbone network unit (such as the U-Net-based structure) processes the current grayscale frame through multiple convolutional layers to extract its features. These convolutional layers process the input information through activation functions (such as ReLU functions) and normalization layers to enhance the network's expressiveness.

[0067] After a series of convolutions and downsampling, the backbone network will generate the first feature map to be corrected. This feature map will be further adjusted in the subsequent correction module to achieve higher quality color image generation. At the same time, the color points and mask information input by the user are integrated into the feature map generation process to ensure that the color specified by the user can be correctly reflected in the output.

[0068] Through the above steps, the backbone network unit ensures that it can effectively utilize the current grayscale frame and the user's interactive input to generate a feature map with preliminary color information, providing a basis for subsequent color channel reconstruction and correction.

[0069] S104, obtaining a historical color state of a historical coloring frame from a memory unit; the historical coloring frame is a frame before the current grayscale frame;

[0070] By obtaining the historical color state from the memory unit and making reference adjustments to the current grayscale frame, the model can effectively reduce the color discontinuity between video frames, especially in the case of fast scene switching or significant color changes, ensuring the overall consistency of video color and smoothness of viewing experience.

[0071] S105, evaluating the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame by a correction unit, and performing a correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map;

[0072] The difference evaluation mechanism of the correction unit during the correction process ensures the effective alignment between the historical color state and the current frame feature map. In this way, the color of each frame of the video is more continuous in time sequence, thereby reducing color flickering or jumping, greatly improving the visual coherence of the video.

[0073] S106, reconstructing the color channel of the current grayscale frame using the modified second feature map to generate a colored color frame, and the memory unit stores the color state of the current frame after the coloring is completed;

[0074] S107, repeat the above steps to colorize the target grayscale video clip frame by frame to generate a color video. T Frame target grayscale video clip Each frame is represented as a brightness channel in the Lab color space. The goal of the network model is to generate high-quality color video clips frame by frame in the Lab color space. The backbone network is a simple grayscale image colorization model, which is pre-trained by the image colorization dataset and receives grayscale frames. and the color point entered by the user and its mask The network is represented by the intermediate feature tensor φ in the figure t Separated into Due to the defects of the current colorization algorithm, color inconsistency may occur during the colorization process, including the instability of the interactive image colorization model itself and the instability caused by the time dimension of the video clip. These problems ultimately manifest as discontinuity in the color of the output results. This factor has been established during the network processing process, that is, the feature tensors of each layer of the network have been incorrectly offset. Therefore, this embodiment proposes to correct the feature map before reconstructing the color channel.

[0075] Feature map φ t The adjustment is done by the correction unit Completed, the correction unit makes full use of the historical coloring information and the semantic features of the current frame, comprehensively evaluates the feature differences in the image space dimension and channel dimension, and finally outputs the corrected feature tensor Continue to use Colorize the current frame.

[0076] The coloring state of the historical frame is stored in the memory unit Storage, where N M is the size of the memory unit, each vector Both compress and save the status information of the historical coloring frames. This information is used by the correction unit to correct the feature map. The memory unit is updated using the current memory state and the latest frame coloring output. and a memory subnetwork The composed write controller implements the update operation, which is directly read by the correction unit when in use, and the memory unit is updated after each frame is rendered.

[0077] In the method according to some embodiments of the present disclosure, the backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask, including:

[0078] Using a pre-trained grayscale image-based backbone network unit model, the grayscale frame, color points and mask are combined to generate a first feature map to be corrected.

[0079] In the method according to some embodiments of the present disclosure, the backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask, including:

[0080] A backbone network unit model based on U-Net is used, which contains ten convolution modules; among them, the first to fourth convolution modules increase the number of channels and reduce the size of feature maps by downsampling, the fifth to sixth convolution modules are hole convolutions, the seventh to tenth convolution modules reduce the number of channels and restore the size of feature maps by upsampling, and the first to fourth convolution modules are connected to the sixth to tenth convolution modules by jump connection.

[0081] The grayscale frame, color points and mask splicing are combined and input into the backbone network unit model to generate the first feature map to be corrected.

[0082] Specifically, the backbone network unit model uses the U-Net structure. The backbone network has a total of 10 convolutional modules. The first four convolutional blocks are used with the ReLu activation function to halve the spatial size of the feature map, while doubling the number of channels, and adding a normalization layer at the end. Next, the 5th and 6th convolutional blocks maintain the size of the feature map unchanged. In order to expand the perceptual range of the convolution and capture more comprehensive image features, a dilated convolution with a stride of 2 is used. Compared with the previous modules, the 7th to 10th modules gradually increase the spatial size and halve the number of channels to restore the original spatial dimension of the image. The model connects the symmetrical front and back four convolutional modules through the jump connection technology, so that in the image reconstruction stage, the model can not only use high-level features, but also effectively use low-level information such as image edges to promote the recovery of color channels and prevent problems such as color overflow. The size change is achieved through downsampling and upsampling techniques. Each convolutional layer uses a convolution kernel with a step size of 1.

[0083] The grayscale video frame, the color point input by the user and its mask are spliced ​​without any processing and directly input into the backbone network to obtain the feature map to be corrected. The 10th convolution block is usually regarded as the color channel reconstruction stage. Therefore, this embodiment proposes to correct the image feature map before the color channel reconstruction. Since the upsampled feature map output by the 9th convolution block will be short-linked with the first layer, the correction unit adjusts the feature map after the short link.

[0084] In the method according to some embodiments of the present disclosure, the method for obtaining the historical color state of the historical coloring frame includes:

[0085] Divide each frame of image into rectangular blocks, calculate the visual code and add the position code, input it into the Transformer-based visual encoder, and obtain the visual embedding tensor;

[0086] Iteratively update the memory units one by one using the visual embedding tensor, and generate candidate memory units based on the current memory state and the visual embedding tensor;

[0087] Through the current memory state and the candidate memory units, a gating vector for controlling the update degree of each memory unit is obtained, and the memory unit updated after this iteration is obtained through weighted summation;

[0088] The memory state output by the last iteration is used as the historical color state of the historical coloring frame to participate in subsequent feature correction.

[0089] In a specific example, the memory unit M t And its update process is as follows Figure 4 As shown, write to the controller The processing process can be divided into two stages: frame encoding and updating. The first stage is a Transformer-based visual encoder, which first divides the image into rectangular blocks, then calculates the visual encoding (Embedding) and adds the position encoding, and then inputs it into the 6-layer Transformer structure to obtain the visual embedding tensor. N represents the number of embedding tensors.

[0090] Then in the second stage, each encoding in the visual embedding Each memory cell will be updated and iterated one by one, with the number of iterations being N. At each iteration, the current memory state and Generate candidate memory units together The first iteration uses the memory state M of the previous frame t-1 To initialize Each candidate memory cell The calculation process can be given by the following formula:

[0091]

[0092] in,[·] p Indicates the concatenation operation on the variable p, AP is the mean pooling, represents the learnable parameters in the multilayer perceptron. Next, the current memory state and candidate memory cells Participate in the calculation together to obtain a gating vector used to control the update degree of each memory unit The calculation process can be defined by the following formula:

[0093]

[0094] Among them, W g Represents a learnable parameter, and the Sigmoid function is used to map the calculation result to the interval [0,1]. At this time, the memory unit updated by this iteration can be obtained through weighted summation. The calculation results can be obtained by the following formula:

[0095]

[0096] After N iterations, the memory unit will be updated by each visual embedding tensor, and the output of the last iteration is Will be used as the latest color memory state M t and participate in the subsequent feature modification phase.

[0097] In the method according to some embodiments of the present disclosure, evaluating the difference between the historical color state of the historical colorization frame and the feature map of the current grayscale frame by the correction unit includes:

[0098] The correction unit obtains the historical color state of the historical coloring frame from the memory unit, and evaluates the difference between the historical color state of the historical coloring frame and the feature map of the current grayscale frame to obtain the difference between the current feature and the historical state, that is, the difference is the offset caused by time.

[0099] The correction unit is used to estimate the difference between the characteristics of the current grayscale frame during the colorization process and the historical colorization state, and solve the instability problem caused by the video time dimension based on these differences. The specific operation process is as follows Figure 5 First, in order to take into account the relationship between adjacent time frames and the global shading state, the memory unit is deformed and upsampled to the image space size and the shading result of the previous moment. The two images are spliced ​​together and mapped to the current frame using optical flow displacement. Then, a lightweight convolutional network with a convolution kernel size of 5×5 and 3 layers is used to obtain the color semantic feature map of the previous moment. The current grayscale frame passes through the front part of the backbone network The calculation output color channel feature map before reconstruction φ t and By subtracting the difference between the current coloring state and the historical memory, we get the difference between the current coloring state and the historical memory, which is the offset caused by time. The model will then assess whether this shift is reasonable and decide whether to retain it or reduce it.

[0100] The addition of the correction unit enables the image colorization algorithm to be easily applied to video colorization tasks. Unlike the existing methods, the model of this embodiment does not make any modifications to the image colorization network, nor does it need to retrain this part of the network. It only needs to train the memory network and the correction unit, which ensures the single-frame colorization effect while enhancing the temporal continuity of the coloring of the entire video clip.

[0101] In the method according to some embodiments of the present disclosure, performing a correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map includes:

[0102] Based on the attention mechanism, the offset correction tensor is calculated through the channel scaling weight and the spatial scaling weight;

[0103] The first feature map is corrected based on the offset correction tensor to obtain a corrected second feature map.

[0104] In one example, inspired by the attention mechanism, With φ t The maximum pooling and mean pooling are calculated separately and spliced ​​together. Here is global pooling. The pooling kernel size is equal to the feature map space size. Then it is sent to the multi-layer perceptron and obtained through the activation function. The channel scaling weight of is the channel retention coefficient of the difference. and The concatenation is then fed into a lightweight spatial encoder consisting of two convolutional layers with a kernel size of 3×3. Each layer of convolution is batch normalized and activated using the ReLu function. The maximum and mean pooling of the channel dimensions are then performed. Finally, two layers of 3×3 convolution and Sigmoid activation are performed to obtain The channel scaling weight and spatial scaling weight act on the offset We can get the offset correction tensor and use it to correct φ t Correct the deviation and finally get the corrected feature map Backbone network Continue to use Completes rendering of the current frame.

[0105] In the method according to some embodiments of the present disclosure, it also includes:

[0106] The interactive video colorization model is pre-trained based on preset optimization objectives and loss functions to obtain a pre-trained interactive colorization model.

[0107] The optimization target of the interactive video coloring model of this embodiment can be given by the following formula:

[0108]

[0109] in, and Respectively represent the correction module and memory subnetwork The learnable parameters of The video dataset used for training, is the loss function, is the shaded output of the previous frame, Represents the true value of the target grayscale frame.

[0110] The backbone network is an interactive image colorization model. First, the necessary reconstruction loss is added. In order to fix the local color point input by the user at a specific location, the color point loss is used To constrain the gradient loss, we also use gradient loss to facilitate the diffusion of color points and maintain the smoothness of the output results. For supervision, the training loss of the backbone network can be summed at each spatial pixel position p:

[0111]

[0112] Among them, Sobel represents the Sobel gradient operator. Although the backbone network is trained using image datasets, this paper still regards these image data as a single video clip with 1 frame number, so t is always equal to 1.

[0113] In the second stage, due to the addition of memory network and correction module, video data set is needed for training. The memory unit records the global and local coloring status. The coloring result of the previous frame is also mapped to the current frame through optical flow in the correction module. This enables the model in this paper to process adjacent frames and long time spans. Therefore, for the training of this stage, it is only necessary to Add the timing loss of the previous and next two frames based on The definition is as follows:

[0114]

[0115] Among them, O t-1→t represents the optical flow between two adjacent frames, Warp is the optical flow mapping, which maps the shading result of the previous moment to the current moment through the optical flow, and M t-1→t The mask representing the non-occluded area is calculated as follows:

[0116]

[0117] Among them, α is set to 50.

[0118] This embodiment proposes a video colorization model based on an interactive image colorization main network, which enhances the quality of video colorization by adjusting the intermediate features of the network.

[0119] The state information of historical frame coloring is memorized through the memory unit, and the Transformer-based sub-network is used to update the memory unit to provide guidance for the coloring of subsequent frames.

[0120] Through the correction unit, the semantic features of the current grayscale frame are adjusted using the recorded historical state and the colorization result of the latest frame, so as to achieve the color consistency of the output video.

[0121] The interactive video colorization method provided in this embodiment can be executed in an intelligent terminal, a computer terminal, a network device, a chip, a chip module or a similar computing device.

[0122] Corresponding to the interactive video colorization method described above, this embodiment also provides an interactive video colorization system. The following will introduce them separately. Specifically, Figure 6 As shown, the interactive video coloring system of this embodiment includes:

[0123] The acquisition unit 1 acquires a target grayscale video clip to be colored and a color point and a mask input by a user; wherein the target grayscale video clip includes a plurality of continuous grayscale frames;

[0124] Input the current grayscale frame, color point and mask into a pre-trained interactive colorization model; wherein the interactive colorization model includes a backbone network unit 2, a memory unit 3 and a correction unit 4;

[0125] The backbone network unit 2 generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask;

[0126] Memory unit 3, storing the historical color state of the historical coloring frame; the historical coloring frame is the previous frame of the current grayscale frame;

[0127] A correction unit 4 evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, and performs a correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map;

[0128] The color channel of the current grayscale frame is reconstructed using the modified second feature map to generate a colored frame, and the memory unit 3 stores the color state of the current frame after the coloring is completed;

[0129] Repeat the above process to colorize the target grayscale video clip frame by frame to generate a color video.

[0130] It should be noted that the interactive video colorization system embodiment provided in the embodiments of the present application and the above-mentioned interactive video colorization method embodiment are based on the same inventive concept.

[0131] It should be noted that the interactive video coloring system of the present embodiment can be, for example: a separate chip, a chip module or an electronic device, or a chip or chip module integrated into an electronic device. Regarding the various modules / units contained in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units contained therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units contained therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; The element can be implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0132] The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements an interactive video coloring method when executing the program. Figure 7 The electronic device 30 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0133] like Figure 7 As shown, the electronic device 30 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0134] The bus 33 includes a data bus, an address bus, and a control bus.

[0135] The memory 32 may include a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0136] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0137] The processor 31 executes various functional applications and data processing by running the computer programs stored in the memory 32, such as the interactive video coloring method of embodiment 1 of the present invention.

[0138] The electronic device 30 may also communicate with one or more external devices 34 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 35. Furthermore, the model generating device 30 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. Figure 7 As shown, the network adapter 36 communicates with other modules of the model-generated device 30 via the bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0139] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0140] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the interactive video colorization method are implemented.

[0141] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0142] In a possible implementation manner, the present invention may also be implemented in the form of a program product, which includes program codes. When the program product is run on a terminal device, the program codes are used to enable the terminal device to execute the steps in the interactive video coloring method.

[0143] Among them, the program code for executing the present invention can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0144] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this embodiment is not limited here.

[0145] It should be understood that the "system", "device", "unit" and / or "module" used in this embodiment is a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0146] In this embodiment, a flow chart is used to illustrate the operations performed by the system according to the embodiment of the present embodiment. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0147] As shown in this embodiment, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate that the steps and elements that have been clearly identified are included, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0148] The definition of including in this embodiment, as used herein, the terms "having", "may have", "including", or "may include" indicate the existence of the corresponding functions, operations, elements, etc. of this embodiment, and do not limit the existence of one or more other functions, operations, elements, etc. In addition, it should be understood that the terms "including" or "having" as used herein indicate the existence of the features, numbers, steps, operations, elements, parts, or a combination thereof described in the specification, and do not exclude the existence or addition of one or more other features, numbers, steps, operations, elements, parts, or a combination thereof.

[0149] In the definition of and / or in this embodiment, the term "A or B", "at least one of A and / or B" or "one or more of A and / or B" as used herein includes any and all combinations of the words listed therewith. For example, "A or B", "at least one of A and / or B" or "one or more of A and / or B" means (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0150] The definitions of first and second in this embodiment, and the descriptions of first, second, etc. in this embodiment are only used for illustration and distinction of the described objects, and have no order, nor do they represent a special limitation on the number of devices in this embodiment, and cannot constitute any limitation on this embodiment. For example, a first element may be referred to as a second element without departing from the scope of this disclosure, and similarly, a second element may be referred to as a first element.

[0151] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. An interactive video colorization method, characterized in that: include: Obtaining a target grayscale video clip to be colored and a color point and a mask input by a user; wherein the target grayscale video clip includes a plurality of continuous grayscale frames; Translating the current grayscale frame, the color point and the mask to a pre-trained interactive colorization model; Wherein, the interactive coloring model includes a backbone network unit, a memory unit and a correction unit; The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask; Acquire a historical color state of a historical coloring frame from the memory unit; the historical coloring frame is a frame before the current grayscale frame; The correction unit evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, and performs a correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map; Reconstructing the color channel of the current grayscale frame using the modified second feature map to generate a colored frame, wherein the memory unit stores the color state of the current frame after the coloring is completed; Repeat the above steps to colorize the target grayscale video clip frame by frame to generate a color video.

2. The interactive video coloring method according to claim 1, characterized in that: The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask, including: Using a pre-trained grayscale image-based backbone network unit model, the grayscale frame, the color point and the mask are combined to generate a first feature map to be corrected.

3. The interactive video coloring method according to claim 1 or 2, characterized in that: The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask, including: A U-Net-based backbone network unit model is used, wherein the backbone network unit model comprises ten convolution modules; wherein the first to fourth convolution modules increase the number of channels and reduce the size of feature maps by downsampling, the fifth to sixth convolution modules are hole convolutions, the seventh to tenth convolution modules reduce the number of channels and restore the size of feature maps by upsampling, the first to fourth convolution modules are connected to the sixth to tenth convolution modules by jump connections, The grayscale frame, the color point and the mask are concatenated and input into the backbone network unit model to generate the first feature map to be corrected.

4. The interactive video coloring method according to claim 1, characterized in that: The method for obtaining the historical color state of the historical coloring frame includes: Divide each frame of image into rectangular blocks, calculate the visual code and add the position code, input it into the Transformer-based visual encoder, and obtain the visual embedding tensor; Iteratively updating the memory unit by the visual embedding tensor one by one, and generating a candidate memory unit based on a current memory state and the visual embedding tensor; Obtaining a gating vector for controlling the update degree of each memory unit through the current memory state and the candidate memory unit, and obtaining the memory unit after the iteration update through weighted summation; The memory state outputted by the last iteration is used as the historical color state of the historical coloring frame to participate in subsequent feature correction.

5. The interactive video coloring method according to claim 1, characterized in that: The step of evaluating the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame by the correction unit includes: The correction unit obtains the historical color state of the historical coloring frame from the memory unit, and evaluates the difference between the historical color state of the historical coloring frame and the feature map of the current grayscale frame to obtain the difference between the current feature and the historical state, that is, the difference is the offset caused by time.

6. The interactive video coloring method according to claim 5, characterized in that: The step of performing a correction process on the first characteristic map to be corrected according to the difference to obtain a corrected second characteristic map includes: Based on the attention mechanism, the offset correction tensor is calculated through the channel scaling weight and the spatial scaling weight; The first feature map is corrected based on the offset correction tensor to obtain a corrected second feature map.

7. The interactive video coloring method according to claim 1, characterized in that: Also includes: The interactive video colorization model is pre-trained based on preset optimization objectives and loss functions to obtain a pre-trained interactive colorization model.

8. An interactive video colorization system, characterized in that: include: An acquisition unit, which acquires a target grayscale video clip to be colored and a color point and a mask input by a user; wherein the target grayscale video clip includes a plurality of continuous grayscale frames; Inputting the current grayscale frame, the color point and the mask into a pre-trained interactive colorization model; Wherein, the interactive coloring model includes a backbone network unit, a memory unit and a correction unit; A backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color point and the mask; A memory unit for storing a historical color state of a historical coloring frame; the historical coloring frame is a frame before the current grayscale frame; A correction unit, which evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, and performs a correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map; Reconstructing the color channel of the current grayscale frame using the modified second feature map to generate a colored frame, wherein the memory unit stores the color state of the current frame after the coloring is completed; The above process is repeated to colorize the target grayscale video clip frame by frame to generate a color video.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images