Interactive Video Coloring Method, System, Device and Medium

Abstract

The present disclosure provides an interactive video coloring method, system, device and medium. The interactive video coloring method adjusts the feature map of the image coloring network before reconstructing the color channel to achieve the purpose of improving the video coloring quality. During the coloring process, no changes are made to the structure of the image coloring network, and only corresponding adjustments are made to one layer of the feature map. A memory unit and a correction unit are used to record the semantic and color information of the historical coloring frames respectively, and the intermediate structure of the current frame is modified using the coloring result of the latest frame, thereby significantly improving the visual quality of the output video.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video processing, and in particular, to an interactive video coloring method, system, device, and medium. Background Art

[0002] In recent years, with the development of hardware devices and 5G networks, short-video-based social applications have emerged in large numbers. Traditional social software and e-commerce software have also had to make adjustments and have added dedicated pages for displaying short videos. The emergence of such new things has profoundly changed people's production and life. Software applications that are limited to using text and images as information carriers are gradually declining and cannot fully meet the social needs of the vast number of users. At the same time, ordinary users, especially the younger generation, are increasingly inclined to display personalized life segments. However, the video software on the market can only provide limited video editing functions such as color grading or deformation, such as beauty effects. These fixed functions greatly limit the flexibility of video processing and also reduce the user's imagination space for customized video processing. Further, for the film and television industry, editing recorded videos requires a large amount of manpower and material resources, and often requires professional personnel to perform complex human-computer interactions to obtain the desired results. Therefore, developing a simple and highly flexible video editing method can bring great convenience to both ordinary users and those engaged in the film and television industry.

[0003] Currently, many interactive image coloring methods have achieved good results, such as methods based on color points and methods that support multi-modal input. However, there is little research in the direction of interactive video coloring. If the image coloring model is directly applied to video coloring without modification, the color reconstruction effect will be greatly reduced. The reason is that videos have an additional time dimension compared to image data. The final output result not only needs to have excellent performance on a single video frame but also needs to ensure the coherence between video frames. Since users cannot tolerate the interaction burden of inputting before coloring each frame, the migration of user guidance information over time also hinders the development of interactive video coloring. One migration method is to provide a path for the flow of interactive information between frames through optical flow. Unfortunately, all current optical flow estimation methods cannot obtain accurate results, and inaccurate optical flow leads to error accumulation as the video duration increases. At this time, the colored strokes input by the user will inevitably be mapped to the offset position. Therefore, the video coloring model also needs to adapt to the situation where the position of the guidance information is inaccurate. Summary of the Invention

[0004] The present disclosure provides a method, system, device, and storage medium for an interactive video coloring method.

[0005] According to one aspect of the present disclosure, an interactive video coloring method is provided, including:

[0006] Obtain a target grayscale video clip to be colored, as well as color points and masks input by the user; wherein, the target grayscale video clip includes a number of consecutive grayscale frames;

[0007] Input the current grayscale frame, the color points, and the masks into a pre-trained interactive coloring model;

[0008] Wherein, the interactive coloring model includes a backbone network unit, a memory unit, and a correction unit;

[0009] The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color points, and the masks;

[0010] Obtain the historical color state of the historical colored frame from the memory unit; the historical colored frame is the previous frame of the current grayscale frame;

[0011] Evaluate the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame through the correction unit, and perform bias correction on the first feature map to be corrected according to the difference to obtain a corrected second feature map;

[0012] Reconstruct the color channels of the current grayscale frame using the corrected second feature map to generate a colored frame, and the memory unit stores the color state of the current frame after coloring is completed;

[0013] Repeat the above steps to color the target grayscale video clip frame by frame to generate a color video.

[0014] In the method according to some embodiments of the present disclosure, the backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color points, and the masks, including:

[0015] Use a pre-trained backbone network unit model based on grayscale images to combine the grayscale frame, the color points, and the masks to generate a first feature map to be corrected.

[0016] In the method according to some embodiments of the present disclosure, the backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color points, and the masks, including:

[0017] Use a backbone network unit model based on U-Net, and the backbone network unit model includes ten convolutional modules; wherein, the first to fourth convolutional modules increase the number of channels and reduce the size of the feature map through downsampling, the fifth to sixth convolutional modules are dilated convolutions, the seventh to tenth convolutional modules reduce the number of channels and restore the size of the feature map through upsampling, and the first to fourth convolutional modules are connected to the sixth to tenth convolutional modules through skip connections,

[0018] Input the splicing combination of the grayscale frame, the color points, and the mask into the backbone network unit model to generate the first feature map to be corrected.

[0019] In the method according to some embodiments of the present disclosure, the method for obtaining the historical color state of the historical coloring frame includes:

[0020] Perform rectangular partitioning on each frame of image, calculate the visual encoding and add the positional encoding, and input it into the vision encoder based on Transformer to obtain the visual embedding tensor;

[0021] Iteratively update the memory unit with the visual embedding tensor successively, and generate a candidate memory unit based on the current memory state and the visual embedding tensor;

[0022] Obtain a gating vector for controlling the update degree of each memory unit through the current memory state and the candidate memory unit, and obtain the memory unit after this iterative update through weighted summation;

[0023] Take the memory state output by the last iteration as the historical color state of the historical coloring frame to participate in subsequent feature correction.

[0024] In the method according to some embodiments of the present disclosure, the method for the correction unit to evaluate the difference between the historical color state of the historical coloring frame and the feature map of the current grayscale frame includes:

[0025] The correction unit obtains the historical color state of the historical coloring frame from the memory unit, and evaluates the difference between the historical color state of the historical coloring frame and the feature map of the current grayscale frame to obtain the difference between the current feature and the historical state, that is, this difference is the offset caused by time.

[0026] In the method according to some embodiments of the present disclosure, the method for performing deviation correction on the first feature map to be corrected according to the difference to obtain the corrected second feature map includes:

[0027] Based on the attention mechanism, calculate the offset correction tensor through the channel scaling weight and the spatial scaling weight;

[0028] Correct the first feature map based on the offset correction tensor to obtain the corrected second feature map.

[0029] In the method according to some embodiments of the present disclosure, it further includes:

[0030] Pre-train the interactive video coloring model based on a preset optimization objective and loss function to obtain a pre-trained interactive coloring model.

[0031] According to another aspect of the present disclosure, an interactive video coloring system is provided, including:

[0032] An acquisition unit that acquires a target grayscale video clip to be colored, as well as color points and masks input by a user; wherein, the target grayscale video clip includes a plurality of consecutive grayscale frames;

[0033] Input the current grayscale frame, the color points, and the masks into a pre-trained interactive coloring model;

[0034] Wherein, the interactive coloring model includes a backbone network unit, a memory unit, and a correction unit;

[0035] The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color points, and the masks;

[0036] The memory unit stores the historical color states of historical colored frames; the historical colored frame is the previous frame of the current grayscale frame;

[0037] The correction unit evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, and performs a deviation correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map;

[0038] Reconstruct the color channels of the current grayscale frame by using the corrected second feature map to generate a colored frame, and the memory unit stores the color state of the current frame after coloring is completed;

[0039] Repeat the above process to color the target grayscale video clip frame by frame to generate a color video.

[0040] According to another aspect of the present disclosure, a computer device is provided, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the above-described method is implemented.

[0041] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-described method are implemented.

[0042] The present disclosure adjusts the feature map before reconstructing the color channels of the image coloring network to achieve the purpose of improving the video coloring quality. During the coloring process, no changes are made to the structure of the image coloring network, and only corresponding adjustments are made to one layer of the feature map. The memory unit and the correction unit are used to record the semantic and color information of the historical colored frames respectively, and the coloring result of the latest frame is used to modify the intermediate structure of the current frame, thereby significantly improving the visual quality of the output video.

[0043] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings are used to better understand the present solution and do not constitute a limitation to the present disclosure. Among them:

[0045] Figure 1 is a schematic flowchart according to an exemplary embodiment of the present disclosure;

[0046] Figure 2 is a schematic overview diagram of the framework according to an exemplary embodiment of the present disclosure;

[0047] Figure 3 is a schematic diagram of the generation process of user input according to an exemplary embodiment of the present disclosure;

[0048] Figure 4 is a schematic diagram of a memory unit according to an exemplary embodiment of the present disclosure;

[0049] Figure 5 is a schematic diagram of a correction unit according to an exemplary embodiment of the present disclosure;

[0050] Figure 6 is a block diagram of an interactive video coloring system according to an exemplary embodiment of the present disclosure;

[0051] Figure 7 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] The following describes exemplary embodiments of the present disclosure in conjunction with the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0053] To more clearly illustrate the technical solutions of the embodiments of this specification, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some examples or embodiments of this specification. For those of ordinary skill in the art, without creative efforts, this specification can also be applied to other similar scenarios based on these drawings. Unless obvious from the language context or otherwise stated, the same reference numerals in the figures represent the same structure or operation.

[0054] Reference to "embodiment" in the present application means that the specific features, structures or characteristics described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears at various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application may be combined with other embodiments.

[0055] Existing methods for solving the problem of video coloring instability can be summarized into three aspects: network structure, loss function, and post-processing. First, some methods cascade multiple networks using optical flow or feature matching in the latent space to align the features of adjacent frames. Other methods improve the network structure by proposing multiple sub-networks that extract features with learning priors at different scales. Second, in addition to many loss functions directly applied to video coloring such as reconstruction loss, perceptual loss, and segmentation loss in image processing, new loss functions such as short-term loss and long-term loss have been proposed and widely applied. These loss functions can improve the color similarity of adjacent frames by matching the pixel-level non-occluded regions of the warped frames and the current output, and model each long-range connection between two generated frames to enforce the temporal consistency of the correct warped regions. Finally, researchers have proposed methods for refining colored video frames through post-processing, including spatio-temporal smoothing and two-stage processing combined with super-resolution models.

[0056] However, most of these methods rely too much on optical flow data, and existing optical flow estimation algorithms based on color videos cannot provide accurate estimates and their performance on grayscale videos will be greatly reduced. In addition, post-processing methods cannot fundamentally solve the problem of temporal consistency. Based on this, as Figure 1 and Figure 2 shown, this embodiment provides an interactive video coloring method, including:

[0057] S101. Obtain a target grayscale video clip to be colored and color points and masks input by the user; wherein, the target grayscale video clip includes a plurality of consecutive grayscale frames;

[0058] In one implementation, obtaining sufficient video data for training and diverse user input data will cost a huge amount of manpower and material resources. Therefore, artificial synthetic data can be used to simulate the interactive input of users, which is also a common method in current interactive coloring literature. The generation process is as Figure 3As shown (since it is necessary to distinguish between grayscale frames and original frames, color images are used). First, an optical flow estimation method is used to calculate the optical flow between two adjacent frames of a grayscale video clip. Subsequently, a pixel point is selected at an arbitrary position in the first frame of the ground truth video clip to simulate the user's color strokes, and the mask of the pixel point is saved. The number of pixel points used during training is evenly distributed in the interval [0, 50] to cover real user interactions in a wide range of situations as much as possible. Finally, the pre-computed optical flow is used to map the simulated user input and its mask on the first frame to all frames of the entire grayscale video clip.

[0059] Some definitions:

[0060] A grayscale frame refers to an image frame that contains only luminance information. It consists of different levels of gray brightness and has no color information. Grayscale frames are usually the black-and-white versions of videos, and the luminance values typically vary from 0 (black) to 255 (white). During the video coloring process, grayscale frames are the basis of the original video, and color information needs to be added to each frame through algorithms. Each frame in the video can be regarded as a separate grayscale image.

[0061] Color points of user input: Color points of user input refer to the color information manually marked by the user on the grayscale frame. These points contain the colors specified by the user (such as red, blue, green, etc.), and are usually applied to a specific object or area. Through selective marking, the user can provide initial color clues for the grayscale frame. These color points provide necessary hints for the automatic coloring algorithm, and the algorithm will decide how to color the same or similar areas in other frames based on these points.

[0062] Mask: It is used to calibrate the position of user input. 1 indicates that there is a color point of user input at this pixel position, and 0 indicates that there is no user input at this pixel position.

[0063] S102. Input the current grayscale frame, color points, and mask into a pre-trained interactive coloring model; among them, the interactive coloring model includes a backbone network unit, a memory unit, and a correction unit;

[0064] S103. The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, color points, and mask;

[0065] Extract the grayscale image information of the current frame from the video clip as the basic input of the model. The user selects specific pixel points through interactive input, specifying the colors that these points should present when generating a color image. The mask corresponding to the color points is used to identify which areas should apply the colors of user input, thus ensuring accuracy in the subsequent generation of feature maps.

[0066] The backbone network unit (such as a U-Net-based structure) processes the current grayscale frame through multiple convolutional layers to extract its features. These convolutional layers process the input information through activation functions (such as the ReLU function) and normalization layers to enhance the network's expressive power.

[0067] After a series of convolutions and downsamplings, the backbone network will generate a first feature map to be corrected. This feature map will be further adjusted in the subsequent correction module to achieve higher-quality color image generation. At the same time, the color points and their mask information input by the user are incorporated into the generation process of the feature map to ensure that the colors specified by the user can be correctly reflected in the output.

[0068] Through the above steps, the backbone network unit ensures that it can effectively utilize the current grayscale frame and the user's interactive input to generate a feature map with preliminary color information, providing a basis for subsequent color channel reconstruction and correction.

[0069] S104. Obtain the historical color state of the historical colored frame from the memory unit; the historical colored frame is the previous frame of the current grayscale frame;

[0070] By obtaining the historical color state from the memory unit and making reference adjustments to the current grayscale frame, the model can effectively reduce the color discontinuity between video frames, especially in the case of rapid scene changes or significant color variations, ensuring the overall consistency of video colors and the smoothness of the visual experience.

[0071] S105. Evaluate the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame through the correction unit, and perform bias correction on the first feature map to be corrected according to the difference to obtain a corrected second feature map;

[0072] The difference evaluation mechanism of the correction unit in the bias correction process ensures the effective alignment between the historical color state and the current frame feature map. In this way, the colors of each frame of the video are more continuous in time sequence, thereby reducing color flickering or jumping phenomena and greatly enhancing the visual coherence of the video.

[0073] S106. Use the corrected second feature map to reconstruct the color channels of the current grayscale frame to generate a colored frame, and the memory unit stores the color state of the current frame after coloring;

[0074] S107. Repeat the above steps to color the target grayscale video clip frame by frame to generate a color video. In one example, for a given N-frame target grayscale video clip with a resolution of H×W T frame target grayscale video clip Each frame is represented as the luminance channel in the Lab color space, and the goal of the network model is to generate a high-quality color video segment frame by frame in the Lab color space. Among them, the backbone network is a simple grayscale image coloring model, which is pre-trained by an image coloring dataset and receives grayscale frames and color points input by the user and their masks This network is separated by the intermediate feature tensor φ in the figure t into front and back parts. Due to the defects of the current coloring algorithm, color inconsistency problems may occur during the coloring process, including the instability of the interactive image coloring model itself and the instability brought by the time dimension of video clips. These problems are ultimately manifested as the discontinuity of the output results in terms of color, and this factor has been established during the network processing, that is, the feature tensors of each layer of the network have incorrect offsets. Therefore, this embodiment proposes to correct the feature map before reconstructing the color channel

[0075] The adjustment of the feature map φ t is completed by the correction unit This correction unit makes full use of historical coloring information and the semantic features of the current frame, evaluates the feature differences in all aspects of the image spatial dimension and channel dimension, and finally outputs the corrected feature tensor Continue to use to realize the colorization of the current frame

[0076] The coloring state of the historical frame is stored by the memory unit where N M is the size of the memory unit, and each vector in it compresses and stores the state information of the historical coloring frames. This information is used by the correction unit for the correction of the feature map. The update of the memory unit uses the current memory state and the coloring output of the latest frame and is implemented by a write controller composed of a memory sub-network to perform the update operation, which is directly read by the correction unit when used, and the memory unit is updated after each frame of coloring is completed

[0077] In the method according to some embodiments of the present disclosure, the backbone network unit generates a first feature map to be corrected based on the current grayscale frame, color points, and mask, including:

[0078] Using a pre-trained backbone network unit model based on grayscale images, combine the grayscale frame, color points, and mask to generate a first feature map to be corrected

[0079] In the method according to some embodiments of the present disclosure, the backbone network unit generates a first feature map to be corrected based on the current grayscale frame, color points, and mask, including:

[0080] Use a backbone network unit model based on U-Net, and the backbone network unit model includes ten convolutional modules; among them, the first to fourth convolutional modules increase the number of channels and reduce the feature map size through downsampling, the fifth to sixth convolutional modules are dilated convolutions, and the seventh to tenth convolutional modules reduce the number of channels and restore the feature map size through upsampling. The first to fourth convolutional modules are connected to the sixth to tenth convolutional modules through skip connections.

[0081] Concatenate the grayscale frame, color points, and mask and input them into the backbone network unit model to generate a first feature map to be corrected.

[0082] Specifically, the backbone network unit model uses a U-Net structure. The backbone network has a total of 10 convolutional modules. The first 4 convolutional blocks are combined with the ReLu activation function, the spatial size of the feature map is halved, while the number of channels is doubled, and a normalization layer is added at the end. Next, the 5th and 6th convolutional blocks maintain the size of the feature map unchanged. In order to expand the perception range of the convolution and capture more comprehensive image features, dilated convolution with a stride of 2 is adopted. Compared with the previous modules, the 7th to 10th modules gradually increase the spatial size and halve the number of channels to restore the original spatial dimension of the image. The model connects the symmetric first 4 and last 4 convolutional modules through skip connection technology. In this way, in the image reconstruction stage, the model can not only utilize high-level features, but also effectively utilize low-level information such as image edges, promote the restoration of color channels, and prevent problems such as color overflow. The size change is achieved through downsampling and upsampling techniques, and each convolutional layer uses a convolutional kernel with a stride set to 1.

[0083] The grayscale video frame, the color points input by the user, and their masks are concatenated without any processing and directly input into the backbone network to obtain a feature map to be corrected. Usually, the 10th convolutional block is regarded as the color channel reconstruction stage. Therefore, this embodiment proposes to correct the image feature map before color channel reconstruction. Since the upsampled feature map output by the 9th convolutional block will have a short connection with the first layer, the correction unit adjusts the feature map after this short connection.

[0084] In the method according to some embodiments of the present disclosure, the way to obtain the historical color state of the historical colored frame includes:

[0085] Divide each frame of the image into rectangular blocks, calculate the visual encoding and add the positional encoding, and input it into the vision encoder based on Transformer to obtain the visual embedding tensor;

[0086] Iteratively update the memory unit with the visual embedding tensor successively, and generate a candidate memory unit based on the current memory state and the visual embedding tensor;

[0087] Based on the current memory state and candidate memory units, a gating vector for controlling the update degree of each memory unit is obtained, and the memory unit after this iteration update is obtained through weighted summation;

[0088] The memory state output in the last iteration is used as the historical color state of the historical coloring frame and participates in subsequent feature correction.

[0089] In a specific example, the memory unit M t and its update process are as Figure 4 shown. The processing process of the write controller can be divided into two stages: frame encoding and update. The first stage is a Transformer-based visual encoder. First, the image is divided into rectangular blocks, and then the visual encoding (Embedding) is calculated and the position encoding is added. Subsequently, it is input into a 6-layer Transformer structure to obtain the visual embedding tensor N represents the number of embedding tensors.

[0090] Then, in the second stage, each encoding in the visual embedding will participate in the update iteration process of the memory unit one by one, and the number of iterations is N. In each iteration, the current memory state and together generate the candidate memory unit In the first iteration, the memory state M of the previous frame t-1 is used to initialize each candidate memory unit The calculation process can be given by the following formula:

[0091]

[0092] where, [·] p represents the concatenation operation on variable p, AP is average pooling, represents the learnable parameters in the multi-layer perceptron. Immediately afterwards, the current memory state and the candidate memory unit jointly participate in the calculation to obtain a gating vector for controlling the update degree of each memory unit. Its calculation process can be defined by the following formula:

[0093]

[0094] where, W g represents the learnable parameters, and the Sigmoid function is used to map the calculation result to the interval [0, ightarrow 1]. At this time, the memory unit after this iteration update can be obtained through weighted summation The calculation result can be obtained by the following formula:

[0095]

[0096] After N iterations, the memory unit will be updated by each visual embedding tensor, and the output of the last iteration will be used as the latest colored memory state M t and participate in the subsequent feature correction stage.

[0097] In the method according to some embodiments of the present disclosure, the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame is evaluated by a correction unit, including:

[0098] The correction unit obtains the historical color state of the historical colored frame from the memory unit, and evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, obtaining the difference between the current feature and the historical state, that is, this difference is the offset caused by time.

[0099] The correction unit is used to estimate the difference between the features of the current grayscale frame during the coloring process and the historical coloring state, and solve the instability problem caused by the video time dimension according to these differences. The specific operation process is as Figure 5 shown. First, in order to take into account the relationship between adjacent time frames and the global coloring state, the memory unit is deformed and upsampled to the image space size and spliced together with the coloring result of the previous moment and mapped to the current frame using the optical flow displacement. Subsequently, a lightweight convolutional network with a convolutional kernel size of 5×5 and 3 layers is passed through to obtain the color semantic feature map of the previous moment The current grayscale frame passes through the front part of the backbone network and the feature map before the reconstruction of the color channel is output φ t and to get the difference between the current coloring state and the historical memory, which is the offset caused by time Next, the model will evaluate whether this offset is reasonable and decide whether to retain or weaken it.

[0100] The addition of the correction unit enables the image coloring algorithm to be easily applied to the video coloring task. Different from the existing methods, the model of this embodiment does not modify the image coloring network at all, nor does it need to retrain this part of the network. Only the memory network and the correction unit need to be trained, ensuring the single-frame coloring effect while enhancing the temporal continuity of the entire video clip coloring.

[0101] In the method according to some embodiments of the present disclosure, the first feature map to be corrected is corrected according to the difference to obtain the corrected second feature map, including:

[0102] Based on the attention mechanism, an offset correction tensor is calculated through channel scaling weights and spatial scaling weights;

[0103] The first feature map is corrected based on the offset correction tensor to obtain a corrected second feature map.

[0104] In one example, inspired by the attention mechanism, and φ t Calculate the maximum pooling and average pooling respectively and splice them together. Here it is global pooling, and the pooling kernel size is equal to the spatial size of the feature map. Then it is fed into a multi-layer perceptron, and after passing through the activation function, the channel scaling weights of are obtained, that is, the channel retention coefficients of the differences. At the same time, and are spliced and fed into a lightweight spatial encoder composed of convolutional layers. The number of convolutional layers is 2, and the convolutional kernel size is 3×3. After each layer of convolution, batch normalization is performed and the ReLu function is used for activation. Then, maximum pooling and average pooling are performed on the channel dimension. Finally, after 2 layers of 3×3 convolution and the Sigmoid activation function, the spatial scaling weights of are obtained. The channel scaling weights and spatial scaling weights act on the offset respectively to obtain an offset correction tensor. This tensor is used to correct φ t and finally a corrected feature map is obtained The latter part of the backbone network continues to use to complete the coloring of the current frame.

[0105] In the method according to some embodiments of the present disclosure, it further includes:

[0106] Based on a preset optimization objective and loss function, the interactive video coloring model is pre-trained to obtain a pre-trained interactive coloring model.

[0107] The optimization objective of the interactive video coloring model in this embodiment can be given by the following formula:

[0108]

[0109] where, and respectively represent the learnable parameters of the correction module and the memory sub-network , is the video data set used for training, is the loss function, is the coloring output of the previous frame, represents the ground truth of the target grayscale frame.

[0110] The backbone network is an interactive image coloring model. First, the necessary reconstruction loss is added. To fix the local color points input by the user at specific positions, the color point loss is used. For constraints, and to facilitate the diffusion of color points and maintain the smoothness of the output results, the gradient loss is also used. For supervision, the training loss of the backbone network can be obtained by summing over each spatial pixel position p:

[0111]

[0112] where Sobel represents the Sobel gradient operator. Although the backbone network is trained using an image dataset, in this paper, these image data are still regarded as a single video clip with 1 frame number, so t is always equal to 1.

[0113] In the second stage, due to the addition of the memory network and the correction module, the video dataset needs to be used for training. The memory unit records the global and local coloring states, and the coloring result of the previous frame will also be mapped to the current frame through optical flow in the correction module. This enables the model in this paper to have the ability to process adjacent frames and long time spans. Therefore, for the training of this stage, only the temporal loss between the previous and the next frames needs to be added on the basis of which is defined as follows:

[0114]

[0115] where O t-1→t represents the optical flow between two adjacent frames, Warp is the optical flow mapping that maps the coloring result of the previous moment to the current moment through optical flow, and M t-1→t represents the mask of the non-occluded area, and its calculation is given by the following formula:

[0116]

[0117] where α is set to 50.

[0118] This embodiment proposes a video coloring model based on the main network of interactive image coloring, which enhances the quality of video colorization by adjusting the intermediate features of the network.

[0119] The state information of the coloring of historical frames is memorized by the memory unit, and the memory unit is updated using the sub-network based on Transformer to provide guidance for the coloring of subsequent frames.

[0120] Through the correction unit, the semantic features of the current grayscale frame are adjusted by using the recorded historical state and the coloring result of the latest frame, so as to achieve the purpose of color consistency of the output video.

[0121] The interactive video coloring method provided in this embodiment can be executed in a smart terminal, a computer terminal, a network device, a chip, a chip module, or a similar computing device.

[0122] Corresponding to the interactive video coloring method introduced above, this embodiment also provides an interactive video coloring system. The following will be introduced separately. Specifically, as Figure 6 shown, the interactive video coloring system of this embodiment includes:

[0123] An acquisition unit 1 that acquires a target grayscale video clip to be colored, as well as color points and masks input by the user; among them, the target grayscale video clip includes a number of consecutive grayscale frames;

[0124] Input the current grayscale frame, color points, and masks into a pre-trained interactive coloring model; among them, the interactive coloring model includes a backbone network unit 2, a memory unit 3, and a correction unit 4;

[0125] The backbone network unit 2 generates a first feature map to be corrected based on the current grayscale frame, color points, and masks;

[0126] The memory unit 3 stores the historical color states of historical colored frames; the historical colored frame is the previous frame of the current grayscale frame;

[0127] The correction unit 4 evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, and performs a deviation correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map;

[0128] Use the corrected second feature map to reconstruct the color channels of the current grayscale frame to generate a colored frame after coloring, and the memory unit 3 stores the color state of the current frame after coloring is completed;

[0129] Repeat the above process to color the target grayscale video clip frame by frame to generate a color video.

[0130] It should be noted that the interactive video coloring system embodiment provided in this application is based on the same inventive concept as the above-mentioned interactive video coloring method embodiment.

[0131] It should be noted that the interactive video coloring system in this embodiment can be, for example: a separate chip, a chip module, or an electronic device, or it can also be a chip or a chip module integrated within an electronic device. Regarding the various devices and products described in the above embodiments and the various modules / units they contain, they can be software modules / units, hardware modules / units, or they can also be partly software modules / units and partly hardware modules / units. For example, for the various devices and products applied to or integrated into a chip, the various modules / units they contain can all be implemented in the form of hardware such as circuits. Or, at least some of the modules / units can be implemented in the form of software programs that run on the processor integrated within the chip, and the remaining (if any) part of the modules / units can be implemented in the form of hardware such as circuits. For the various devices and products applied to or integrated into a chip module, the various modules / units they contain can all be implemented in the form of hardware such as circuits. Different modules / units can be located in the same component (such as a chip, a circuit module, etc.) or different components of the chip module. Or, at least some of the modules / units can be implemented in the form of software programs that run on the processor integrated within the chip module, and the remaining (if any) part of the modules / units can be implemented in the form of hardware such as circuits. For the various devices and products applied to or integrated into a terminal, the various modules / units they contain can all be implemented in the form of hardware such as circuits. Different modules / units can be located in the same component (such as a chip, a circuit module, etc.) or different components within the terminal. Or, at least some of the modules / units can be implemented in the form of software programs that run on the processor integrated within the terminal, and the remaining (if any) part of the modules / units can be implemented in the form of hardware such as circuits.

[0132] FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the interactive video coloring method. Figure 7 The illustrated electronic device 30 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0133] As Figure 7 shown, the electronic device 30 can be presented in the form of a general computing device. For example, it can be a server device. The components of the electronic device 30 can include, but are not limited to: the at least one processor 31 described above, the at least one memory 32 described above, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0134] The bus 33 includes a data bus, an address bus, and a control bus.

[0135] The memory 32 may include volatile memory such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0136] The memory 32 may also include a program / utilities 325 having a set (at least one) of program modules 324. Such program modules 324 include but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0137] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the interactive video coloring method of Embodiment 1 of the present invention.

[0138] The electronic device 30 may also communicate with one or more external devices 34 (such as a keyboard, a pointing device, etc.). Such communication may be performed through an input / output (I / O) interface 35. Moreover, the model generation device 30 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 36. As Figure 7 shown, the network adapter 36 communicates with other modules of the model generation device 30 through the bus 33. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in combination with the model generation device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (redundant array of independent disks) systems, tape drives, and data backup storage systems, etc.

[0139] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present invention, the features and functions of two or more of the above-mentioned units / modules may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided and embodied by multiple units / modules.

[0140] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps in the interactive video coloring method are implemented.

[0141] Among them, the more specific forms that the readable storage medium may adopt may include but are not limited to: portable disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories, optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0142] In a possible implementation, the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps in the interactive video coloring method.

[0143] Among them, the program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, executed as an independent software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0144] It should be understood that various forms of processes shown above can be used, steps can be reordered, added, or deleted. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present disclosure can be achieved. This embodiment does not limit this here.

[0145] It should be understood that the "system", "device", "unit", and / or "module" used in this embodiment is a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0146] In this embodiment, a flowchart is used to illustrate the operations performed by the system according to the embodiments of this embodiment. It should be understood that the previous or subsequent operations do not necessarily need to be executed precisely in order. On the contrary, they can be executed in reverse order or each step can be processed simultaneously. At the same time, other operations can also be added to these processes, or one or several steps of operations can be removed from these processes.

[0147] As shown in this embodiment, unless the context clearly indicates an exceptional situation, words such as "a", "one", "a kind of", and / or "the" do not specifically refer to the singular and can also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0148] The definition of "include" in this embodiment, as the terms "have", "may have", "include", or "may include" used here indicate the existence of the corresponding functions, operations, elements, etc. of this embodiment, and do not limit the existence of one or more other functions, operations, elements, etc. In addition, it should be understood that as the terms "include" or "have" used here indicate the existence of the features, numbers, steps, operations, elements, parts, or combinations thereof described in the specification, and do not exclude the existence or addition of one or more other features, numbers, steps, operations, elements, parts, or combinations thereof.

[0149] In this embodiment, the definition of "and / or": As used herein, the terms "A or B", "at least one of A and / or B", or "one or more of A and / or B" include any and all combinations of the words listed therewith. For example, "A or B", "at least one of A and / or B", or "one or more of A and / or B" means (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0150] In this embodiment, the definition of "first" and "second": The descriptions such as "first" and "second" that appear in this embodiment are only for the purpose of illustration and distinguishing the described objects, without an order, and do not represent a special limitation on the number of devices in this embodiment, and cannot constitute any limitation to this embodiment. For example, the first element can be referred to as the second element without departing from the scope of the present disclosure. Similarly, the second element can be referred to as the first element.

[0151] Although the specific embodiments of the present invention have been described above, those skilled in the art should understand that this is only an example, and the protection scope of the present invention is defined by the appended claims. Without departing from the principles and essence of the present invention, those skilled in the art can make various changes or modifications to these embodiments, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. An interactive video coloring method, characterized in that, Including: Obtain a target grayscale video clip to be colored, as well as color points and masks input by the user; wherein, the target grayscale video clip includes a number of consecutive grayscale frames; Input the current grayscale frame, the color points, and the masks into a pre-trained interactive coloring model; Wherein, the interactive coloring model includes a backbone network unit, a memory unit, and a correction unit; The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color points, and the masks; Obtain the historical color state of the historical colored frame from the memory unit; the historical colored frame is the previous frame of the current grayscale frame; Evaluate the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame through the correction unit, and perform a correction process on the first feature map to be corrected according to the difference to obtain a corrected second feature map; Use the corrected second feature map to reconstruct the color channels of the current grayscale frame to generate a colored frame after coloring, and the memory unit stores the color state of the current frame after coloring is completed; Repeat the above steps to color the target grayscale video clip frame by frame to generate a color video.

2. The interactive video coloring method according to claim 1, characterized in that The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color points, and the masks, including: Use a pre-trained backbone network unit model based on grayscale images to combine the grayscale frame, the color points, and the masks to generate a first feature map to be corrected.

3. The interactive video coloring method according to claim 1 or 2, characterized in that, The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color points, and the masks, including: Use a backbone network unit model based on U-Net. The backbone network unit model contains ten convolutional modules; wherein, the first to fourth convolutional modules increase the number of channels and reduce the feature map size through downsampling, the fifth to sixth convolutional modules are dilated convolutions, the seventh to tenth convolutional modules reduce the number of channels and restore the feature map size through upsampling, and the first to fourth convolutional modules are connected to the sixth to tenth convolutional modules through skip connections, Input the combined grayscale frame, the color points, and the masks into the backbone network unit model to generate the first feature map to be corrected.

4. The interactive video coloring method according to claim 1, wherein The method for obtaining the historical color state of the historical colored frame includes: Perform rectangular partitioning on each frame of the image, calculate the visual encoding and add the position encoding, and input it into a vision encoder based on Transformer to obtain a visual embedding tensor; Iteratively update the memory unit with the visual embedding tensor successively, and generate a candidate memory unit based on the current memory state and the visual embedding tensor; Obtain a gating vector for controlling the update degree of each memory unit through the current memory state and the candidate memory unit, and obtain the memory unit after this iterative update through weighted summation; Use the memory state output by the last iteration as the historical color state of the historical colored frame to participate in subsequent feature correction.

5. The interactive video coloring method according to claim 1, characterized in that, The evaluation of the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame through the correction unit includes: The correction unit obtains the historical color state of the historical colored frame from the memory unit, and evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, obtaining the difference between the current feature and the historical state, that is, this difference is the offset caused by time.

6. The interactive video coloring method according to claim 5, wherein The correcting the first feature map to be corrected according to the difference to obtain a corrected second feature map includes: Based on the attention mechanism, calculating an offset correction tensor through a channel scaling weight and a spatial scaling weight; Correcting the first feature map based on the offset correction tensor to obtain a corrected second feature map.

7. The interactive video coloring method according to claim 1, characterized in that It further includes: Pre-training the interactive video coloring model based on a preset optimization objective and loss function to obtain a pre-trained interactive coloring model.

8. An interactive video coloring system, characterized in that, It includes: An acquisition unit that acquires a target grayscale video clip to be colored, as well as color points and masks input by a user; wherein, the target grayscale video clip includes a plurality of consecutive grayscale frames; Inputting the current grayscale frame, the color points and the mask into the pre-trained interactive coloring model; Wherein, the interactive coloring model includes a backbone network unit, a memory unit and a correction unit; The backbone network unit generates a first feature map to be corrected based on the current grayscale frame, the color points and the mask; The memory unit stores the historical color state of the historical colored frame; the historical colored frame is the previous frame of the current grayscale frame; The correction unit evaluates the difference between the historical color state of the historical colored frame and the feature map of the current grayscale frame, and corrects the first feature map to be corrected according to the difference to obtain a corrected second feature map; Reconstructing the color channels of the current grayscale frame by using the corrected second feature map to generate a colored frame, and the memory unit stores the color state of the current frame after coloring; Repeating the above process to color the target grayscale video clip frame by frame to generate a color video.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images