Video processing method and device, and storage medium

Abstract

The present disclosure relates to a video processing method, device and storage medium. The video processing method comprises: acquiring a first video, the first video being a video to be processed; determining a video colorization model corresponding to the first video based on a set of video colorization models, wherein the set of video colorization models comprises a plurality of video colorization models, different video colorization models being used for video colorization processing of different types of videos, the video colorization processing representing conversion of black-and-white image frames of a video into color image frames; and performing video colorization processing on the first video using the video colorization model corresponding to the first video to obtain a second video. The present disclosure can improve the efficiency and effect of colorization processing of a video.

Description

Technical Field

[0001] This disclosure relates to the field of video, and in particular to video processing methods, apparatus and storage media. Background Technology

[0002] Video colorization is a branch of computer vision whose main goal is to convert black-and-white video into color video. Since video is composed of image frames, current techniques that manually add color to each frame are inefficient. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a video processing method, apparatus and storage medium.

[0004] According to a first aspect of the present disclosure, a video processing method is provided, comprising: acquiring a first video, wherein the first video is a video to be processed; determining a video coloring model corresponding to the first video based on a set of video coloring models, wherein the set of video coloring models includes multiple video coloring models, and different video coloring models among the multiple video coloring models are used to perform video coloring processing on different types of videos, wherein the video coloring processing represents converting black and white image frames of a video into color image frames; and performing video coloring processing on the first video using the video coloring model corresponding to the first video to obtain a second video.

[0005] In one implementation, determining the video coloring model corresponding to the first video based on the set of video coloring models includes: acquiring a first image frame, wherein the first image frame is one or more image frames of the first video;

[0006] Obtain the training dataset corresponding to each video colorization model in the video colorization model set, the training dataset including training data frames; determine the similarity between the first image frame and the training dataset based on the similarity between the first image frame and each training data frame in the training dataset; determine the video colorization model corresponding to the first video based on the similarity between the first image frame and the training dataset.

[0007] In one implementation, determining the video colorization model corresponding to the first video based on the similarity between the first image frame and the training dataset includes: determining the video colorization model corresponding to the training dataset with a similarity greater than or equal to a similarity threshold as the video colorization model corresponding to the first video.

[0008] In one implementation, the video coloring model corresponding to the first video includes at least two video coloring models;

[0009] The step of performing video colorization processing on the first video using the video colorization model corresponding to the first video to obtain the second video includes: colorizing the first video based on each of the at least two video colorization models to obtain multiple third videos; and fusing the multiple third videos based on the similarity between the first image frame and the training dataset corresponding to each of the at least two video colorization models to obtain the second video.

[0010] In one embodiment, the first image frame is a keyframe; the step of performing video colorization processing on the first video using the video colorization model corresponding to the first video includes: performing video colorization processing on the first image frame using the video colorization model corresponding to the first video; performing video colorization processing on a second image frame using the colorized first image frame and a reference model, wherein the second image frame is an image frame in the first video other than the first image frame, and the reference model is used to perform video colorization processing on uncolorized image frames using the colorized image frames; and obtaining the second video based on the colorized first image frame and the colorized second image frame.

[0011] In one embodiment, the step of performing video colorization processing on the second image frame using the colorized first image frame and the reference model includes: using N consecutive image frames in the first video and the reference model, performing video colorization processing on the next second image frame adjacent to the N consecutive image frames, until the video colorization processing is completed for each second image frame, where N is a positive integer; the N image frames include any one or more of the following: the colorized first image frame; the colorized second image frame.

[0012] In one embodiment, the second video includes a third image frame and a fourth image frame, wherein the third image frame is one or more image frames in the second video, and the fourth image frame is an image frame in the second video other than the third image frame. The method further includes: performing video smoothing processing on the fourth image frame based on the third image frame to obtain a second video after video smoothing.

[0013] In one embodiment, the step of performing video smoothing processing on the fourth image frame based on the third image frame to obtain a smoothed second video includes: using the fourth image frame, performing image registration on M consecutive image frames adjacent to the fourth image frame, where M is a positive integer; determining a second weight corresponding to each image frame in the fourth image frame and the M consecutive image frames after image registration; performing image fusion on the fourth image frame and the M consecutive image frames after image registration based on each of the second weights to obtain a fused fourth image frame; until image fusion is completed on each of the fourth image frames in the second video to obtain a smoothed second video; wherein the M consecutive image frames in the second video include one or more of the following: the third image frame; the fused fourth image frame.

[0014] In one embodiment, the video colorization model set includes a first video colorization model and a second video colorization model; the first video colorization model is trained based on a first training dataset, and the second video colorization model is trained based on a second training dataset, wherein the second training dataset is determined based on the first training dataset.

[0015] In one implementation, the reference model is trained as follows: a third training dataset is acquired; the third training dataset includes multiple sets of adjacent image frames; image registration is performed on the (N+1)th image frame in the adjacent image frames using N image frames in a preset order; video desaturation is performed on the (N+1)th image frame, wherein the video desaturation refers to converting a color video into a black and white video; the first N image frames after image registration and the (N+1)th image frame after desaturation are input into an untrained reference model to obtain an output result; the untrained reference model is trained using the loss function between the output result and the (N+1)th image frame before desaturation.

[0016] According to a second aspect of the present disclosure, a video processing apparatus is provided, characterized in that the video processing apparatus comprises: an acquisition module, configured to acquire a first video, the first video being a video to be processed; a determination module, configured to determine a video coloring model corresponding to the first video based on a set of video coloring models, wherein the set of video coloring models includes multiple video coloring models, different video coloring models among the multiple video coloring models are used to perform video coloring processing on different types of videos, the video coloring processing representing the conversion of black and white image frames of a video into color image frames; and a processing module, configured to perform video coloring processing on the first video using the video coloring model corresponding to the first video to obtain a second video.

[0017] In one embodiment, the acquisition module is further configured to acquire a first image frame, wherein the first image frame is one or more image frames of the first video; acquire a training dataset corresponding to each video colorization model in the video colorization model set, wherein the training dataset includes training data frames; the processing module is further configured to determine the similarity between the first image frame and the training dataset based on the similarity between the first image frame and each training data frame in the training dataset; the determining module determines the video colorization model corresponding to the first video based on the video colorization model set in the following manner: determining the video colorization model corresponding to the first video based on the similarity between the first image frame and the training dataset.

[0018] In one implementation, the determining module determines the video colorization model corresponding to the first video based on the similarity in the following manner: determining the video colorization model corresponding to the training dataset with a similarity greater than or equal to a similarity threshold as the video colorization model corresponding to the first video.

[0019] In one implementation, the video coloring model corresponding to the first video includes at least two video coloring models;

[0020] The determining module determines the video coloring model corresponding to the first video based on the video coloring model set in the following manner: Based on the video coloring model set, at least two video coloring models corresponding to the first video are determined. The processing module performs video coloring processing on the first video using the video coloring model corresponding to the first video to obtain the second video in the following manner: Based on each of the at least two video coloring models, the first video is colorized to obtain multiple third videos; Based on the similarity between the first image frame and the training dataset corresponding to each of the at least two video coloring models, the multiple third videos are fused to obtain the second video.

[0021] In one implementation, the first image frame is a keyframe; the processing module performs video coloring processing on the first video using the video coloring model corresponding to the first video in the following manner: performing video coloring processing on the first image frame using the video coloring model corresponding to the first video; performing video coloring processing on the second image frame using the colorized first image frame and the reference model, wherein the second image frame is an image frame in the first video other than the first image frame, and the reference model is used to perform video coloring processing on the uncolorized image frame using the colorized image frame; and obtaining the second video based on the colorized first image frame and the colorized second image frame.

[0022] In one embodiment, the processing module performs video colorization processing on the second image frame using the first image frame after colorization and the reference model as follows: using N consecutive image frames in the first video and the reference model, the next second image frame adjacent to the N consecutive image frames is subjected to video colorization processing until the video colorization processing of each second image frame is completed, where N is a positive integer; the N image frames include any one or more of the following: the first image frame after colorization; the second image frame after colorization.

[0023] In one embodiment, the second video includes a third image frame and a fourth image frame, wherein the third image frame is one or more image frames in the second video, and the fourth image frame is an image frame in the second video other than the third image frame. The processing module is further configured to: perform video smoothing processing on the fourth image frame based on the third image frame to obtain a second video after video smoothing.

[0024] In one embodiment, the processing module performs video smoothing processing on the fourth image frame based on the third image frame as follows: using the fourth image frame, image registration is performed on the M consecutive image frames adjacent to the fourth image frame, where M is a positive integer; a second weight is determined for each image frame in the fourth image frame and the M consecutive image frames after image registration; based on each of the second weights, the fourth image frame and the M consecutive image frames after image registration are image fused to obtain a fourth image frame after image fusion; until image fusion is completed for each of the fourth image frames in the second video, a second video after video smoothing is obtained;

[0025] The M consecutive image frames in the second video include one or more of the following:

[0026] The third image frame; the fourth image frame after image fusion.

[0027] In one embodiment, the video colorization model set includes a first video colorization model and a second video colorization model; the first video colorization model is trained based on a first training dataset, and the second video colorization model is trained based on a second training dataset, wherein the second training dataset is determined based on the first training dataset.

[0028] In one embodiment, the apparatus further includes: a training module, configured to train the reference model in the following manner: acquiring a third training dataset; the third training dataset includes multiple sets of adjacent image frames; performing image registration on the (N+1)th image frame in the adjacent image frames according to a preset order using N image frames; performing video desaturation processing on the (N+1)th image frame, wherein the video desaturation processing refers to converting a color video into a black and white video; inputting the first N image frames after image registration and the (N+1)th image frame after desaturation into an untrained reference model to obtain an output result; and training the untrained reference model using a loss function between the output result and the (N+1)th image frame before desaturation.

[0029] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory for storing instructions; and a processor for invoking the instructions stored in the memory to execute a video processing method of the first aspect or any embodiment of the first aspect.

[0030] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions, which, when executed by a processor, perform a video processing method according to the first aspect or any embodiment of the first aspect.

[0031] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: by determining the video coloring model corresponding to the video to be processed in the video coloring model set, the video coloring model is used to colorize the video to be processed, so as to improve the efficiency and effect of colorizing the video.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0034] Figure 1 This is a flowchart illustrating a video processing method according to an exemplary embodiment.

[0035] Figure 2 This is a flowchart illustrating a method for determining a video colorization model according to an exemplary embodiment.

[0036] Figure 3 This is a flowchart illustrating another video processing method according to an exemplary embodiment.

[0037] Figure 4aThis is a schematic diagram illustrating a video processing method according to an exemplary embodiment.

[0038] Figure 4b This is a schematic diagram illustrating another video processing method according to an exemplary embodiment.

[0039] Figure 4c This is a schematic diagram illustrating another video processing method according to an exemplary embodiment.

[0040] Figure 5 This is a flowchart illustrating another video processing method according to an exemplary embodiment.

[0041] Figure 6 This is a schematic diagram illustrating another video processing method according to an exemplary embodiment.

[0042] Figure 7 This is a schematic diagram illustrating another video processing method according to an exemplary embodiment.

[0043] Figure 8 This is a flowchart illustrating another video processing method according to an exemplary embodiment.

[0044] Figure 9 This is a schematic diagram illustrating another video processing method according to an exemplary embodiment.

[0045] Figure 10 This is a schematic diagram illustrating another video processing method according to an exemplary embodiment.

[0046] Figure 11 This is a schematic diagram illustrating another video processing method according to an exemplary embodiment.

[0047] Figure 12 This is a flowchart illustrating another video processing method according to an exemplary embodiment.

[0048] Figure 13 This is a schematic diagram illustrating a reference model training method according to an exemplary embodiment.

[0049] Figure 14 This is a schematic diagram illustrating another video processing method according to an exemplary embodiment.

[0050] Figure 15 This is a block diagram illustrating a video processing apparatus according to an exemplary embodiment.

[0051] Figure 16 This is a block diagram illustrating another video processing apparatus according to an exemplary embodiment. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0053] Video colorization is a branch of computer vision whose main goal is to convert black and white video into color video. Since video is composed of image frames, current techniques that manually add color to each frame are inefficient.

[0054] Among related technologies, automated methods, based on rules and traditional image processing techniques, are used to achieve video colorization. The steps are as follows:

[0055] Image segmentation: dividing the image frames of the video to be processed into different regions, such as background, people, objects, etc.

[0056] Object tracking: For dynamic scenes, objects in image frames are tracked to maintain consistent color in subsequent processing.

[0057] Color matching: Match each area or object with the corresponding color based on manually selected rules or a color library.

[0058] Color fill: Based on the color matching results, fill each area or object with the corresponding color.

[0059] Post-processing: Post-processing is performed on the filled image, such as noise reduction.

[0060] However, the effectiveness of this video colorization method largely depends on manually specified rules and parameters, making it difficult to adapt to videos in different scenarios. Furthermore, this colorization technique is less efficient at processing complex scenes and large-scale videos.

[0061] In another related technology, deep learning models are used for video colorization. Deep learning models can automatically learn how to convert black and white videos to color videos without manually specifying colors or rules. Commonly used deep learning models include convolutional neural networks (CNNs), generative adversarial networks (GANs), and variational autoencoders (VAEs). The steps are as follows:

[0062] Data preparation: Prepare a large number of blank videos and corresponding color videos as training data.

[0063] Network Design: Design a neural network model suitable for video colorization tasks.

[0064] Model training: The neural network model is trained using training data to learn how to convert black and white video into color video.

[0065] Model testing: The trained neural network model is tested using test data to evaluate its performance and effectiveness.

[0066] Colorization: Input the black and white video to be colored into the trained neural network model and get a color video as the output.

[0067] However, this video colorization method still has many problems. For example, it requires a lot of time for data screening and labeling, and training on specific training datasets limits the model's generalization ability, potentially leading to unexpected errors when processing different videos. Furthermore, while deep learning-based video colorization techniques can learn complex features, the colorization results may not be natural enough in some details and textures, resulting in strange colors and imperfections, especially in video, where frame-to-frame fluctuations are common, making the results look very unnatural. Finally, since videos are composed of image frames, video colorization requires processing a large amount of image data, resulting in very high computational costs and necessitating the use of high-performance equipment to complete the task.

[0068] Therefore, this disclosure provides a video processing method that determines the video coloring model corresponding to the video to be processed from a set of video coloring models, and then uses the video coloring model to colorize the video to be processed, so as to improve the efficiency and effect of video coloring processing.

[0069] The video processing method is applied to the terminal, which can also be called a terminal device, mobile station (MS), mobile terminal (MT), etc. It is a device that provides voice and / or data connectivity to users. For example, a terminal can be a handheld device with wireless connectivity, an in-vehicle device, etc. Currently, examples of terminals include: smartphones, pocket personal computers (PPCs), handheld computers, personal digital assistants (PDAs), laptops, tablets, wearable devices, or in-vehicle devices. Furthermore, when it is a vehicle-to-everything (V2X) communication system, the terminal device can also be an in-vehicle device. It should be understood that the embodiments of this disclosure do not limit the specific technology or device form used in the terminal.

[0070] Figure 1 This is a flowchart illustrating a video processing method according to an exemplary embodiment, such as... Figure 1 As shown, the video processing method includes the following steps.

[0071] In step S11, the first video is acquired.

[0072] In one implementation, the terminal can acquire a first video, which is a video that has not undergone colorization processing. Video colorization processing refers to converting a black-and-white video into a color video.

[0073] In step S12, the video coloring model corresponding to the first video is determined based on the set of video coloring models.

[0074] In one implementation, a set of video colorization models can be pre-configured, which may include multiple video colorization models. Different video colorization models are used to perform video colorization processing on different videos.

[0075] For example, video colorization model A is used to colorize videos of common scenes. It can be understood that video colorization model A is more effective at colorizing videos of common scenes. Video colorization model B is used to colorize videos of uncommon scenes (scenes that are less frequently encountered). Video colorization model C is used to colorize both common and uncommon scenes. It can be understood that compared to video colorization model A, video colorization model C is less effective at colorizing videos of common scenes but more effective at colorizing videos of uncommon scenes; compared to video colorization model B, it is less effective at colorizing videos of uncommon scenes but more effective at colorizing videos of common scenes.

[0076] Of course, the video colorization models exemplified in this disclosure are merely exemplary examples, and video colorization models can include many other types. For example, video colorization models for food videos, video colorization models for natural landscape videos, etc., are not listed in this disclosure.

[0077] For example, videos of common scenarios may include any one or more of the following: people, animals, vehicles, furniture, plants, appliances, food, buildings, toys, tableware, stationery, etc. Videos of uncommon scenarios may include any one or more of the following: physical experimental equipment, artworks, natural landscapes, historical buildings, traditional clothing, sports equipment, musical instruments, electronic components, agricultural tools, scientific charts, etc.

[0078] It is understood that the common and uncommon scenarios in this disclosure are exemplary examples, and users can set common and uncommon scenarios according to actual circumstances. This disclosure does not limit them.

[0079] In this embodiment, the terminal can determine the video coloring model corresponding to the first video from a pre-configured set of video coloring models. It is understood that different video coloring models have different capabilities in colorizing videos. Therefore, the determined video coloring model corresponding to the first video...

[0080] In step S13, the first video is colored using the video coloring model corresponding to the first video to obtain the second video.

[0081] In one implementation, the terminal can input a first video into a video colorization model corresponding to the first video, perform video colorization processing on the first video, and obtain a second video. It can be understood that the second video is the first video after video colorization processing. Inputting the first video into the video colorization model means inputting each image frame of the first video into the video colorization model. The video colorization model sequentially performs video colorization processing on each image frame, obtaining various colorized image frames. It is clear that the various colorized image frames constitute the second video.

[0082] The pre-configured set of video colorization models in this disclosure features different capabilities for each model. Therefore, video colorization processing can be performed on different videos across multiple scenes, effectively improving the generalization ability of the video colorization method. Furthermore, using the video colorization model corresponding to the first video to be processed for targeted colorization improves the efficiency of video colorization, especially for videos with complex scenes or large-scale videos, significantly enhancing efficiency. Moreover, targeted video colorization improves the final colorization effect.

[0083] In some implementations, the video is composed of image frames, and the terminal can determine the video colorization model corresponding to the first video based on the image frames of the first video to be processed. Therefore, this disclosure provides a video processing method, such as... Figure 2 As shown, the video processing method includes the following steps.

[0084] In step S21, the first image frame is acquired.

[0085] In one implementation, the terminal can acquire one or more image frames of the video to be processed. This disclosure refers to the acquired one or more image frames of the first video as a first image frame. Here, an image frame refers to the smallest unit that makes up a video. It can be understood that a video is composed of a series of image frames. References used to describe video include frames per second (FPS), which refers to the number of image frames transmitted per second. For example, 24 FPS means 24 image frames are transmitted per second. It is clear that the more frames transmitted per second, the smoother the video playback and the better the user's viewing experience.

[0086] In step S22, the training dataset corresponding to the video colorization model is obtained.

[0087] In one implementation, the terminal can obtain the training dataset corresponding to each video colorization model in a pre-configured set of video colorization models. Alternatively, the terminal can selectively obtain the training dataset corresponding to a subset of the video colorization models in the set. For example, the terminal can select a subset of video colorization models from the pre-configured set based on factors such as the duration and complexity of the first video to be processed, thereby obtaining the training dataset corresponding to that subset. The training dataset corresponding to a video colorization model can be understood as the training dataset used to train that video colorization model. The training dataset includes a large number of video clips, each consisting of multiple image frames.

[0088] It is understood that there is no fixed execution order between steps S21 and S22. The terminal may acquire the first image frame first and then acquire the dataset corresponding to the video colorization model, or it may acquire the dataset corresponding to the video colorization model first and then acquire the first image frame. This disclosure does not limit this.

[0089] In step S23, the similarity between the first image frame and the training dataset is calculated.

[0090] In one implementation, the terminal can calculate the similarity between the first image frame and the training dataset. For example, the terminal can extract multiple image frames from the training dataset, calculate the similarity between each of the extracted image frames and the first image frame, and use the maximum value as the similarity between the first image frame and the training dataset. When extracting multiple image frames from the training dataset, the terminal can extract image frames for multiple video segments in the training dataset separately, rather than extracting multiple image frames for the same video segment in the training dataset, to make the final similarity result more accurate.

[0091] In one implementation, the terminal can calculate the similarity between two image frames in several ways. For example, the terminal can extract the feature vectors of the two image frames separately and calculate the distance between the vectors to obtain the similarity between the two image frames. Alternatively, the terminal can input the two image frames into a pre-configured deep learning model to obtain the similarity between the two image frames.

[0092] In step S24, the video colorization model corresponding to the first video is determined based on similarity.

[0093] In one implementation, the terminal can determine the video colorization model corresponding to the first video based on the similarity between the first image frame and the training dataset. A higher similarity between the first image frame and a given training dataset indicates a better performance of the video colorization model corresponding to that training dataset in colorizing the first video. For example, the terminal can match the first image frame with the training dataset corresponding to each video colorization model in a pre-configured set of video colorization models, obtaining multiple similarity scores. The terminal can then sort these similarity scores by numerical value, determine the training dataset corresponding to the highest similarity score, and identify the video colorization model corresponding to that training dataset as the video colorization model for the first video.

[0094] This disclosure determines the video coloring model corresponding to the first video by performing similarity matching between the image frames of the first video to be processed and the training dataset of the video coloring model. This accurately selects the video coloring model suitable for the video to be processed, thereby improving the efficiency and effect of video coloring the first video to be processed.

[0095] In some embodiments, this disclosure provides a video colorization method: in response to a similarity greater than or equal to a similarity threshold, the video colorization model corresponding to the training dataset corresponding to the similarity is determined as the video colorization model corresponding to the first video.

[0096] In one implementation, the terminal can select a video colorization model from a pre-configured set of video colorization models in a predetermined order. The terminal then obtains the training dataset corresponding to the selected video colorization model and calculates the similarity between the first image frame and the training dataset. If the similarity is greater than or equal to a similarity threshold, the obtained video colorization model is determined as the video colorization model corresponding to the first video.

[0097] Of course, the terminal may also randomly select a video coloring model from a pre-configured set of video coloring models, and this disclosure does not impose any limitations.

[0098] This announcement identifies the video colorization model corresponding to the training dataset with a similarity greater than or equal to a similarity threshold as the video colorization model corresponding to the first video, thus enabling rapid determination of the video colorization model corresponding to the first video.

[0099] In some implementations, the terminal can determine at least two video coloring models corresponding to the first video from a pre-configured set of video coloring models. Figure 3 This is a flowchart illustrating a video processing method according to an exemplary embodiment. Figure 3 As shown, the video processing method includes the following steps.

[0100] In step S31, based on the set of video colorization models, at least two video colorization models corresponding to the first video are determined.

[0101] In step S32, based on each of the at least two video coloring models, the first video is colorized to obtain the third video.

[0102] In step S33, based on the similarity between the first image frame and the training dataset corresponding to each of the at least two video colorization models, the first weight of the third video corresponding to each video colorization model is determined.

[0103] In step S34, the third video corresponding to each video colorization model is fused using the first weight of the third video corresponding to each video colorization model to obtain the second video.

[0104] In one implementation, the terminal can determine one or more video coloring models from a set of video coloring models.

[0105] In one implementation, if the terminal determines that the first video corresponds to at least two video coloring models, the terminal can apply the at least two video coloring models to the first video for coloring, obtaining a third video corresponding to each video coloring model. The third videos are then merged to obtain the second video. Here, merging at least two third videos means merging the image frames in the at least two third videos one-to-one. Merging image frames means merging the pixel values ​​of each pixel in the image frame one-to-one.

[0106] In one implementation, the terminal can calculate the similarity between the first image frame and the training dataset corresponding to each video colorization model of the first video. That is, the terminal can determine the similarity corresponding to each video colorization model, thereby determining the first weight of the third video corresponding to each video colorization model. It can be considered that the higher the similarity corresponding to the video colorization model, the greater the weight of its corresponding third video. The first weight is used to fuse at least two third videos. For example, the first weight can represent the proportion of the third video in the fused video when fusing the third videos. For instance, when the terminal fuses third video A and third video B, it determines the weight based on the first weight P of third video A and the second weight Q of third video B.

[0107] In one implementation, the terminal fuses at least two third videos according to a first weight. Specifically, each image frame in the at least two third videos is fused one-to-one. The image fusion method can be referred to in related technologies, and will not be described in detail here.

[0108] This disclosure performs video colorization processing based on at least two video colorization models corresponding to the first video, and then merges the processed at least two third videos to obtain the second video. Since different videos are used to perform video colorization processing on different types of videos, and the colorization results of multiple video colorization models are referenced and the results are merged according to weights, the video colorization effect can be improved, and the effect of handling complex video scenes is significant.

[0109] In some implementations, the terminal may determine at least two video coloring models corresponding to the first video from a pre-configured set of video coloring models, as follows.

[0110] In one implementation, the terminal can randomly select at least two video coloring models from a pre-configured set of video coloring models.

[0111] In one implementation, the terminal can calculate the similarity between the first image frame and the training dataset corresponding to the video colorization model, and determine at least two video colorization models corresponding to the first video based on the similarity. For example, if at least two of the calculated similarities are greater than or equal to a similarity threshold, the video colorization model corresponding to the training dataset with that similarity is determined as the video colorization model corresponding to the first video. If only one of the calculated similarities is greater than or equal to the similarity threshold, or if no similarity is greater than or equal to the similarity threshold, the terminal can sort the calculated similarities and determine the video colorization models corresponding to the training datasets with the highest similarity as the video colorization model corresponding to the first video.

[0112] In some embodiments, in the video processing method provided in this disclosure, the terminal may acquire the first image frame by acquiring any one or more image frames from the first video.

[0113] In some cases, the first image frame can be any image frame in the first video. The terminal arbitrarily selects one image frame as the first image frame and determines the video colorization model corresponding to the first image frame. (Reference) Figure 4a Assume the first video consists of 5 image frames. Figure 4a Each square in the diagram represents an image frame of the first video. The terminal can select any one of image frames 1 to 5 as the first image frame. Correspondingly, the video colorization model can be a video colorization model determined by the terminal based on any one of the image frames 1 to 5. The terminal performs video colorization processing on each image frame in the first video according to the video colorization model corresponding to the first image frame to obtain the second video.

[0114] In other cases, the first image frame can also be any number of image frames in the first video. The terminal can perform video colorization processing on the first image frame using the video colorization model corresponding to the first image frame. Simultaneously, for image frames adjacent to or close to the first image frame, video colorization processing is also performed using the video colorization model corresponding to the first image frame. For example, refer to... Figure 4b The terminal can select image frames 1 and 4 from the first video as the first image frames. The terminal can determine the video colorization model corresponding to each of image frames 1 and 4, and perform video colorization processing on image frames 1 and 4. For image frames 2 and 3 adjacent to image frame 1, video colorization processing is performed using the video colorization model corresponding to image frame 1. For image frame 5 adjacent to image frame 4, video colorization processing is performed using the video colorization model corresponding to image frame 4.

[0115] In other cases, the first image frame can be all the image frames of the first video. The terminal can determine the video colorization model corresponding to each image frame in the first video. (See reference) Figure 4c The terminal can determine image frames 1 to 5 as the first image frame and determine their respective corresponding video coloring models. The terminal uses the video coloring model corresponding to image frame 1 to perform video coloring processing on image frame 1; and uses the video coloring model corresponding to image frame 2 to perform video coloring processing on image frame 2. This disclosure does not provide examples of each.

[0116] It is understood that the first video in this embodiment, including image frames 1 to 5, is merely an exemplary example, and this disclosure does not limit it.

[0117] This disclosure improves the efficiency of video colorization processing by quickly determining the corresponding video colorization model for the first video by acquiring any one image frame from the first video as the first image frame. Alternatively, by acquiring any multiple image frames from the first video as the first image frame, targeted video colorization processing can be performed on different image frames in the first video, thereby improving the effect of video colorization processing on the first video. Finally, by acquiring all image frames from the first video as the first image frame, targeted video colorization processing can be performed on each frame of the first video, effectively improving the effect of video colorization processing on the first video, but the processing efficiency is relatively low.

[0118] In some embodiments, this disclosure provides a video colorization method, such as... Figure 5 As shown, it includes the following steps.

[0119] In step S41, the video coloring model corresponding to the first video is used to perform video coloring processing on the first image frame.

[0120] In step S42, the second image frame is subjected to video colorization processing using the first image frame after colorization and the pre-configured reference model.

[0121] In step S43, the second video is obtained based on the first image frame after colorization and the second image frame after colorization.

[0122] In one implementation, the terminal can extract a first image frame based on a pre-configured keyframe extraction algorithm. The first image frame is a keyframe in the first video. A keyframe refers to an image frame in the video that has representative significance. For example, the frame containing a key action in the movement of a character or object in the video. The keyframe extraction algorithm can be a motion analysis-based algorithm, a video frame clustering-based method, or a color or color histogram frame difference-based method, etc.

[0123] In one implementation, the terminal can determine the video colorization model corresponding to the first image frame based on the first image frame. If the first video includes multiple first image frames, each first image frame has its corresponding video colorization model; that is, if there are multiple first image frames, there may also be multiple video colorization models corresponding to the first video. The terminal can perform video colorization processing on each first image frame using the video colorization model corresponding to that first image frame.

[0124] In one implementation, the terminal can perform video colorization processing on a second image frame using a first colorized image frame and a pre-configured reference model. The reference model is a neural network model pre-trained by the terminal, which uses the colorized image frame to perform video colorization processing on the uncolorized image frame. It can be understood that the input of the video colorization model in this disclosure is the uncolorized image frame A, and the output is the colorized image frame A. The input of the reference model is the uncolorized image frame C and the colorized image frame D; typically, image frames C and D are adjacent image frames. The output of the reference model is the colorized image frame C. In other words, the reference model uses the colorized image frame to perform video colorization processing on the uncolorized image frame.

[0125] For example, by inputting the colored first image frame and the second image frame into the reference model, video colorization processing can be performed on the second image frame. Here, the second image frame refers to the image frame in the first video other than the first image frame.

[0126] In one implementation, the terminal performs video colorization processing on the first image frame and the second image frame respectively to obtain a second video. The second video is the first video after video colorization processing.

[0127] This disclosure extracts keyframes from a first video to be processed, and then performs video coloring processing on the first video based on the video coloring model corresponding to the keyframes, so as to achieve more targeted video coloring processing and better video coloring processing results.

[0128] This disclosure uses a pre-configured reference model to perform video colorization processing on image frames other than keyframes, which can reduce computation and save power. Furthermore, by processing keyframes and other image frames simultaneously, this disclosure can effectively improve the efficiency of video colorization processing.

[0129] In some embodiments, this disclosure provides a video colorization method, comprising: using N consecutive image frames in a first video and a pre-configured reference model, performing video colorization processing on the next second image frame adjacent to the N consecutive image frames, until video colorization processing is completed for each second image frame, where N is a positive integer.

[0130] In one implementation, N is 1, and the terminal performs video colorization processing on the second image frame in the first video each time, using a certain image frame as a reference. The terminal can perform video colorization processing on the second image frame using the colorized first image frame as a reference, or it can perform video colorization processing on the uncolorized second image frame using the colorized second image frame as a reference.

[0131] For example, the terminal uses the first image frame after colorization as a reference to perform video colorization processing on the next second image frame adjacent to the first image frame; the terminal uses the second image frame after colorization as a reference to perform video colorization processing on the next second image frame adjacent to the second image frame after colorization, until video colorization processing is completed for each second image frame in the first video. Figure 6 Assume the first video consists of 5 image frames, where image frames 1 and 4 are the first image frames determined by the terminal, and image frames 2, 3, and 5 are the second image frames in the terminal besides the first image frames. For image frames 1 and 4, the terminal can perform video colorization processing using their corresponding video colorization models. For image frame 2, which is adjacent to image frame 1, the terminal can input the colorized image frame 1 and the uncolorized image frame 2 into a pre-configured reference model to obtain the colorized image frame 2. Similarly, for image frame 3, which is adjacent to image frame 2, the terminal can input the colorized image frame 2 and the uncolorized image frame 3 into a pre-configured reference model to obtain the colorized image frame 3. Likewise, for image frame 5, which is adjacent to image frame 4, the terminal can input the colorized image frame 4 and the uncolorized image frame 5 into a pre-configured reference model to obtain the colorized image frame 5.

[0132] In one implementation, N is a positive integer greater than 1. The terminal performs video colorization processing on a second image frame in the first video, using multiple consecutive image frames as references each time. For example, the terminal can use multiple colored first image frames as references to perform video colorization processing on the second image frame. Alternatively, the terminal can use at least one of the colored first image frames and a colored second image frame as references to perform video colorization processing on an uncolored second image frame. The terminal can also perform video colorization processing on an uncolored second image frame using multiple colored second image frames.

[0133] For example, assume N is 2. The terminal uses the first image frame after colorization as a reference to perform video colorization processing on the second image frame adjacent to the first image frame. Here, the first image frame consists of two consecutive image frames. The terminal selects the first image frame adjacent to the second image frame from the colorized first image frames, and uses this first image frame and the colorized second image frame as references to perform video colorization processing on the next image frame adjacent to the colorized second image frame. The terminal uses two adjacent colorized second image frames as references to perform video colorization processing on the next second image frame after those two adjacent colorized second image frames. Figure 7Assume the first video consists of 5 image frames, where image frames 1 and 2 are the first image frames determined by the terminal, and image frames 3, 4, and 5 are the second image frames in the terminal besides the first image frames. The terminal uses a video colorization model to colorize image frames 1 and 2. It can be understood that the video colorization model can be either the model corresponding to image frame 1 or the model corresponding to image frame 2. The terminal inputs the colorized image frames 1 and 2, along with the uncolorized image frame 3, into a pre-configured reference model. Using the colorized image frames 1 and 2 as a reference, the terminal obtains the colorized image frame 3. The terminal inputs the colorized image frames 2 and 3, along with the uncolorized image frame 4, into the pre-configured reference model to obtain the colorized image frame 4. The terminal inputs the colorized image frames 3 and 4, along with the uncolorized image frame 5, into the pre-configured reference model to obtain the colorized image frame 5.

[0134] It is understood that the first video in this embodiment includes image frames 1 to 5 and the video coloring method is merely an example, and this disclosure does not limit it.

[0135] This disclosure improves the efficiency of video colorization processing by utilizing N consecutive image frames from the first video to perform video colorization processing on the second image frame. When N is an integer greater than 1, the terminal performs more detailed video colorization processing on the second image frame, resulting in better video colorization effects.

[0136] In some embodiments, the second video may include a third image frame and a fourth image frame, wherein the third image frame is one or more image frames in the first video, and the fourth image frame is an image frame in the second video other than the third image frame. This disclosure provides a video processing method, including: performing video smoothing processing on the fourth image frame based on the third image frame.

[0137] In one implementation, the terminal can determine a third image frame. The third image frame is one or more image frames from the first video. For example, the terminal can arbitrarily select one or more image frames as the third image frame. Exemplarily, the terminal can select the first or last image frame from the second video in chronological order as the third image frame. As another example, the terminal can use a keyframe extraction algorithm to extract keyframes from the second video and use the extracted keyframes as the third image frame.

[0138] In one embodiment, the terminal can perform video smoothing processing on other image frames in the second video, excluding the third image frame, based on a determined third image frame. This disclosure refers to the image frames in the second video, excluding the third image frame, as fourth image frames.

[0139] In this embodiment, the terminal performs video smoothing processing on the fourth image frame based on the third image frame, including: the terminal can perform image registration on the third image frame based on the fourth image frame. Image registration is also known as image alignment. Image registration refers to the process of matching and superimposing two or more images from different times, different sensors, or different conditions. The terminal fuses the image-registered third image frame and the fourth image frame to obtain the fused fourth image frame. The third image frame and the fused fourth image frame constitute the second video after video smoothing processing.

[0140] This disclosure increases the stability between images by performing video smoothing on the second video, making the smoothed second video more natural and realistic.

[0141] In some implementations, the terminal provides a video processing method, such as Figure 8 As shown, it includes the following steps.

[0142] In step S51, the fourth image frame is used to perform image registration on the M consecutive image frames adjacent to the fourth image frame, where M is a positive integer.

[0143] In step S52, the second weights corresponding to each image frame in the fourth image frame and the M consecutive image frames after image registration are determined.

[0144] In step S53, based on each second weight, the fourth image frame and the M consecutive image frames after image registration are fused to obtain the fourth image frame after image fusion.

[0145] In step S54, the image fusion is completed for each fourth image frame in the second video to obtain the second video after video smoothing.

[0146] In one implementation, M is 1, and the terminal performs image registration and image fusion on the fourth image frame based on a certain image frame each time. For example, the terminal performs image registration on the third image frame based on the next fourth image frame adjacent to the third image frame. That is, the third image frame is aligned with the next fourth image frame adjacent to the third image frame. The terminal performs image fusion on the image-registered third and fourth image frames. The terminal performs image registration on the image-fused fourth image frame based on the next fourth image frame adjacent to the image-fused fourth image frame. The terminal performs image fusion on the image-fused and image-registered fourth image frame and the next adjacent fourth image frame until all fourth image frames in the second video have been image-fused.

[0147] For example, refer to Figure 9Assume the second video consists of 5 image frames. Image frame 1 can be the third image frame determined by the terminal, and image frames 2 through 5 are the fourth image frames in the second video, excluding the third image frame. The terminal can perform image registration and image fusion operations on image frame 1 and image frame 2 to obtain the fused image frame 2. The terminal can perform image registration and image fusion operations on the fused image frame 2 and image frame 3 to obtain the fused image frame 3. Taking image frame 1 and image frame 2 as an example, the process of image registration and image fusion can be found by referring to... Figure 10 .like Figure 10 As shown, the terminal can perform image registration on image frame 1 based on image frame 2. That is, using image frame 2 as a standard, image frame 1 is aligned to image frame 2. The terminal then performs image fusion on the image-registered image frame 1 and image frame 2 to obtain the fused image frame 2.

[0148] It is understood that the second video included in this embodiment, including image frames 1 to 5, and the video smoothing processing method described herein are merely exemplary examples, and this disclosure does not limit them. For example, the third image frame determined by the terminal may also be image frame 5, and image registration and image fusion operations may be performed on image frame 5 and image frame 4 to obtain the image frame 4 after image fusion. This disclosure does not provide examples of all possible scenarios.

[0149] In one implementation, M is a positive integer greater than 1. The terminal performs image registration and image fusion on the fourth image frame each time based on multiple adjacent image frames. For example, if M is 2, refer to... Figure 11 Assume the second video consists of 5 image frames. Image frames 1 and 2 are the third image frames determined by the terminal, and image frames 3 to 5 are the fourth image frames in the second video, excluding the third image frame. The terminal can perform image registration and image fusion operations on image frames 1, 2, and 3 to obtain the fused image frame 3. The terminal can also perform image registration and image fusion operations on image frame 2, the fused image frame 3, and image frame 4 to obtain the fused image frame 4. For example, the terminal performs image registration on image frames 1 and 2 based on image frame 3. The registered image frames 1, 2, and 3 are then fused to obtain the fused image frame 3.

[0150] In one implementation, the terminal can determine a second weight. This second weight refers to the proportion of each image frame in the fusion process.

[0151] For example, the second weight can be preset. For instance, when fusing image frame 1 and image frame 2 after image registration, the second weight can be (X, Y). Here, X represents the proportion of image frame 1 after image registration to image frame 2 after fusion, and Y represents the proportion of image frame 2 to image frame 2 after fusion.

[0152] For example, the terminal can also derive a second weight based on the consistency of the second video. Consistency can be understood as the frequency of object movement in the second video. If the objects in the second video move frequently, the consistency of the second video can be considered low. Conversely, if the objects in the second video move infrequently, the consistency of the second video can be considered high. For instance, the terminal can determine the consistency of the second video by obtaining its color distribution consistency (CDC) index. Alternatively, the terminal can also determine whether objects in the second video move frequently using other feasible methods; this disclosure does not limit this approach.

[0153] In this embodiment, the fusion of image frame 1 and image frame 2 after image registration is taken as an example. If the consistency of the second video is high, then in the second weighting, the proportion of image frame 1 after image registration in the fused image frame 2 is small, and the proportion of image frame 2 in the fused image frame 2 is large, to prevent ghosting. Conversely, if the consistency of the second video is low, then in the second weighting, the proportion of image frame 1 after image registration in the fused image frame 2 is large, and the proportion of image frame 2 in the fused image frame 2 is small.

[0154] This disclosure increases the stability between image frames by performing video smoothing processing on the second video, making the smoothed second video more natural and realistic. When M is an integer greater than 1, the terminal performs video smoothing processing on the second image frames more meticulously, resulting in better video smoothing effects.

[0155] In some implementations, the video colorization models in the video colorization model set correspond to different training datasets. In the video processing method provided in this disclosure, if the video colorization model set includes a first video colorization model and a second video colorization model, the second training dataset corresponding to the second video colorization model can be obtained based on the first training dataset corresponding to the first video colorization model.

[0156] In one implementation, the terminal can pre-acquire a first training dataset, which includes a large number of uncolored videos. For example, the terminal can acquire a large number of color videos, perform video desaturation processing on the color videos to obtain black and white videos. The black and white videos are uncolored videos, and the large number of black and white videos constitutes the first training dataset. The terminal can then filter from the first training dataset to obtain a second training dataset. The terminal can filter the first training dataset based on pre-set rules to obtain the second training dataset.

[0157] In one implementation, the terminal can train an untrained deep learning model using a first training dataset to obtain a first video colorization model. The deep learning model can be any one of a convolutional neural network, a generative adversarial network, and a variational autoencoder. For example, the terminal can input the first training dataset into the untrained deep learning model to obtain an output result. The terminal can then train the untrained deep learning model using a loss function between the color video corresponding to the first training dataset and the output result to obtain the first video colorization model.

[0158] It is understood that the terminal can use the second training dataset to train the second video colorization model. The specific training method is the same as that used for the first video colorization model, and will not be repeated here. Users can also use training methods from other related technologies to train the first and second video colorization models, and this disclosure does not limit them.

[0159] This disclosure reduces the time required to obtain training datasets for another video colorization model by filtering the training dataset of a certain video colorization model, thereby improving efficiency.

[0160] In some implementations, such as Figure 12 As shown, the terminal can train the reference model in the following ways.

[0161] In step S61, a third training dataset is obtained, which includes multiple sets of adjacent image frames.

[0162] In step S62, image registration is performed on the (N+1)th image frame in the adjacent image frames using N image frames in the adjacent image frames.

[0163] In step S63, video desaturation processing is performed on the (N+1)th image frame.

[0164] In step S64, the first N image frames after image registration and the (N+1)th image frame after desaturation are input into the untrained reference model to obtain the output result.

[0165] In step S65, the untrained reference model is trained using the loss function between the output result and the (N+1)th undesaturated image frame.

[0166] In one implementation, the terminal can acquire a third training dataset, which includes multiple sets of adjacent image frames. The terminal can crawl a large number of random video clips from the network and extract several adjacent image frames from each video clip as the third training dataset.

[0167] In one implementation, reference Figure 13The terminal can use N image frames from the third training dataset to perform image registration on the (N+1)th image frame and then perform video desaturation on the (N+1)th image frame. Video desaturation can be understood as converting a color video to a black and white video. The terminal then uses the image-registered N image frames and the desaturated (N+1)th image frame to train an untrained reference model. For example, the image-registered N image frames and the desaturated (N+1)th image frame are input into the untrained reference model to obtain the output. The loss function between the input and the undesaturated (N+1)th image frame is then used to train the untrained reference model so that its output is the same as or similar to that of the undesaturated (N+1)th image frame.

[0168] This disclosure obtains a reference model through training, thereby enabling the colorization of a first video based on the reference model to obtain a second video, thus improving the efficiency of video colorization.

[0169] This disclosure will illustrate a video colorization method with an exemplary embodiment. Figure 14 As shown, the video colorization method includes the following steps.

[0170] In step S71, keyframes are extracted from the video to be processed.

[0171] In one implementation, after the terminal acquires the video to be processed, it uses a keyframe extraction algorithm to extract keyframes from the video.

[0172] In step S72, the keyframes and the third video colorization model are matched.

[0173] In one implementation, the terminal can match keyframes with a third video colorization model. The third video colorization model is a video colorization model that performs colorization processing on videos of common scenes. For example, the terminal can crawl a large number of random video clips from the web to obtain a fourth training dataset. The fourth training dataset includes videos of common scenes and videos of non-common scenes. The terminal can filter videos of common scenes from the fourth training dataset. This filtering can be done by extracting features from the image frames of the videos in the fourth training dataset, identifying the extracted features, and determining whether the video is a video of a common scene, thus obtaining the third training dataset. For example, if videos of animals are pre-defined as videos of common scenes, then by extracting and identifying features from the image frames, it can be determined that the video is a video of an animal, thus determining that the video is a video of a common scene. Alternatively, the terminal can also input the fourth training dataset into a pre-trained judgment model, which is used to determine whether the input video is a video of a common scene, thus obtaining the third training dataset. The terminal uses the third training dataset to pre-train a third video colorization model.

[0174] In this embodiment, the terminal matches keyframes with image frames of any video segment in the third training dataset to obtain a similarity score. This similarity score is used to determine whether the match is successful. For example, if the similarity score is greater than a similarity threshold, the match is considered successful; if the similarity score is less than the similarity threshold, the match is considered unsuccessful.

[0175] In step S73, in response to a successful match, the keyframe is colorized using the third video colorization model.

[0176] In step S74, in response to the failure of matching, the keyframe is colored using the fourth video coloring model.

[0177] In one implementation, if a match is successful, the terminal can use a third video colorization model to colorize the keyframes. If a match is unsuccessful, it can be assumed that the video to be processed contains uncommon scenes, and the terminal can use a fourth video colorization model to colorize the keyframes. The fourth video colorization model is a video colorization model trained by the terminal using a fourth training dataset. The fourth video colorization model is a video colorization model that can colorize videos of both common and uncommon scenes.

[0178] In step S75, the non-key frames in the video to be processed are colorized using the reference model.

[0179] In one implementation, the terminal can use the colored keyframe as a reference, and input the adjacent non-keyframes and the colored keyframe into the reference model to obtain the colored non-keyframes. The non-keyframes are image frames in the video to be processed, excluding the keyframes.

[0180] In step S76, video smoothing is performed on the processed video.

[0181] In one implementation, the terminal can perform video smoothing on the processed video. For example, starting from the first image frame of the processed video, video smoothing is performed on each image frame sequentially. For instance, taking image frame 1 and image frame 2 as examples, the terminal performs image registration on image frame 1 based on image frame 2, and then performs image fusion on the image-registered image frame 1 and image frame 2 to obtain the image-fused image frame 2.

[0182] This disclosure improves the efficiency and effectiveness of video coloring by determining a video coloring model corresponding to the video to be processed from a set of video coloring models, and then using the video coloring model to colorize the video to be processed.

[0183] It should be noted that those skilled in the art will understand that the various implementation methods / embodiments described above in this disclosure can be used in conjunction with the foregoing embodiments, or they can be used independently. Whether used alone or in conjunction with the foregoing embodiments, the implementation principle is similar. In this disclosure, some embodiments are described as implementations used together. Of course, those skilled in the art will understand that such illustrative examples are not intended to limit the embodiments of this disclosure.

[0184] Based on the same concept, embodiments of this disclosure also provide a video processing apparatus.

[0185] It is understood that the video processing apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0186] Figure 15 This is a block diagram illustrating a video processing apparatus 100 according to an exemplary embodiment. (Refer to...) Figure 15 The device includes an acquisition module 101, a determination module 102, a processing module 103, and a training module 104.

[0187] The system includes an acquisition module 101 for acquiring a first video, which is the video to be processed. A determination module 102 is used to determine the video colorization model corresponding to the first video based on a set of video colorization models. This set includes multiple video colorization models, each used to colorize different types of videos. Video colorization processing involves converting black-and-white frames into color frames. A processing module 103 is used to colorize the first video using the corresponding video colorization model to obtain a second video.

[0188] In one implementation, the acquisition module 101 is further configured to acquire a first image frame, wherein the first image frame is one or more image frames of the first video. The training dataset corresponding to the video colorization model is acquired. The processing module 103 is further configured to calculate the similarity between the first image frame and the training dataset. The determination module 102 determines the video colorization model corresponding to the first video based on a pre-configured set of video colorization models in the following manner: determining the video colorization model corresponding to the first video based on similarity.

[0189] In one implementation, the determining module 102 determines the video colorization model corresponding to the first video based on similarity in the following manner: in response to a similarity greater than or equal to a similarity threshold, the video colorization model corresponding to the training dataset corresponding to the similarity is determined as the video colorization model corresponding to the first video.

[0190] In one implementation, the determining module 102 determines the video colorization model corresponding to the first video based on the video colorization model set as follows: Based on the video colorization model set, at least two video colorization models corresponding to the first video are determined. The processing module 103 performs video colorization processing on the first video using the video colorization model corresponding to the first video to obtain the second video as follows: Based on each of the at least two video colorization models, the first video is colorized to obtain the third video. Based on the similarity between the first image frame and the training dataset corresponding to each of the at least two video colorization models, a first weight is determined for the third video corresponding to each video colorization model. Using the first weight of the third video corresponding to each video colorization model, the third videos corresponding to each video colorization model are fused to obtain the second video.

[0191] In one implementation, the first image frame is a keyframe. The processing module 103 performs video colorization processing on the first video using the video colorization model corresponding to the first video: The first image frame is colorized using the video colorization model corresponding to the first video. Using the colorized first image frame and a pre-configured reference model, a second image frame is colorized. The second image frame is any image frame in the first video other than the first image frame. The reference model is used to colorize uncolorized image frames using the colorized image frames. Based on the colorized first image frame and the colorized second image frame, a second video is obtained.

[0192] In one implementation, the processing module 103 performs video colorization processing on the second image frame using the colorized first image frame and a pre-configured reference model as follows: Using N consecutive image frames in the first video and the pre-configured reference model, video colorization processing is performed on the next second image frame adjacent to the N consecutive image frames, until video colorization processing is completed for each second image frame, where N is a positive integer. The N image frames include any one or more of the following: the colorized first image frame; the colorized second image frame.

[0193] In one embodiment, the second video includes a third image frame and a fourth image frame. The third image frame is one or more image frames in the second video, and the fourth image frame is an image frame in the second video other than the third image frame. The processing module 103 is further configured to: perform video smoothing processing on the fourth image frame based on the third image frame to obtain a fourth video, which is the second video after video smoothing processing.

[0194] In one embodiment, the processing module 103 performs video smoothing processing on the fourth image frame based on the third image frame as follows: Using the fourth image frame, image registration is performed on M consecutive image frames in the second video, where the fourth image frame is the next adjacent image frame of the M consecutive image frames. A second weight is determined for the M consecutive image frames after image registration and the next adjacent fourth image frame. Based on the second weight, the M consecutive image frames after image registration and the next adjacent fourth image frame are fused to obtain a fused fourth image frame, until image fusion is completed for each fourth image frame in the second video. The fourth video is obtained based on the third image frame and the fused fourth image frame. The first N image frames in the second video include one or more of the following: the third image frame; and the fused fourth image frame.

[0195] In one embodiment, the video colorization model set includes a first video colorization model and a second video colorization model. The first video colorization model is trained based on a first training dataset, and the second video colorization model is trained based on a second training dataset, wherein the second training dataset is determined based on the first training dataset.

[0196] In one embodiment, the apparatus further includes a training module 104, configured to train a reference model in the following manner: acquiring a third training dataset. The third training dataset includes multiple sets of adjacent image frames. Following a preset order, image registration is performed on the (N+1)th image frame among the adjacent image frames using N image frames. Video desaturation processing is performed on the (N+1)th image frame, which involves converting a color video to a black-and-white video. The first N image frames after image registration and the (N+1)th desaturated image frame are input into the untrained reference model to obtain an output result. The untrained reference model is trained using a loss function between the output result and the undesaturated (N+1)th image frame.

[0197] Figure 16 This is a block diagram illustrating a video processing apparatus 200 according to an exemplary embodiment.

[0198] like Figure 16As shown, device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0199] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0200] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0201] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0202] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0203] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0204] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0205] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0206] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0207] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0208] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0209] This disclosure improves the efficiency and effectiveness of video coloring by determining a video coloring model corresponding to the video to be processed from a set of video coloring models, and then using the video coloring model to colorize the video to be processed.

[0210] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0211] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0212] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0213] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0214] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

Claims

1. A method of video processing, the method comprising: The video processing method includes: Obtain the first video, which is the video to be processed; Based on the set of video colorization models, the video colorization model corresponding to the first video is determined. The set of video colorization models includes multiple video colorization models. Different video colorization models are used to perform video colorization processing on different types of videos. The video colorization processing means converting black and white image frames of the video into color image frames. The first video is colored using the video coloring model corresponding to the first video to obtain the second video; The video colorization model corresponding to the first video includes at least two video colorization models; The step of performing video colorization processing on the first video using the video colorization model corresponding to the first video to obtain the second video includes: Based on each of the at least two video coloring models, the first video is colorized to obtain multiple third videos; Based on the similarity between the first image frame and the training dataset corresponding to each of the at least two video colorization models, the plurality of third videos are fused to obtain a second video, wherein the first image frame is one or more image frames of the first video.

2. The method of claim 1, wherein, The step of determining the video colorization model corresponding to the first video based on the set of video colorization models includes: Obtain the first image frame; Obtain the training dataset corresponding to each video colorization model in the video colorization model set, wherein the training dataset includes training data frames; The similarity between the first image frame and each training data frame in the training dataset is determined based on the similarity between the first image frame and the training dataset. Based on the similarity between the first image frame and the training dataset, the video colorization model corresponding to the first video is determined.

3. The method of claim 2, wherein, The step of determining the video colorization model corresponding to the first video based on the similarity between the first image frame and the training dataset includes: The video colorization model corresponding to the training dataset with a similarity greater than or equal to the similarity threshold is determined as the video colorization model corresponding to the first video.

4. The method of claim 2, wherein, The first image frame is a keyframe; The second video, obtained by applying the video colorization model corresponding to the first video to perform video colorization processing, is replaced by: The first image frame is processed by video coloring using the video coloring model corresponding to the first video. Using the first image frame after colorization and the reference model, video colorization processing is performed on the second image frame. The second image frame is the image frame in the first video other than the first image frame. The reference model is used to perform video colorization processing on the uncolored image frame using the colorized image frame. The second video is obtained based on the first image frame after colorization and the second image frame after colorization.

5. The method of claim 4, wherein, The step of performing video colorization processing on the second image frame using the first colorized image frame and the reference model includes: Using N consecutive image frames and a reference model in the first video, video colorization processing is performed on the next second image frame adjacent to the N consecutive image frames until video colorization processing is completed for each second image frame, where N is a positive integer; The N consecutive image frames include: The first image frame after coloring.

6. The method of claim 1, wherein, The second video includes a third image frame and a fourth image frame, wherein the third image frame is one or more image frames in the second video, and the fourth image frame is an image frame in the second video other than the third image frame. The method further includes: Based on the third image frame, the fourth image frame is subjected to video smoothing processing to obtain the second video after video smoothing.

7. The method of claim 6, wherein, The step of performing video smoothing processing on the fourth image frame based on the third image frame to obtain a second video after video smoothing includes: Using the fourth image frame, image registration is performed on the M consecutive image frames preceding the fourth image frame, where M is a positive integer; Determine the second weight corresponding to each image frame in the fourth image frame and the M consecutive image frames after image registration; Based on each of the second weights, the fourth image frame and the M consecutive image frames after image registration are fused to obtain the fourth image frame after image fusion. The image fusion is completed for each of the fourth image frames in the second video to obtain the second video after video smoothing. The second video consists of M consecutive image frames, including: Third image frame.

8. The method according to any one of claims 1 to 7, characterized in that, The video colorization model set includes a first video colorization model and a second video colorization model; The first video colorization model is trained based on the first training dataset, and the second video colorization model is trained based on the second training dataset, wherein the second training dataset is determined based on the first training dataset.

9. The method of claim 4 or 5, wherein, The reference model is trained in the following manner: Obtain a third training dataset; the third training dataset includes multiple sets of adjacent image frames; according to a preset order, use N image frames in the adjacent image frames to perform image registration on the (N+1)th image frame in the adjacent image frames; The N+1th image frame is subjected to video desaturation processing, which refers to converting the color video into a black and white video. Input the N registered image frames and the (N+1)th desaturated image frame into the untrained reference model to obtain the output result; The untrained reference model is trained using the loss function between the output and the (N+1)th undesaturated image frame.

10. A video processing apparatus, characterized in that, The video processing device includes: The acquisition module is used to acquire the first video, which is the video to be processed; The determining module is used to determine the video coloring model corresponding to the first video based on the video coloring model set, wherein the video coloring model set includes multiple video coloring models, and different video coloring models among the multiple video coloring models are used to perform video coloring processing on different types of videos, and the video coloring processing means converting the black and white image frames of the video into color image frames; The processing module is used to perform video coloring processing on the first video using the video coloring model corresponding to the first video to obtain the second video; The video colorization model corresponding to the first video includes at least two video colorization models; The determining module uses the following method based on the set of video coloring models to determine the video coloring model corresponding to the first video: based on the set of video coloring models, at least two video coloring models corresponding to the first video are determined; The processing module performs video colorization processing on the first video using the video colorization model corresponding to the first video to obtain the second video: based on each of the at least two video colorization models, the first video is colorized to obtain multiple third videos; based on the similarity between the first image frame and the training dataset corresponding to each of the at least two video colorization models, the multiple third videos are fused to obtain the second video, wherein the first image frame is one or more image frames of the first video.

11. The apparatus according to claim 10, characterized in that, The acquisition module is further configured to acquire a first image frame; acquire a training dataset corresponding to each video colorization model in the video colorization model set, wherein the training dataset includes training data frames; The processing module is further configured to determine the similarity between the first image frame and the training dataset based on the similarity between the first image frame and each training data frame in the training dataset. The determining module determines the video coloring model corresponding to the first video based on a set of video coloring models in the following manner: Based on the similarity between the first image frame and the training dataset, the video colorization model corresponding to the first video is determined.

12. The apparatus of claim 11, wherein, The determining module determines the video colorization model corresponding to the first video based on the similarity using the following method: The video colorization model corresponding to the training dataset with a similarity greater than or equal to the similarity threshold is determined as the video colorization model corresponding to the first video.

13. The apparatus of claim 11, wherein, The first image frame is a keyframe; The processing module uses the video colorization model corresponding to the first video to perform video colorization processing on the first video, resulting in the second video being replaced with: The first image frame is processed by video coloring using the video coloring model corresponding to the first video. Using the first image frame after colorization and the reference model, video colorization processing is performed on the second image frame. The second image frame is the image frame in the first video other than the first image frame. The reference model is used to perform video colorization processing on the uncolored image frame using the colorized image frame. The second video is obtained based on the first image frame after colorization and the second image frame after colorization.

14. The apparatus of claim 13, wherein, The processing module uses the first image frame after colorization and the reference model to perform video colorization processing on the second image frame in the following manner: Using N consecutive image frames and a reference model in the first video, video colorization processing is performed on the next second image frame adjacent to the N consecutive image frames until video colorization processing is completed for each second image frame, where N is a positive integer; The N consecutive image frames include: The first image frame after coloring.

15. The apparatus of claim 10, wherein, The second video includes a third image frame and a fourth image frame, wherein the third image frame is one or more image frames in the second video, and the fourth image frame is an image frame in the second video other than the third image frame. The processing module is further configured to: Based on the third image frame, the fourth image frame is subjected to video smoothing processing to obtain the second video after video smoothing.

16. The apparatus of claim 15, wherein, The processing module performs video smoothing on the fourth image frame based on the third image frame in the following manner to obtain a second video after video smoothing: Using the fourth image frame, image registration is performed on the M consecutive image frames preceding the fourth image frame, where M is a positive integer; Determine the second weight corresponding to each image frame in the fourth image frame and the M consecutive image frames after image registration; Based on each of the second weights, the fourth image frame and the M consecutive image frames after image registration are fused to obtain the fourth image frame after image fusion. The image fusion is completed for each of the fourth image frames in the second video to obtain the second video after video smoothing. The second video consists of M consecutive image frames, including: Third image frame.

17. The apparatus of any one of claims 10 to 16, wherein, The video colorization model set includes a first video colorization model and a second video colorization model; The first video colorization model is trained based on the first training dataset, and the second video colorization model is trained based on the second training dataset, wherein the second training dataset is determined based on the first training dataset.

18. The apparatus of claim 13 or 14, wherein, The device further includes: The training module is used to train the reference model in the following manner: Obtain a third training dataset; the third training dataset includes multiple sets of adjacent image frames; according to a preset order, use N image frames in the adjacent image frames to perform image registration on the (N+1)th image frame in the adjacent image frames; The N+1th image frame is subjected to video desaturation processing, which refers to converting the color video into a black and white video. Input the N registered image frames and the (N+1)th desaturated image frame into the untrained reference model to obtain the output result; The untrained reference model is trained using the loss function between the output and the (N+1)th undesaturated image frame.

19. An electronic device, comprising: include: Memory, used to store instructions; as well as A processor for invoking instructions stored in the memory to execute the method as described in any one of claims 1-9.

20. A storage medium, characterized by The storage medium stores instructions that, when executed by a processor, perform the method as described in any one of claims 1-9.

Images