Video processing method and device and storage medium

Abstract

The invention relates to a video processing method and device and a storage medium. The video processing method comprises the following steps: acquiring a first video, wherein the first video is a video to be processed; based on a video coloring model set, a video coloring model corresponding to the first video is determined, the video coloring model set comprises a plurality of video coloring models, and different video coloring models are used for performing video coloring processing on different types of videos; the video coloring processing means that a black-and-white image frame of the video is converted into a color image frame; and performing video coloring processing on the first video by adopting the video coloring model corresponding to the first video to obtain a second video. According to the invention, the coloring processing efficiency and effect of the video can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video, and in particular to a video processing method, device and storage medium. Background Art

[0002] Video colorization technology is a branch of computer vision, and its main goal is to convert black-and-white videos into color videos. Since videos are composed of image frames, the related technology manually adds color to each image frame of the video, which has a low processing efficiency. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a video processing method, device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a video processing method is provided, including: obtaining a first video, where the first video is a video to be processed; determining a video colorization model corresponding to the first video based on a video colorization model set, wherein the video colorization model set includes multiple video colorization models, and different video colorization models in the multiple video colorization models are used to perform video colorization processing on different types of videos, and the video colorization processing represents converting black and white image frames of a video into color image frames; and performing video colorization processing on the first video using the video colorization model corresponding to the first video to obtain a second video.

[0005] In one implementation, determining the video colorization model corresponding to the first video based on the video colorization model set includes: acquiring a first image frame, where the first image frame is one or more image frames of the first video;

[0006] Obtain a training data set corresponding to each video colorization model in the video colorization model set, wherein the training data set includes training data frames; based on the similarity between the first image frame and each training data frame in the training data set, determine the similarity between the first image frame and the training data set; based on the similarity between the first image frame and the training data set, determine the video colorization model corresponding to the first video.

[0007] In one embodiment, determining the video colorization model corresponding to the first video based on the similarity between the first image frame and the training data set includes: determining the video colorization model corresponding to the training data set whose similarity is greater than or equal to a similarity threshold as the video colorization model corresponding to the first video.

[0008] In one embodiment, the video colorization model corresponding to the first video includes at least two video colorization models;

[0009] The method of using the video colorization model corresponding to the first video to perform video colorization on the first video to obtain the second video includes: based on each of the at least two video colorization models, coloring the first video separately to obtain multiple third videos; based on the similarity between the first image frame and the training data set corresponding to each of the at least two video colorization models, fusing the multiple third videos to obtain the second video.

[0010] In one embodiment, the first image frame is a key frame; the video colorization model corresponding to the first video is used to perform video colorization on the first video, including: using the video colorization model corresponding to the first video to perform video colorization on the first image frame; using the colored first image frame and a reference model to perform video colorization on the second image frame, the second image frame being an image frame in the first video other than the first image frame, and the reference model is used to perform video colorization on the uncolored image frame using the colored image frame; and obtaining the second video based on the colored first image frame and the colored second image frame.

[0011] In one embodiment, the video colorization processing is performed on the second image frame using the colored first image frame and the reference model, including: using N consecutive image frames in the first video and the reference model to perform video colorization processing on the next second image frame adjacent to the N consecutive image frames, until the video colorization processing is completed on each second image frame, where N is a positive integer; the N image frames include any one or more of the following: the colored first image frame; the colored second image frame.

[0012] In one embodiment, the second video includes a third image frame and a fourth image frame, the third image frame is one or more image frames in the second video, and the fourth image frame is an image frame in the second video except the third image frame. The method also includes: performing video smoothing processing on the fourth image frame according to the third image frame to obtain a second video after video smoothing processing.

[0013] In one embodiment, the video smoothing processing is performed on the fourth image frame according to the third image frame to obtain a second video after the video smoothing processing, including: using the fourth image frame to perform image registration on M consecutive image frames adjacent to the fourth image frame, M is a positive integer; determining the second weight corresponding to each image frame in the fourth image frame and the M consecutive image frames after the image registration; based on each of the second weights, performing image fusion on the fourth image frame and the M consecutive image frames after the image registration to obtain a fourth image frame after image fusion; until image fusion is completed for each of the fourth image frames in the second video to obtain a second video after video smoothing processing; wherein, the M consecutive image frames in the second video include one or more of the following: the third image frame; the fourth image frame after image fusion.

[0014] In one embodiment, the video colorization model set includes a first video colorization model and a second video colorization model; the first video colorization model is trained based on a first training data set, and the second video colorization model is trained based on a second training data set, wherein the second training data set is determined based on the first training data set.

[0015] In one embodiment, the reference model is trained in the following manner: obtaining a third training data set; the third training data set includes multiple groups of adjacent image frames; performing image registration on the N+1th image frame in the adjacent image frames using N image frames in a preset order; performing video decolorization processing on the N+1th image frame, wherein the video decolorization processing refers to converting a color video into a black and white video; inputting the first N image frames after image registration and the N+1th image frame after decolorization into an untrained reference model to obtain an output result; and training the untrained reference model using a loss function between the output result and the N+1th image frame that has not been decolorized.

[0016] According to a second aspect of an embodiment of the present disclosure, a video processing device is provided, characterized in that the video processing device includes: an acquisition module, used to acquire a first video, the first video being a video to be processed; a determination module, used to determine a video colorization model corresponding to the first video based on a video colorization model set, wherein the video colorization model set includes a plurality of video colorization models, and different video colorization models among the plurality of video colorization models are used to perform video colorization processing on different types of videos, and the video colorization processing represents converting black and white image frames of a video into color image frames; a processing module, used to perform video colorization processing on the first video using the video colorization model corresponding to the first video to obtain a second video.

[0017] In one embodiment, the acquisition module is also used to acquire a first image frame, which is one or more image frames of the first video; to acquire a training data set corresponding to each video colorization model in the video colorization model set, wherein the training data set includes a training data frame; the processing module is also used to determine the similarity between the first image frame and the training data set based on the similarity between the first image frame and each training data frame in the training data set; the determination module determines the video colorization model corresponding to the first video based on the video colorization model set in the following manner: based on the similarity between the first image frame and the training data set, determine the video colorization model corresponding to the first video.

[0018] In one embodiment, the determination module determines the video colorization model corresponding to the first video based on the similarity in the following manner, including: determining the video colorization model corresponding to the training data set whose similarity is greater than or equal to a similarity threshold as the video colorization model corresponding to the first video.

[0019] In one embodiment, the video colorization model corresponding to the first video includes at least two video colorization models;

[0020] The determination module determines the video colorization model corresponding to the first video based on the video colorization model set in the following manner: based on the video colorization model set, at least two video colorization models corresponding to the first video are determined; the processing module performs video colorization on the first video using the video colorization model corresponding to the first video in the following manner to obtain a second video: based on each of the at least two video colorization models, the first video is colorized separately to obtain multiple third videos; based on the similarity between the first image frame and the training data set corresponding to each of the at least two video colorization models, the multiple third videos are fused to obtain the second video.

[0021] In one embodiment, the first image frame is a key frame; the processing module performs video colorization on the first video using a video colorization model corresponding to the first video in the following manner: performs video colorization on the first image frame using a video colorization model corresponding to the first video; performs video colorization on a second image frame using the colored first image frame and a reference model, the second image frame being an image frame in the first video other than the first image frame, and the reference model being used to perform video colorization on an uncolored image frame using the colored image frame; and obtains the second video based on the colored first image frame and the colored second image frame.

[0022] In one embodiment, the processing module uses the colored first image frame and the reference model to perform video colorization processing on the second image frame in the following manner: using N consecutive image frames and the reference model in the first video, perform video colorization processing on the next second image frame adjacent to the N consecutive image frames until the video colorization processing is completed on each second image frame, where N is a positive integer; the N image frames include any one or more of the following: the colored first image frame; the colored second image frame.

[0023] In one embodiment, the second video includes a third image frame and a fourth image frame, the third image frame is one or more image frames in the second video, and the fourth image frame is an image frame in the second video except the third image frame, and the processing module is also used to: perform video smoothing processing on the fourth image frame according to the third image frame to obtain the second video after video smoothing processing.

[0024] In one embodiment, the processing module performs video smoothing processing on the fourth image frame according to the third image frame in the following manner: using the fourth image frame, performing image registration on M consecutive image frames adjacent to the fourth image frame, where M is a positive integer; determining the second weight corresponding to each of the fourth image frame and the M consecutive image frames after image registration; based on each of the second weights, performing image fusion on the fourth image frame and the M consecutive image frames after image registration to obtain a fourth image frame after image fusion; until image fusion is completed for each of the fourth image frames in the second video, to obtain a second video after video smoothing processing;

[0025] The M consecutive image frames in the second video include one or more of the following:

[0026] The third image frame; and the fourth image frame after image fusion.

[0027] In one embodiment, the video colorization model set includes a first video colorization model and a second video colorization model; the first video colorization model is trained based on a first training data set, and the second video colorization model is trained based on a second training data set, wherein the second training data set is determined based on the first training data set.

[0028] In one embodiment, the device also includes: a training module, which is used to train the reference model in the following manner: obtain a third training data set; the third training data set includes multiple groups of adjacent image frames; according to a preset order, use N image frames in the adjacent image frames to perform image registration on the N+1th image frame in the adjacent image frames; perform video decolorization processing on the N+1th image frame, and the video decolorization processing refers to converting a color video into a black and white video; input the first N image frames after image registration and the N+1th image frame after decolorization into an untrained reference model to obtain an output result; and train the untrained reference model using a loss function between the output result and the N+1th image frame that has not been decolorized.

[0029] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a memory for storing instructions; and a processor for calling the instructions stored in the memory to execute the video processing method in the first aspect or any one of the implementations of the first aspect.

[0030] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions are executed by a processor, the video processing method in the first aspect or any one of the implementations of the first aspect is executed.

[0031] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by determining a video colorization model corresponding to the video to be processed in a video colorization model set, the video to be processed is colorized using the video colorization model, so as to improve the efficiency and effect of video colorization processing.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0034] Figure 1 The figure is a flowchart of a video processing method according to an exemplary embodiment.

[0035] Figure 2 The figure is a flowchart of a method for determining a video colorization model according to an exemplary embodiment.

[0036] Figure 3 The figure is a flowchart of another video processing method according to an exemplary embodiment.

[0037] Figure 4aIt is a schematic diagram of a video processing method according to an exemplary embodiment.

[0038] Figure 4b is a schematic diagram showing another video processing method according to an exemplary embodiment.

[0039] Figure 4c is a schematic diagram showing another video processing method according to an exemplary embodiment.

[0040] Figure 5 The figure is a flowchart of another video processing method according to an exemplary embodiment.

[0041] Figure 6 is a schematic diagram showing another video processing method according to an exemplary embodiment.

[0042] Figure 7 is a schematic diagram showing another video processing method according to an exemplary embodiment.

[0043] Figure 8 The figure is a flowchart of another video processing method according to an exemplary embodiment.

[0044] Fig. 9 is a schematic diagram showing another video processing method according to an exemplary embodiment.

[0045] Fig.10 is a schematic diagram showing another video processing method according to an exemplary embodiment.

[0046] Fig.11 is a schematic diagram showing another video processing method according to an exemplary embodiment.

[0047] Fig.12 The figure is a flowchart of another video processing method according to an exemplary embodiment.

[0048] Fig.13 It is a schematic diagram of a reference model training method according to an exemplary embodiment.

[0049] Fig.14 is a schematic diagram showing another video processing method according to an exemplary embodiment.

[0050] Fig.15 The invention is a block diagram of a video processing device according to an exemplary embodiment.

[0051] Fig.16 is a block diagram of another video processing device according to an exemplary embodiment. DETAILED DESCRIPTION

[0052] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0053] Video colorization technology is a branch of computer vision, and its main goal is to convert black videos into color videos. Since videos are composed of image frames, related technologies manually add color to each image frame of the video, which has a low processing efficiency.

[0054] In related technologies, video colorization is achieved by using automated methods based on rules and traditional image processing techniques. The steps are as follows:

[0055] Image segmentation: Segment the image frames of the video to be processed into different regions, such as background, characters, objects, etc.

[0056] Object tracking: For dynamic scenes, objects in the image frame are tracked so that the color of the objects remains consistent in subsequent processing.

[0057] Color matching: Match each area or object with a corresponding color based on manually set rules or color libraries.

[0058] Color Fill: Fill each area or object with the corresponding color based on the color matching results.

[0059] Post-processing: Perform post-processing on the filled image, such as denoising.

[0060] However, the effect of this video colorization method depends largely on manually specified rules and parameters, so it is difficult to adapt to videos of different scenes. In addition, this colorization technology is less efficient for complex scenes and large-scale videos.

[0061] In another related technology, a deep learning model is used to implement video colorization. The deep learning model can automatically learn how to convert black and white videos into color videos without manually specifying colors or rules. Commonly used deep learning models include convolutional neural networks (CNN), generative adversarial networks (GAN), and variational auto-encoders (VAE). The steps are as follows:

[0062] Data preparation: Prepare a large number of background-white videos and corresponding color videos as training data.

[0063] Network Design: Design a neural network model suitable for video colorization tasks.

[0064] Model training: The neural network model is trained using the training data to learn how to convert black and white videos to color videos.

[0065] Model testing: Use test data to test the trained neural network model and evaluate the performance and effect of the model.

[0066] Colorization: Input the black-and-white video to be colored into the trained neural network model and obtain a color video as output.

[0067] However, there are still many problems with this video colorization method. For example, it takes a lot of time to screen and label data and train on a specific training data set, which results in limited generalization of the model and may cause unexpected errors when processing different videos. For another example, although the video colorization technology based on deep learning can learn complex features, the colorization results may not be natural enough in some details and textures, and some strange colors and flaws may appear, especially in the video performance, which is prone to frame-to-frame fluctuations, making the results look very unnatural. For another example, since the video is composed of image frames, a large amount of image data needs to be processed when colorizing the video, resulting in very high computational costs and the need to use high-performance equipment to complete the task.

[0068] Therefore, the present disclosure provides a video processing method, which determines a video colorization model corresponding to a video to be processed in a video colorization model set, and then uses the video colorization model to perform colorization processing on the video to be processed, so as to improve the efficiency and effect of video colorization processing.

[0069] Among them, the video processing method is applied to the terminal, which can also be called terminal equipment, mobile station (mobilestation, MS), mobile terminal (mobile terminal, MT), etc. It is a device that provides voice and / or data connectivity to users. For example, the terminal can be a handheld device with wireless connection function, a vehicle-mounted device, etc. At present, some examples of terminals are: smart phones (mobile phones), pocket personal computers (pocket personal computers, PPCs), handheld computers, personal digital assistants (personal digital assistants, PDAs), laptops, tablet computers, wearable devices, or vehicle-mounted devices, etc. In addition, when it is a vehicle to everything (V2X) communication system, the terminal device can also be a vehicle-mounted device. It should be understood that the embodiments of the present disclosure do not limit the specific technology and specific device form adopted by the terminal.

[0070] Figure 1 is a flowchart of a video processing method according to an exemplary embodiment. Figure 1 As shown, the video processing method includes the following steps.

[0071] In step S11, a first video is obtained.

[0072] In one implementation, the terminal may obtain a first video, where the first video is a video that has not been subjected to video colorization processing. Video colorization processing refers to converting a black-and-white video into a color video.

[0073] In step S12, based on the video colorization model set, a video colorization model corresponding to the first video is determined.

[0074] In one implementation, a video colorization model set may be preconfigured, and the video colorization model set may include multiple video colorization models. Different video colorization models are used to perform video colorization processing on different videos.

[0075] For example, video colorization model A is used to colorize videos of common scenes. It can be understood that video colorization model A has a stronger ability to colorize videos of common scenes; video colorization model B is used to colorize videos of uncommon scenes, and uncommon scenes refer to scenes that are less common.; Video colorization model C is used to colorize videos of common scenes and videos of uncommon scenes. It can be understood that compared with video colorization model A, video colorization model C has a weaker ability to colorize videos of common scenes and a stronger ability to colorize videos of uncommon scenes, and compared with video colorization model B, video colorization model C has a weaker ability to colorize videos of uncommon scenes and a stronger ability to colorize videos of common scenes.

[0076] Of course, the video colorization models cited in the present disclosure are only illustrative examples, and the video colorization models may include more types, such as a video colorization model for coloring food videos, a video colorization model for coloring natural landscape videos, etc., which are not listed one by one in the present disclosure.

[0077] For example, videos of common scenes may include any one or more of the following: people, animals, vehicles, furniture, plants, electrical appliances, food, buildings, toys, tableware, stationery, etc. Videos of uncommon scenes may include any one or more of the following: physical experiment equipment, artworks, natural landscapes, historical buildings, traditional costumes, sports equipment, musical instruments, electronic components, agricultural tools, scientific charts, etc.

[0078] It can be understood that the common scenarios and uncommon scenarios in the present disclosure are illustrative examples, and users can set common scenarios and uncommon scenarios according to actual conditions, and the present disclosure does not limit this.

[0079] In this embodiment, the terminal can determine the video coloring model corresponding to the first video from the pre-configured video coloring model set. It can be understood that different video coloring models have different capabilities for coloring videos. Therefore, the determined video coloring model corresponding to the first video

[0080] In step S13, the first video is subjected to video colorization processing using a video colorization model corresponding to the first video to obtain a second video.

[0081] In one embodiment, the terminal may input the first video into a video coloring model corresponding to the first video, perform video coloring on the first video, and obtain a second video. It can be understood that the second video is the first video after the video coloring is performed. Inputting the first video into the video coloring model means inputting each image frame of the first video into the video coloring model. The video coloring model performs video coloring on each image frame in turn to obtain image frames after each video coloring is performed. It can be understood that each image frame after the video coloring constitutes the second video.

[0082] The capabilities of each video colorization model in the pre-configured video colorization model set disclosed in the present invention are different. Therefore, video colorization processing can be performed on different videos of multiple scenes, effectively improving the generalization ability of the video colorization processing method. At the same time, using the video colorization model corresponding to the first video to be processed and performing video colorization processing on it in a targeted manner can improve the efficiency of video colorization processing, especially for videos of complex scenes or large-scale videos, which can significantly improve the efficiency. Moreover, targeted video colorization processing can improve the effect of video colorization processing.

[0083] In some embodiments, the video is composed of image frames, and the terminal can determine the video coloring model corresponding to the first video according to the image frames of the first video to be processed. Therefore, the present disclosure provides a video processing method, such as Figure 2 As shown, the video processing method includes the following steps.

[0084] In step S21, a first image frame is acquired.

[0085] In one embodiment, the terminal can obtain one or more image frames of the video to be processed, and the present disclosure refers to the one or more image frames of the first video obtained as the first image frame. Among them, the image frame refers to the smallest unit that constitutes the video. It can be understood that the video is composed of a series of image frames. The reference used to describe the video includes the number of frames per second (FPS), which is used to refer to the number of image frames transmitted per second. For example, 24FPS means 24 image frames are transmitted per second. It can be understood that the more frames a video transmits per second, the smoother the video is played and the better the user's viewing experience.

[0086] In step S22, a training data set corresponding to the video colorization model is obtained.

[0087] In one embodiment, the terminal may obtain a training data set corresponding to each video colorization model in a preconfigured video colorization model set. Of course, the terminal may also obtain a training data set corresponding to some video colorization models based on some video colorization models in the video colorization model set. For example, the terminal screens out some video colorization models from a preconfigured video colorization model set according to the length and complexity of the first video to be processed, thereby obtaining a training data set corresponding to some video colorization models. The training data set corresponding to the video colorization model can be understood as a training data set for training the video colorization model. The training data set includes a large number of video clips, and each video clip consists of multiple image frames.

[0088] It is understandable that there is no fixed execution order between the above steps S21 and S22. The terminal may first obtain the first image frame and then obtain the data set corresponding to the video colorization model, or may first obtain the data set corresponding to the video colorization model and then obtain the first image frame, which is not limited in the present disclosure.

[0089] In step S23, the similarity between the first image frame and the training data set is calculated.

[0090] In one embodiment, the terminal may calculate the similarity between the first image frame and the training data set. For example, the terminal may extract multiple image frames from the training data set, respectively calculate the similarity between the multiple image frames extracted from the training data set and the first image frame, and use the maximum value thereof as the similarity between the first image frame and the training data set. When extracting multiple image frames from the training data set, the terminal may extract image frames for multiple video clips in the training data set respectively, rather than extracting multiple image frames for the same video clip in the training data set, so that the final similarity is more accurate.

[0091] In one embodiment, the terminal may calculate the similarity between two image frames in a variety of ways. For example, the terminal may extract feature vectors of the two image frames respectively and calculate the distance between the vectors to obtain the similarity between the two image frames. For another example, the terminal may input the two image frames into a pre-configured deep learning model to obtain the similarity between the two image frames.

[0092] In step S24, a video colorization model corresponding to the first video is determined based on the similarity.

[0093] In one embodiment, the terminal may determine the video colorization model corresponding to the first video based on the similarity between the first image frame and the training data set. The higher the similarity between the first image frame and a certain training data set, it can be considered that the video colorization model corresponding to the training data set has a better effect on the video colorization processing of the first video. For example, the terminal may match the first image frame with the training data set corresponding to each video colorization model in the pre-configured video colorization model set to obtain multiple similarities. The terminal may sort the multiple similarities according to the numerical value, determine the training data set corresponding to the similarity with the largest numerical value, and determine that the video colorization model corresponding to the training data set is the video colorization model corresponding to the first video.

[0094] The present invention determines the video colorization model corresponding to the first video by performing similarity matching between the image frames of the first video to be processed and the training data set of the video colorization model, accurately selects the video colorization model suitable for video colorization of the video to be processed, and improves the efficiency and effect of video colorization of the first video to be processed.

[0095] In some embodiments, the present disclosure provides a video colorization method: in response to the similarity being greater than or equal to a similarity threshold, determining a video colorization model corresponding to a training data set corresponding to the similarity as a video colorization model corresponding to a first video.

[0096] In one embodiment, the terminal may select a video colorization model from a preconfigured video colorization model set in a predetermined order, obtain a training data set corresponding to the selected video colorization model, and calculate a similarity between the first image frame and the training data set, and in response to the similarity being greater than or equal to a similarity threshold, determine the obtained video colorization model as the video colorization model corresponding to the first video.

[0097] Of course, the terminal may also randomly select a video colorization model from a pre-configured video colorization model set, which is not limited in the present disclosure.

[0098] This announcement can quickly determine the video colorization model corresponding to the first video by determining the video colorization model corresponding to the training data set corresponding to the similarity as the video colorization model corresponding to the first video in response to the similarity being greater than or equal to a similarity threshold.

[0099] In some implementations, the terminal may determine at least two video coloring models corresponding to the first video from a preconfigured video coloring model set. Figure 3 FIG. 1 is a flow chart of a video processing method according to an exemplary embodiment. Figure 3 As shown, the video processing method includes the following steps.

[0100] In step S31, at least two video colorization models corresponding to the first video are determined based on the video colorization model set.

[0101] In step S32, based on each of the at least two video colorization models, the first video is subjected to video colorization processing to obtain a third video.

[0102] In step S33, based on the similarity between the first image frame and the training data set corresponding to each of the at least two video colorization models, a first weight of the third video corresponding to each video colorization model is determined.

[0103] In step S34, the third videos corresponding to each video coloring model are fused using the first weight of the third videos corresponding to each video coloring model to obtain a second video.

[0104] In one implementation, the terminal may determine one or more video colorization models from a video colorization model set.

[0105] In one implementation, if the terminal determines that the first video corresponds to at least two video coloring models, the terminal may respectively use the at least two video coloring models to perform video coloring processing on the first video, obtain a third video corresponding to each video coloring model, and fuse the third videos to obtain a second video. Fusion of at least two third videos refers to fusing image frames in at least two third videos one by one. Fusion of image frames refers to fusing pixel values ​​of each pixel in the image frames one by one.

[0106] In one embodiment, the terminal can calculate the similarity between the first image frame and the training data set corresponding to each video coloring model corresponding to the first video. That is, the terminal can determine the similarity corresponding to each video coloring model, thereby determining the first weight of the third video corresponding to each video coloring model. It can be considered that the higher the similarity corresponding to the video coloring model, the greater the weight of the corresponding third video. Among them, the first weight is used to fuse at least two third videos. For example, the first weight can represent the proportion of the third video in the fused video when the third video is fused. Exemplarily, when the terminal fuses the third video A and the third video B, according to the first weight P of the third video A and the second weight Q of the third video B

[0107] In one implementation, the terminal fuses at least two third videos according to the first weight. Specifically, each image frame in the at least two third videos is fused one by one. The image fusion method can refer to the method in the related art, and the present disclosure will not repeat it.

[0108] The present invention performs video colorization processing based on at least two video colorization models corresponding to the first video, and fuses at least two third videos obtained after the processing to obtain a second video. Since different videos are used to perform video colorization processing on different types of videos, and the colorization results of multiple video colorization models are referred to at the same time, each result is fused according to the weight, which can improve the effect of video colorization, and the video effect of processing complex scenes is remarkable.

[0109] In some implementations, the terminal may determine at least two video coloring models corresponding to the first video from a preconfigured video coloring model set in the following manner.

[0110] In one implementation, the terminal may randomly extract at least two video colorization models from pre-configured video colorization models.

[0111] In one embodiment, the terminal can calculate the similarity between the first image frame and the training data set corresponding to the video colorization model, and determine at least two video colorization models corresponding to the first video based on the similarity. Exemplarily, if there are at least two similarities greater than or equal to a similarity threshold among the calculated similarities, the video colorization model corresponding to the training data set corresponding to the similarity is determined as the video colorization model corresponding to the first video. If there is only one similarity greater than or equal to the similarity threshold among the calculated similarities, or there is no similarity greater than or equal to the similarity threshold. The terminal can sort the calculated similarities, and determine the video colorization model corresponding to the training data set corresponding to at least two similarities with the highest similarity as the video colorization model corresponding to the first video.

[0112] In some implementations, in the video processing method provided by the present disclosure, the terminal may obtain the first image frame by obtaining any one or more image frames from the first video.

[0113] In some cases, the first image frame may be any image frame in the first video. The terminal arbitrarily selects one of the image frames as the first image frame and determines the video coloring model corresponding to the first image frame. Figure 4a , assuming that the first video includes 5 image frames, Figure 4a Each square in represents an image frame of the first video. The terminal may select any one of the image frames 1 to 5 as the first image frame, and accordingly, the video coloring model may be a video coloring model determined by the terminal according to any one of the image frames 1 to 5. The terminal performs video coloring processing on each image frame in the first video according to the video coloring model corresponding to the first image frame to obtain a second video.

[0114] In other cases, the first image frame may also be any multiple image frames in the first video. The terminal may use the video coloring model corresponding to the first image frame to perform video coloring processing on the first image frame. At the same time, the video coloring model corresponding to the first image frame is also used to perform video coloring processing on image frames adjacent to or close to the first image frame. For example, referring to Figure 4b , the terminal may select image frame 1 and image frame 4 in the first video as the first image frame. The terminal may determine the video coloring models corresponding to image frame 1 and image frame 4, respectively, and perform video coloring on image frame 1 and image frame 4. For image frame 2 and image frame 3 adjacent to image frame 1, the video coloring model corresponding to image frame 1 is used to perform video coloring. For image frame 5 adjacent to image frame 4, the video coloring model corresponding to image frame 4 is used to perform video coloring.

[0115] In other cases, the first image frame may be all image frames of the first video. The terminal may determine the video coloring model corresponding to each image frame in the first video. Figure 4c , the terminal can determine image frames 1 to 5 as the first image frame, and determine their corresponding video colorization models respectively. The terminal uses the video colorization model corresponding to image frame 1 to perform video colorization processing on image frame 1; and uses the video colorization model corresponding to image frame 2 to perform video colorization processing on image frame 2. This disclosure does not give examples one by one.

[0116] It can be understood that the image frames 1 to 5 included in the first video exemplified in this embodiment are only illustrative examples, and the present disclosure does not limit this.

[0117] The present disclosure obtains any one image frame in the first video as the first image frame, and can quickly determine the video colorization model corresponding to the first video, thereby improving the efficiency of video colorization processing on the first video to be processed. The present disclosure obtains any multiple image frames in the first video as the first image frame, and can specifically perform video colorization processing on different image frames in the first video, thereby improving the effect of video colorization processing on the first video to be processed. The present disclosure obtains all image frames in the first video as the first image frame, and can specifically perform video colorization processing on each frame in the first video, effectively improving the effect of video colorization processing on the first video to be processed, but the processing efficiency is low.

[0118] In some embodiments, the present disclosure provides a video colorization method, such as Figure 5 As shown, the following steps are included.

[0119] In step S41, a video colorization model corresponding to the first video is used to perform video colorization processing on the first image frame.

[0120] In step S42, the second image frame is subjected to video colorization processing using the colored first image frame and a pre-configured reference model.

[0121] In step S43, a second video is obtained according to the colored first image frame and the colored second image frame.

[0122] In one embodiment, the terminal may extract a first image frame based on a preconfigured key frame extraction algorithm, where the first image frame is a key frame in the first video. A key frame refers to an image frame in a video that has a representative meaning. For example, the frame in which a key action is located in a motion change of a character or object in a video. The key frame extraction algorithm may be an algorithm based on motion analysis, a method based on video frame clustering, a method based on color or color histogram frame difference, etc.

[0123] In one implementation, the terminal may determine a video coloring model corresponding to the first image frame based on the first image frame. If the first video includes multiple first image frames, each first image frame has its corresponding video coloring model, that is, if the first image frame includes multiple first image frames, the video coloring model corresponding to the first video may also be multiple. The terminal may perform video coloring processing on each first image frame using the video coloring model corresponding to the first image frame.

[0124] In one embodiment, the terminal can use the colored first image frame and a pre-configured reference model to perform video colorization processing on the second image frame. The reference model is a neural network model pre-trained by the terminal, and uses the colored image frame to perform video colorization processing on the uncolored image frame. It can be understood that the input of the video colorization model in the present disclosure is the uncolored image frame A, and the output is the colored image frame A. The input of the reference model is the uncolored image frame C and the colored image frame D. Usually, the image frame C and the image frame D are adjacent image frames. The output of the reference model is the colored image frame C. In other words, the reference model uses the colored image frame to perform video colorization processing on the uncolored image frame.

[0125] Exemplarily, the colored first image frame and the second image frame are input into the reference model, and then the second image frame can be subjected to video colorization processing, wherein the second image frame refers to an image frame other than the first image frame in the first video.

[0126] In one implementation, after the terminal performs video colorization processing on the first image frame and the second image frame respectively, a second video is obtained, which is the first video after the video colorization processing.

[0127] The present disclosure extracts key frames from a first video to be processed, and then performs video colorization processing on the first video according to a video colorization model corresponding to the key frames, so as to achieve more targeted video colorization processing on the first video and achieve better effect of the video colorization processing.

[0128] The present disclosure uses a pre-configured reference model to perform video colorization processing on image frames other than key frames, which can reduce the amount of calculation and save power consumption. In addition, the present disclosure can effectively improve the efficiency of video colorization processing by processing key frames and image frames other than key frames at the same time.

[0129] In some embodiments, the present disclosure provides a video colorization method, comprising: utilizing N consecutive image frames in a first video and a preconfigured reference model to perform video colorization processing on the next second image frame adjacent to the N consecutive image frames, until the video colorization processing is completed on each second image frame, where N is a positive integer.

[0130] In one implementation, N is 1, and the terminal performs video colorization processing on the second image frame in the first video with a certain image frame as a reference each time. The terminal may perform video colorization processing on the second image frame with the colored first image frame as a reference, or may perform video colorization processing on the uncolored second image frame with the colored second image frame as a reference.

[0131] Exemplarily, the terminal uses the colored first image frame as a reference to perform video colorization processing on the next second image frame adjacent to the first image frame; the terminal uses the colored second image frame as a reference to perform video colorization processing on the next second image frame adjacent to the colored second image frame, until the video colorization processing is completed for each second image frame in the first video. Figure 6 , assuming that the first video includes 5 image frames, among which image frame 1 and image frame 4 are the first image frames determined by the terminal, and image frame 2, image frame 3 and image frame 5 are the second image frames other than the first image frame in the terminal. For image frame 1 and image frame 4, the terminal can respectively use the corresponding video colorization models to perform video colorization processing. For image frame 2 adjacent to image frame 1, the terminal can input the colored image frame 1 and the uncolored image frame 2 into the reference model pre-configured by the terminal to obtain the colored image frame 2; for image frame 3 adjacent to image frame 2, the terminal can input the colored image frame 2 and the uncolored image frame 3 into the reference model pre-configured by the terminal to obtain the colored image frame 3; for image frame 5 adjacent to image frame 4, the terminal can input the colored image frame 4 and the uncolored image frame 5 into the reference model pre-configured by the terminal to obtain the colored image frame 5.

[0132] In one embodiment, N is a positive integer greater than 1. The terminal uses a plurality of continuous image frames as a reference each time to perform video colorization processing on the second image frame in the first video. For example, the terminal may use the plurality of first image frames after colorization as a reference to perform video colorization processing on the second image frame. The terminal may also use at least one of the plurality of first image frames after colorization and the second image frame after colorization as a reference to perform video colorization processing on the uncolored second image frame. The terminal may also use the plurality of second image frames after colorization to perform video colorization processing on the uncolored second image frame.

[0133] Exemplarily, assume that N is 2. The terminal uses the colored first image frame as a reference to perform video colorization processing on the second image frame adjacent to the first image frame. The first image frame is two consecutive image frames. The terminal selects the first image frame adjacent to the second image frame in the colored first image frame, uses the first image frame and the colored second image frame as a reference, and performs video colorization processing on the next image frame adjacent to the colored second image frame. The terminal uses the two adjacent colored second image frames as a reference to perform video colorization processing on the next second image frame of the two adjacent colored second image frames. Reference Figure 7, assuming that the first video includes 5 image frames, wherein image frame 1 and image frame 2 are the first image frames determined by the terminal, and image frame 3, image frame 4 and image frame 5 are the second image frames other than the first image frame in the terminal. The terminal performs video colorization processing on image frame 1 and image frame 2 using a video colorization model. It can be understood that the video colorization model can be a video colorization model corresponding to image frame 1, or a video colorization model corresponding to image frame 2. The terminal inputs the colored image frame 1 and image frame 2 and the uncolored image frame 3 into a preconfigured reference model. Using the colored image frame 1 and image frame 2 as a reference, a colored image frame 3 is obtained. The terminal inputs the colored image frame 2, the colored image frame 3 and the uncolored image frame 4 into a preconfigured reference model to obtain a colored image frame 4. The terminal inputs the colored image frame 3, the colored image frame 4 and the uncolored image frame 5 into a preconfigured reference model to obtain a colored image frame 5.

[0134] It can be understood that the image frames 1 to 5 included in the first video and the video coloring method cited in this embodiment are only illustrative examples, and the present disclosure does not limit this.

[0135] The present disclosure can improve the efficiency of video colorization processing by using N consecutive image frames in the first video to perform video colorization processing on the second image frame. If N is an integer greater than 1, the terminal performs more detailed video colorization processing on the second image frame, and the effect of video colorization processing is better.

[0136] In some embodiments, the second video may include a third image frame and a fourth image frame, wherein the third image frame is one or more image frames in the first video. The fourth image frame is an image frame other than the third image frame in the second video. The present disclosure provides a video processing method, comprising: performing video smoothing processing on the fourth image frame according to the third image frame.

[0137] In one implementation, the terminal may determine a third image frame. The third image frame is one or more image frames in the first video. For example, the terminal may arbitrarily select one or more image frames as the third image frame. Exemplarily, the terminal may select the first image frame or the last image frame in the second video as the third image frame in chronological order. For another example, the terminal may use a key frame extraction algorithm to extract a key frame in the second video, and use the extracted key frame as the third image frame.

[0138] In one implementation, the terminal may perform video smoothing processing on other image frames in the second video except the third image frame according to the determined third image frame. In the present disclosure, image frames in the second video except the third image frame are referred to as fourth image frames.

[0139] In this embodiment, the terminal performs video smoothing processing on the fourth image frame based on the third image frame, including: the terminal can perform image registration on the third image frame based on the fourth image frame. Image registration is also called image alignment. Image registration refers to the process of matching and superimposing two or more images at different times, different sensors or under different conditions. The terminal fuses the third image frame and the fourth image frame after image registration to obtain a fused fourth image frame. The third image frame and the fourth image frame after image fusion constitute the second video after video smoothing processing.

[0140] The present disclosure performs video smoothing on the second video to increase the stability of the image, so that the second video after smoothing is more natural and more realistic.

[0141] In some implementations, the terminal provides a video processing method, such as Figure 8 As shown, the following steps are included.

[0142] In step S51, the fourth image frame is used to perform image registration on M consecutive image frames adjacent to the fourth image frame, where M is a positive integer.

[0143] In step S52, the second weight corresponding to the fourth image frame and each of the M consecutive image frames after image registration is determined.

[0144] In step S53, based on the second weights, the fourth image frame and the M consecutive image frames after image registration are fused to obtain a fourth image frame after image fusion.

[0145] In step S54, until the image fusion is completed for each fourth image frame in the second video, the second video after video smoothing is obtained.

[0146] In one embodiment, M is 1, and the terminal performs image registration and image fusion on the fourth image frame based on a certain image frame each time. Exemplarily, the terminal performs image registration on the third image frame based on the next fourth image frame adjacent to the third image frame. That is, the third image frame is aligned to the next fourth image frame adjacent to the third image frame. The terminal performs image fusion on the third image frame after image registration. The terminal performs image registration on the fourth image frame after image fusion based on the next fourth image frame adjacent to the fourth image frame after image fusion. The terminal performs image fusion on the fourth image frame after image fusion and image registration with the next adjacent fourth image frame until image fusion of each fourth image frame in the second video is completed.

[0147] For example, reference Fig. 9, assuming that the second video includes 5 image frames. Among them, image frame 1 can be the third image frame determined by the terminal, and image frames 2 to 5 are the fourth image frames in the second video except the third image frame. The terminal can perform image registration and image fusion operations on image frame 1 and image frame 2 to obtain a fused image frame 2. The terminal can perform image registration and image fusion operations on the fused image frame 2 and image frame 3 to obtain a fused image frame 3. Among them, taking image frame 1 and image frame 2 as an example, the operation process of performing image registration and image fusion can refer to Fig.10 .like Fig.10 As shown, the terminal can perform image registration on image frame 1 based on image frame 2. That is, using image frame 2 as a standard, image frame 1 is aligned with image frame 2. The terminal fuses image frame 1 and image frame 2 after image registration to obtain image frame 2 after image fusion.

[0148] It can be understood that the image frames 1 to 5 included in the second video and the video smoothing processing method exemplified in this embodiment are only illustrative examples, and the present disclosure does not limit this. For example, the third image frame determined by the terminal can also be image frame 5, and image frame 5 and image frame 4 are imaged and fused to obtain image frame 4 after image fusion. The present disclosure does not exemplify one by one.

[0149] In one implementation, M is a positive integer greater than 1. The terminal performs image registration and image fusion on the fourth image frame based on multiple adjacent image frames each time. Fig.11 , assuming that the second video includes 5 image frames. Among them, image frame 1 and image frame 2 are the third image frames determined by the terminal, and image frames 3 to 5 are the fourth image frames in the second video except the third image frame. The terminal can perform image registration and image fusion operations on image frames 1, 2 and 3 to obtain image frame 3 after image fusion. The terminal can perform image registration and image fusion operations on image frame 2, image frame 3 after image fusion, and image frame 4 to obtain image frame 4 after image fusion. Exemplarily, the terminal performs image registration on image frames 1 and 2 based on image frame 3. Image frame 1 after image registration, image frame 2 after image registration, and image frame 3 are fused to obtain fused image frame 3.

[0150] In one implementation, the terminal may determine a second weight, wherein the second weight refers to the proportion of each image frame in the fusion.

[0151] For example, the second weight may be pre-set, and illustratively, when the image frame 1 and the image frame 2 after image registration are fused, the second weight may be (X, Y), wherein X represents the ratio of the image frame 1 after image registration to the fused image frame 2, and Y represents the ratio of the image frame 2 to the fused image frame 2.

[0152] For another example, the terminal may also obtain a second weight based on the consistency of the second video. The consistency may be understood as whether the objects in the second video move frequently. If the objects in the second video move frequently, it may be considered that the consistency of the second video is low. On the contrary, if the objects in the second video move less frequently, it may be considered that the consistency of the second video is high. Exemplarily, the terminal may determine the consistency of the second video by obtaining a color distribution consistency (CDC) indicator of the second video. Alternatively, the terminal may determine whether the objects in the second video move frequently by other feasible methods, which is not limited in the present disclosure.

[0153] In this implementation, the fusion of image frame 1 and image frame 2 after image registration is taken as an example. If the consistency of the second video is high, then in the second weight, the image frame 1 after image registration accounts for a smaller proportion of the fused image frame 2, and the image frame 2 accounts for a larger proportion of the fused image frame 2, so as to prevent the occurrence of the smear phenomenon. On the contrary, if the consistency of the second video is low, then in the second weight, the image frame 1 after image registration accounts for a larger proportion of the fused image frame 2, and the image frame 2 accounts for a smaller proportion of the fused image frame 2.

[0154] The present disclosure performs video smoothing on the second video to increase the stability between image frames, so that the smoothed second video is more natural and more realistic. If M is an integer greater than 1, the terminal performs video smoothing on the second image frame more meticulously, and the effect of video smoothing is better.

[0155] In some embodiments, the video colorization models in the video colorization model set correspond to different training data sets. In the video processing method provided by the present disclosure, if the video colorization model set includes a first video colorization model and a second video colorization model, the second training data set corresponding to the second video colorization model can be obtained based on the first training data set corresponding to the first video colorization model.

[0156] In one implementation, the terminal may pre-acquire a first training data set, and the first training data set includes a large number of uncolored videos. Exemplarily, the terminal may acquire a large number of color videos, perform video decolorization processing on the color videos, and obtain black-and-white videos. Black-and-white videos are also uncolored videos, and a large number of black-and-white videos constitute the first training data set. The terminal may filter the first training data set to obtain a second training data set. The terminal may filter the first training data set based on a pre-set rule to obtain the second training data set.

[0157] In one embodiment, the terminal may train an untrained deep learning model using the first training data set to obtain a first video colorization model. The deep learning model may be any one of a convolutional neural network, a generative adversarial network, and a variational autoencoder. Exemplarily, the terminal may input the first training data set into the untrained deep learning model to obtain an output result. The terminal may train the untrained deep learning model using a loss function between the color video corresponding to the first training data set and the output result to obtain the first video colorization model.

[0158] It is understandable that the terminal can use the second training data set to train the second video colorization model. The specific training method refers to the training method of the first video colorization model mentioned above, and the present disclosure will not repeat it here. The user can also use the training method in other related technologies to train the first video colorization model and the second video colorization model, which is not limited in the present disclosure.

[0159] The present disclosure obtains a training data set of another video colorization model by screening a training data set of a certain video colorization model, thereby reducing the time for obtaining the training data set and improving efficiency.

[0160] In some embodiments, Fig.12 As shown, the terminal can train the reference model in the following manner.

[0161] In step S61, a third training data set is acquired, where the third training data set includes a plurality of groups of adjacent image frames.

[0162] In step S62, image registration is performed on the N+1th image frame among the adjacent image frames using the N image frames among the adjacent image frames.

[0163] In step S63, video decolorization processing is performed on the N+1th image frame.

[0164] In step S64, the first N image frames after image registration and the N+1th image frame after decolorization are input into an untrained reference model to obtain an output result.

[0165] In step S65, the untrained reference model is trained using the loss function between the output result and the N+1th image frame without decolorization.

[0166] In one implementation, the terminal may obtain a third training data set, which includes multiple groups of adjacent image frames. The terminal may crawl a large number of random video clips from the network, and extract several adjacent image frames from each video clip as the third training data set.

[0167] In one embodiment, reference Fig.13, the terminal can use the N image frames in the third training data set to perform image registration on the N+1th image frame, and perform video decolorization processing on the N+1th image frame. Among them, video decolorization processing can be understood as converting a color video into a black and white video. The terminal trains an untrained reference model with the N image frames after image registration and the N+1th image frame after decolorization processing. Exemplarily, the N image frames after image registration and the N+1th image frame after decolorization processing are input into the untrained reference model to obtain an output result. The untrained reference model is trained using the loss function between the input result and the non-decolorized N+1th image frame, so that the output result of the reference model is the same or similar to the non-decolorized N+1th image frame.

[0168] The present invention obtains a reference model through training, thereby performing video colorization processing on a first video according to the reference model to obtain a second video, thereby improving the efficiency of video colorization processing.

[0169] This disclosure will illustrate the video colorization method with an exemplary embodiment. Fig.14 As shown, the video colorization method includes the following steps.

[0170] In step S71, key frames are extracted from the video to be processed.

[0171] In one implementation, after the terminal obtains the video to be processed, it uses a key frame extraction algorithm to extract key frames in the video to be processed.

[0172] In step S72, the key frame is matched with the third video colorization model.

[0173] In one embodiment, the terminal may match the key frame with the third video coloring model. The third video coloring model is a video coloring model for performing video coloring on videos of common scenes. Exemplarily, the terminal may crawl a large number of random video clips from the network to obtain a fourth training data set. The fourth training data set includes videos of common scenes and videos of uncommon scenes. The terminal may filter videos of common scenes from the fourth training data set. The screening method may be to extract features from the image frames of the videos in the fourth training data set, identify the extracted features, and determine whether the video is a video of common scenes, thereby obtaining the third training data set. For example, if the video of an animal is pre-set as a video of a common scene. By extracting and identifying features from the image frames, it is determined that the video is a video of an animal, so that the video can be determined to be a video of a common scene. Alternatively, the terminal may also input the fourth training data set into a pre-trained judgment model, and the judgment model is used to determine whether the input video is a video of a common scene, thereby obtaining the third training data set. The terminal pre-trains the third video coloring model using the third training data set.

[0174] In this embodiment, the terminal matches the key frame with the image frame of any video clip in the third training data set to obtain a similarity, and the similarity is used to determine whether the match is successful. For example, if the similarity is greater than the similarity threshold, the match can be considered successful, and if the similarity is less than the similarity threshold, the match can be considered unsuccessful.

[0175] In step S73, in response to successful matching, the key frame is subjected to video colorization processing using the third video colorization model.

[0176] In step S74, in response to the matching being unsuccessful, the key frame is subjected to video colorization processing using the fourth video colorization model.

[0177] In one implementation, if the match is successful, the terminal may perform video colorization processing on the key frame using the third video colorization model. If the match is unsuccessful, it may be considered that the video to be processed includes an uncommon scene, and the terminal may perform video colorization processing on the key frame using the fourth video colorization model. The fourth video colorization model is a video colorization model obtained by training the terminal using the fourth training data set. The fourth video colorization model is a video colorization model that can perform video colorization processing on videos of common scenes and videos of uncommon scenes.

[0178] In step S75, the reference model is used to perform video colorization processing on non-key frames in the video to be processed.

[0179] In one implementation, the terminal may use the colored key frame as a reference, input the non-key frame adjacent to the key frame and the colored key frame into a reference model, and obtain the colored non-key frame, wherein the non-key frame is an image frame other than the key frame in the video to be processed.

[0180] In step S76, video smoothing is performed on the processed video.

[0181] In one implementation, the terminal may perform video smoothing on the processed video. For example, starting from the first image frame of the processed video, the image frames are sequentially smoothed. For example, taking image frame 1 and image frame 2 as an example, the terminal performs image registration on image frame 1 based on image frame 2, and performs image fusion on image frame 1 and image frame 2 after image registration to obtain image frame 2 after image fusion.

[0182] The present disclosure determines a video colorization model corresponding to a video to be processed in a video colorization model set, and then uses the video colorization model to perform colorization processing on the video to be processed, so as to improve the efficiency and effect of colorization processing on the video.

[0183] It should be noted that those skilled in the art can understand that the various implementation methods / embodiments involved in the embodiments of the present disclosure can be used in conjunction with the aforementioned embodiments or can be used independently. Whether used alone or in conjunction with the aforementioned embodiments, the implementation principle is similar. In the implementation of the present disclosure, some embodiments are described in terms of implementation methods used together. Of course, those skilled in the art can understand that such examples are not limitations of the embodiments of the present disclosure.

[0184] Based on the same concept, an embodiment of the present disclosure also provides a video processing device.

[0185] It is understandable that, in order to realize the above functions, the video processing device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0186] Fig.15 is a block diagram of a video processing device 100 according to an exemplary embodiment. Fig.15 The device includes an acquisition module 101, a determination module 102, a processing module 103 and a training module 104.

[0187] Among them, the acquisition module 101 is used to obtain a first video, and the first video is a video to be processed. The determination module 102 is used to determine the video colorization model corresponding to the first video based on the video colorization model set, wherein the video colorization model set includes multiple video colorization models, and different video colorization models are used to perform video colorization processing on different types of videos, and video colorization processing means converting the black and white image frames of the video into color image frames. The processing module 103 is used to perform video colorization processing on the first video using the video colorization model corresponding to the first video to obtain a second video.

[0188] In one embodiment, the acquisition module 101 is further used to acquire a first image frame, where the first image frame is one or more image frames of the first video. A training data set corresponding to the video colorization model is acquired. The processing module 103 is further used to calculate the similarity between the first image frame and the training data set. The determination module 102 determines the video colorization model corresponding to the first video based on a preconfigured video colorization model set in the following manner: Determine the video colorization model corresponding to the first video based on the similarity.

[0189] In one embodiment, the determination module 102 determines the video colorization model corresponding to the first video based on similarity in the following manner, including: in response to the similarity being greater than or equal to a similarity threshold, determining the video colorization model corresponding to the training data set corresponding to the similarity as the video colorization model corresponding to the first video.

[0190] In one embodiment, the determination module 102 determines the video coloring model corresponding to the first video based on the video coloring model set in the following manner: at least two video coloring models corresponding to the first video are determined based on the video coloring model set. The processing module 103 performs video coloring on the first video using the video coloring model corresponding to the first video to obtain a second video in the following manner: based on each of the at least two video coloring models, the first video is colorized to obtain a third video. Based on the similarity between the first image frame and the training data set corresponding to each of the at least two video coloring models, the first weight of the third video corresponding to each video coloring model is determined. Using the first weight of the third video corresponding to each video coloring model, the third videos corresponding to each video coloring model are fused to obtain a second video.

[0191] In one embodiment, the first image frame is a key frame. The processing module 103 performs video colorization processing on the first video using a video colorization model corresponding to the first video in the following manner: the first image frame is subjected to video colorization processing using a video colorization model corresponding to the first video. The second image frame is subjected to video colorization processing using the colored first image frame and a pre-configured reference model. The second image frame is an image frame other than the first image frame in the first video. The reference model is used to perform video colorization processing on an uncolored image frame using the colored image frame. The second video is obtained based on the colored first image frame and the colored second image frame.

[0192] In one embodiment, the processing module 103 performs video colorization processing on the second image frame using the colored first image frame and the pre-configured reference model in the following manner: using N consecutive image frames in the first video and the pre-configured reference model, the next second image frame adjacent to the N consecutive image frames is subjected to video colorization processing until the video colorization processing is completed for each second image frame, where N is a positive integer. The N image frames include any one or more of the following: the colored first image frame. The colored second image frame.

[0193] In one embodiment, the second video includes a third image frame and a fourth image frame, the third image frame is one or more image frames in the second video, and the fourth image frame is an image frame in the second video except the third image frame. The processing module 103 is also used to: perform video smoothing processing on the fourth image frame according to the third image frame to obtain a fourth video, and the fourth video is the second video after video smoothing processing.

[0194] In one embodiment, the processing module 103 performs video smoothing processing on the fourth image frame according to the third image frame in the following manner: using the fourth image frame, image registration is performed on M consecutive image frames in the second video, and the fourth image frame is the next image frame adjacent to the M consecutive image frames. Determine the second weights of the M consecutive image frames after image registration and the next adjacent fourth image frame. Based on the second weight, the M consecutive image frames after image registration and the next adjacent fourth image frame are image fused to obtain the fourth image frame after image fusion, until image fusion is completed for each fourth image frame in the second video. A fourth video is obtained based on the third image frame and the fourth image frame after image fusion. Among them, the first N image frames in the second video include one or more of the following: the third image frame. The fourth image frame after image fusion.

[0195] In one embodiment, it is characterized in that the video colorization model set includes a first video colorization model and a second video colorization model. The first video colorization model is trained based on a first training data set, and the second video colorization model is trained based on a second training data set, wherein the second training data set is determined based on the first training data set.

[0196] In one embodiment, the device also includes: a training module 104, which is used to train a reference model in the following manner: obtain a third training data set. The third training data set includes multiple groups of adjacent image frames. According to a preset order, use N image frames in the adjacent image frames to perform image registration on the N+1th image frame in the adjacent image frames. Perform video decolorization processing on the N+1th image frame, and video decolorization processing refers to converting a color video into a black and white video. The first N image frames after image registration and the N+1th image frame after decolorization are input into an untrained reference model to obtain an output result. The untrained reference model is trained using a loss function between the output result and the N+1th image frame that has not been decolorized.

[0197] Fig.16 is a block diagram of a video processing device 200 according to an exemplary embodiment.

[0198] like Fig.16As shown, apparatus 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0199] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0200] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0201] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0202] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0203] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0204] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0205] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0206] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0207] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0208] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0209] The present disclosure determines a video colorization model corresponding to a video to be processed in a video colorization model set, and then uses the video colorization model to perform colorization processing on the video to be processed, so as to improve the efficiency and effect of colorization processing on the video.

[0210] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0211] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0212] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0213] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0214] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

Claims

1. A video processing method, characterized in that: The video processing method comprises: Acquire a first video, where the first video is a video to be processed; Determining a video colorization model corresponding to the first video based on a video colorization model set, wherein the video colorization model set includes a plurality of video colorization models, and different video colorization models among the plurality of video colorization models are used to perform video colorization processing on different types of videos, and the video colorization processing represents converting a black and white image frame of a video into a color image frame; The first video is subjected to video colorization processing using a video colorization model corresponding to the first video to obtain a second video.

2. The method according to claim 1, characterized in that The determining, based on the video colorization model set, a video colorization model corresponding to the first video includes: Acquire a first image frame, where the first image frame is one or more image frames of the first video; Acquire a training data set corresponding to each video colorization model in the video colorization model set, wherein the training data set includes a training data frame; Determining the similarity between the first image frame and the training data set based on the similarity between the first image frame and each training data frame in the training data set; Based on the similarity between the first image frame and the training data set, a video colorization model corresponding to the first video is determined.

3. The method according to claim 2, characterized in that The determining, based on the similarity between the first image frame and the training data set, a video colorization model corresponding to the first video includes: The video colorization model corresponding to the training data set whose similarity is greater than or equal to the similarity threshold is determined as the video colorization model corresponding to the first video.

4. The method according to claim 1, characterized in that: The video colorization model corresponding to the first video includes at least two video colorization models; The step of performing video colorization processing on the first video using a video colorization model corresponding to the first video to obtain a second video includes: Based on each of the at least two video colorization models, coloring the first video respectively to obtain a plurality of third videos; Based on the similarity between the first image frame and the training data set corresponding to each of the at least two video colorization models, the multiple third videos are fused to obtain the second video.

5. The method according to claim 2, characterized in that: The first image frame is a key frame; The step of performing video colorization processing on the first video by using the video colorization model corresponding to the first video includes: Performing video colorization processing on the first image frame using a video colorization model corresponding to the first video; Using the colored first image frame and the reference model, performing video colorization processing on a second image frame, where the second image frame is an image frame other than the first image frame in the first video, and the reference model is used to perform video colorization processing on an uncolored image frame using the colored image frame; The second video is obtained according to the colored first image frame and the colored second image frame.

6. The method according to claim 5, characterized in that The step of performing video colorization processing on the second image frame by using the colored first image frame and the reference model includes: Using N consecutive image frames in the first video and a reference model, perform video colorization processing on the next second image frame adjacent to the N consecutive image frames until the video colorization processing is completed on each second image frame, where N is a positive integer; The N image frames include any one or more of the following: The first image frame after colorization; The second image frame after colorization.

7. The method according to claim 1, characterized in that The second video includes a third image frame and a fourth image frame, the third image frame is one or more image frames in the second video, and the fourth image frame is an image frame in the second video except the third image frame, and the method further includes: The fourth image frame is subjected to video smoothing processing according to the third image frame to obtain a second video after the video smoothing processing.

8. The method according to claim 7, characterized in that The step of performing video smoothing processing on the fourth image frame according to the third image frame to obtain a second video after the video smoothing processing includes: Using the fourth image frame, performing image registration on M consecutive image frames adjacent to the fourth image frame, where M is a positive integer; Determine a second weight corresponding to each image frame in the fourth image frame and the M consecutive image frames after image registration; Based on each of the second weights, fusing the fourth image frame with the M consecutive image frames after the image registration to obtain a fourth image frame after image fusion; until image fusion is completed for each of the fourth image frames in the second video to obtain a second video after video smoothing; The M consecutive image frames in the second video include one or more of the following: a third image frame; The fourth image frame after image fusion.

9. The method according to any one of claims 1 to 8, characterized in that The video colorization model set includes a first video colorization model and a second video colorization model; The first video colorization model is trained based on a first training data set, and the second video colorization model is trained based on a second training data set, wherein the second training data set is determined based on the first training data set.

10. The method according to any one of claims 1 to 8, characterized in that The reference model is trained in the following way: Acquire a third training data set; the third training data set includes a plurality of groups of adjacent image frames; perform image registration on the N+1th image frame in the adjacent image frames using N image frames in a preset order; Performing video decolorization processing on the N+1th image frame, wherein the video decolorization processing refers to converting a color video into a black-and-white video; Input the first N image frames after image registration and the N+1th image frame after decolorization into the untrained reference model to obtain the output result; The untrained reference model is trained using a loss function between the output result and the (N+1)th image frame that has not been decolorized.

11. A video processing device, characterized in that: The video processing device comprises: An acquisition module, used for acquiring a first video, where the first video is a video to be processed; a determination module, configured to determine a video colorization model corresponding to the first video based on a video colorization model set, wherein the video colorization model set includes a plurality of video colorization models, and different video colorization models among the plurality of video colorization models are used to perform video colorization processing on different types of videos, wherein the video colorization processing represents converting a black and white image frame of a video into a color image frame; A processing module is used to perform video colorization processing on the first video using a video colorization model corresponding to the first video to obtain a second video.

12. The device according to claim 11, characterized in that The acquisition module is further used to acquire a first image frame, which is one or more image frames of the first video; acquire a training data set corresponding to each video colorization model in the video colorization model set, wherein the training data set includes a training data frame; The processing module is further used to determine the similarity between the first image frame and the training data set based on the similarity between the first image frame and each training data frame in the training data set; The determination module determines the video colorization model corresponding to the first video based on the video colorization model set in the following manner: Based on the similarity between the first image frame and the training data set, a video colorization model corresponding to the first video is determined.

13. The device according to claim 12, characterized in that The determination module determines the video colorization model corresponding to the first video based on the similarity in the following manner, including: The video colorization model corresponding to the training data set whose similarity is greater than or equal to the similarity threshold is determined as the video colorization model corresponding to the first video.

14. The device according to claim 11, characterized in that The video colorization model corresponding to the first video includes at least two video colorization models; The determination module determines the video colorization model corresponding to the first video based on the video colorization model set in the following manner: Determining at least two video colorization models corresponding to the first video based on a video colorization model set; The processing module performs video colorization processing on the first video using a video colorization model corresponding to the first video to obtain a second video in the following manner: Based on each of the at least two video colorization models, coloring the first video respectively to obtain a plurality of third videos; Based on the similarity between the first image frame and the training data set corresponding to each of the at least two video colorization models, the multiple third videos are fused to obtain the second video.

15. The device according to claim 12, characterized in that The first image frame is a key frame; The processing module performs video colorization processing on the first video using the video colorization model corresponding to the first video in the following manner: Performing video colorization processing on the first image frame using a video colorization model corresponding to the first video; Using the colored first image frame and the reference model, performing video colorization processing on a second image frame, where the second image frame is an image frame other than the first image frame in the first video, and the reference model is used to perform video colorization processing on an uncolored image frame using the colored image frame; The second video is obtained according to the colored first image frame and the colored second image frame.

16. The device according to claim 15, characterized in that The processing module performs video colorization processing on the second image frame by using the colored first image frame and the reference model in the following manner: Using N consecutive image frames in the first video and a reference model, perform video colorization processing on the next second image frame adjacent to the N consecutive image frames until the video colorization processing is completed on each second image frame, where N is a positive integer; The N image frames include any one or more of the following: The first image frame after colorization; The second image frame after colorization.

17. The device according to claim 11, characterized in that The second video includes a third image frame and a fourth image frame, the third image frame is one or more image frames in the second video, and the fourth image frame is an image frame other than the third image frame in the second video, and the processing module is further used for: The fourth image frame is subjected to video smoothing processing according to the third image frame to obtain a second video after the video smoothing processing.

18. The device according to claim 17, characterized in that The processing module performs video smoothing processing on the fourth image frame according to the third image frame in the following manner: Using the fourth image frame, performing image registration on M consecutive image frames adjacent to the fourth image frame, where M is a positive integer; Determine a second weight corresponding to each image frame in the fourth image frame and the M consecutive image frames after image registration; Based on each of the second weights, fusing the fourth image frame with the M consecutive image frames after the image registration to obtain a fourth image frame after image fusion; until image fusion is completed for each of the fourth image frames in the second video to obtain a second video after video smoothing; The M consecutive image frames in the second video include one or more of the following: a third image frame; The fourth image frame after image fusion.

19. The device according to any one of claims 11 to 18, characterized in that The video colorization model set includes a first video colorization model and a second video colorization model; The first video colorization model is trained based on a first training data set, and the second video colorization model is trained based on a second training data set, wherein the second training data set is determined based on the first training data set.

20. The device according to any one of claims 11 to 18, characterized in that: The device also includes: The training module is used to train the reference model in the following manner: Acquire a third training data set; the third training data set includes a plurality of groups of adjacent image frames; perform image registration on the N+1th image frame in the adjacent image frames using N image frames in a preset order; Performing video decolorization processing on the N+1th image frame, wherein the video decolorization processing refers to converting a color video into a black-and-white video; Input the first N image frames after image registration and the N+1th image frame after decolorization into the untrained reference model to obtain the output result; The untrained reference model is trained using a loss function between the output result and the (N+1)th image frame that has not been decolorized.

21. An electronic device, characterized in that: include: A memory for storing instructions; as well as A processor, configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 10.

22. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed by the processor, the method according to any one of claims 1 to 10 is executed.

Images