Video processing methods and apparatus

Abstract

This application provides a video processing method, the method comprising: acquiring a first video, the first video including a jittery low bit depth video; converting the first video into a second video using a jitter-free model, the second video including a jitter-free high bit depth video, wherein the first bit depth of the low bit depth video is lower than the second bit depth of the high bit depth video; wherein the jitter-free model includes a trained deep neural network model, the deep neural network model being used for jitter-free and bit depth conversion. This application also provides a video processing apparatus, a computer device, and a computer-readable storage medium. The technical solution provided by this application can convert jittery low bit depth video into jitter-free high bit depth video, jitter reduction will reduce the bit rate to a certain extent, and bit depth conversion will bring a certain degree of image quality improvement, enhancing the viewing experience.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video processing method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] With the development of computer technology, electronic products such as mobile phones, computers, and tablets have become everyday necessities. Users' needs for processing media information such as audio and video are becoming increasingly diverse, such as converting low-bit-depth video to high-bit-depth video.

[0003] Because most devices on the market only support low bit depth video, dithering is added during the compression (preprocessing) stage of the video source to optimize color transitions, allowing the video to exhibit subtle color gradations similar to those in high bit depth videos, even at low bit depth. However, with the emergence and increasing popularity of monitors supporting high bit depth video, the demand for high bit depth video has become increasingly apparent. At high bit depth, excellent color transitions and picture quality can be achieved without dithering. Currently, many low bit depth videos with dithering lack corresponding high bit depth versions. Therefore, how to achieve high-quality conversion from low bit depth videos with dithering to high bit depth videos without dithering, and improve picture quality, has become a key technical problem to be solved.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0005] The purpose of this application is to provide a video processing method, apparatus, computer device, and computer-readable storage medium to solve the above-mentioned problems.

[0006] One aspect of this application provides a video processing method, the method comprising:

[0007] Acquire the first video, which includes a low-resolution video with shaky graphics;

[0008] The first video is converted into a second video using a stabilization model. The second video includes a stabilized high bit depth video, where the first bit depth of the low bit depth video is lower than the second bit depth of the high bit depth video. The stabilization model includes a trained deep neural network model used for stabilization and bit depth conversion.

[0009] Optionally, the method further includes:

[0010] Once the de-jitter model is obtained, it is accelerated and optimized to obtain the target format file.

[0011] The target format file is associated with the inference engine so that the debouncing model can be run by calling the inference engine.

[0012] Optionally, the method further includes:

[0013] The target filter is obtained by encapsulating the inference engine associated with the target format file using a custom filter.

[0014] Optionally, the deep neural network model is trained through the following operations:

[0015] The sample input image is input into the deep neural network model to obtain the predicted image; wherein, the sample input image is a low-bit depth image with jitter.

[0016] Obtain the average absolute error of pixels between the predicted image and the sample reference image; wherein, the sample reference image is a high bit depth jitter-free image corresponding to the sample input image;

[0017] Obtain the total variation loss of the predicted image;

[0018] The deep neural network model is optimized based on the mean absolute difference and the total variation loss.

[0019] Optionally, the method further includes:

[0020] Obtain a sample video group, which includes an input video and a reference video corresponding to the same video. The input video is a low-bit-depth sample video with jitter, and the reference video is a high-bit-depth sample video without jitter.

[0021] Multiple low-bit deep frames are obtained from the input video at a preset number of frame intervals, and multiple corresponding frames are obtained from the reference video to obtain multiple corresponding high-bit deep frames.

[0022] Image frames conforming to the target size are extracted from each low-bit-depth frame to obtain multiple sample input frames; and

[0023] Image frames conforming to the target size are cropped from the corresponding area of ​​each high-bit-depth frame to obtain multiple sample reference frames.

[0024] Optionally, the acquisition of the sample video group includes:

[0025] Multiple sample video groups are obtained, wherein the multiple input videos corresponding to the multiple sample video groups correspond to several color dithering strategies.

[0026] Optionally, the acquisition of the sample video group includes:

[0027] Acquire multiple high-depth sample videos, which correspond to several content types.

[0028] Based on different color-dithering strategies, the multiple high-bit-depth sample videos are converted into low-bit-depth sample videos with dithering; and

[0029] Based on the multiple high-depth sample videos and the multiple low-depth sample videos, multiple sample video groups are obtained.

[0030] Another aspect of this application provides a video processing apparatus, the apparatus comprising:

[0031] The acquisition module is used to acquire a first video, which includes a low-bit depth video with jitter.

[0032] A conversion module is used to convert the first video into a second video using a shading model. The second video includes a shading high bit depth video, wherein the first bit depth of the low bit depth video is lower than the second bit depth of the high bit depth video. The shading model includes a trained deep neural network model, which is used for shading and bit depth conversion.

[0033] Another aspect of this application provides a computer device, including:

[0034] At least one processor; and

[0035] A memory communicatively connected to the at least one processor; wherein:

[0036] The memory stores computer instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0037] Another aspect of this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.

[0038] The technical solution provided in this application has the following advantages:

[0039] Using a trained deep neural network model for de-jittering and bit depth conversion can minimize color jitter (global and local jitter) in the image and convert the video from low bit depth to high bit depth without compromising image quality.

[0040] The technical solution described in this application can convert low bit depth video with jitter into high bit depth video without jitter. Jitter reduction will reduce the bit rate to a certain extent, and bit depth conversion will bring a certain degree of image quality improvement, thus enhancing the viewing experience. Attached Figure Description

[0041] Figure 1 The diagram illustrates the operating environment of the video processing method according to Embodiment 1 of this application.

[0042] Figure 2 A flowchart illustrating a video processing method according to Embodiment 1 of this application is shown schematically.

[0043] Figure 3 The flowchart illustrating the additional steps of the video processing method according to Embodiment 1 of this application is shown in the schematic diagram.

[0044] Figure 4 This schematically illustrates the model deployment process;

[0045] Figure 5 This schematically illustrates a flowchart of another additional step in the video processing method according to Embodiment 1 of this application;

[0046] Figure 6 This schematically illustrates the model training process;

[0047] Figure 7 This diagram illustrates the sample data acquisition process.

[0048] Figure 8 A block diagram of a video processing apparatus according to Embodiment 2 of this application is schematically shown; and

[0049] Figure 9 A schematic diagram of the hardware architecture of a computer device according to Embodiment 3 of this application is shown. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0051] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0052] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0053] Explanation of terms used in this application:

[0054] Deep Neural Networks (DNNs) are one of the fundamental network structures in deep learning. The neural network layers within a DNN can be categorized into several types, such as input layers, hidden layers, and output layers.

[0055] Inference Engine provides users with services for deploying and inferring artificial intelligence models.

[0056] Dithering: Based on the characteristics of human vision, it uses color dithering in the time or spatial domain to simulate high-precision color using low-precision color chromatography.

[0057] De-jittering: This application relates to the elimination of color jitter in the spatial domain, which refers to restoring an 8-bit deep video with a color depth of 10 bits after spatial color jittering processing into a high-precision video, and eliminating color jitter in the original video.

[0058] Bit depth refers to the number of bits used to store the color of a single pixel in a bitmap or video frame buffer in computer graphics, such as 8 bits or 10 bits. The higher the color bit depth, the more colors are available.

[0059] ffmpeg is an audio and video encoding and decoding tool, as well as a set of audio and video encoding development kits. As an encoding development kit, it provides developers with a rich set of calling interfaces for audio and video processing.

[0060] AVFilter: A library for FFmpeg that provides a general framework for filtering audio, video, subtitles, and other data.

[0061] To facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below:

[0062] In some cases, the captured video source is 10-bit, but since most devices on the market only support 8-bit color depth, and some services only support uploading and playing 8-bit video sources, dithering is added during the compression (preprocessing) stage to optimize color transitions. This allows the video to exhibit as subtle color gradations as possible, even with an 8-bit color depth, as if it were 10-bit. When dithering from 8-bit to 10-bit color, subpixels may borrow colors from neighboring pixels to create new colors when displaying only a portion of the color. When the pixels are magnified, it can be seen that neighboring pixels tend to blend, producing colors that don't actually exist in the image, thus achieving a compensation purpose. Common spatial dithering algorithms include Random Dithering, Ordered Dithering, and Error Diffusion.

[0063] However, with the emergence and gradual popularization of 10-bit displays, and the anticipated future support for 10-bit video uploads and playback, the market demand for native 10-bit video is gradually emerging. In 10-bit conditions, excellent color transitions are no longer necessary. Furthermore, color dithering consumes a significant amount of bitrate and also leads to a certain degree of image quality degradation. Under suitable external conditions, 10-bit video sources will provide a better viewing experience. However, due to various reasons, many videos with jitter lack 10-bit source material, necessitating a smart and fast method for efficient conversion from 8-bit jittery video to 10-bit jitter-free video.

[0064] To address this, this application provides an intelligent solution for eliminating color jitter in video processing. The solution utilizes a deep learning neural network-based method to completely eliminate color jitter, including both global and local jitter, while simultaneously converting the source video from 8-bit to 10-bit without compromising image quality. Finally, the algorithm can be encapsulated as an ffmpeg filter, allowing for direct call to ffmpeg filters to perform video stabilization. This algorithm can achieve a processing speed of 6.5fps on a CPU using OpenVino for 1080p video, including encoding and decoding time. See below for details.

[0065] Figure 1 An environmental operation diagram of a video processing method according to an embodiment of this application is illustrated schematically.

[0066] like Figure 1 As shown in the diagram, the environment includes server 2, network 4, and electronic device 6, wherein:

[0067] Server 2, as a digital streaming media service platform, can consist of one or more computing devices. These devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as emulations of computer systems, operating systems, servers, etc. The computing devices can load virtual machines based on virtual images and / or other data that define specific software used for emulation (e.g., operating systems, dedicated applications). As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.

[0068] Server 2 can provide digital streaming media services, such as providing audio and video data to electronic devices. Server 2 can be configured to communicate with electronic devices 6 via network 4. Electronic device 6 can be any type of computing device, such as mobile devices, tablets, laptops, virtual reality devices, gaming devices, set-top boxes, in-vehicle terminals, smart TVs, or headphones. In some embodiments, it can also be a virtual terminal.

[0069] Electronic device 6 may include computing devices using operating systems such as Windows and Android, such as mobile devices, tablets, laptops, virtual reality devices, gaming devices, set-top boxes, in-vehicle terminals, and smart TVs.

[0070] Electronic device 6 can be used to run video editing or playback programs to provide video processing and playback functions.

[0071] The following describes a video processing scheme using electronic device 6 as the execution entity through several embodiments. It should be noted that this scheme can also utilize server 2, which generates mixed audio and returns it to electronic device 6.

[0072] Example 1

[0073] Figure 2 A flowchart illustrating a video processing method according to Embodiment 1 of this application is shown schematically.

[0074] like Figure 2 As shown, the video processing method may include steps S200 to S202, wherein:

[0075] Step S200: Obtain the first video, which includes a low-bit depth video with shakiness.

[0076] Step S202: The first video is converted into a second video using a stabilization model. The second video includes a stabilized high bit depth video, where the first bit depth of the low bit depth video is lower than the second bit depth of the high bit depth video. The stabilization model includes a trained deep neural network model used for stabilization and bit depth conversion.

[0077] The bit depth of a video is compatible with the hardware and software, such as the display. Older versions of hardware and software may not be backward compatible with high bit depth videos. Therefore, after a video is produced, it is often converted to a low bit depth version to be compatible with mainstream devices. During the low bit depth video production process, dithering is added in the compression (preprocessing) stage to optimize color transitions, allowing the video to exhibit as subtle color gradations as possible, even at low bit depths. Of course, dithering consumes a significant amount of bitrate and also results in some degradation in image quality.

[0078] With the upgrade of software and hardware versions, more and more mainstream devices support high bit depth video.

[0079] Taking an 8-bit color depth as an example, it can display 16.7 million colors.

[0080] For example, a 10-bit color depth can display 1.07 billion colors. The higher the bit depth, the more colors can be displayed, resulting in more natural color transitions on the monitor and avoiding striped color transitions.

[0081] Therefore, if the device supports it, previously created low bit depth videos can be converted into jitter-free high bit depth videos, improving the viewing experience.

[0082] In this embodiment, the first video is a low-bit-depth video with jitter, such as an 8-bit jitter video. The second video is a high-bit-depth video with jitter removed and bit depth increased, such as a 10-bit jitter-free video. Low-bit-depth and high-bit-depth videos are relative, not absolute, concepts. In some embodiments, the first video can be, for example, a 10-bit jitter video, and the second video can be, for example, a 12-bit jitter-free video. If applied to 10-bit to 12-bit conversion, the corresponding model can be retrained using the same technique.

[0083] Because color jitter consumes a significant amount of bitrate and also degrades image quality to some extent, it's necessary to remove as much color jitter as possible during the conversion from low bit depth to high bit depth video—that is, to perform a jitter removal operation. Furthermore, if we consider color jitter blocks as a special type of noise, the distribution of this "noise" in the video does not conform to common Gaussian or Poisson distributions, making traditional jitter removal algorithms ineffective. In this embodiment, a trained deep neural network model is used for jitter removal and bit depth conversion, which can minimize color jitter (global and local jitter) and achieve video conversion from low bit depth to high bit depth without compromising image quality during the conversion process.

[0084] As can be seen from the above, the video processing method described in the embodiments of this application can convert low bit depth video with jitter into high bit depth video without jitter. Jitter reduction will reduce the bit rate to a certain extent, and bit depth conversion will bring a certain degree of image quality improvement, thus improving the viewing experience.

[0085] The following combination Figure 2 The steps in steps S200 to S202 and other newly added steps are described in detail.

[0086] Step S200 Acquire the first video, which includes a low-resolution video with shaky graphics.

[0087] The first video can be in various formats, such as AVI (Audio Video Interleaved) format.

[0088] The first video is a low-depth video with shakiness, and we try to maintain the same subtle gradation as a high-depth video.

[0089] Dithering, or color mixing, involves sub-pixels borrowing colors from neighboring pixels to create new colors when displaying only a portion of the original color. When the pixel is magnified, it becomes apparent that neighboring pixels tend to mix, producing colors that don't actually exist in the image, thus achieving a compensation purpose. Common spatial dithering algorithms include RandomDither, OrderedDither, and Error Diffusion.

[0090] Step S202 The first video is converted into a second video using a stabilization model. The second video includes a stabilized high bit depth video, where the first bit depth of the low bit depth video is lower than the second bit depth of the high bit depth video. The stabilization model includes a trained deep neural network model used for stabilization and bit depth conversion.

[0091] Because color jitter consumes a significant amount of bitrate and causes a certain degree of image quality degradation, the color jitter in the first video should be removed as much as possible during the conversion from the first video to the second video; this is called de-jittering. Furthermore, using a trained deep neural network model for de-jittering and bit depth conversion can minimize color jitter (global and local jitter) and achieve video conversion from low bit depth to high bit depth without compromising image quality.

[0092] For shaking removal models, the process can be divided into operational stages such as sample data preparation, model training, and model deployment.

[0093] During the model deployment phase, to improve the efficiency of calling the de-jitter model and its scalability and ease of use, the following optional implementation methods can be provided.

[0094] In optional embodiments, such as Figure 3 As shown, the method may further include: step S300, in the case of obtaining the de-jitter model, accelerating and optimizing the de-jitter model to obtain a target format file; step S302, associating the target format file with the inference engine so as to run the de-jitter model by calling the inference engine.

[0095] like Figure 4 As shown, after training a de-jitter model, the model optimizer is used to accelerate and optimize the model (e.g., a .pth file), generating a .bin file recording the model parameters and a .xml file recording the network model configuration in the target format. Then, the target format file is loaded into the inference engine. The inference engine, based on the .bin and .xml files, can obtain the de-jitter model, thus enabling its deployment. During use, the inference engine is invoked to perform network inference, and the results are returned to the interface. In the exemplary application, the target format file is, for example, an Openvino IR format file. Openvino is a visual inference acceleration toolkit from Intel, applicable to computer vision, enabling neural network model optimization and inference acceleration. In this application, inference acceleration toolkits such as Openvino can be used to quickly deploy the trained de-jitter model on the device. The inference acceleration toolkit accelerates the model's inference process; its model optimizer converts the trained model into a unified IR format through various operator fusions and memory optimizations for subsequent calls. That is, the inference engine can accelerate the model and improve calling efficiency.

[0096] In optional embodiments, such as Figure 5 As shown, the method may further include:

[0097] Step S500: Encapsulate the inference engine associated with the target format file using a custom filter to obtain the target filter.

[0098] For example, the shaking service can be encapsulated as an FFmpeg filter through a custom filter. Users can then easily perform shaking and conversion processing on videos using FFmpeg, just as conveniently as using FFmpeg's built-in filters. The target filter uses an accelerated IR format file.

[0099] FFmpeg is an audio and video codec tool and also an audio and video encoding development kit. As an encoding development kit, it provides developers with a rich set of APIs for audio and video processing. To better deploy anti-shake functionality, the method for eliminating color distortion can be encapsulated into an FFmpeg filter based on the FFmpeg AVFilter library, allowing users to easily process shaky videos using FFmpeg. Ultimately, the target filter based on the inference acceleration toolkit can achieve a very high FPS (Frames Per Second) speed when processing 1080p video sources on a CPU, including encoding and decoding speed. If the target filter based on the inference acceleration toolkit is deployed on a GPU, the speed will be further improved.

[0100] FFmpeg's AVFilter library provides a general framework for audio, video, and subtitle filtering. FFmpeg itself also has a large number of built-in filters, which can be used to process audio and video with various special effects, and supports custom filters. In this embodiment, by encapsulating the target filter (FFmpeg filter), it can be arbitrarily combined with other filters in FFmpeg to achieve specific composite functions, improving extensibility.

[0101] During the model training phase, to improve the accuracy of model shading and the image quality of converted high bit depth videos, the following optional implementation methods can be provided.

[0102] In optional embodiments, such as Figure 6 As shown, the deep neural network model is trained through the following operations:

[0103] Step S600: Input the sample input image into the deep neural network model to obtain the predicted image.

[0104] The sample input image is a low-resolution image with jitter.

[0105] Considering the practical deployment scenario, where a large number of videos require shaking removal, the model's processing speed is a crucial factor. A simplified convolutional structure can be used to ensure a lower overall computational cost. Since shaking removal is a low-level image quality task, its requirements for global semantic information are not as high as for higher-level tasks like image recognition. Therefore, the feature map size remains consistent with the input / output map during processing. Thus, in this embodiment, the sample input image and the predicted image are the same size.

[0106] Step S602: Obtain the average absolute error of pixels between the predicted image and the sample reference image.

[0107] The sample reference image is a high-bit-depth jitter-free image corresponding to the sample input image.

[0108] Mean absolute error (L1 loss) is used to calculate the absolute interpolation between the target pixel and the predicted pixel. The calculation formula is as follows:

[0109]

[0110] Where n is the total number of pixels, y i These are the pixel values ​​of each pixel in the sample reference image; It predicts the pixel value of each pixel in the image.

[0111] Step S604: Obtain the total variation loss of the predicted image.

[0112] Total Variation Loss (TV loss) addresses the difference in values ​​between adjacent pixels in an image, thus maintaining image smoothness to some extent. Denoising and dejittering tasks share significant similarities; jittery footage is also unsmooth. TV loss can smooth the entire image, making it appear as if no color grading was applied. The specific formula is as follows:

[0113]

[0114] Where x represents the pixel value, and β can be set to 2 by default.

[0115] Step S606: Optimize the deep neural network model based on the mean absolute difference and the total variation loss.

[0116] Based on the mean absolute difference and the total variation loss, the parameters in the deep neural network module can be adjusted to gradually adjust the deep neural network model to meet the expected direction until the effect is greater than the expected target.

[0117] The above describes an exemplary training method. It should be noted that other training methods can also be used for training.

[0118] Besides the training method, the sample data used for training also plays a significant role in the speed and effectiveness of model training. Therefore, to improve the speed and effectiveness of model training, the following steps can be taken during the sample data preparation stage.

[0119] In optional embodiments, such as Figure 7 As shown, the method may further include: step S700, obtaining a sample video group, the sample video group including an input video and a reference video corresponding to the same video, the input video being a low bit depth sample video with jitter, and the reference video being a high bit depth sample video without jitter; step S702, obtaining multiple frames from the input video at a preset number of frame intervals to obtain multiple low bit depth frame frames, and obtaining multiple corresponding frames from the reference video to obtain multiple corresponding high bit depth frame frames; step S704, cropping an image frame that meets the target size from each low bit depth frame frame to obtain multiple sample input frames; and step S706, cropping an image frame that meets the target size from the corresponding area of ​​each high bit depth frame frame to obtain multiple sample reference frames.

[0120] In the exemplary application, to closely resemble real-world scenarios, a set of video groups with corresponding 8-bit videos and their 10-bit sources is first collected. These corresponding video groups can form the basic data source for model training. Simultaneously, because adjacent frames in a video often have a high degree of correlation, to ensure dataset diversity, frames from the corresponding 8-bit and 10-bit videos are extracted every 300 frames during dataset creation, and 512x512 images are randomly cropped from these extracted frames to form the training dataset.

[0121] In an optional embodiment, step S700 "acquiring a sample video group" may include:

[0122] Multiple sample video groups are obtained, wherein the multiple input videos corresponding to the multiple sample video groups correspond to several color dithering strategies.

[0123] As mentioned earlier, color dithering strategies can be categorized into local and global dithering on one dimension, and random, ordered, and error diffusion methods on another. In other words, color dithering strategies are diverse. Therefore, to enable the model to handle various types of visuals and maintain the expected good results, a series of videos using different color dithering strategies and schemes are needed for training, simulating situations that might occur in real-world applications as closely as possible.

[0124] Model training typically requires a large number of samples. When there are not enough 8-bit to 10-bit video groups, video groups can be constructed by adding dithering to some videos with 10-bit source material, thus obtaining a large number of sample video groups. See below for details.

[0125] In an optional embodiment, step S700 "acquiring a sample video group" may include:

[0126] Acquire multiple high-depth sample videos, which correspond to several content types.

[0127] Based on different color-dithering strategies, the multiple high-bit-depth sample videos are converted into low-bit-depth sample videos with dithering; and

[0128] The multiple high-depth sample videos and the multiple low-depth sample videos are combined to obtain multiple sample video groups.

[0129] The limited number and concentration of video groups can also restrict the training performance of the model. Therefore, other 10-bit videos with different content types can be selected, such as 3D videos, 2D videos, and game videos. For example, by using a multi-dimensional color-shifting strategy, these 10-bit videos can be converted into 8-bit videos with some shaking, resulting in a video group consisting of 8-bit videos with shaking and 10-bit videos. This video group can be used for model training to improve the training effect.

[0130] Example 2

[0131] Figure 8 A block diagram of a video processing apparatus according to Embodiment 2 of this application is schematically shown. This video processing apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of this application. The program module referred to in this embodiment is a series of computer instruction segments capable of performing a specific function. The following description will specifically introduce the functions of each program module in this embodiment. The video processing apparatus 800 may include an acquisition module 810 and a conversion module 820, wherein:

[0132] The acquisition module 810 is used to acquire a first video, which includes a low-bit depth video with jitter.

[0133] The conversion module 820 is used to convert the first video into a second video using a shading model. The second video includes a shading high bit depth video, wherein the first bit depth of the low bit depth video is lower than the second bit depth of the high bit depth video. The shading model includes a trained deep neural network model, which is used for shading and bit depth conversion.

[0134] In an optional embodiment, the apparatus further includes a deployment module (unidentified) for:

[0135] Once the de-jitter model is obtained, it is accelerated and optimized to obtain the target format file.

[0136] The target format file is associated with the inference engine so that the debouncing model can be run by calling the inference engine.

[0137] In an optional embodiment, the device further includes a packaging module (unidentified) for:

[0138] The target filter is obtained by encapsulating the inference engine associated with the target format file using a custom filter.

[0139] In an optional embodiment, the apparatus further includes a training module (unidentified) for:

[0140] The sample input image is input into the deep neural network model to obtain the predicted image; wherein, the sample input image is a low-bit depth image with jitter.

[0141] Obtain the average absolute error of pixels between the predicted image and the sample reference image; wherein, the sample reference image is a high bit depth jitter-free image corresponding to the sample input image;

[0142] Obtain the total variation loss of the predicted image;

[0143] The deep neural network model is optimized based on the mean absolute difference and the total variation loss.

[0144] In an optional embodiment, the training module is further configured to:

[0145] Obtain a sample video group, which includes an input video and a reference video corresponding to the same video. The input video is a low-bit-depth sample video with jitter, and the reference video is a high-bit-depth sample video without jitter.

[0146] Multiple low-bit deep frames are obtained from the input video at a preset number of frame intervals, and multiple corresponding frames are obtained from the reference video to obtain multiple corresponding high-bit deep frames.

[0147] Image frames conforming to the target size are extracted from each low-bit-depth frame to obtain multiple sample input frames; and

[0148] Image frames conforming to the target size are cropped from the corresponding area of ​​each high-bit-depth frame to obtain multiple sample reference frames.

[0149] In an optional embodiment, the training module is further configured to:

[0150] Multiple sample video groups are obtained, wherein the multiple input videos corresponding to the multiple sample video groups correspond to several color dithering strategies.

[0151] In an optional embodiment, the training module is further configured to:

[0152] Acquire multiple high-depth sample videos, which correspond to several content types.

[0153] Based on different color-dithering strategies, the multiple high-bit-depth sample videos are converted into low-bit-depth sample videos with dithering; and

[0154] The multiple high-depth sample videos and the multiple low-depth sample videos are combined to obtain multiple sample video groups.

[0155] Example 3

[0156] Figure 9 This illustration schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a video processing method according to Embodiment 3 of this application. The computer device 10000 can be part of a server 2 or an electronic device 6. In this embodiment, the computer device 10000 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a smartphone, tablet computer, laptop, personal computer, virtual device, set-top box, television, projector, vehicle terminal, headphones, etc. In other embodiments, the computer device 10000 can also be a rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 9 As shown, the computer device 10000 includes, but is not limited to, a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked to each other via a system bus. Wherein:

[0157] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for video processing methods. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0158] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0159] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0160] It should be pointed out that, Figure 9 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0161] In this embodiment, the video processing method stored in the memory 10010 can be further divided into one or more program modules and executed by one or more processors (processor 10020 in this embodiment) to complete the embodiment of this application.

[0162] Example 4

[0163] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the video processing method in the embodiment.

[0164] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium may include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the video processing method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0165] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0166] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A video processing method, characterized in that, The method includes: Acquire the first video, which includes a low-resolution video with shaky graphics; The first video is converted into a second video using a stabilization model. The second video includes a stabilized high bit depth video, where the first bit depth of the low bit depth video is lower than the second bit depth of the high bit depth video. The stabilization model includes a trained deep neural network model, which is used for stabilization and bit depth conversion. The stabilization operation is used to remove color jitter.

2. The video processing method according to claim 1, characterized in that, The method further includes: Once the de-jitter model is obtained, it is accelerated and optimized to obtain the target format file. The target format file is associated with the inference engine so that the debouncing model can be run by calling the inference engine.

3. The video processing method according to claim 2, characterized in that, The method further includes: The target filter is obtained by encapsulating the inference engine associated with the target format file using a custom filter.

4. The video processing method according to any one of claims 1 to 3, characterized in that, The deep neural network model is trained through the following operations: The sample input image is input into the deep neural network model to obtain the predicted image; wherein, the sample input image is a low-bit depth image with jitter. Obtain the average absolute error of pixels between the predicted image and the sample reference image; wherein, the sample reference image is a high bit depth jitter-free image corresponding to the sample input image; Obtain the total variation loss of the predicted image; The deep neural network model is optimized based on the mean absolute difference and the total variation loss.

5. The video processing method according to claim 4, characterized in that, The method further includes: Obtain a sample video group, which includes an input video and a reference video corresponding to the same video. The input video is a low-bit-depth sample video with jitter, and the reference video is a high-bit-depth sample video without jitter. Multiple low-bit deep frames are obtained from the input video at a preset number of frame intervals, and multiple corresponding frames are obtained from the reference video to obtain multiple corresponding high-bit deep frames. Image frames conforming to the target size are extracted from each low-bit-depth frame to obtain multiple sample input frames; and Image frames conforming to the target size are cropped from the corresponding area of ​​each high-bit-depth frame to obtain multiple sample reference frames.

6. The video processing method according to claim 4, characterized in that, The acquired sample video group includes: Multiple sample video groups are obtained, wherein the multiple input videos corresponding to the multiple sample video groups correspond to several color dithering strategies.

7. The video processing method according to claim 4, characterized in that, The acquired sample video group includes: Acquire multiple high-depth sample videos, which correspond to several content types. Based on different color-dithering strategies, the multiple high-bit-depth sample videos are converted into low-bit-depth sample videos with dithering; and Based on the multiple high-depth sample videos and the multiple low-depth sample videos, multiple sample video groups are obtained.

8. A video processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire a first video, which includes a low-bit depth video with jitter. A conversion module is used to convert the first video into a second video using a stabilization model. The second video includes a stabilized high bit depth video, wherein the first bit depth of the low bit depth video is lower than the second bit depth of the high bit depth video. The stabilization model includes a trained deep neural network model, which is used for stabilization and bit depth conversion. The stabilization operation is used to remove color jitter.

9. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images