Video noise reduction methods and apparatus, video processing methods and apparatus, computer equipment

Abstract

This application discloses a video denoising method and apparatus, a video processing method and apparatus, a computer device, and a computer-readable storage medium. In one specific embodiment, the video denoising method includes: decoding acquired video data to obtain YUV format data; using a trained denoising model to denoise the Y channel data in the YUV format data to obtain denoised Y channel data; and merging the UV channel data in the YUV format data with the denoised Y channel data to complete video denoising. This embodiment, by directly decoding the YUV format data and extracting only the Y channel data for denoising, can significantly reduce the computational load of video denoising processing.

Description

Technical Field

[0001] This application relates to the field of computer technology. More specifically, it relates to a video noise reduction method and apparatus, a video processing method and apparatus, a computer device, and a computer-readable storage medium. Background Technology

[0002] Currently, video noise reduction solutions typically decode video data into RGB format and perform noise reduction processing on all data from the R, G, and B channels separately. This results in a large amount of data computation, consumes a lot of video memory, and has low processing efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a video noise reduction method and apparatus, a video processing method and apparatus, a computer device, and a computer-readable storage medium to solve at least one of the problems existing in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] The first aspect of this application provides a video noise reduction method, including:

[0006] The acquired video data is decoded to obtain YUV format data;

[0007] The Y channel data in YUV format data is denoised using a trained denoising model to obtain denoised Y channel data.

[0008] The UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete the video noise reduction.

[0009] In some optional embodiments, before decoding the acquired video data to obtain YUV format data, the method further includes: establishing a noise reduction model, which includes: a downsampling unit, a convolutional network, and a dimension restoration unit, wherein,

[0010] The downsampling unit downsamples the Y channel data to four channels with half the resolution;

[0011] The convolutional network is a three-layer, four-channel convolutional network that performs convolution calculations on four-channel data;

[0012] The dimension restoration unit restores the denoised data after convolution calculation to the original size of the Y channel data, thus obtaining the denoised Y channel data.

[0013] In some optional embodiments, the convolutional network further includes: a first convolutional layer, a first merging layer, a second convolutional layer, a summing layer, a second merging layer, and a third convolutional layer.

[0014] The first convolutional layer performs the first four-channel convolution on the four-channel data to obtain the first feature data;

[0015] The first merging layer merges the first feature data with the Y channel data to obtain the second feature data;

[0016] The second convolutional layer performs a second four-channel convolution on the second feature data to obtain the third feature data;

[0017] The summation layer adds the first feature data to the third feature data to obtain the fourth feature data;

[0018] The second merging layer merges the fourth feature data with the Y channel data to obtain the fifth feature data;

[0019] The third convolutional layer performs a third four-channel convolution on the fifth feature data to obtain the denoised data.

[0020] In some optional embodiments, after establishing the denoising model and before using the trained denoising model to denoise the Y channel data in the YUV format data to obtain the denoised Y channel data, the method further includes:

[0021] Training YUV data is obtained by decoding pre-stored high-definition video data;

[0022] Based on the training YUV data, high-definition frame sequences and low-definition frame sequences of the Y channel were generated as training data pairs.

[0023] The trained noise reduction model is obtained by training the noise reduction model using a complex network model based on the training data.

[0024] In some optional embodiments, the training YUV data includes lossless YUV data and compressed YUV data.

[0025] The training YUV data obtained by decoding pre-stored high-definition video data further includes:

[0026] The pre-stored high-definition video data is compressed to obtain compressed data, and the compressed data is decoded to obtain compressed YUV data;

[0027] Decoding the pre-stored high-definition video data yields lossless YUV data.

[0028] Based on the training YUV data, a high-definition frame sequence and a low-definition frame sequence of the Y channel are generated as training data, which further include:

[0029] The Y channel data is extracted from the compressed YUV data to obtain the Y channel low-resolution frame sequence, and the Y channel data is extracted from the lossless YUV data to obtain the Y channel high-resolution frame sequence, which are then used as training data pairs.

[0030] In some optional embodiments, training the denoising model using a complex network model based on training data to obtain a trained denoising model further includes:

[0031] Based on the training data, obtain the trained high-definition frame sequence of the complex network model;

[0032] The trained noise reduction model is obtained by using distilled training data composed of trained high-definition frame sequences and low-definition frame sequences.

[0033] A second aspect of this application provides a video processing method, comprising:

[0034] Acquire video data;

[0035] The noise reduction module uses a multi-threaded processing mode to perform video noise reduction on the video data;

[0036] The adjustment module, operating in multi-threaded processing mode, adjusts the brightness and / or saturation of the video after noise reduction to obtain the processed video data.

[0037] The video noise reduction process includes: decoding the acquired video data to obtain YUV format data;

[0038] The Y channel data in YUV format data is denoised using a trained denoising model to obtain denoised Y channel data.

[0039] The UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete the video noise reduction.

[0040] A third aspect of this application provides a video noise reduction apparatus, comprising:

[0041] The acquisition module is used to acquire video data;

[0042] The noise reduction module decodes the acquired video data to obtain YUV format data; it uses a trained noise reduction model to perform noise reduction on the Y channel data in the YUV format data to obtain the noise-reduced Y channel data; and it merges the UV channel data in the YUV format data with the noise-reduced Y channel data to complete the video noise reduction.

[0043] A fourth aspect of this application provides a video processing apparatus, comprising:

[0044] The input module is used to acquire video data;

[0045] The noise reduction processing module is used to perform video noise reduction processing on video data in a multi-threaded processing mode.

[0046] The adjustment module, operating in a multi-threaded processing mode, adjusts the brightness and / or saturation of the video after noise reduction to obtain the processed video data.

[0047] The video noise reduction process includes: decoding the acquired video data to obtain YUV format data;

[0048] The Y channel data in YUV format data is denoised using a trained denoising model to obtain denoised Y channel data.

[0049] The UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete the video noise reduction.

[0050] A fifth aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to achieve:

[0051] The method described in the first aspect above, or the method described in the second aspect above.

[0052] A sixth aspect of this application provides a computer-readable storage medium having a computer program stored thereon.

[0053] When the program is executed by the processor, it implements the method described in the first aspect above, or

[0054] When the program is executed by the processor, it implements the method described in the second aspect above.

[0055] The beneficial effects of this application are as follows:

[0056] This application addresses existing problems by providing a video denoising method and apparatus, a video processing method and apparatus, a computer device, and a computer-readable storage medium. Specifically, the video denoising method, after acquiring video data, directly obtains YUV format data through decoding, and extracts only the Y channel data for denoising processing using a denoising model. This leverages the human eye's greater sensitivity to changes in Y channel data compared to changes in UV channel data, achieving effective video denoising while reducing computational load by two-thirds, thus improving denoising processing efficiency and offering broad applications. Attached Figure Description

[0057] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0058] Figure 1 A schematic flowchart illustrating a video noise reduction method according to an embodiment of this application is shown;

[0059] Figure 2This diagram illustrates a signal flow graph of a video noise reduction process according to a specific embodiment of this application.

[0060] Figure 3 This diagram illustrates the network framework of a noise reduction model according to an embodiment of the present application.

[0061] Figure 4 A schematic flowchart of a video noise reduction method according to an embodiment of this application is shown;

[0062] Figure 5 This diagram illustrates the signal flow of the noise reduction model training process in a noise reduction processing method according to an embodiment of the present application.

[0063] Figure 6 A schematic block diagram of a video noise reduction apparatus according to an embodiment of this application is shown;

[0064] Figure 7 A schematic flowchart illustrating a video processing method according to an embodiment of this application is shown;

[0065] Figure 8 A detailed processing flowchart of a video processing method according to an embodiment of this application is shown;

[0066] Figure 9 A schematic block diagram of a video processing apparatus according to an embodiment of this application is shown; and

[0067] Figure 10 A schematic diagram of the structure of a computer device according to an embodiment of this application is shown. Detailed Implementation

[0068] To more clearly illustrate this application, the following description, in conjunction with embodiments and accompanying drawings, further clarifies the application. Similar components in the drawings are indicated by the same reference numerals. Those skilled in the art should understand that the specific description below is illustrative rather than restrictive and should not be construed as limiting the scope of protection of this application.

[0069] It should be noted that in the description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0070] To solve one of the above problems, refer to Figure 1 As shown, one embodiment of this application provides a video noise reduction method, including:

[0071] Step S1: Decode the acquired video data to obtain YUV format data;

[0072] Step S2: Use the trained denoising model to denoise the Y channel data in the YUV format data to obtain the denoised Y channel data.

[0073] Step S3: Merge the UV channel data in the YUV format data with the noise-reduced Y channel data to complete the video noise reduction.

[0074] In this embodiment, YUV format data is directly obtained by decoding, and only the Y channel data is extracted for noise reduction processing using a noise reduction model. This leverages the fact that the human eye is more sensitive to changes in Y channel data than to changes in UV channel data, thereby achieving effective video noise reduction while reducing the computational load by 2 / 3 and improving noise reduction processing efficiency.

[0075] The video noise reduction method provided in this embodiment can be implemented through a terminal device equipped with an intelligent system. Specifically, the terminal device can be a device with data computing capabilities, such as a smartphone, smart screen, smart cloud box, laptop, etc., and this application does not limit it. A direct implementation of the video denoising method can be, for example, a software development kit (SDK) loaded into the aforementioned terminal. Specifically, it can act as a video decoder in the intelligent system loaded on the terminal device, performing denoising processing on the video data received by the device during the decoding process, and then directly rendering and playing the denoised video data after the denoising processing is completed. In this case, supported application scenarios include system video playback, video conferencing, video calls, and other situations where the video is played directly on the device. Alternatively, the direct implementation of the video denoising method can be as a SSD loaded into the aforementioned terminal system or software application, performing denoising processing on the video data received by the device during the decoding process, and then further encoding and outputting the data after the denoising processing is completed. In this case, supported application scenarios include, for example, during live streaming, the broadcaster needs to record video, and after the recorded video is denoised, it needs to be encoded into video data packets and sent to the cloud server, or after the recorded video is sent to the cloud server, the video data is denoised and encoded on the cloud server, and then distributed and played to various users watching the live stream via the cloud server.

[0076] It should be noted that the specific implementation methods and application scenarios mentioned above are merely exemplary and not intended to be exhaustive. In practical applications, any device and scenario that can use the noise reduction processing method of this application embodiment to perform noise reduction processing on video is acceptable.

[0077] Next, from the perspective of a terminal device with video processing capabilities, the video noise reduction method provided in this embodiment will be described.

[0078] In step S1, the acquired video data is decoded to obtain YUV format data.

[0079] Specifically, refer to Figure 2 As shown, the terminal device acquires video data, for example, by inputting video from an external source, and decodes the video data to directly obtain YUV format data. In this example, the decoding process can use the currently mainstream audio and video decoding framework ffmpeg, but this application does not impose any specific restrictions.

[0080] In step S2, the trained denoising model is used to denoise the Y channel data in the YUV format data to obtain the denoised Y channel data.

[0081] In particular, in the embodiments of this application, the Y channel data in the YUV format data is directly used for processing, instead of the traditional method of further converting to RGB format data and performing noise reduction processing on the three color channel data in the RGB format. This can save the time of converting to RGB format data, and at the same time, it can use only one channel data in the YUV format data for noise reduction processing, thereby reducing the amount of data computation.

[0082] This approach takes into account that in YUV format data, "Y" represents luminance (or luma), which is the grayscale value, while "U" and "V" represent chrominance (or chroma). Accordingly, the Y channel data is the luminance channel data, and the U and V channel data are the chrominance channel data. Unlike RGB format data, where each channel affects the human eye's viewing experience, the human eye is more sensitive to changes in luminance than to changes in chrominance values. Therefore, when using YUV format data for noise reduction, processing the Y channel data can meet the requirements of video noise reduction. Thus, in the embodiments of this application, by extracting the Y channel data from the YUV format data and performing noise reduction processing, the computational load is reduced by 2 / 3.

[0083] To be more specific, refer to Figure 2As shown, one way to extract Y channel data is to use the memcpy algorithm to copy the Y channel data separately from the YUV format data and then use a trained denoising model for denoising processing.

[0084] Those skilled in the art should understand that a noise reduction model training step should be included before the specific noise reduction processing is performed in step S2.

[0085] In particular, considering that terminal boards of terminal devices often have insufficient computing power, in order to further reduce the amount of computation, in addition to using Y channel data for separate noise reduction processing to reduce the amount of computation, this application preferably adopts a lightweight network framework to further reduce the amount of computation and memory occupancy of noise reduction processing and improve processing efficiency. Therefore, the training process of the noise reduction model of the embodiment of this application will be further described in detail below.

[0086] It should be noted that the training process of the denoising model needs to be performed before the denoising process, that is, before step S2. Therefore, it is not necessary to limit it to be performed before step S1. However, if the training process of the denoising model is completed in advance on another computer device, then the training process can be performed before step S1. Of course, this application does not limit the specific device for executing the training process of the denoising model. If conditions permit, this step can be performed on a terminal device.

[0087] The first step in training a noise reduction model is to establish the noise reduction model, referring to... Figure 3 The data flow diagram shown illustrates a lightweight network framework built in the embodiments of this application.

[0088] and Figure 3 Corresponding to the data flow diagram shown, the denoising model includes: downsampling unit, convolutional network, and dimension restoration unit.

[0089] Combination Figure 3As shown, to reduce computational load, in the embodiments of this application, a downsampling unit that reduces video resolution is introduced before the denoising process within the denoising model. Specifically, the downsampling unit downsamples the Y-channel data into four-channel data with half the resolution. For example, the downsampling unit can use Space_to_depth downsampling to downsample the Y-channel data input to the denoising model into four-channel data with half the resolution. The Space_to_depth downsampling process involves sampling at positions (1,0), (0,1), (1,1), and (0,0) with a step size of 1. For example, if the input Y-channel data is 512×512×1, after Space_to_depth downsampling, it will become 256×256×4. After this processing, the resolution is reduced to half of the original, and the number of channels increases to four times the original. By introducing downsampling processing into the denoising model, the computational load and memory usage can be significantly reduced.

[0090] More preferably, in this embodiment, the convolutional network is a three-layer, four-channel convolutional network that performs convolution calculations on the four-channel data. Continuing to refer to... Figure 3 As shown, the convolutional network specifically includes: a first convolutional layer, a first merging layer, a second convolutional layer, a summing layer, a second merging layer, and a third convolutional layer.

[0091] The process involves several steps. First, the first convolutional layer performs a first four-channel convolution on the four-channel data to obtain the first feature data. The operator can be represented as Conv1(4 channels). This first four-channel convolution extracts the first feature data from the four-channel data. Next, the first merging layer merges the first feature data with the Y-channel data to obtain the second feature data. The operator can be represented as Concat(y_channel, conv1). This first merging process merges the original, unsampled Y-channel data input to the noise reduction model with the first feature data obtained after the first feature extraction in a memory-based manner. This merging is not simply an addition of data; it aims to preserve the original information of the video data. This process allows data lost or abstracted after the first convolution to be retained by merging with the original data, improving the feature extraction effect. Finally, the second convolutional layer performs a second four-channel convolution on the merged second feature data to obtain the third feature data. This is the second layer of feature processing, and the operator can be Conv2(4 channels). This process further supplements the feature extraction process by adding the original information from the video data. Next, an addition layer is used to add the first and third feature data to obtain the fourth feature data. The operator can be Add(conv1+conv2), which aims to sum the results of the first and second convolutions. Following this, a second merging layer merges the fourth feature data with the Y-channel data to obtain the fifth feature data. This further merges the original feature information based on the second feature extraction, using the operator Concat(add+y_channel). Finally, a third convolutional layer performs a final feature extraction process, applying a third four-channel convolution to the fifth feature data to obtain the denoised data.

[0092] In the embodiments of this application, by specially setting a three-convolutional-layer four-channel processing process, the number of convolutional layers and the number of data processing channels are greatly reduced compared with the prior art. The four channels share weights, which reduces the occupancy of GPU memory and realizes a lightweight design of the denoising model network. In combination with the hierarchical order of the above layers, the feature extraction accuracy of the denoising process can be ensured while reducing the weight, and a good denoising effect can be achieved while reducing the amount of computation.

[0093] Continue to refer to Figure 3As shown, after the convolutional network completes its computation, four channels of Y-channel denoised data are obtained: Y-channel denoised data 1, Y-channel denoised data 2, Y-channel denoised data 3, and Y-channel denoised data 4. These data are not the original size, but rather low-resolution denoised data. For example, the actual operators can be represented as: Y_channel0_ed, Y_channel1_ed, Y_channel2_ed, and Y_channel3_ed.

[0094] Next, the denoised data after convolution calculation is restored to the original size of the Y channel data using a dimension restoration unit, resulting in the denoised Y channel data. This restoration process can be the inverse of the downsampling process. For example, when the downsampling process uses space_to_depth to downsample 512×512×1 data to 256×256×4 data, the restoration process uses depth_to_space to restore it to 512×512×1 data, thus obtaining the denoised Y channel data.

[0095] Furthermore, after establishing the denoising model, it needs to be trained. In the embodiments of this application, in order to make the denoising results of the established lightweight network structure denoising model more accurate, that is, to achieve the denoising effect of complex denoising networks with a simpler data calculation process, the denoising model in this application can be trained using the complex network model distillation learning process.

[0096] Specifically, refer to Figure 4 As shown, after establishing the denoising model and before using the trained denoising model to denoise the Y channel data in the YUV format data to obtain the denoised Y channel data, the following training process is also included. It should be noted that in the distillation learning process of this application, the most important process is creating training data pairs, where... Figure 5 The training data pair generation process of an embodiment of this application is illustrated.

[0097] In step S2-1, training YUV data is obtained by decoding the pre-stored high-definition video data.

[0098] Those skilled in the art will understand that the pre-stored high-definition video data can typically be sample high-definition video data stored in a data sample library. These sample high-definition data are generally high-quality, publicly available high-definition data, intended to be used to train complex network models.

[0099] It is worth mentioning that the inventors discovered during the research process that if the training data format used during training is inconsistent with the data format decoded during noise reduction, the noise reduction effect will be unsatisfactory and affect the noise reduction effect.

[0100] Therefore, preferably, in the embodiments of this application, the pre-stored high-definition video data is also decoded into YUV format data during the training process. For example, the ffmpeg processing tool is used to decode the training YUV data, that is, the same rules as the noise reduction process are used to decode the training YUV data in YUV format.

[0101] In the embodiments of this application, in order to perform distillation learning, a set of low-quality low-resolution frame sequences (LR video data) and lossless high-resolution frame sequences (HR video data) are created using pre-stored high-resolution video data as training data pairs. In other words, referring to... Figure 5 As shown, the training YUV data obtained after decoding also includes lossless YUV data and compressed YUV data.

[0102] Specifically, the process of decoding pre-stored high-definition video data to obtain training YUV data further includes: compressing the pre-stored high-definition video data to obtain compressed data, and decoding the compressed data to obtain compressed YUV data. The compression command line is not the Fgmpeg command line, and the decoding can be performed using Fgmpeg's C++ API to obtain compressed YUV data (i.e., LR YUV data). In addition, the pre-stored high-definition video data is decoded to obtain lossless YUV data. This process does not require compression and directly uses Fgmpeg's C++ API for decoding, resulting in lossless YUV data (i.e., HR YUV data).

[0103] In step S2-2, a high-definition frame sequence and a low-definition frame sequence of the Y channel are generated based on the training YUV data and used as training data pairs.

[0104] Specifically, refer to Figure 5 As shown, Y channel data is extracted from compressed YUV data to obtain a low-resolution Y channel frame sequence, and Y channel data is extracted from lossless YUV data to obtain a high-resolution Y channel frame sequence, which are then used as training data pairs.

[0105] Specifically, the memcpy algorithm can be used to copy the low-resolution frame sequence of the Y channel from the self-compressed YUV data, and the memcpy algorithm can be used to copy the high-resolution frame sequence of the Y channel from the lossless YUV data. The purpose of this is to use the training data pair made from the high-resolution and low-resolution frame sequences of the Y channel to train the model, so as to ensure that the extracted Y channel data can be effectively denoised in the subsequent actual denoising process. Then, the extracted low-resolution and high-resolution frame sequences of the Y channel are combined as training data pairs. Thus, the training data pair used in the training process of this application embodiment is completed.

[0106] In steps S2-3, the denoising model is trained using a complex network model based on the training data to obtain the trained denoising model.

[0107] In distillation learning, the complex network model used for distillation training of the lightweight network model is also called the "teacher model," and the lightweight network model to be trained is also called the "student model."

[0108] The specific training process is as follows: Based on the training data, a training high-definition frame sequence is obtained by training a complex network model; the training high-definition frame sequence and the low-definition frame sequence are used to form distilled training data to train the denoising model and obtain the trained denoising model.

[0109] In other words, to achieve the same denoising effect as the complex network model trained using the aforementioned training data, and to obtain a trained lightweight network while simplifying the training process, this application first trains an existing mature complex network model using the training data prepared in the above process. This training results in a high-definition frame sequence generated by the trained complex network model that closely approximates pre-stored high-definition video data (i.e., sample high-definition video data). Then, the trained high-definition frame sequence and the aforementioned low-definition frame sequence are used to form distilled training data to train the denoising model of the lightweight network in this embodiment. This achieves the acquisition of a trained denoising model from the Y-channel training data prepared using the above method for the complex network model. The distillation process ensures that the trained denoising model has no color cast after data processing. It should be noted that this application does not aim to limit the specific model structure of the complex network model; existing mature complex network models, such as EDVR and U-NET, can be used, which will not be elaborated upon here.

[0110] The trained denoising model has Figure 3 The lightweight network shown reduces memory usage and further reduces the computational cost of denoising processing compared to one-third of existing technologies, while achieving denoising performance comparable to or even better than complex network models.

[0111] Returning to step S2, combining Figure 1 and Figure 2 As shown in the embodiment of this application, the trained denoising model obtained by the above method is used to denoise the Y channel data in YUV format data to obtain denoised Y channel data.

[0112] The above settings significantly reduce the computational load compared to RGB three-channel denoising because they only denoise the Y channel data in the YUV format. Furthermore, due to the lightweight structure of the denoising model, the training process of the denoising model also uses the same data format as the denoising process, further reducing the computational load while ensuring good denoising results.

[0113] In step S3, the UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete the video noise reduction.

[0114] exist Figure 2 In the example shown, after merging the UV channel data in the YUV format data with the denoised Y channel data, further encoding and packaging are required before outputting the data. Those skilled in the art should understand that this step is not mandatory. In some application scenarios, if the video denoising method of this application embodiment is implemented as a video decoder loaded in the terminal device, the merged data can be directly rendered and played. However, in scenarios requiring further encoding and output, such as the distribution processing needed for live streaming, the encoding process will use the same audio and video encoding / decoding framework as the decoding, and Fgmpeg encoding can still be used, which will not be elaborated further here.

[0115] Corresponding to video noise reduction methods, refer to Figure 6 As shown, embodiments of this application also provide a video noise reduction device 10, comprising:

[0116] Acquisition module 101 is used to acquire video data;

[0117] The noise reduction module 102 is used to decode the acquired video data to obtain YUV format data; to perform noise reduction processing on the Y channel data in the YUV format data using a trained noise reduction model to obtain the noise-reduced Y channel data; and to merge the UV channel data in the YUV format data with the noise-reduced Y channel data to complete the video noise reduction.

[0118] It should be noted that the principle and workflow of the video noise reduction device provided in this embodiment are similar to the video noise reduction method described above. The relevant parts can be referred to the above description and will not be repeated here.

[0119] In this embodiment, YUV format data is directly obtained by decoding through the noise reduction module, and only the Y channel data is extracted for noise reduction processing using the noise reduction model. This takes advantage of the fact that the human eye is more sensitive to changes in Y channel data than to changes in UV channel data, thereby achieving effective video noise reduction while reducing the amount of computation by 2 / 3 and improving the efficiency of noise reduction processing.

[0120] Based on the same inventive concept, embodiments of this application also provide a video processing method, referring to... Figure 7As shown, it includes:

[0121] Step S1': Obtain video data;

[0122] Step S2': The noise reduction processing module in multi-threaded processing mode performs video noise reduction processing on the video data;

[0123] Step S3': The adjustment module in multi-threaded processing mode adjusts the brightness and / or saturation of the video noise-reduced data to obtain the video processing data.

[0124] The video noise reduction process includes: decoding the acquired video data to obtain YUV format data;

[0125] The Y channel data in YUV format data is denoised using a trained denoising model to obtain denoised Y channel data.

[0126] The UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete the video noise reduction.

[0127] It is worth mentioning that, in order to improve the playback effect of video data, the video processing includes not only video noise reduction but also further adjustments to the noise-reduced video data, such as saturation and brightness adjustments. Considering the computing power and memory limitations of terminal devices, this embodiment optimizes the computational load and memory occupancy of video noise reduction processing by adopting a multi-threaded pipeline processing method, referring to... Figure 8 As shown, solid lines represent the current processing thread, while dashed lines indicate that once the previous thread of a module finishes processing, it immediately connects to the next thread of the current module without waiting for all subsequent adjustment modules to complete their processing. This achieves multi-threaded pipeline operation for each module. Furthermore, the order of saturation and brightness adjustments in the diagram is not intended to be limited; in practical applications, their order can be interchanged, or only one type of processing can be performed as needed. Additionally, the output module is not mandatory, for reasons similar to those in the above embodiments, and will not be repeated here.

[0128] It should also be noted that the principle and workflow of the noise reduction process in the video processing method provided in this embodiment are similar to the video noise reduction method described above. The relevant parts can be referred to the above description, and will not be repeated here.

[0129] The above settings, by employing a multi-threaded pipeline processing approach, maximize resource utilization and improve video processing efficiency.

[0130] Corresponding to video processing methods, refer to Figure 8 and Figure 9 As shown, embodiments of this application also provide a video processing apparatus 20, comprising:

[0131] The input module is used to acquire video data;

[0132] The noise reduction processing module is used to perform video noise reduction processing on video data in a multi-threaded processing mode.

[0133] The adjustment module, operating in a multi-threaded processing mode, adjusts the brightness and / or saturation of the video after noise reduction to obtain the processed video data.

[0134] The video noise reduction process includes: decoding the acquired video data to obtain YUV format data;

[0135] The Y channel data in YUV format data is denoised using a trained denoising model to obtain denoised Y channel data.

[0136] The UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete the video noise reduction.

[0137] It should be noted that the video processing method and workflow provided in this embodiment are similar to the video processing method described above. Relevant details can be found in the above description and will not be repeated here.

[0138] The above settings, by employing a multi-threaded pipeline processing approach, maximize resource utilization and improve video processing efficiency.

[0139] Another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following: decoding acquired video data to obtain YUV format data; using a trained noise reduction model to perform noise reduction processing on the Y channel data in the YUV format data to obtain noise-reduced Y channel data; and merging the UV channel data in the YUV format data with the noise-reduced Y channel data to complete video noise reduction.

[0140] Another embodiment of this application provides a computer-readable storage medium for acquiring video data; performing video noise reduction processing on the video data using a noise reduction processing module in a multi-threaded processing mode; and adjusting the brightness and / or saturation of the video noise-reduced data using an adjustment module in a multi-threaded processing mode to obtain video processed data. The video noise reduction process includes: decoding the acquired video data to obtain YUV format data; performing noise reduction processing on the Y channel data in the YUV format data using a trained noise reduction model to obtain noise-reduced Y channel data; and merging the UV channel data in the YUV format data with the noise-reduced Y channel data to complete video noise reduction.

[0141] In practical applications, a computer-readable storage medium can take any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0142] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0143] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0144] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] like Figure 10As shown, another embodiment of this application provides a structural schematic diagram of a computer device. Figure 10 The computer device 1 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0146] like Figure 10 As shown, computer device 1 is represented in the form of a general-purpose computing device. The components of computer device 1 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0147] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0148] Computer device 1 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 1, including volatile and non-volatile media, removable and non-removable media.

[0149] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 1 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 10 Not shown; usually referred to as a "hard drive"). Although Figure 10 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0150] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this application.

[0151] Computer device 1 can also communicate with one or more external devices 15 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with computer device 1, and / or with any device that enables computer device 1 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 22. Furthermore, computer device 1 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 2. Figure 10 As shown, network adapter 2 communicates with other modules of computer device 1 via bus 18. It should be understood that, although... Figure 10 As not shown, it can be used in conjunction with computer device 1 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0152] The processor unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the video noise reduction method or video processing method provided in the embodiments of this application.

[0153] This application addresses existing problems by providing a video denoising method and apparatus, a video processing method and apparatus, a computer device, and a computer-readable storage medium. Specifically, the video denoising method, after acquiring video data, directly obtains YUV format data through decoding, and extracts only the Y channel data for denoising processing using a denoising model. This leverages the human eye's greater sensitivity to changes in Y channel data compared to changes in UV channel data, achieving effective video denoising while reducing computational load by two-thirds, thus improving denoising processing efficiency and offering broad applications.

[0154] Obviously, the above embodiments of this application are merely examples for clearly illustrating this application, and are not intended to limit the implementation of this application. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all implementation methods here. Any obvious variations or modifications derived from the technical solutions of this application are still within the protection scope of this application.

Claims

1. A video noise reduction method, characterized in that, include: The acquired video data is decoded to obtain YUV format data; The Y channel data in the YUV format data is denoised using a trained denoising model to obtain denoised Y channel data. The UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete video noise reduction. Before decoding the acquired video data to obtain YUV format data, the method further includes: establishing a noise reduction model, wherein the noise reduction model includes: a downsampling unit, a convolutional network, and a dimension restoration unit, wherein... The downsampling unit uses the Space_to_depth method to downsample the Y channel data into four-channel data with half the resolution; The convolutional network is a three-layer, four-channel convolutional network that performs convolution calculations on the four-channel data. The dimension restoration unit restores the denoised data after convolution calculation to the original size of the Y channel data, thus obtaining the denoised Y channel data. The convolutional network further includes: a first convolutional layer, a first merging layer, a second convolutional layer, a summing layer, a second merging layer, and a third convolutional layer. The first convolutional layer performs a first four-channel convolution on the four-channel data to obtain the first feature data; The first merging layer merges the first feature data with the Y channel data to obtain the second feature data, thereby merging the original, unsampled Y channel data input into the noise reduction model with the first feature data in a memory sense. The second convolutional layer performs a second four-channel convolution on the second feature data to obtain the third feature data; The summation layer adds the first feature data to the third feature data to obtain the fourth feature data; The second merging layer merges the fourth feature data with the Y channel data to obtain the fifth feature data, thereby merging the original feature data once more based on the fourth feature data. The third convolutional layer performs a third four-channel convolution on the fifth feature data to obtain the denoised data.

2. The method according to claim 1, characterized in that, After establishing the denoising model and before using the trained denoising model to denoise the Y channel data in the YUV format data to obtain the denoised Y channel data, the method further includes: Training YUV data is obtained by decoding pre-stored high-definition video data; Based on the training YUV data, a high-definition frame sequence and a low-definition frame sequence of the Y channel were generated as training data pairs. The trained noise reduction model is obtained by training the noise reduction model using a complex network model based on the training data.

3. The method according to claim 2, characterized in that, The training YUV data includes lossless YUV data and compressed YUV data. The step of decoding pre-stored high-definition video data to obtain training YUV data further includes: The pre-stored high-definition video data is compressed to obtain compressed data, and the compressed data is decoded to obtain compressed YUV data; The lossless YUV data is obtained by decoding the pre-stored high-definition video data. The step of generating a high-resolution Y-channel frame sequence and a low-resolution Y-channel frame sequence from the training YUV data as training data further includes: The Y channel data is extracted from the compressed YUV data to obtain the Y channel low-resolution frame sequence, and the Y channel data is extracted from the lossless YUV data to obtain the Y channel high-resolution frame sequence, so that the two can be used as training data pairs.

4. The method according to claim 2, characterized in that, The step of training the denoising model using a complex network model based on the training data to obtain the trained denoising model further includes: Based on the training data, the trained high-definition frame sequence is obtained by training the complex network model. The noise reduction model is trained by using the trained high-definition frame sequence and the low-definition frame sequence to form distilled training data.

5. A video processing method, characterized in that, include: Acquire video data; The noise reduction processing module, operating in a multi-threaded processing mode, performs video noise reduction processing on the video data. The adjustment module, operating in multi-threaded processing mode, adjusts the brightness and / or saturation of the video after noise reduction to obtain processed video data. The video noise reduction process includes: decoding the acquired video data to obtain YUV format data; The Y channel data in the YUV format data is denoised using a trained denoising model to obtain denoised Y channel data. The UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete video noise reduction. Before decoding the acquired video data to obtain YUV format data, the video noise reduction process further includes: establishing a noise reduction model, which includes a downsampling unit, a convolutional network, and a dimension restoration unit. The downsampling unit uses a space-to-depth method to downsample the Y channel data into four-channel data with half the resolution. The convolutional network is a three-layer, four-channel convolutional network that performs convolution calculations on the four-channel data. The dimension restoration unit restores the noise-reduced data after convolution calculations to the original size of the Y channel data, thus obtaining the noise-reduced Y channel data. The convolutional network further includes: a first convolutional layer, a first merging layer, a second convolutional layer, an summing layer, a second merging layer, and a third convolutional layer. The first convolutional layer performs a first four-channel convolution on the four-channel data to obtain first feature data. The first merging layer merges the first feature data with the Y-channel data to obtain second feature data, thereby merging the original, unsampled Y-channel data input to the denoising model with the first feature data in a memory sense. The second convolutional layer performs a second four-channel convolution on the second feature data to obtain third feature data. The summing layer adds the first feature data and the third feature data to obtain fourth feature data. The second merging layer merges the fourth feature data with the Y-channel data to obtain fifth feature data, thereby merging the original feature data once more based on the fourth feature data. The third convolutional layer performs a third four-channel convolution on the fifth feature data to obtain the denoised data.

6. A video noise reduction device, characterized in that, include: The acquisition module is used to acquire video data; The noise reduction module is used to decode the acquired video data to obtain YUV format data; The Y channel data in the YUV format data is denoised using a trained denoising model to obtain denoised Y channel data. The UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete video noise reduction. The noise reduction module is further used to establish a noise reduction model before decoding the acquired video data to obtain YUV format data. The noise reduction model includes a downsampling unit, a convolutional network, and a dimension restoration unit. The downsampling unit uses the Space_to_depth method to downsample the Y channel data into four-channel data with half the resolution; The convolutional network is a three-layer, four-channel convolutional network that performs convolution calculations on the four-channel data. The dimension restoration unit restores the denoised data after convolution calculation to the original size of the Y channel data, thus obtaining the denoised Y channel data. The convolutional network further includes: a first convolutional layer, a first merging layer, a second convolutional layer, an summing layer, a second merging layer, and a third convolutional layer. The first convolutional layer performs a first four-channel convolution on the four-channel data to obtain first feature data. The first merging layer merges the first feature data with the Y-channel data to obtain second feature data, thereby merging the original, unsampled Y-channel data input to the denoising model with the first feature data in a memory sense. The second convolutional layer performs a second four-channel convolution on the second feature data to obtain third feature data. The summing layer adds the first feature data and the third feature data to obtain fourth feature data. The second merging layer merges the fourth feature data with the Y-channel data to obtain fifth feature data, thereby merging the original feature data once more based on the fourth feature data. The third convolutional layer performs a third four-channel convolution on the fifth feature data to obtain the denoised data.

7. A video processing apparatus, characterized in that, include: The input module is used to acquire video data; A noise reduction processing module is used to perform video noise reduction processing on the video data in a multi-threaded processing mode. The adjustment module is used to adjust the brightness and / or saturation of the video noise-reduced data in a multi-threaded processing mode to obtain video processing data. The video noise reduction process includes: decoding the acquired video data to obtain YUV format data; The Y channel data in the YUV format data is denoised using a trained denoising model to obtain denoised Y channel data. The UV channel data in the YUV format data is merged with the noise-reduced Y channel data to complete video noise reduction. Before decoding the acquired video data to obtain YUV format data, the video noise reduction process further includes: establishing a noise reduction model, which includes a downsampling unit, a convolutional network, and a dimension restoration unit. The downsampling unit uses a space-to-depth method to downsample the Y channel data into four-channel data with half the resolution. The convolutional network is a three-layer, four-channel convolutional network that performs convolution calculations on the four-channel data. The dimension restoration unit restores the noise-reduced data after convolution calculations to the original size of the Y channel data, thus obtaining the noise-reduced Y channel data. The convolutional network further includes: a first convolutional layer, a first merging layer, a second convolutional layer, an summing layer, a second merging layer, and a third convolutional layer. The first convolutional layer performs a first four-channel convolution on the four-channel data to obtain first feature data. The first merging layer merges the first feature data with the Y-channel data to obtain second feature data, thereby merging the original, unsampled Y-channel data input to the denoising model with the first feature data in a memory sense. The second convolutional layer performs a second four-channel convolution on the second feature data to obtain third feature data. The summing layer adds the first feature data and the third feature data to obtain fourth feature data. The second merging layer merges the fourth feature data with the Y-channel data to obtain fifth feature data, thereby merging the original feature data once more based on the fourth feature data. The third convolutional layer performs a third four-channel convolution on the fifth feature data to obtain the denoised data.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the following: The method as described in any one of claims 1-4, or The method as described in claim 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4, or When the program is executed by the processor, it implements the method as described in claim 5.

Images