Video processing method, system, device, equipment and medium

Abstract

The present application relates to a video processing method, system, device, equipment and medium. The method comprises: obtaining a video to be processed, and extracting the first brightness channel information of the Y channel in the video to be processed; processing the brightness channel information within the specified filter window range using a bilateral filtering algorithm to obtain spatial proximity and pixel values; determining the corresponding processing method for the weight of the kernel function for the pixel value based on the comparison result between the pixel value and the pixel threshold; calculating the target pixel value using the obtained kernel function weight; replacing the pixel value in the first brightness channel information with the target pixel value to obtain the second brightness channel information. By optimizing the bilateral filtering algorithm, the commonly used convolutional neural network accelerator can efficiently accelerate the bilateral filtering, releasing the computing resources of the GPU and CPU. By using the accelerator quantization mode, the pre-processing and post-processing processes are eliminated, while avoiding the disadvantage of the accelerator's low performance in matrix mathematical operations, giving full play to the accelerator performance.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video processing method, system, device, equipment and medium. Background Art

[0002] The main purpose of image denoising is to remove noise from an image while preserving image details and important information as much as possible. Specifically, denoising technology needs to consider several key issues, including noise identification and separation, how to preserve details and edges as much as possible, and at the same time, improving algorithm efficiency while ensuring denoising quality so that it can be applied to image processing in real time has become a current challenge.

[0003] As for most noise reduction technologies currently available on the market, simple algorithms such as linear spatial domain filtering (mean filtering, median filtering, Gaussian filtering) will mostly cause image blur, fail to retain the required details, and fail to meet the requirements. Nonlinear filtering methods (bilateral filtering) combine the similarity of spatial domain and pixel value domain, and although they are effective in preserving image edges and details and removing various types of noise, they are computationally complex and slow in processing speed. Summary of the invention

[0004] The purpose of this application is to provide a video processing method, system, device, equipment and medium, which can solve the problem of efficient noise reduction of images.

[0005] According to a first aspect of an embodiment of the present application, a video processing method is provided, including:

[0006] Obtain a video to be processed, and extract first brightness channel information of a Y channel in YUV information contained in the video to be processed;

[0007] In the process of calculating the target pixel using the bilateral filtering algorithm, the first brightness channel information is processed within the specified filtering window range to obtain the spatial proximity and the pixel value;

[0008] Based on the pixel value, determining a processing method for a weight of a kernel function corresponding to the pixel value;

[0009] Utilizing the obtained weight of the kernel function to optimize the operator in the bilateral filtering algorithm, and calculating the target pixel value;

[0010] After the pixel value in the first brightness channel information is replaced with the target pixel value, the second brightness channel information after denoising is obtained.

[0011] Optionally, the determining, based on the pixel value, a processing method of a weight of a kernel function corresponding to the pixel value includes:

[0012] The weight of the kernel function corresponding to the pixel value is determined by the relative position of the pixel point and the target pixel, and the difference between the pixel value of each point in the filter window and the pixel value of the target position; wherein the weight is implemented using a convolutional neural network accelerator.

[0013] Optionally, the convolutional neural network accelerator comprises: at least one of a plurality of convolutional layers, division layers and multiplication layers; wherein the first convolutional layer is used to calculate a pixel difference between an adjacent pixel value and a current pixel value;

[0014] The division layer is used to determine a processing method for the weight of the kernel function corresponding to the pixel value according to the size relationship between the pixel difference and the pixel threshold;

[0015] The second convolution layer is used to calculate the target pixel value using the obtained weight of the kernel function. Optionally, the method of determining the corresponding processing method of the weight of the kernel function for the pixel value based on the comparison result of the pixel value and the pixel threshold includes:

[0016] Obtaining the pixel value of each pixel point in the image of the video to be processed;

[0017] Calculate pixel differences between adjacent pixels according to the pixel values;

[0018] When the pixel difference is between the first pixel threshold and the second pixel threshold, the weight assignment of the kernel function is calculated using a fitting function within a valid interval; the fitting function formula is:

[0019] ;

[0020] in, is the hyperparameter value of the fitting.

[0021] Optionally, before using a bilateral filtering algorithm to process the first brightness channel information within a specified filtering window range, a weighted technical approach for calculating the spatial proximity includes:

[0022] The weight of the spatial proximity is calculated based on the spatial distance and standard deviation between pixels, and the calculation formula is:

[0023] ;

[0024] Among them, i represents the abscissa of the filter window, j represents the ordinate of the filter window, k represents the abscissa range, and l represents the ordinate range. Represents the spatial domain standard deviation.

[0025] Optionally, the diameter range of the filter window is ;

[0026] The spatial domain standard deviation is , pixel domain standard deviation .

[0027] Optionally, the weights of the bilateral filtering algorithm are optimized as follows:

[0028] ;

[0029] in is a fixed value calculated in advance;

[0030] Normalization is performed according to the weights to obtain the bilateral filtering algorithm formula:

[0031] .

[0032] Optionally, the data format of the video to be processed is UINT8 data format.

[0033] According to a second aspect of an embodiment of the present application, a video processing system is provided, including:

[0034] The client is used to provide a video to be processed; wherein the video to be processed contains YUV information;

[0035] The server side includes a preprocessing unit and an artificial intelligence model; wherein the preprocessing unit is used to extract the first brightness channel information of the Y channel from the YUV information;

[0036] The artificial intelligence model comprises at least one of a plurality of convolutional layers, division layers and multiplication layers; wherein the first convolutional layer is used to calculate the pixel difference between the adjacent pixel value and the current pixel value;

[0037] The division layer is used to determine a processing method for the weight of the kernel function corresponding to the pixel value according to the size relationship between the pixel difference and the pixel threshold;

[0038] The second convolution layer is used to calculate the target pixel value using the obtained weight of the kernel function; after replacing the pixel value in the first brightness channel information with the target pixel value, the denoised second brightness channel information is obtained.

[0039] Optionally, the preprocessing unit is further used to fill the extracted Y channel according to the size of the filter window to ensure that the size of the output image is consistent with the input image after the subsequent convolution operation.

[0040] Optionally, the multiplication layer is used to perform square calculation on the pixel difference to obtain a square calculation result;

[0041] The third convolutional layer and the division layer are used to calculate the weight of the spatial proximity using the square calculation result and the spatial standard deviation.

[0042] Optionally, the fourth convolutional layer is used to calculate the original target kernel function by using the weight of the spatial proximity and the product of the function variable value;

[0043] Use the fifth convolutional layer to calculate the sum of the original values ​​of each channel;

[0044] A division calculation result of the original value and the sum of the original values ​​is calculated using a division layer to obtain a target kernel function corresponding to the original target kernel function.

[0045] Optionally, the sixth convolutional layer is used to perform kernel function calculation on input channels corresponding to other pixels in the filter window;

[0046] Perform weighted summation according to the kernel function calculation results to obtain a target pixel value;

[0047] The target pixel value is used to replace the pixel value in the first brightness channel information of the Y channel to obtain the second brightness channel information.

[0048] According to a third aspect of an embodiment of the present application, a video processing device is provided, including:

[0049] An acquisition module is used to acquire a video to be processed and extract first brightness channel information of a Y channel in YUV information contained in the video to be processed;

[0050] A processing module, used for processing the brightness channel information within a specified filtering window range by using a bilateral filtering algorithm to obtain spatial proximity and pixel values;

[0051] A determination module, used for determining a processing method of a weight of a kernel function corresponding to the pixel value based on a comparison result between the pixel value and the pixel threshold;

[0052] A calculation module, used to calculate a target pixel value using the obtained weight of the kernel function;

[0053] A replacement module is used to replace the pixel value in the first brightness channel information with the target pixel value to obtain the second brightness channel information after denoising.

[0054] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the method described in the first aspect above.

[0055] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the executable computer program in the storage medium is executed by a processor, the method described in the first aspect above can be implemented.

[0056] Compared with the prior art, the beneficial effects of the present application are: obtaining a video to be processed, and extracting the first brightness channel information of the Y channel in the YUV information of the video to be processed; using a bilateral filtering algorithm to process the brightness channel information within a specified filtering window range to obtain spatial proximity and pixel values; based on the comparison result of the pixel value and the pixel threshold, determining the corresponding processing method for the weight of the kernel function for the pixel value; using the obtained weight of the kernel function to calculate the target pixel value; replacing the pixel value in the first brightness channel information with the target pixel value, obtaining the second brightness channel information after denoising. By modifying and optimizing the bilateral filtering algorithm, a commonly used convolutional neural network accelerator can be used to efficiently accelerate bilateral filtering, thereby releasing the computing resources of the GPU and CPU. The quantization mode of the accelerator is utilized to remove the pre-processing and post-processing processes, while avoiding the disadvantages of the accelerator's low performance in matrix mathematical operations, and giving full play to the performance of the accelerator. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of a video processing method provided in an embodiment of the present application.

[0058] Figure 2 A schematic diagram of the structure of a video processing system provided in an embodiment of the present application.

[0059] Figure 3 A schematic diagram of the system processing flow provided as an example in an embodiment of the present application.

[0060] Figure 4 is a block diagram of a video processing device according to an exemplary embodiment.

[0061] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] Unless otherwise defined, the technical terms or scientific terms used in this specification and claims shall have the usual meanings understood by persons with ordinary skills in the technical field to which the invention belongs. The specific embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be noted that in the specific description of these embodiments, in order to provide a concise description, it is impossible for this specification to provide a detailed description of all the features of the actual embodiments. Without departing from the spirit and scope of the present invention, those skilled in the art may modify and replace the embodiments of the present invention, and the resulting embodiments are also within the scope of protection of the present invention.

[0063] In the current digital age, images play an extremely important role. They not only occupy a core position in information transmission and communication, but are also widely used in various fields. There are many ways to obtain images, but due to the instability of image sources, noise in images has become a problem that cannot be ignored in the current era. The presence of noise will lead to a decrease in image quality, data loss during transmission, waste of excess bandwidth, etc. It will also introduce errors in the image processing process, seriously affecting the codec algorithm and subsequent image analysis and recognition. Therefore, image denoising has become an extremely important preprocessing method.

[0064] The main purpose of image denoising is to remove noise from an image while preserving image details and important information as much as possible. Specifically, denoising technology needs to consider several key issues, including noise identification and separation, how to preserve details and edges as much as possible, and at the same time, improving algorithm efficiency while ensuring denoising quality so that it can be applied to image processing in real time has become a current challenge.

[0065] As for most noise reduction technologies currently available on the market, simple algorithms such as linear spatial domain filtering (mean filtering, median filtering, Gaussian filtering) will mostly cause image blur, fail to retain the required details, and fail to meet the requirements. Nonlinear filtering methods (bilateral filtering) combine the similarity of spatial domain and pixel value domain, with high computational complexity and slow processing speed.

[0066] Because multiple frames need to be denoised during the encoding and decoding process, the performance and real-time requirements are very high, and there are also certain requirements for the quality of the encoded and decoded video. Therefore, the filtering and denoising mentioned above are not suitable for direct CPU operation. In addition, with the widespread use of video encoding and decoding hardware, transferring each frame to the host CPU for processing will cause serious delays and waste a lot of memory bandwidth.

[0067] For example, during the encoding and decoding process, multiple frames usually need to be denoised, which places high demands on performance and real-time performance while also ensuring video quality. In application scenarios with high throughput, low latency, and minimal power consumption, the use of customized hardware accelerators is a common choice. However, due to the limitations of chip size and power consumption, the performance of the built-in processors of these hardware accelerators is often not powerful enough to achieve real-time complex algorithm denoising.

[0068] If the host CPU is used for preprocessing, although the performance problem can be solved, it requires frequent data exchange between the accelerator and the host, which not only increases latency but is also limited by the DDR performance bottleneck. Therefore, relying solely on the CPU to achieve filtering and denoising is not an ideal solution.

[0069] The ideal implementation is based on a hardware decoder combined with hardware acceleration of the denoising algorithm. This method can complete the entire process of video decoding, denoising and encoding in one chip without the involvement of the host, thereby significantly reducing latency while improving performance.

[0070] With the continuous development of artificial intelligence, especially deep learning technology, its application in the fields of image and audio processing has become more and more extensive. The main implementation method of deep learning is based on deep convolutional neural network (DCNN). Due to its wide range of applications, various deep learning inference hardware accelerators have emerged in the market. Compared with traditional GPU-based inference engines, these hardware accelerators perform better in terms of efficiency and power consumption.

[0071] To further improve inference efficiency, hardware accelerators usually use fixed-point calculations (while GPUs use floating-point calculations), with 8-bit fixed-point calculations being the most commonly used to strike a balance between computational efficiency and accuracy. These accelerators mainly use neural processing units (NPUs) or tensor processing units (TPUs) to implement common operations such as convolution and activation.

[0072] To increase applicability, NPUs are usually used in conjunction with some general-purpose digital signal processors (DSPs). DSPs are responsible for processing some operations that NPUs cannot handle, such as complex mathematical calculations (such as SOFTMAX) and pre-processing and post-processing. However, since NPUs and DSPs belong to different computing units, data handling and data format conversion become bottlenecks, which seriously affect the overall performance of hardware inference accelerators.

[0073] At the same time, with the widespread promotion of deep learning applications, many PCs, servers, and mobile phones have built-in dedicated convolutional neural network accelerators. Unlike traditional general-purpose GPUs, this type of accelerator is cheap and has low power consumption. However, the operators it supports are limited, and the performance is not high for uncommon operators. It is precisely because of these limitations that this type of accelerator is often in standby mode when AI functions such as convolutional neural networks are not used, wasting computing resources. Therefore, in order to solve the above problems, the present application proposes a solution to implement bilateral filtering based on the characteristics of convolutional neural network accelerators, which can effectively improve processing efficiency.

[0074] like Figure 1 A schematic diagram of a video processing method provided in an embodiment of the present application. The method comprises the following steps:

[0075] Step 101: Obtain a video to be processed, and extract first brightness channel information of a Y channel in YUV information contained in the video to be processed.

[0076] Step 102: Process the brightness channel information within the specified filtering window using a bilateral filtering algorithm to obtain spatial proximity and pixel values.

[0077] Step 103: Based on the comparison result between the pixel value and the pixel threshold, determine a corresponding processing method for the weight of the kernel function for the pixel value.

[0078] Step 104: Calculate the target pixel value using the obtained weight of the kernel function.

[0079] Step 105: After replacing the pixel value in the first brightness channel information with the target pixel value, the denoised second brightness channel information is obtained.

[0080] In video images, the YUV channel is an important concept in image processing. It represents a color encoding method, also known as the "Luma-Chroma model". In this model, color information is decomposed into three channels: Y, U, and V. Among them, the Y channel: represents brightness (Luminance or Luma), which is the grayscale value of the image. It describes the brightness of the image without containing any color information. The value range of the Y channel is usually [0, 255] (in 8-bit representation), where 0 represents black and 255 represents white. U channel: represents a component of blue chromaticity (Chrominance or Chroma), which describes the difference in intensity of the blue part of the image relative to the brightness. V channel: represents a component of red chromaticity, which describes the difference in intensity of the red part of the image relative to the brightness.

[0081] Among them, the Y channel extraction process is as follows:

[0082] First, you need to determine the specific format of the YUV data being processed, such as YUV420P, YUV422P, YUV444P, etc. Different formats mean that the arrangement and sampling rate of Y, U, and V data are different.

[0083] Use appropriate programming languages ​​and libraries (such as file operation functions in C / C++ or image processing libraries such as OpenCV) to read YUV data files or images. Make sure to open the file in binary mode because YUV data is raw pixel values ​​and does not contain any file format information. For example, for the YUV420P format (taking the most commonly used format as an example): YUV420P is a planar format in which the Y, U, and V components are stored in different planes. The Y component is stored first, followed by the U component, and finally the V component. In YUV420P, the resolution of the Y component is the same as the original image, while the resolution of the U and V components is half of the Y component (both horizontally and vertically). During the extraction process, calculate the size of the Y component: the size of the Y component is equal to the width of the image multiplied by the height (in bytes). Read the Y component data: read the corresponding number of bytes from the beginning of the YUV data into the data buffer of the Y channel, which is the first brightness channel information of the Y channel mentioned above.

[0084] After extracting the first brightness channel information of the Y channel, the first brightness channel information within the specified filter window is processed using a bilateral filtering algorithm, that is, the spatial domain and the pixel domain are processed simultaneously.

[0085] The bilateral filtering algorithm mentioned here is a type of nonlinear filtering. This is a processing method that combines the spatial proximity of an image with the similarity of pixel values. This method considers both spatial proximity information and color similarity information during the filtering process, thereby removing noise and smoothing the image while retaining the edge details of the image.

[0086] Bilateral filtering combines the effects of two Gaussian filters. The first Gaussian filter is used to calculate the weight of spatial proximity, based on the traditional Gaussian filter principle; the second Gaussian filter is used to calculate the weight of pixel value similarity. With the combined effect of these two Gaussian filters, the effect of bilateral filtering is achieved. In short, bilateral filtering performs a weighted average on each pixel, and its formula can be expressed as:

[0087] ;

[0088] in:

[0089] is the target pixel;

[0090] is the size range of (2N+1)×(2N+1) centered at pixel (i, j) in the original image;

[0091] represents the input pixel points in multiple original images;

[0092] is the weight calculated by two Gaussian functions.

[0093] The weight is determined by the product of two Gaussian filters, namely

[0094] ;

[0095] in:

[0096] is the spatial proximity Gaussian function (spatial domain Gaussian function), that is .

[0097] is the pixel value similarity Gaussian function, that is .

[0098] When the image is in an area with a gentle change, the adjacent pixel values ​​are not much different. It approaches 1 infinitely, so bilateral filtering can be regarded as ordinary Gaussian filtering, thereby achieving image smoothing. When the gap between adjacent pixels is large (such as edges), it will cause Close to 0, the weight approaches 0, causing the area to maintain the original pixel value without processing.

[0099] Notice It is only related to the relative position, so its kernel function is fixed. It is related to the value of the pixel, so for each pixel, the weight of the kernel function needs to be calculated in real time according to the values ​​of other pixels in the window where it is located. This greatly increases the complexity of the calculation. It also makes it difficult to implement using a general convolutional neural network accelerator.

[0100] Transform the bilateral filtering formula and assume is the coefficient , the formula can be expressed as:

[0101] ;

[0102] make , the formula can be expressed as:

[0103] ;

[0104] This can be regarded as obtaining the final value by convolution operator with the image matrix and the kernel.

[0105] It should be noted that, in the present application, in order to avoid being limited by the type of hardware accelerator, the UINT8 data format can be used to process the video.

[0106] In one or more embodiments of the present application, as described in step 103, based on the comparison result between the pixel value and the pixel threshold, the corresponding processing method of the kernel function weight for the pixel value is determined, including the following three situations, which will be described in detail below through specific embodiments. Specifically, it includes:

[0107] Obtaining the pixel value of each pixel in the image of the video to be processed; calculating the pixel difference between adjacent pixels according to the pixel value; and assigning the weight of the kernel function to 1 when the pixel difference is greater than a first pixel threshold. Or,

[0108] Obtaining the pixel value of each pixel point in the image of the video to be processed; calculating the pixel difference between adjacent pixel points according to the pixel value; and assigning the weight of the kernel function to 0 when the pixel difference is less than a second pixel threshold.

[0109] In practical applications, when the image is in a region with a gentle change, the adjacent pixel values ​​are not much different. In this case, Wr approaches 1 infinitely, so bilateral filtering can be regarded as ordinary Gaussian filtering, thereby achieving image smoothing. When the difference between adjacent pixels is large (such as edges), Wr will be close to 0, making the weight approach 0, resulting in the original pixel value of the area being maintained without processing.

[0110] The calculation process of adjacent pixel differences is as follows:

[0111] In image processing, the difference between adjacent pixels is calculated by comparing the pixel values ​​of the current pixel with those of its surrounding pixels. In the context of bilateral filtering, this difference is often used to measure the similarity or distance between pixels.

[0112] Assume that when a 3x3 convolution kernel (or filter window) slides on the image, it covers a 3x3 pixel area. In this area, in addition to the central pixel (called f(i,j)), there are 8 neighboring pixels around it.

[0113] In order to calculate the difference between adjacent pixels, it is necessary to compare the central pixel f(i,j) with the eight adjacent pixels. For each adjacent pixel f(k,l) (where k and l represent the row and column index of the adjacent pixel, respectively), the difference between them can be calculated. This difference can be a simple difference in pixel values, that is:

[0114] Difference = f(i,j) − f(k,l)

[0115] However, in the weight calculation process of bilateral filtering, we may actually be more concerned with the square of these differences, because the squared differences can better reflect the similarity (or distance) between pixels and avoid the appearance of negative values. Therefore, it is more common to calculate the square of the difference:

[0116] Difference Squared=(f(i,j)−f(k,l)) 2

[0117] In addition, a filter window of 3×3 means that the convolutional layer will output 9 channels, each channel corresponding to a specific difference calculation method (although in the actual bilateral filtering implementation, these differences may not be directly output as channels, but they are further processed).

[0118] However, to simplify the processing, the convolutional layer may be designed to directly calculate the square of the difference, or multiple convolutional layers may be used to complete the difference calculation and square operation respectively.

[0119] In the calculation of the weights for the bilateral filter, the squares of these differences (or more generally, a similarity measure between pixels) are taken into account along with the spatial distance (determined by the geometric position of the convolution kernel) and the difference in pixel values ​​(determined by the squares of the differences) to generate the final filter weights.

[0120] After the pixel difference is calculated, in addition to the two cases described above, where Wr is assigned a value of 0 or 1, there is another case where a fitting calculation is performed within a certain range.

[0121] When the pixel difference is between the first pixel threshold and the second pixel threshold, the weight assignment of the kernel function is calculated using a fitting function within a valid interval; the fitting function formula is:

[0122] ;

[0123] in, is the hyperparameter value of the fitting.

[0124] For example, the original interval is , the interval to be fitted is [0, 9], .

[0125] Through the above scheme, a fitting method within a valid interval can be used to approximate the exponential calculation of Wr, which can effectively reduce the calculation complexity and thus improve the calculation efficiency.

[0126] In one or more embodiments of the present application, the weighted technical approach of the spatial proximity includes: calculating the weighted value of the spatial proximity according to the spatial distance and standard deviation between pixels, and the calculation formula is:

[0127] ;

[0128] Among them, i represents the abscissa of the filter window, j represents the ordinate of the filter window, k represents the abscissa range, and l represents the ordinate range. Represents the spatial domain standard deviation.

[0129] As an optional solution, the diameter range of the filter window is ; The spatial domain standard deviation is , pixel domain standard deviation .

[0130] Edge filter window diameter: The noise pixels in the current example image are usually smaller than the 3x3 pixel area. In this example, the default bilateral filter window diameter is set to 3, that is, (i, j) = (3, 3). The specific parameters can be adjusted according to the characteristics of the actual image.

[0131] Pixel domain standard deviation and spatial domain standard deviation of bilateral filtering: In the experimental example, we hope to retain edge details of the denoised image, so the two standard deviations are selected to be small, that is, σs=5, σr=5. The specific parameters can be adjusted according to the characteristics of the actual image. In practical applications, for example, a 1080P image (1920x1080 pixel size) can be selected as the original image to be processed, and the filter window size is defined as 3x3.

[0132] After using the above scheme to calculate the kernel function weight Wr and the spatial proximity weight Ws, we can further use Wr*Ws, that is:

[0133] ;

[0134] in, .

[0135] The above formula can be further optimized, and the optimized bilateral filter weight formula can be expressed as:

[0136] ;

[0137] After splitting and deducing the formula, we get:

[0138] ;

[0139] in, It is a fixed value and can be calculated in advance.

[0140] Based on the same idea, the embodiment of the present application also proposes a video processing system solution. Figure 2 The schematic diagram of the structure of the video processing system provided in the embodiment of the present application is as follows. Figure 2 As can be seen in the figure, the video processing system includes:

[0141] The client 21 is used to provide a video to be processed; wherein the video to be processed contains YUV information.

[0142] The server side 22 includes a preprocessing unit 221 and an artificial intelligence model 222; wherein the preprocessing unit is used to extract the first brightness channel information of the Y channel from the YUV information;

[0143] The artificial intelligence model 222 includes at least one of a plurality of convolutional layers, division layers, and multiplication layers; wherein the first convolutional layer is used to calculate the pixel difference between the adjacent pixel value and the current pixel value;

[0144] The division layer is used to determine a processing method for the weight of the kernel function corresponding to the pixel value according to the size relationship between the pixel difference and the pixel threshold;

[0145] The second convolution layer is used to calculate the target pixel value using the obtained weight of the kernel function; after replacing the pixel value in the first brightness channel information with the target pixel value, the denoised second brightness channel information is obtained.

[0146] Optionally, the preprocessing unit is further used to fill the extracted Y channel according to the size of the filter window (corresponding to Figure 3 Padding_A in the convolution operation to ensure that the size of the output image is consistent with the input image after the subsequent convolution operation.

[0147] Optionally, the multiplication layer Multiply_Layer_C is used to perform square calculation on the pixel difference to obtain a square calculation result;

[0148] The third convolutional layer (Conv2d_D) and the division layer (Divide_Layer_E) are used to calculate the weight of the spatial proximity using the square calculation result and the spatial standard deviation.

[0149] Optionally, the fourth convolutional layer (Conv2d_F) is used to utilize the weights of the spatial proximity and the function variable values ​​(i.e. The original target kernel function is calculated by multiplying .

[0150] Use the fifth convolutional layer (Conv2d_G) to calculate the sum of the original values ​​of each channel;

[0151] The division layer is used to calculate the division result of the original value and the sum of the original values ​​to obtain the target kernel function corresponding to the original target kernel function. .

[0152] Optionally, the sixth convolutional layer (Conv2d_I) is used to perform kernel function calculation on input channels corresponding to other pixels in the filter window;

[0153] Perform weighted summation based on the kernel function calculation results to obtain the target pixel value g(i,j);

[0154] The target pixel value is used to replace the pixel value in the first brightness channel information of the Y channel to obtain the second brightness channel information.

[0155] For ease of understanding, the following will be based on the system Figure 2 The process of using artificial intelligence models to process videos is described in detail. Figure 3 The following is a schematic diagram of the system processing flow provided by the embodiment of the present application. Figure 3 As can be seen in the figure, the YUV color frames of each frame image in the input video frame are split and the Y channel is extracted separately. This is because the Y channel is a brightness channel and the human eye is most sensitive to the brightness information in the Y channel. Therefore, when performing noise reduction processing, only the Y channel can be subjected to noise reduction processing, which can effectively reduce the amount of calculation and thus improve the calculation efficiency.

[0156] Pad the extracted Y channel (Padding_A) according to the size of the filter window to ensure that the size of the output image is consistent with the input image after the subsequent convolution operation. The number of paddings is ((k-1) / 2), where k is the diameter of the filter window. In this example, the filter window is 3x3, so the number of paddings is ((3-1) / 2 = 1). Pad the original input image of size 1920x1080 to 1922x1082, and the padding value is 0. Through the above padding operation, the input and output sizes are consistent.

[0157] Next, the convolution layer (Conv2d_B) is used for calculation processing. The purpose is to calculate the pixel difference between the neighboring pixel value and the current pixel value, that is, to calculate the pixel difference between the target pixel and the surrounding pixels within the filter window. According to the filter window size, the neighboring pixel values ​​in this example are the 8 points around the input point and itself. The size of this convolution layer is 3×3×1×9, there is one input channel, and the shape of the convolution kernel is 3x3. The pixel difference between all neighboring pixels and the self pixel is output as 9 channels. For example, the parameters of the first channel are: [1, 0, 0; 0, -1, 0; 0, 0, 0].

[0158] It should be noted that the 8 pixels refer to the neighboring pixels around the target pixel in the filter window. For example, when there is a 3x3 convolution kernel (or filter window), it will slide on the image, covering a 3x3 area each time. In this 3x3 area, in addition to the central pixel (the pixel currently being processed), there are 8 pixels around it. These 8 pixels are located at:

[0159] The eight pixels above, below, left, and right of the center pixel, as well as the upper left, upper right, lower left, and lower right diagonal directions, together with the center pixel, constitute all the pixels in the 3x3 area.

[0160] The 9 channels mentioned above refer to the 9 “channels” artificially created during the processing process in order to calculate the difference between the current pixel and the surrounding 8 pixels and process these differences as independent features.

[0161] Specifically, these 9 channels correspond to:

[0162] The difference between the current pixel and the pixel above it;

[0163] The difference between the current pixel and the pixel below it;

[0164] The difference between the current pixel and the pixel to its left;

[0165] The difference between the current pixel and the pixel to its right;

[0166] The difference between the current pixel and its upper left pixel;

[0167] The difference between the current pixel and its upper right pixel;

[0168] The difference between the current pixel and its lower left pixel;

[0169] The difference between the current pixel and its lower right pixel;

[0170] The difference between the current pixel and itself (although this difference is usually 0, it may be retained in some processing for consistency or for special purposes).

[0171] However, in the actual bilateral filtering implementation, the 9th channel (i.e., the difference between the current pixel and itself) may not be explicitly calculated, because it is always 0 and does not contribute to subsequent processing. But for the consistency of processing and the convenience of convolutional layer design, this channel is sometimes still retained. In subsequent steps, these differences will be further processed (such as squaring, normalization, etc.) to calculate the weights required for bilateral filtering and ultimately used to generate the denoised image.

[0172] The output of the convolutional layer (Conv2d_B) is squared (Multiply_Layer_C). The calculation can be regarded as the formula:

[0173] ;

[0174] Use the convolutional layer (Conv2d_D). The purpose is to divide the output of Multiply_Layer_C by The size of this convolutional layer is 1×1×1×9, with one input channel, the shape of the convolution kernel is 1x1, and its weight parameter is , the convolution kernel convolves with each channel of the input data to generate new values ​​as 9 output channels. The bias value (Bias) of this layer is set to 1, so that the output of the final Conv2d_D becomes This step is to calculate The variable value of .

[0175] Use the divide layer (Divide_Layer_E) to complete the division operation. The output of this layer can be regarded as That is, the computational approximation .

[0176] Use the convolutional layer (Conv2d_F) to calculate Ws and The input of this convolutional layer is the result of Divide_Layer_E. The weight parameter size is 1×1 ×1 ×9, which is the pre-calculated spatial domain Gaussian function Ws. The output is the product of each channel and Ws, with a total of 9 channels. Set the offset value of this layer to the pre-calculated To complete the addition operation. So far, the output of Conv2d_F is the weight of the bilateral filter in this bilateral filter calculation, that is, W(i,j,k,l). Through the above steps, the original target kernel function of the kernel function used in bilateral filtering is calculated.

[0177] Use the convolution layer (Conv2d_G) instead of the addition function to get the sum of the bilateral filter weights:

[0178] ;

[0179] The size of this convolutional layer is 1×1×9×9, the input is Conv2d_F, a total of 9 input channels, the weights are all convolution kernels with parameters of 1, and the output is 9 identical output channels, each of which is the sum of the weights. Calculate the sum of the original target kernel function of the kernel function used in bilateral filtering to prepare for the next step of normalization).

[0180] Use the divide layer (Divide_Layer_H) to complete the division operation, dividing Conv2d_F and Conv2d_G. The output of this layer is the final weight of the bilateral filter:

[0181] ;

[0182] The target kernel function used in normalized bilateral filtering.

[0183] Use the convolution layer (Conv2d_I) to copy the current pixel value and the neighboring pixel values. According to the filter window size, in this example, the 8 points around the input point and itself will be copied. The size of this convolution layer is 3×3×1×9, there is one input channel, and the shape of the convolution kernel is 3x3. All neighboring pixels and itself are output as 9 channels. For example, the parameters of the first channel are: [1, 0, 0; 0, 0, 0; 0, 0, 0]. This step is to rearrange the original pixels and prepare the input arrangement format required for bilateral filtering in this application.

[0184] Use the multiply layer (Multiply_Layer_J) to complete the multiplication, multiply the results of Conv2d_I and Divide_Layer_H. The output of this layer can be expressed as:

[0185] ;

[0186] That is, the target kernel function is used to perform weighted processing on the 9 channel outputs.

[0187] Use the convolution layer (Conv2d_K) instead of the addition function to get the sum of the products just now. The input of this convolution layer is Multiply_Layer_J, with a total of nine input channels, the weights are all 1, the convolution kernel size is 1×1×9×1, and the output is a total of one output channel. It is the target pixel value of the Y channel after bilateral filtering.

[0188] The target pixel value of the Y channel after bilateral filtering replaces the pixel value of the Y channel of the original input, that is, the denoised YUV image is generated.

[0189] Figure 4 FIG. 1 is a block diagram of a video processing device according to an exemplary embodiment. Figure 4 As shown, in this embodiment, the video processing device includes:

[0190] The acquisition module 41 is used to acquire the video to be processed and extract the first brightness channel information of the Y channel in the YUV information of the video to be processed;

[0191] A processing module 42, configured to process the brightness channel information within a specified filtering window range using a bilateral filtering algorithm to obtain spatial proximity and pixel values;

[0192] A determination module 43, configured to determine a processing method for a weight of a kernel function corresponding to the pixel value based on a comparison result between the pixel value and the pixel threshold;

[0193] A calculation module 44, used to calculate a target pixel value using the obtained weight of the kernel function;

[0194] The replacement module 45 is used to replace the pixel value in the first brightness channel information with the target pixel value to obtain the second brightness channel information after denoising.

[0195] A determination module 43 is used to obtain the pixel value of each pixel point in the image of the video to be processed;

[0196] Calculate pixel differences between adjacent pixels according to the pixel values;

[0197] When the pixel difference is greater than the first pixel threshold, the weight of the kernel function is assigned to 1.

[0198] A determination module 43 is used to obtain the pixel value of each pixel point in the image of the video to be processed;

[0199] Calculate pixel differences between adjacent pixels according to the pixel values;

[0200] When the pixel difference is less than the second pixel threshold, the weight of the kernel function is assigned to 0.

[0201] A determination module 43 is used to obtain the pixel value of each pixel point in the image of the video to be processed;

[0202] Calculate pixel differences between adjacent pixels according to the pixel values;

[0203] When the pixel difference is between the first pixel threshold and the second pixel threshold, the weight assignment of the kernel function is calculated using a fitting function within a valid interval; the fitting function formula is:

[0204] ;

[0205] in, is the hyperparameter value of the fitting.

[0206] Optionally, the weighting technology of the spatial proximity includes:

[0207] The weight of the spatial proximity is calculated based on the spatial distance and standard deviation between pixels, and the calculation formula is:

[0208] ;

[0209] Among them, i represents the abscissa of the filter window, j represents the ordinate of the filter window, k represents the abscissa range, and l represents the ordinate range. Represents the spatial domain standard deviation.

[0210] Optionally, the diameter range of the filter window is ;

[0211] The spatial domain standard deviation is , pixel domain standard deviation .

[0212] Optionally, the data format of the video to be processed is UINT8 data format.

[0213] Obtain the video to be processed, and extract the first brightness channel information of the Y channel in the YUV information of the video to be processed; use the bilateral filtering algorithm to process the brightness channel information within the specified filtering window range to obtain spatial proximity and pixel value; based on the comparison result of the pixel value and the pixel threshold, determine the corresponding processing method of the kernel function weight for the pixel value; use the obtained kernel function weight to calculate the target pixel value; replace the pixel value in the first brightness channel information with the target pixel value, and obtain the second brightness channel information after denoising. By modifying and optimizing the bilateral filtering algorithm, the commonly used convolutional neural network accelerator can be used to efficiently accelerate the bilateral filtering, thereby releasing the computing resources of the GPU and CPU. The quantization mode of the accelerator is utilized to remove the pre-processing and post-processing processes, while avoiding the disadvantage of the accelerator's low performance in matrix mathematical operations, and giving full play to the performance of the accelerator. Through the above scheme, the technical characteristics of the deep learning hardware inference accelerator are fully utilized, which has the advantages of high efficiency and flexibility. Compared with the traditional use of CPU and other implementation methods, the efficiency is significantly improved. It can achieve maximum throughput and minimum delay, and does not require host participation at all.

[0214] An embodiment of the present application also proposes an electronic device, comprising a processor and a memory; the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the video processing method described in any of the above embodiments.

[0215] The embodiments of the present application further provide a computer-readable storage medium. When the executable computer program in the storage medium is executed by a processor, the video processing method described in any of the above embodiments can be implemented.

[0216] Regarding the device in the above embodiment, the specific manner in which the processor performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0217] Figure 5 900 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 900 may be provided as a server. Figure 5 , the device 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as applications, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the above-mentioned video processing method.

[0218] The device 900 may also include a power component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0219] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 932 including instructions, which can be executed by the processing component 922 of the device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0220] In the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

[0221] The above description of the embodiments is to facilitate those of ordinary skill in the art to understand and apply the present application. It is obvious that those familiar with the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without having to pay creative labor. Therefore, the present application is not limited to the embodiments herein, and improvements and modifications made by those skilled in the art based on the contents disclosed in the present application without departing from the scope and spirit of the present application are within the scope of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Obtain a video to be processed, and extract first brightness channel information of a Y channel in YUV information contained in the video to be processed; In the process of calculating the target pixel using the bilateral filtering algorithm, the first brightness channel information within the specified filtering window is processed to obtain the weight and pixel value of the spatial proximity; Calculating the weight of the spatial proximity includes: calculating the weight of the spatial proximity according to the spatial distance and standard deviation between pixels, and the calculation formula is: ； Among them, i represents the abscissa of the filter window, j represents the ordinate of the filter window, k represents the abscissa range, and l represents the ordinate range. represents the standard deviation in the spatial domain; Based on the pixel value, determining a processing method for a weight of a kernel function corresponding to the pixel value; Specifically, the method includes: obtaining the pixel value of each pixel point in the image of the video to be processed; calculating the pixel difference between adjacent pixel points according to the pixel value; when the size of the pixel difference is between the first pixel threshold and the second pixel threshold, calculating the weight assignment of the kernel function by using the fitting function within the effective interval; the fitting function formula is: ； in, is the hyperparameter value of the fitting; The operator in the bilateral filtering algorithm is optimized by using the weight of the kernel function obtained to calculate the target pixel value; specifically comprising: performing kernel function calculation on the input channels corresponding to other pixels in the filter window; performing weighted summation according to the kernel function calculation results to obtain the target pixel value; The weights for optimizing the bilateral filtering algorithm are as follows: ； in is a fixed value calculated in advance; Normalization is performed according to the weights to obtain the bilateral filtering algorithm formula: ； in: is the target pixel; is the size range of (2N+1)×(2N+1) centered at pixel (i, j) in the original image; represents the input pixel points in multiple original images; After the pixel value in the first brightness channel information is replaced with the target pixel value, the second brightness channel information after denoising is obtained.

2. The method according to claim 1, characterized in that The method of determining the weight of the kernel function corresponding to the pixel value based on the pixel value includes: The weight of the kernel function corresponding to the pixel value is determined by the relative position of the pixel point and the target pixel, and the difference between the pixel value of each point in the filter window and the pixel value of the target position; wherein the weight is implemented using a convolutional neural network accelerator.

3. The method according to claim 2, characterized in that The convolutional neural network accelerator includes: at least one of a plurality of convolutional layers, division layers and multiplication layers; wherein the first convolutional layer is used to calculate a pixel difference between an adjacent pixel value and a current pixel value; The division layer is used to determine a processing method for the weight of the kernel function corresponding to the pixel value according to the size relationship between the pixel difference and the pixel threshold; The second convolution layer is used to calculate the target pixel value using the obtained weight of the kernel function.

4. The method according to claim 1, characterized in that The diameter range of the filter window ; The spatial domain standard deviation is , pixel domain standard deviation .

5. The method according to claim 1, characterized in that The data format of the video to be processed is UINT8 data format.

6. A video processing system, characterized in that: The system comprises: The client is used to provide a video to be processed; wherein the video to be processed contains YUV information; The server side includes a preprocessing unit and an artificial intelligence model; wherein the preprocessing unit is used to extract the first brightness channel information of the Y channel from the YUV information; The artificial intelligence model comprises at least one of a plurality of convolutional layers, division layers and multiplication layers; wherein the first convolutional layer is used to calculate the pixel difference between the adjacent pixel value and the current pixel value; The division layer is used to determine the processing method of the weight of the kernel function corresponding to the pixel value according to the size relationship between the pixel difference and the pixel threshold; specifically including: obtaining the pixel value of each pixel point in the image of the video to be processed; calculating the pixel difference between adjacent pixel points according to the pixel value; when the size of the pixel difference is between the first pixel threshold and the second pixel threshold, using the fitting function to calculate the weight assignment of the kernel function within the effective interval; the fitting function formula is: ； in, is the hyperparameter value of the fitting; Calculating the weight of the spatial proximity includes: calculating the weight of the spatial proximity according to the spatial distance and standard deviation between pixels, and the calculation formula is: ； Among them, i represents the abscissa of the filter window, j represents the ordinate of the filter window, k represents the abscissa range, and l represents the ordinate range. represents the standard deviation in the spatial domain; The second convolutional layer is used to calculate the target pixel value using the obtained weight of the kernel function; after replacing the pixel value in the first brightness channel information with the target pixel value, the second brightness channel information after denoising is obtained, wherein the calculation method of the target pixel value includes: performing kernel function calculation on the input channel corresponding to other pixels in the filter window; performing weighted summation according to the calculation result of the kernel function to obtain the target pixel value; the weights of the optimized bilateral filtering algorithm are as follows: ； in is a fixed value calculated in advance; Normalization is performed according to the weights to obtain the bilateral filtering algorithm formula: ； in: is the target pixel; is the size range of (2N+1)×(2N+1) centered at pixel (i, j) in the original image; Represents the input pixels in multiple original images.

7. The system according to claim 6, characterized in that The preprocessing unit is further used to fill the extracted Y channel according to the size of the filter window to ensure that the size of the output image is consistent with the input image after the subsequent convolution operation.

8. The system according to claim 6, characterized in that The multiplication layer is used to perform square calculation on the pixel difference to obtain a square calculation result; The third convolutional layer and the division layer are used to calculate the weight of the spatial proximity using the square calculation result and the spatial standard deviation.

9. The system according to claim 8, characterized in that The fourth convolutional layer is used to calculate the original target kernel function by using the weight of the spatial proximity and the product of the function variable value; Use the fifth convolutional layer to calculate the sum of the original values ​​of each channel; A division calculation result of the original value and the sum of the original values ​​is calculated using a division layer to obtain a target kernel function corresponding to the original target kernel function.

10. The system according to claim 9, characterized in that The sixth convolutional layer is used to calculate the kernel function of the input channels corresponding to other pixels in the filter window; Perform weighted summation according to the kernel function calculation results to obtain a target pixel value; The target pixel value is used to replace the pixel value in the first brightness channel information of the Y channel to obtain the second brightness channel information.

11. A video processing device, characterized in that: The device comprises: An acquisition module is used to acquire a video to be processed and extract first brightness channel information of a Y channel in YUV information contained in the video to be processed; A processing module, used for processing the brightness channel information within the specified filter window range in the process of calculating the target pixel value using the bilateral filtering algorithm to obtain the weight and pixel value of the spatial proximity; Calculating the weight of the spatial proximity includes: calculating the weight of the spatial proximity according to the spatial distance and standard deviation between pixels, and the calculation formula is: ； Among them, i represents the abscissa of the filter window, j represents the ordinate of the filter window, k represents the abscissa range, and l represents the ordinate range. represents the standard deviation in the spatial domain; A determination module, used for determining a processing method of a weight of a kernel function corresponding to the pixel value based on the pixel value; Specifically, the method includes: obtaining the pixel value of each pixel point in the image of the video to be processed; calculating the pixel difference between adjacent pixel points according to the pixel value; when the size of the pixel difference is between the first pixel threshold and the second pixel threshold, calculating the weight assignment of the kernel function by using the fitting function within the effective interval; the fitting function formula is: ； in, is the hyperparameter value of the fitting; A calculation module is used to optimize the operator in the bilateral filtering algorithm by using the obtained weight of the kernel function to calculate the target pixel value; specifically comprising: performing kernel function calculation on the input channels corresponding to other pixels in the filter window; performing weighted summation according to the kernel function calculation results to obtain the target pixel value; The weights for optimizing the bilateral filtering algorithm are as follows: ； in is a fixed value calculated in advance; Normalization is performed according to the weights to obtain the bilateral filtering algorithm formula: ； in: is the target pixel; is the size range of (2N+1)×(2N+1) centered at pixel (i, j) in the original image; represents the input pixel points in multiple original images; A replacement module is used to replace the pixel value in the first brightness channel information with the target pixel value to obtain the second brightness channel information after denoising.

12. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 5.

13. A computer-readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 5.

Images