Image compression noise removal method and device, electronic equipment and storage medium

Abstract

The invention discloses an image compression noise removal method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a to-be-processed compressed image frame; detecting the compressed image frame to be processed based on a target noise detection model to obtain a target noise intensity matrix; constructing a target filtering template corresponding to the first pixel point based on the element corresponding to the first pixel point and a plurality of target pixel points including the first pixel point; and performing filtering processing on the first pixel point by applying the target filtering template to obtain a target compressed image frame. According to the invention, noise detection is carried out on the to-be-processed compressed image frame through the target noise detection model to obtain the target noise intensity matrix used for representing the noise distribution condition of the to-be-processed compressed image frame, the detection precision of image compression noise is improved, and denoising processing is carried out on the to-be-processed compressed image frame based on the target noise intensity matrix. While the image compression noise is removed, more detail information of the image is reserved, so that the display effect of the image is optimized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for removing image compression noise. Background Art

[0002] With the development of digital multimedia technology, image compression technology has emerged to improve data transmission rate and storage efficiency. Image compression technology is a technology that represents the original pixel data of an image with fewer bits, either losslessly or losslessly.

[0003] Image compression technology is used to optimize image data transmission to increase the transmission rate. However, this process also introduces compression noise, which manifests as visible banding noise on the image, thereby degrading the image's display quality and affecting the user's visual experience. Summary of the Invention

[0004] The present invention provides a method, device, electronic device and storage medium for removing image compression noise, which are used to solve the problem in the prior art that the use of image compression technology leads to the introduction of compression noise into the compressed image, resulting in a reduction in the display effect of the image.

[0005] In a first aspect, an embodiment of the present invention provides a method for removing image compression noise, comprising:

[0006] Obtaining a compressed image frame to be processed;

[0007] Based on the target noise detection model, the compressed image frame to be processed is detected to obtain a target noise intensity matrix, wherein the target noise intensity matrix includes multiple elements corresponding to multiple pixels included in the compressed image frame to be processed, and the elements are used to represent the compression noise intensity of the corresponding pixels;

[0008] Constructing a target filtering template corresponding to a first pixel based on an element corresponding to a first pixel and a plurality of target pixels including the first pixel, wherein the first pixel is any one of the plurality of pixels, and the plurality of target pixels are selected from pixels included in the compressed image frame to be processed according to a preset window size, and the preset window size is used to represent an effective range of the target filtering template;

[0009] Apply the target filtering template to perform filtering processing on the first pixel point to obtain a target compressed image frame.

[0010] In the image compression noise removal method provided by an embodiment of the present invention, noise detection is performed on the compressed image frame to be processed through a target noise detection model to obtain a target noise intensity matrix for characterizing the noise distribution of the compressed image frame to be processed, thereby improving the detection accuracy of image compression noise, and based on the target noise intensity matrix, compression noise is removed from the compressed image frame to be processed. While removing the image compression noise, more detailed information of the image is retained, thereby optimizing the image display effect and improving the user's visual experience.

[0011] In an optional embodiment, the target noise detection model includes an encoding network layer, a decoding network layer and an output layer;

[0012] The target noise detection model is used to detect the compressed image frame to be processed to obtain a target noise intensity matrix, including:

[0013] Performing feature extraction on the compressed image frame to be processed through the coding network layer to obtain feature vectors corresponding to a plurality of pixel points included in the compressed image frame to be processed;

[0014] Parsing the feature vectors corresponding to the plurality of pixel points respectively through the decoding network layer to obtain the compression noise intensities corresponding to the plurality of pixel points respectively;

[0015] Processing the compressed noise intensities corresponding to the plurality of pixel points respectively through the output layer to obtain the target noise intensity matrix;

[0016] Among them, for any pixel point among the multiple pixel points, the compression noise intensity corresponding to the pixel point is used as an element in the target noise intensity matrix, and the position information corresponding to the pixel point in the compressed image frame to be processed is used as the position information corresponding to the element in the target noise intensity matrix.

[0017] The above method extracts high-dimensional feature information of the compressed image frame to be processed through the encoding network layer, and extracts the compression noise intensity corresponding to each pixel point from the aforementioned high-dimensional feature information through the decoding network layer, integrates the determined compression noise intensities into a target noise intensity matrix and outputs it, thereby achieving high-precision detection of image compression noise, with good detection effect and wide applicability.

[0018] In an optional implementation, constructing a target filtering template corresponding to the first pixel based on the element corresponding to the first pixel and a plurality of target pixels including the first pixel includes:

[0019] Determining, according to the preset window size, the plurality of target pixels within an area centered around the first pixel;

[0020] Determining a plurality of filter coefficients according to an element corresponding to the first pixel point and distances between the plurality of target pixels and the first pixel point;

[0021] The target filtering template is constructed based on the preset window size and the determined multiple filtering coefficients.

[0022] In an optional implementation, applying the target filtering template to perform filtering processing on the first pixel includes:

[0023] Using the target filter template, convolution processing is performed on the pixel value of the first pixel to obtain a target pixel value;

[0024] Update the pixel value of the first pixel point to the target pixel value.

[0025] The above method constructs a target filter template corresponding to any pixel based on multiple selected elements in the target noise intensity matrix. This target filter template is then used to filter the corresponding pixel to achieve denoising of the compressed image frame being processed. Because the denoising process is based on the detected target noise intensity matrix, it can simultaneously remove image compression noise while preserving more image detail, thereby optimizing image display and improving the user's visual experience.

[0026] In an optional implementation, the target noise detection model is determined by:

[0027] Obtaining a preset set of training sample pairs, wherein each training sample pair consists of an image frame including compression noise and an image frame not including compression noise;

[0028] Based on the set of training sample pairs, the noise detection model to be trained is trained in an iterative manner until a preset convergence condition is met, and the noise detection model output in the last round is used as the target noise detection model, wherein the following operations are performed during one round of iteration:

[0029] Inputting the image frames including the compressed noise in the training sample pair set into the noise detection model to be trained to obtain a predicted noise intensity matrix corresponding to the image frames including the compressed noise;

[0030] Based on the predicted noise intensity matrix, filtering the image frame including the compression noise to obtain a predicted denoised image frame;

[0031] Determining a loss value based on the predicted denoised image frame and the image frame that does not include compression noise in the training sample pair set;

[0032] Based on the loss value, model parameters of the noise detection model to be trained are adjusted.

[0033] The above method converts the image compression noise detection problem into a sub-link of the compressed image restoration problem. It uses a large number of real sample data sets in the compressed image restoration task to train the noise detection model so that the noise detection model can learn the real compression noise characteristics, thereby improving the detection accuracy of compressed image noise detection using the trained target noise detection model, and achieving the purpose of high-precision compression noise detection.

[0034] In an optional embodiment, the loss value is determined by:

[0035] Determining an absolute difference between a pixel value of a second pixel and a pixel value of a third pixel, wherein the second pixel is any one of a plurality of pixels included in the predicted denoised image frame, and the third pixel is a pixel corresponding to the second pixel in the image frame not including compression noise, and the image frame including compression noise used to generate the predicted denoised image frame and the image frame not including compression noise belong to a sample training pair;

[0036] The weighted sum of all determined absolute differences is used as the loss value.

[0037] The above method determines the loss value based on the predicted denoised image frame and the image frame that does not include compressed noise. Since the image frame that does not include compressed noise includes display image features, the guidance of display image features is introduced in the process of training the noise detection model to reduce the difficulty of compressed noise prediction and improve the prediction accuracy of compressed noise.

[0038] In a second aspect, an embodiment of the present invention provides an image compression noise removal device, comprising:

[0039] An acquisition module, used for acquiring a compressed image frame to be processed;

[0040] a noise detection module, configured to detect the compressed image frame to be processed based on a target noise detection model to obtain a target noise intensity matrix, wherein the target noise intensity matrix includes a plurality of elements corresponding to a plurality of pixels included in the compressed image frame to be processed, and the elements are used to represent the compression noise intensity of the corresponding pixels;

[0041] a filtering construction module, configured to construct a target filtering template corresponding to a first pixel point based on an element corresponding to the first pixel point and a plurality of target pixels including the first pixel point, wherein the first pixel point is any pixel point among the plurality of pixels points, and the plurality of target pixels points are selected from the pixels included in the compressed image frame to be processed according to a preset window size, and the preset window size is used to represent the scope of action of the target filtering template;

[0042] A processing module is used to apply the target filtering template to perform filtering processing on the first pixel point to obtain a target compressed image frame.

[0043] In an optional embodiment, the target noise detection model includes an encoding network layer, a decoding network layer and an output layer;

[0044] The noise detection module is specifically used for:

[0045] Performing feature extraction on the compressed image frame to be processed through the coding network layer to obtain feature vectors corresponding to a plurality of pixel points included in the compressed image frame to be processed;

[0046] Parsing the feature vectors corresponding to the plurality of pixel points respectively through the decoding network layer to obtain the compression noise intensities corresponding to the plurality of pixel points respectively;

[0047] Processing the compressed noise intensities corresponding to the plurality of pixel points respectively through the output layer to obtain the target noise intensity matrix;

[0048] Among them, for any pixel point among the multiple pixel points, the compression noise intensity corresponding to the pixel point is used as an element in the target noise intensity matrix, and the position information corresponding to the pixel point in the compressed image frame to be processed is used as the position information corresponding to the element in the target noise intensity matrix.

[0049] In an optional implementation, the filtering construction module is specifically configured to:

[0050] Determining, according to the preset window size, the plurality of target pixels within an area centered around the first pixel;

[0051] Determining a plurality of filter coefficients according to an element corresponding to the first pixel point and distances between the plurality of target pixels and the first pixel point;

[0052] The target filtering template is constructed based on the preset window size and the determined multiple filtering coefficients.

[0053] In an optional embodiment, the processing module is specifically configured to:

[0054] Using the target filter template, convolution processing is performed on the pixel value of the first pixel point to obtain a target pixel value;

[0055] Update the pixel value of the first pixel point to the target pixel value.

[0056] In an optional implementation, the target noise detection model is determined by:

[0057] Obtaining a preset set of training sample pairs, wherein each training sample pair consists of an image frame including compression noise and an image frame not including compression noise;

[0058] Based on the set of training sample pairs, the noise detection model to be trained is trained in an iterative manner until a preset convergence condition is met, and the noise detection model output in the last round is used as the target noise detection model, wherein the following operations are performed during one round of iteration:

[0059] Inputting the image frames including the compressed noise in the training sample pair set into the noise detection model to be trained to obtain a predicted noise intensity matrix corresponding to the image frames including the compressed noise;

[0060] Based on the predicted noise intensity matrix, filtering the image frame including the compression noise to obtain a predicted denoised image frame;

[0061] Determining a loss value based on the predicted denoised image frame and the image frame that does not include compression noise in the training sample pair set;

[0062] Based on the loss value, model parameters of the noise detection model to be trained are adjusted.

[0063] In an optional embodiment, the loss value is determined by:

[0064] Determining an absolute difference between a pixel value of a second pixel and a pixel value of a third pixel, wherein the second pixel is any one of a plurality of pixels included in the predicted denoised image frame, and the third pixel is a pixel corresponding to the second pixel in the image frame not including compression noise, and the image frame including compression noise used to generate the predicted denoised image frame and the image frame not including compression noise belong to a sample training pair;

[0065] The weighted sum of all determined absolute differences is used as the loss value.

[0066] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0067] a memory for storing executable instructions;

[0068] A processor is used to read and execute the executable instructions stored in the memory to implement the steps of the image compression noise removal method as described in any one of the embodiments of the first aspect above.

[0069] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the steps of the image compression noise removal method as described in any one of the embodiments of the first aspect above.

[0070] For the technical effects that may be achieved by the image compression noise removal device disclosed in the second aspect, the electronic device disclosed in the third aspect, and the computer-readable storage medium disclosed in the fourth aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible solutions in the first aspect, and no further details will be given here.

[0071] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0073] Figure 1 A schematic diagram of an application scenario of an image compression noise removal method provided by an embodiment of the present invention;

[0074] Figure 2 A schematic diagram of an application scenario of another method for removing image compression noise provided by an embodiment of the present invention;

[0075] Figure 3 A schematic diagram of the workflow of a method for removing image compression noise provided by an embodiment of the present invention;

[0076] Figure 4 A schematic diagram of a network architecture of a target noise detection model provided by an embodiment of the present invention;

[0077] Figure 5A schematic flow chart of a method for obtaining a target noise intensity matrix provided by an embodiment of the present invention;

[0078] Figure 6 A schematic flow chart of a method for constructing a target filtering template provided by an embodiment of the present invention;

[0079] Figure 7 A schematic diagram of a structure for selecting a target pixel point from a compressed image frame to be processed provided by an embodiment of the present invention;

[0080] Figure 8 A schematic diagram of a structure for processing a target pixel provided by an embodiment of the present invention;

[0081] Figure 9 A schematic structural diagram of a target filtering template provided by an embodiment of the present invention;

[0082] Figure 10 A flowchart of a method for filtering a compressed image frame to be processed provided by an embodiment of the present invention;

[0083] Figure 11 A flowchart of a noise detection model training method provided by an embodiment of the present invention;

[0084] Figure 12 A flowchart of a method for performing filtering processing during a model training process provided by an embodiment of the present invention;

[0085] Figure 13 A schematic flow chart of a method for determining a loss value during model training provided by an embodiment of the present invention;

[0086] Figure 14 A schematic diagram of a complete training process of a model training phase provided by an embodiment of the present invention;

[0087] Figure 15 A complete flowchart of a specific implementation stage provided by an embodiment of the present invention;

[0088] Figure 16 A schematic diagram of the module structure of an image compression noise removal device provided by an embodiment of the present invention;

[0089] Figure 17 A schematic structural diagram of an electronic device provided by an embodiment of the present invention;

[0090] Figure 18 A schematic diagram of a program product of an image compression noise removal method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0091] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0092] It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0093] As mentioned earlier, the application of image compression technology introduces compression noise, which affects the image display quality. Therefore, how to detect and remove image compression noise to achieve compressed image restoration is an urgent problem to be solved.

[0094] To achieve high-precision detection of image compression noise, existing techniques typically employ artificially added simulated noise. Specifically, various artificially configured simulated noises are added to uncompressed images to simulate real image compression noise. Algorithms are then designed to detect this simulated noise and predict the image noise distribution. However, real image compression noise is diverse and its distribution is complex. Significant differences exist between artificially simulated noise and real noise. Therefore, these methods fail to achieve ideal noise detection results in real-world noise detection scenarios.

[0095] Existing solutions for detecting real-world image compression noise typically employ noise detection methods based on mathematical modeling. However, this method is only suitable for detecting simple image compression noise, not complex ones. Furthermore, supporting more complex image compression noise detection would incur significant computational overhead, making it unsuitable for practical application on most hardware platforms.

[0096] On the other hand, existing technologies typically use filtering-based image compression noise removal solutions to remove image compression noise. However, due to the limited receptive field of the filter, this solution cannot accurately distinguish image compression noise from image detail information, especially details similar to image compression noise. Furthermore, for various application scenarios, a single-parameter filter cannot effectively remove compression noise at various image quality levels.

[0097] The development of neural network technology and the introduction of neural network models, such as convolutional neural networks (CNNs), have addressed some of the challenges of filtering-based methods for removing image compression noise. Specifically, as the number of layers in a neural network increases, the receptive field of the neural network model significantly increases compared to the receptive field of traditional filters. This allows the neural network model to globally learn the difference between image details and compression noise.

[0098] However, using this method to achieve pixel-level image denoising requires a large number of parameters for the neural network model, which makes it impossible to deploy the model on the hardware side. Specifically, because image resolutions in real applications are often 2K, 4K, and so on, and the amount of image data is large, the hardware side lacks sufficient computing power and bandwidth to support the operation of the neural network model. To ensure that the neural network model meets the hardware's operational requirements, it needs to be cropped and quantized, which will significantly reduce the model's accuracy. Furthermore, because the neural network model uses a black-box training method, it will produce more artifacts when faced with images with unfamiliar distributions, resulting in poor robustness of the overall algorithm output.

[0099] Based on this, in order to solve the problems of poor detection accuracy, high computational consumption, and inconvenience for hardware deployment in traditional methods for detecting and removing image compression noise, the embodiments of the present invention provide a method, device, electronic device, and storage medium for removing image compression noise. By converting the image compression noise detection problem into a sub-link of the image compression noise removal problem, the detection accuracy of image compression noise is improved, and the compression noise is removed based on the distribution of the detected compression noise. While removing the image compression noise, more detailed information of the image is retained, thereby optimizing the image display effect and improving the user's visual experience.

[0100] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to illustrate the present invention and are not intended to limit the present invention.

[0101] An image compression noise removal method provided in an embodiment of the present invention is applicable to electronic devices. The electronic device uses the above method to detect and remove compression noise from multiple compressed image frames to be processed included in a video stream, thereby restoring the compressed image. This allows the compressed image frames after denoising to be decompressed during subsequent image display, and image display is performed based on the decompressed image frames to optimize the image display effect.

[0102] Exemplarily, the electronic device in the embodiment of the present invention may be an electronic device having an image display device (such as a display panel); for example, it may be a smart terminal, a smart mobile terminal, a tablet computer, a laptop computer, a smart handheld device, a personal computer (PC), a computer, a smart screen, a display device, a vehicle-mounted device, various wearable devices, a personal digital assistant (PDA), etc.; among them, wearable devices such as virtual reality (VR) devices, augmented reality (AR) devices, etc.

[0103] Another exemplary embodiment, the electronic device can also be a server connected to a device with a display function (such as a display device). For example, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, and a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; wherein the server can be connected to the device with a display function by wired or wireless means, and the present invention does not limit the connection method.

[0104] It is understandable that the present invention does not limit the specific type of the above electronic devices.

[0105] The following describes possible application scenarios of the image compression noise removal method provided by the embodiment of the present invention with reference to the accompanying drawings:

[0106] like Figure 1 As shown in FIG, a schematic diagram of an application scenario in an embodiment of the present invention is shown by taking the electronic device as a display device as an example. Figure 1As shown, the application scenario includes a display device 10; the display device 10 executes the process of the image compression noise removal method provided by an embodiment of the present invention (the specific process will be introduced later), obtains a target compressed image frame with compression noise removed, decompresses the target compressed image frame, and displays an image based on the decompressed image frame to optimize the display effect of the image display.

[0107] like Figure 2 As shown in FIG, another application scenario diagram in an embodiment of the present invention is shown by taking the electronic device as a server as an example. Figure 2 As shown, the application scenario includes a display device 10 and a server 20. The server 20 executes the process of the image compression noise removal method provided by the embodiment of the present invention (the specific process will be described later) to obtain a target compressed image frame with compression noise removed.

[0108] In one or more embodiments, after the server 20 obtains the target compressed image frame, it decompresses the target compressed image frame, and then transmits the decompressed image frame to the display device 10 through the communication network, so that the display device 10 displays the image according to the decompressed image frame.

[0109] In one or more embodiments, the server 20 directly transmits the target compressed image frame with compression noise removed to the display device 10 through the communication network, so that the display device 10 decompresses the target compressed image frame and displays the image based on the decompressed image frame to optimize the display effect of the image display.

[0110] Of course, the method provided by the embodiment of the present invention is not limited to Figure 1 、 Figure 2 The application scenario shown can also be used in other possible application scenarios, and the embodiment of the present invention does not impose any limitation thereto.

[0111] After introducing the application scenarios of the embodiments of the present invention, the preferred implementation methods of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention. In addition, the embodiments of the present invention and the features in the embodiments can be combined with each other if there is no conflict.

[0112] The following is a detailed description of the image compression noise removal method provided by the embodiment of the present invention with reference to the accompanying drawings:

[0113] Figure 3 FIG. 4 shows a schematic diagram of a workflow of a method for removing image compression noise provided by an embodiment of the present invention. Figure 3 As shown, the specific process of this method is as follows:

[0114] Step S301: Obtain a compressed image frame to be processed.

[0115] In one or more embodiments, the compressed image frame to be processed includes a plurality of pixels, and the number of pixels is associated with the resolution of the compressed image frame to be processed. For example, for a compressed image frame to be processed with a resolution of 640×480, it includes 640×480=307200 pixels.

[0116] Step S302: Based on the target noise detection model, the compressed image frame to be processed is detected to obtain a target noise intensity matrix;

[0117] The target noise intensity matrix includes a plurality of elements corresponding to a plurality of pixel points included in the compressed image frame to be processed, and the elements are used to represent the compression noise intensity of the corresponding pixel points.

[0118] In one or more embodiments, the target noise detection model may be a U-Net (U-shaped Deep Neural Network).

[0119] Figure 4 A schematic diagram of the network architecture of the aforementioned U-Net model is shown in FIG. Figure 4 As shown in Figure 1, the network architecture includes an encoding network layer, a decoding network layer, and an output layer, where:

[0120] The encoding network layer and the decoding network layer form an "encoder-decoder" structure. The encoding network layer is used to extract features from the compressed image frames to be processed and obtain feature information; the decoding network layer is used to perform feature fusion on the feature information output by the encoding network layer to obtain the compressed noise intensity; the output layer is a single-channel output convolution layer, which is used to perform dimensionality transformation on the compressed noise intensity output by the decoding network layer to obtain the target noise intensity matrix.

[0121] In an optional embodiment, based on Figure 4 The network architecture of the target noise detection model shown in FIG. 3 , the target noise intensity matrix can be specifically implemented through steps S302 - 1 to S302 - 3 , as shown in FIG. Figure 5 The specific process is as follows:

[0122] Step S302 - 1 : performing feature extraction on the compressed image frame to be processed through the coding network layer to obtain feature vectors corresponding to a plurality of pixel points included in the compressed image frame to be processed.

[0123] In a specific implementation, the compressed image frame to be processed includes m×n pixels. The compressed image frame to be processed is input to the encoding network layer, and the encoding network layer accordingly outputs the feature vectors corresponding to each pixel to the decoding network layer.

[0124] Step S302-2: parse the feature vectors corresponding to the multiple pixel points through the decoding network layer to obtain the compression noise intensities corresponding to the multiple pixel points.

[0125] In a specific implementation, the decoding network layer receives the feature vectors corresponding to each pixel point, and performs feature fusion processing on the feature vectors corresponding to each pixel point to extract the compressed noise intensity sigma corresponding to each pixel point, and outputs the compressed noise intensity sigma corresponding to each pixel point to the output layer.

[0126] In one or more embodiments, the compression noise intensity sigma is used to indicate the compression noise intensity of the corresponding pixel. A larger value of the compression noise intensity sigma indicates a higher compression noise intensity of the corresponding pixel; a smaller value of the compression noise intensity sigma indicates a lower compression noise intensity of the corresponding pixel.

[0127] Exemplarily, the compressed image frame to be processed includes m×n pixels, and m×n compression noise intensities sigma can be obtained through step S302-2.

[0128] Step S302-3: Process the compressed noise intensities corresponding to the multiple pixel points through the output layer to obtain a target noise intensity matrix.

[0129] Among them, for any pixel point among multiple pixel points, the compression noise intensity corresponding to the pixel point is used as an element in the target noise intensity matrix, and the position information corresponding to the pixel point in the compressed image frame to be processed is used as the position information corresponding to an element in the target noise intensity matrix.

[0130] In a specific implementation, the output layer receives the compression noise intensity sigma corresponding to each pixel and performs a dimensionality transformation on it to obtain a target noise intensity matrix corresponding to the compressed image frame to be processed. Since the compressed image frame to be processed includes m×n pixels, the dimension of the target noise intensity matrix is ​​m×n, that is, the target noise intensity matrix can be expressed as: sigma(m,n).

[0131] Exemplarily, the compressed noise intensity sigma corresponding to the pixel point Pixel (1,1) in the compressed image frame to be processed is used as the element at the position (1,1) in the target noise intensity matrix sigma (m, n); the compressed noise intensity sigma corresponding to the pixel point Pixel (2,3) in the compressed image frame to be processed is used as the element at the position (2,3) in the target noise intensity matrix sigma (m, n); and so on, to obtain the target noise intensity matrix sigma (m, n) corresponding to the compressed image frame to be processed.

[0132] The high-dimensional feature information of the compressed image frame to be processed is extracted through the encoding network layer, and the compression noise intensity corresponding to each pixel point is extracted from the aforementioned high-dimensional feature information through the decoding network layer. The determined compression noise intensities are integrated into a target noise intensity matrix and output, thereby achieving high-precision detection of image compression noise. The detection effect is good and the application scenarios are wide.

[0133] In the embodiment of the present invention, the target noise detection model is obtained by training the noise detection model to be trained based on multiple training sample pairs included in the training sample pair set (the specific model training process is detailed in the subsequent process).

[0134] During the training process, the image frames input into the noise detection model are image frames including compressed noise in the training sample pairs. The adjustment of the model parameters of the noise detection model is based on the predicted denoised image frames and the image frames not including compressed noise in the training sample pairs. Since the image frames not including compressed noise include display image features, the guidance of display image features is introduced in the process of noise detection model training to reduce the difficulty of compressed noise prediction and improve the prediction accuracy of compressed noise.

[0135] After the noise detection model to be trained is trained to obtain the target noise detection model, in a specific implementation, taking a display device such as a TV playing a video screen as an example:

[0136] After the TV is turned on, the received compressed image frame to be processed is directly input into the encoding network layer of the target noise detection model. The encoding network layer extracts image features from the compressed image frame to be processed and inputs the extracted image features into the decoding network layer of the target noise detection model. The decoding network layer performs feature fusion processing on the image features to obtain the compressed noise intensity. The compressed noise intensity passes through the output layer of the target noise detection model and outputs the target noise intensity matrix corresponding to the compressed image frame to be processed.

[0137] After the target noise detection model outputs the target noise intensity matrix, the target noise intensity matrix can be used to perform denoising on the compressed image frame to be processed to restore the compressed image, as follows:

[0138] Step S303: constructing a target filtering template corresponding to the first pixel based on the element corresponding to the first pixel and a plurality of target pixels including the first pixel;

[0139] Among them, the first pixel point is any pixel point among the multiple pixel points, and the multiple target pixel points are selected from the pixel points included in the compressed image frame to be processed according to a preset window size. The preset window size is used to characterize the scope of action of the target filtering template.

[0140] It should be noted that the preset window size in the embodiment of the present invention is an odd-dimensional size, for example, 3×3, 5×5, 7×7, 15×15, etc. The larger the preset window size, the better the denoising effect of the target filter template constructed by it, but the corresponding computational cost is also greater. Therefore, in practical applications, an appropriate preset window size can be selected according to actual business needs, and the embodiment of the present invention does not impose any restrictions on this.

[0141] In an optional implementation, the target filtering template can be implemented through steps S303-1 to S303-3. Figure 6 The specific process is as follows:

[0142] Step S303 - 1 : determining a plurality of target pixels within an area centered around a first pixel according to a preset window size.

[0143] Figure 7 A schematic diagram of a compressed image frame to be processed comprising m×n pixels is shown, as shown in FIG. Figure 7 As shown, the compressed image frame to be processed includes pixel points pixel(1,1) to pixel points pixel(m,n). Take the first pixel point as pixel point pixel(4,5) and the preset window size as 3×3 as an example for explanation:

[0144] like Figure 7 As shown in the figure, according to the 3×3 window, the 9 target pixels determined are: pixel (3,4), pixel (3,5), pixel (3,6), pixel (4,4), pixel (4,5), pixel (4,6), pixel (5,4), pixel (5,5), and pixel (5,6).

[0145] Step S303 - 2 : determining a plurality of filter coefficients according to the element corresponding to the first pixel and the distances between the plurality of target pixels and the first pixel.

[0146] In a specific implementation, for the 9 target pixels determined in the above embodiment, a rectangular coordinate system is established with the first pixel pixel (4,5) as the origin, where x is the horizontal axis and y is the vertical axis. Figure 8 As shown, the position information (x, y) of the above 9 target pixel points corresponding to the rectangular coordinate system are:

[0147] The position information corresponding to pixel (3,4) is (-1,1), the position information corresponding to pixel (3,5) is (0,1), and the position information corresponding to pixel (3,6) is (1,1);

[0148] The position information corresponding to pixel (4,4) is (-1,0), the position information corresponding to pixel (4,5) is (0,0), and the position information corresponding to pixel (4,6) is (1,0);

[0149] The position information corresponding to pixel (5,4) is (-1,-1), the position information corresponding to pixel (5,5) is (0,-1), and the position information corresponding to pixel (5,6) is (1,-1).

[0150] After determining the above position information, the filter coefficient corresponding to each target pixel can be determined according to the following formula:

[0151]

[0152] Wherein, (x, y) is the position information corresponding to the target pixel; δ is the standard deviation, and if the element corresponding to the first pixel is sigma1, the standard deviation in the above formula can be expressed as: δ=1-sigma1.

[0153] Step S303-3: constructing a target filtering template based on the preset window size and the determined multiple filtering coefficients.

[0154] In one or more embodiments, the target filter template may be a Gaussian filter template.

[0155] In a specific implementation, based on the multiple filter coefficients determined in step S303-2, a target filter template corresponding to the first pixel pixel (4, 5) can be constructed, such as Figure 9 As shown in , the size of the target filter template is 3×3, and the filter coefficients in the target filter template are the largest at the center and decrease in order towards the surrounding areas. Figure 9 In the filter template shown, filter coefficient k1>filter coefficient k2>filter coefficient k3.

[0156] It should be noted that the method for determining the target filter template corresponding to other pixel points included in the compressed image frame to be processed is similar to the method for determining the target filter template corresponding to the first pixel point. Please refer to the above implementation method and will not be repeated here.

[0157] The target filter template is constructed in the above manner. When the compression noise intensity sigma is larger, the constructed target filter template is closer to the mean filter. That is, the target filter template processes the first pixel point as the mean value, which indicates that the first pixel point contains noise.

[0158] When the compression noise intensity sigma is smaller, the central filter coefficient of the constructed target filter template is larger, that is, the target filter template processes the first pixel point without changing its original value, thereby indicating that the first pixel point does not contain noise.

[0159] After determining the target filter template corresponding to each pixel of the compressed image frame to be processed, the Gaussian filter processing is performed using the target filter template, as follows:

[0160] Step S304 : Apply the target filtering template to perform filtering processing on the first pixel to obtain a target compressed image frame.

[0161] In one or more embodiments, the target compressed image frame may be obtained by applying a target filtering template and performing Gaussian filtering on the first pixel.

[0162] In a specific implementation, the target filtering template corresponding to each pixel point is applied to perform filtering processing on the corresponding pixel points respectively until each pixel point in the compressed image frame to be processed is processed to obtain the target compressed image frame.

[0163] In an optional implementation, the above filtering operation can be implemented through steps S304-1 to S304-2. Figure 10 As shown, the specific process is as follows:

[0164] Step S304 - 1 : Using the target filtering template, perform convolution processing on the pixel value of the first pixel to obtain a target pixel value.

[0165] In a specific implementation, the target pixel value is obtained by multiplying the multiple filter coefficients included in the target filter template and the pixel values ​​of the corresponding multiple pixel points and adding them together.

[0166] For example, taking the first pixel as pixel (4, 5) as an example, the target filter template corresponding to the first pixel pixel (4, 5) is filter (4, 5), where:

[0167] Assuming the target filter template is filter(4,5), it can be expressed as: Assume that the pixel values ​​of multiple pixels centered at the first pixel pixel (4,5) can be expressed as: Then the target pixel value can be expressed as:

[0168]

[0169] Step S304-2: Update the pixel value of the first pixel to the target pixel value.

[0170] For example, assuming that the pixel value of the first pixel point pixel (4, 5) is p45=226, and the target pixel value p45'=164 determined by step S304-1, 226 is replaced by 164 to update the pixel value of the first pixel point and achieve the purpose of denoising the first pixel point.

[0171] It should be noted that the filtering method for other pixel points included in the compressed image frame to be processed is similar to the filtering method for the first pixel point mentioned above, and can be referred to the above implementation method, which will not be repeated here.

[0172] After filtering is completed on all pixels included in the compressed image frame to be processed, a target compressed image with compression noise removed can be obtained, thereby realizing the restoration operation of the compressed image.

[0173] Based on multiple selected elements in the target noise intensity matrix, a target filter template corresponding to any pixel is constructed and used to filter the corresponding pixel to achieve denoising of the compressed image frame being processed. Because denoising is performed based on the detected target noise intensity matrix, it can simultaneously remove image compression noise while retaining more image detail, thereby optimizing image display and improving the user's visual experience.

[0174] In a specific implementation, take a display device such as a TV playing a video screen as an example:

[0175] Suppose a TV needs to display a 4K landscape image. In this landscape image, there is only a strip of compression noise in the upper left corner of the mountain due to image compression. This landscape image is input into the target noise detection model, which outputs a target noise intensity matrix. In this target noise intensity matrix, the element corresponding to the upper left corner of the mountain in the landscape image has a larger value, while the other elements have a smaller value.

[0176] Based on the distribution of element values ​​in the target noise intensity matrix, the subsequent filtering and denoising process can specifically remove the compression noise in the upper left corner of the mountain, while retaining detailed information in other locations. Of course, the filtering and denoising process in this embodiment of the present invention is not limited to using the Gaussian filtering algorithm; other suitable filtering algorithms can also be used. Furthermore, the degree of noise removal at different locations can be adjusted to improve the quality of compressed image restoration.

[0177] In the image compression noise removal method provided by an embodiment of the present invention, noise detection is performed on the compressed image frame to be processed through a target noise detection model to obtain a target noise intensity matrix for characterizing the noise distribution of the compressed image frame to be processed, thereby improving the detection accuracy of image compression noise, and based on the target noise intensity matrix, compression noise is removed from the compressed image frame to be processed. While removing the image compression noise, more detailed information of the image is retained, thereby optimizing the image display effect and improving the user's visual experience.

[0178] After introducing the image compression noise removal method provided by the embodiment of the present invention, the training process of the target noise detection model in the embodiment of the present invention is described. The training process in the embodiment of the present invention can be applied to the server, such as Figure 2 Server 20 is shown.

[0179] In the embodiment of the present invention, before training the model, first, a network architecture of the noise detection model to be trained is constructed, such as Figure 4 Then, the specific training process of the noise detection model is described below:

[0180] like Figure 11 As shown, the training process of the noise detection model in the embodiment of the present invention is as follows:

[0181] Step S1101, obtaining a preset training sample pair set;

[0182] Each training sample pair consists of an image frame including compression noise and an image frame not including compression noise.

[0183] In one or more embodiments, the preset set of training sample pairs may be a public dataset for a compressed image restoration task.

[0184] Step S1102: Based on the training sample pair set, the noise detection model to be trained is trained in an iterative manner until a preset convergence condition is met, and the noise detection model outputted in the last round is used as the target noise detection model;

[0185] It should be noted that, in the embodiment of the present invention, the preset convergence condition may be any one of the following conditions:

[0186] Condition 1: The loss value determined in this round of training is less than the loss threshold;

[0187] Condition 2: The number of model training times reaches the preset threshold.

[0188] In step S1102, the following operations are performed during one round of iteration:

[0189] Step S1102-1: inputting the image frames including compressed noise in the training sample pair set into the noise detection model to be trained to obtain a predicted noise intensity matrix corresponding to the image frames including compressed noise;

[0190] The predicted noise intensity matrix includes a plurality of prediction elements, which correspond to a plurality of pixel points in an image frame including compression noise, and the prediction elements are used to represent the predicted compression noise intensity of the corresponding pixel points.

[0191] In a specific implementation, multiple image frames including compressed noise are input into the noise detection model to be trained, the encoding network layer in the noise detection model to be trained performs image feature extraction on the image frames including compressed noise, and the extracted predicted image features are input into the decoding network layer in the noise detection model to be trained; the decoding network layer performs feature fusion processing on the predicted image features to obtain predicted compressed noise intensity; the predicted compressed noise intensity passes through the output layer in the noise detection model to be trained, and outputs a predicted noise intensity matrix corresponding to the image frames including compressed noise.

[0192] Step S1102-2: performing filtering processing on the image frame including the compression noise based on the predicted noise intensity matrix to obtain a predicted denoised image frame.

[0193] In one or more embodiments, Figure 12 As shown, the predicted denoised image frame can be determined as follows:

[0194] Step S1201, determining a plurality of middle pixel points within an area centered on a second pixel point according to a preset window size;

[0195] The second pixel point is any pixel point in the image frame including compression noise.

[0196] Step S1202 : determining a plurality of prediction filter coefficients according to the prediction element corresponding to the second pixel point and the distances between the second pixel point and a plurality of intermediate pixels.

[0197] Step S1203: constructing a prediction filter template based on the preset window size and the determined multiple prediction filter coefficients.

[0198] Step S1204: Using the prediction filter template, perform convolution processing on the pixel value of the second pixel to obtain a predicted pixel value.

[0199] Step S1205: Update the pixel value of the second pixel point to the predicted pixel value.

[0200] It should be noted that, during the model training process, the specific implementation of step S1102-2 can refer to the implementation of steps S303 to S304 in the aforementioned image compression noise removal method, and will not be repeated here.

[0201] After filtering each pixel in the image frame including the compression noise, a predicted denoised image frame with the compression noise removed can be obtained.

[0202] Step S1102-3: determining a loss value based on the predicted denoised image frame and the image frames that do not include compression noise in the training sample pair set.

[0203] In a specific implementation, for a training sample pair (I1, G1), I1 is used to represent an image frame including compression noise in the training sample pair, and G1 is used to represent an image frame not including compression noise in the training sample pair; after executing steps S1102-1 to S1102-2 on the image frame I1 including compression noise in the training sample pair (I1, G1), a predicted denoised image frame D1 can be obtained, and the loss value can be determined using the predicted denoised image frame D1 and the image frame G1 not including compression noise, as follows:

[0204] In an optional implementation, the loss value can be specifically realized through steps S1301 to S1302, such as Figure 13 As shown, the specific process is as follows:

[0205] Step S1301, determining the absolute difference between the pixel value of the second pixel point and the pixel value of the third pixel point;

[0206] Among them, the second pixel point is any pixel point among the multiple pixel points included in the predicted denoised image frame, and the third pixel point is a pixel point corresponding to the second pixel point in the image frame that does not include compressed noise. The image frame including compressed noise that is used to generate the predicted denoised image frame belongs to a sample training pair with the image frame that does not include compressed noise.

[0207] Step S1302: The weighted sum of all the determined absolute differences is used as the loss value.

[0208] In a specific implementation, the loss value can be determined according to the constructed loss function, which can be expressed as:

[0209]

[0210] Among them, I (i,j) Used to represent the pixel value of the (i, j)th pixel of the image frame I containing compression noise; G (i.j)The pixel value of the (i, j)th pixel in the image frame G that does not contain compression noise is used to represent the pixel value. The image frame I containing compression noise and the image frame G that does not contain compression noise both belong to the training sample pair (I, G).

[0211] filter (i,j) A prediction filter template corresponding to the (i, j)th pixel point is used to represent the prediction filter template, which is constructed based on the predicted compressed noise intensity matrix determined by the noise detection model to be trained;

[0212] D (i,j) =I (i,j) *filter (i,j) Used to represent the pixel value of the (i, j)th pixel of the predicted denoised image frame D. The predicted denoised image frame D is obtained by filtering and denoising the image frame I containing compression noise based on the prediction filter template filter;

[0213] |I (i,j) *filter (i,j) -G (i.j) |Used to represent the absolute difference corresponding to the (i, j)th pixel; Used to represent the loss value, which is the sum of the absolute differences of all pixels.

[0214] The loss value is determined based on the predicted denoised image frame and the image frame excluding compressed noise. Since the image frame excluding compressed noise includes display image features, the guidance of display image features is introduced in the process of noise detection model training to reduce the difficulty of compressed noise prediction and improve the prediction accuracy of compressed noise.

[0215] Step S1102-4: Based on the loss value, adjust the model parameters of the noise detection model to be trained.

[0216] In the specific implementation, after the loss value is determined through step S1102-3, if the loss value of this round of training is less than the loss threshold, the loss value is backpropagated to adjust and optimize the model parameters; otherwise, the model parameters obtained in this round of training are saved, the training is ended, and the target noise detection model is obtained.

[0217] The problem of detecting image compression noise is converted into a sub-link of the compressed image restoration problem. A large number of real sample data sets in the compressed image restoration task are used to train the noise detection model so that the noise detection model can learn the real compression noise characteristics, thereby improving the detection accuracy of compressed image noise using the trained target noise detection model and achieving the purpose of high-precision compression noise detection.

[0218] The following uses specific examples to further explain the model training stage and the specific implementation stage in detail.

[0219] Example 1: Model training phase.

[0220] In the embodiment of the present invention, Figure 14 As shown in the figure, the specific training process of the model training phase is as follows:

[0221] Step S1401: construct a noise detection model and initialize the noise detection model.

[0222] In one or more embodiments, the network architecture of the noise detection model can still refer to Figure 4 .

[0223] Step S1402, obtaining a preset training sample pair set;

[0224] Each training sample pair consists of an image frame including compression noise and an image frame not including compression noise.

[0225] Step S1403: Based on the set of training sample pairs, the noise detection model to be trained is trained in an iterative manner until a preset convergence condition is met, and the noise detection model outputted in the last round is used as the target noise detection model. The following operations are performed during one round of iteration:

[0226] Step S1403-1: inputting the image frames including compressed noise in the training sample pair set into the noise detection model to be trained to obtain a predicted noise intensity matrix corresponding to the image frames including compressed noise;

[0227] The predicted noise intensity matrix includes a plurality of prediction elements, which correspond to a plurality of pixel points in an image frame including compression noise, and the prediction elements are used to represent the predicted compression noise intensity of the corresponding pixel points.

[0228] Step S1403-2, according to the preset window size, determine multiple intermediate pixels in the area centered on the second pixel, and determine multiple prediction filter coefficients according to the prediction elements corresponding to the second pixel and the distances between the multiple intermediate pixels and the second pixel.

[0229] The second pixel point is any pixel point in the image frame including compression noise.

[0230] Step S1403-3: construct a prediction filter template based on the preset window size and the determined multiple prediction filter coefficients.

[0231] Step S1403-4: Using the prediction filter template, perform convolution processing on the pixel value of the second pixel to obtain a predicted pixel value, and update the pixel value of the second pixel to the predicted pixel value to obtain a predicted denoised image frame.

[0232] Step S1403-5: determining a loss value based on the predicted denoised image frame and the image frames that do not include compression noise in the training sample pair set.

[0233] Step S1403-6, determine whether the loss value meets the preset convergence condition, if so, execute step S1404, otherwise, execute step S1403-1.

[0234] Step S1404: save the model parameters of the noise detection model and end the training.

[0235] In the embodiment of the present invention, after executing step 1404, the training is ended and the target noise detection model is obtained.

[0236] Example 2: Specific implementation stage.

[0237] In the embodiment of the present invention, Figure 15 The specific process of the implementation phase is as follows:

[0238] Step S1501: construct a target noise detection network and load the model parameters saved after training to obtain a target noise detection model.

[0239] Step S1502: Obtain the compressed image frame to be processed.

[0240] Step S1503 : Based on the target noise detection model, the compressed image frame to be processed is detected to obtain a target noise intensity matrix.

[0241] The target noise intensity matrix includes a plurality of elements corresponding to a plurality of pixel points included in the compressed image frame to be processed, and the elements are used to represent the compression noise intensity of the corresponding pixel points.

[0242] Step S1504: determining a plurality of target pixels within an area centered on the first pixel according to a preset window size, and determining a plurality of filter coefficients according to elements corresponding to the first pixel and distances between the plurality of target pixels and the first pixel.

[0243] The first pixel point is any pixel point among multiple pixel points.

[0244] Step S1505 : constructing a target filtering template based on the preset window size and the determined multiple filtering coefficients.

[0245] Step S1506: Using the target filtering template, perform convolution processing on the pixel value of the first pixel to obtain a target pixel value.

[0246] Step S1507: Update the pixel value of the first pixel point to the target pixel value to obtain a target compressed image frame.

[0247] Step S1508: decompress the target compressed image frame to obtain an image frame to be displayed.

[0248] Step S1509: display the image frame to be displayed.

[0249] Based on the same concept, an embodiment of the present invention also provides an image compression noise removal device. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0250] like Figure 16 As shown, the above device includes the following modules:

[0251] An acquisition module 1601 is configured to acquire a compressed image frame to be processed;

[0252] The noise detection module 1602 is configured to detect the compressed image frame to be processed based on a target noise detection model to obtain a target noise intensity matrix;

[0253] The target noise intensity matrix includes a plurality of elements corresponding to a plurality of pixel points included in the compressed image frame to be processed, and the elements are used to represent the compression noise intensity of the corresponding pixel points;

[0254] A filter construction module 1603 is configured to construct a target filter template corresponding to the first pixel based on the element corresponding to the first pixel and a plurality of target pixels including the first pixel;

[0255] The first pixel point is any pixel point among the plurality of pixel points, and the plurality of target pixel points are selected from the pixel points included in the compressed image frame to be processed according to a preset window size, and the preset window size is used to represent the scope of action of the target filtering template;

[0256] The processing module 1604 is configured to apply a target filtering template to perform filtering processing on the first pixel point to obtain a target compressed image frame.

[0257] In an optional embodiment, the target noise detection model includes an encoding network layer, a decoding network layer and an output layer;

[0258] The noise detection module 1602 is specifically used for:

[0259] Extract features from the compressed image frame to be processed through the coding network layer to obtain feature vectors corresponding to multiple pixel points included in the compressed image frame to be processed;

[0260] The feature vectors corresponding to the multiple pixels are analyzed through the decoding network layer to obtain the compression noise intensity corresponding to the multiple pixels;

[0261] The compressed noise intensity corresponding to multiple pixel points is processed through the output layer to obtain the target noise intensity matrix;

[0262] Among them, for any pixel point among multiple pixel points, the compression noise intensity corresponding to the pixel point is used as an element in the target noise intensity matrix, and the position information corresponding to the pixel point in the compressed image frame to be processed is used as the position information corresponding to an element in the target noise intensity matrix.

[0263] In an optional implementation, the filtering construction module 1603 is specifically configured to:

[0264] Determining, according to a preset window size, a plurality of target pixel points within an area centered on the first pixel point;

[0265] Determining a plurality of filter coefficients according to an element corresponding to the first pixel and distances between a plurality of target pixels and the first pixel;

[0266] A target filtering template is constructed based on a preset window size and a plurality of determined filtering coefficients.

[0267] In an optional implementation, the processing module 1604 is specifically configured to:

[0268] Using the target filter template, convolution processing is performed on the pixel value of the first pixel to obtain the target pixel value;

[0269] Update the pixel value of the first pixel to the target pixel value.

[0270] In an optional implementation, the target noise detection model is determined by:

[0271] Obtaining a preset set of training sample pairs, wherein each training sample pair consists of an image frame including compression noise and an image frame not including compression noise;

[0272] Based on the set of training sample pairs, the noise detection model to be trained is trained in an iterative manner until the preset convergence condition is met, and the noise detection model output in the last round is used as the target noise detection model. The following operations are performed during one round of iteration:

[0273] Inputting the image frames including the compressed noise in the training sample pair set into the noise detection model to be trained to obtain a predicted noise intensity matrix corresponding to the image frames including the compressed noise;

[0274] Based on the predicted noise intensity matrix, filtering the image frame including the compression noise is performed to obtain a predicted denoised image frame;

[0275] Determining a loss value based on the predicted denoised image frame and the image frame that does not include compression noise in the set of training sample pairs;

[0276] Based on the loss value, the model parameters of the noise detection model to be trained are adjusted.

[0277] In an optional embodiment, the loss value is determined by:

[0278] Determining an absolute difference between a pixel value of a second pixel and a pixel value of a third pixel, wherein the second pixel is any pixel among a plurality of pixels included in the predicted denoised image frame, and the third pixel is a pixel corresponding to the second pixel in the image frame not including compression noise, and the image frame including compression noise used to generate the predicted denoised image frame and the image frame not including compression noise belong to a sample training pair;

[0279] The weighted sum of all the absolute differences determined is used as the loss value.

[0280] Based on the same concept, an embodiment of the present invention also provides an electronic device. Since the electronic device is the electronic device in the method in the embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0281] Refer to the following Figure 17 The electronic device 170 according to this embodiment of the present invention will be described. Figure 17 The electronic device 170 shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0282] like Figure 17 As shown, the electronic device 170 can be implemented as a general-purpose computing device, such as a terminal device. Components of the electronic device 170 may include, but are not limited to, the at least one processor 171 described above, the at least one memory 172 storing instructions executable by the processor 171, and a bus 173 connecting different system components (including the memory 172 and the processor 171). The processor 171 is a processor of the smart device.

[0283] The processor 171 executes the executable instructions to implement the following steps:

[0284] Obtaining a compressed image frame to be processed;

[0285] Based on the target noise detection model, the compressed image frame to be processed is detected to obtain the target noise intensity matrix;

[0286] The target noise intensity matrix includes a plurality of elements corresponding to a plurality of pixel points included in the compressed image frame to be processed, and the elements are used to represent the compression noise intensity of the corresponding pixel points;

[0287] Constructing a target filtering template corresponding to the first pixel based on the element corresponding to the first pixel and a plurality of target pixels including the first pixel;

[0288] The first pixel point is any pixel point among the plurality of pixel points, and the plurality of target pixel points are selected from the pixel points included in the compressed image frame to be processed according to a preset window size, and the preset window size is used to represent the scope of action of the target filtering template;

[0289] Apply the target filtering template to perform filtering processing on the first pixel to obtain a target compressed image frame.

[0290] In an optional embodiment, the target noise detection model includes an encoding network layer, a decoding network layer and an output layer;

[0291] The processor 171 is specifically configured to:

[0292] Extract features from the compressed image frame to be processed through the coding network layer to obtain feature vectors corresponding to multiple pixel points included in the compressed image frame to be processed;

[0293] The feature vectors corresponding to the multiple pixels are analyzed through the decoding network layer to obtain the compression noise intensity corresponding to the multiple pixels;

[0294] The compressed noise intensity corresponding to multiple pixel points is processed through the output layer to obtain the target noise intensity matrix;

[0295] Among them, for any pixel point among multiple pixel points, the compression noise intensity corresponding to the pixel point is used as an element in the target noise intensity matrix, and the position information corresponding to the pixel point in the compressed image frame to be processed is used as the position information corresponding to an element in the target noise intensity matrix.

[0296] In an optional implementation, the processor 171 is specifically configured to:

[0297] Determining, according to a preset window size, a plurality of target pixel points within an area centered on the first pixel point;

[0298] Determining a plurality of filter coefficients according to an element corresponding to the first pixel and distances between a plurality of target pixels and the first pixel;

[0299] A target filtering template is constructed based on a preset window size and a plurality of determined filtering coefficients.

[0300] In an optional implementation, the processor 171 is specifically configured to:

[0301] Using the target filter template, convolution processing is performed on the pixel value of the first pixel to obtain the target pixel value;

[0302] Update the pixel value of the first pixel to the target pixel value.

[0303] In an optional implementation, the target noise detection model is determined by:

[0304] Obtaining a preset set of training sample pairs, wherein each training sample pair consists of an image frame including compression noise and an image frame not including compression noise;

[0305] Based on the set of training sample pairs, the noise detection model to be trained is trained in an iterative manner until the preset convergence condition is met, and the noise detection model output in the last round is used as the target noise detection model. The following operations are performed during one round of iteration:

[0306] Inputting the image frames including the compressed noise in the training sample pair set into the noise detection model to be trained to obtain a predicted noise intensity matrix corresponding to the image frames including the compressed noise;

[0307] Based on the predicted noise intensity matrix, filtering the image frame including the compression noise is performed to obtain a predicted denoised image frame;

[0308] Determining a loss value based on the predicted denoised image frame and the image frame that does not include compression noise in the set of training sample pairs;

[0309] Based on the loss value, the model parameters of the noise detection model to be trained are adjusted.

[0310] In an optional embodiment, the loss value is determined by:

[0311] Determining an absolute difference between a pixel value of a second pixel and a pixel value of a third pixel, wherein the second pixel is any pixel among a plurality of pixels included in the predicted denoised image frame, and the third pixel is a pixel corresponding to the second pixel in the image frame not including compression noise, and the image frame including compression noise used to generate the predicted denoised image frame and the image frame not including compression noise belong to a sample training pair;

[0312] The weighted sum of all the absolute differences determined is used as the loss value.

[0313] Bus 173 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0314] The memory 172 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1721 and / or a cache memory 1722 , and may further include a read-only memory (ROM) 1723 .

[0315] The memory 172 may also include a program / utility 1725 having a set (at least one) of program modules 1724, such program modules 1724 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0316] The electronic device 170 can also communicate with one or more external devices 174 (e.g., a keyboard, pointing device, etc.), one or more devices that enable a user to interact with the electronic device 170, and / or any device that enables the electronic device 170 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 175. Furthermore, the electronic device 170 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 176. As shown, the network adapter 176 communicates with other modules of the electronic device 170 via a bus 173. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the electronic device 170, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0317] In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of each module in the image compression noise removal device according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0318] For example, a compressed image frame to be processed is obtained; based on a target noise detection model, the compressed image frame to be processed is detected to obtain a target noise intensity matrix, wherein the target noise intensity matrix includes multiple elements corresponding to multiple pixel points included in the compressed image frame to be processed, and the elements are used to characterize the compression noise intensity of the corresponding pixel points; based on the elements corresponding to the first pixel point and multiple target pixel points including the first pixel point, a target filtering template corresponding to the first pixel point is constructed, wherein the first pixel point is any pixel point among the multiple pixel points, and the multiple target pixel points are selected from the pixel points included in the compressed image frame to be processed according to a preset window size, and the preset window size is used to characterize the scope of the target filtering template; the target filtering template is applied to filter the first pixel point to obtain a target compressed image frame.

[0319] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0320] like Figure 18 As shown, a program product 180 for a method for removing image compression noise according to an embodiment of the present invention is described. The program product 180 may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0321] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0322] Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0323] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0324] It should be noted that although several modules or submodules of the system are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0325] Furthermore, although the operations of the various modules of the system of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some operations may be omitted, multiple operations may be combined into one operation, and / or one operation may be decomposed into multiple operations.

[0326] The present invention is described above with reference to block diagrams and / or flow charts illustrating methods, apparatus (systems) and / or computer program products according to embodiments of the present invention. It should be understood that a block of a block diagram and / or flow chart, as well as a combination of blocks of a block diagram and / or flow chart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer and / or other programmable data processing device to produce a machine such that the instructions executed by the computer processor and / or other programmable data processing device create a method for implementing the functions / actions specified in the block diagram and / or flow chart block.

[0327] Accordingly, the present invention may also be implemented in hardware and / or software (including firmware, resident software, microcode, etc.). Furthermore, the present invention may take the form of a computer program product on a computer-usable or computer-readable storage medium having computer-usable or computer-readable program code embodied in the medium for use by or in conjunction with an instruction execution system. In the context of the present invention, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, transmit, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0328] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for removing image compression noise, characterized in that: include: Obtaining a compressed image frame to be processed; Based on the target noise detection model, the compressed image frame to be processed is detected to obtain a target noise intensity matrix, wherein the target noise intensity matrix includes multiple elements corresponding to multiple pixels included in the compressed image frame to be processed, and the elements are used to represent the compression noise intensity of the corresponding pixels; Constructing a target filtering template corresponding to a first pixel based on an element corresponding to a first pixel and a plurality of target pixels including the first pixel, wherein the first pixel is any one of the plurality of pixels, and the plurality of target pixels are selected from pixels included in the compressed image frame to be processed according to a preset window size, and the preset window size is used to represent an effective range of the target filtering template; Apply the target filtering template to perform filtering processing on the first pixel point to obtain a target compressed image frame.

2. The method according to claim 1, wherein The target noise detection model includes an encoding network layer, a decoding network layer and an output layer; The target noise detection model is used to detect the compressed image frame to be processed to obtain a target noise intensity matrix, including: Performing feature extraction on the compressed image frame to be processed through the coding network layer to obtain feature vectors corresponding to a plurality of pixel points included in the compressed image frame to be processed; Parsing the feature vectors corresponding to the plurality of pixel points respectively through the decoding network layer to obtain the compression noise intensities corresponding to the plurality of pixel points respectively; Processing the compressed noise intensities corresponding to the plurality of pixel points respectively through the output layer to obtain the target noise intensity matrix; Among them, for any pixel point among the multiple pixel points, the compression noise intensity corresponding to the pixel point is used as an element in the target noise intensity matrix, and the position information corresponding to the pixel point in the compressed image frame to be processed is used as the position information corresponding to the element in the target noise intensity matrix.

3. The method according to claim 1, wherein The step of constructing a target filtering template corresponding to the first pixel based on an element corresponding to the first pixel and a plurality of target pixels including the first pixel includes: Determining, according to the preset window size, the plurality of target pixels within an area centered around the first pixel; Determining a plurality of filter coefficients according to an element corresponding to the first pixel point and distances between the plurality of target pixels and the first pixel point; The target filtering template is constructed based on the preset window size and the determined multiple filtering coefficients.

4. The method according to claim 3, wherein The applying the target filtering template to perform filtering processing on the first pixel point includes: Using the target filter template, convolution processing is performed on the pixel value of the first pixel point to obtain a target pixel value; Update the pixel value of the first pixel point to the target pixel value.

5. The method according to any one of claims 1 to 4, characterized in that: The target noise detection model is determined by: Obtaining a preset set of training sample pairs, wherein each training sample pair consists of an image frame including compression noise and an image frame not including compression noise; Based on the set of training sample pairs, the noise detection model to be trained is trained in an iterative manner until a preset convergence condition is met, and the noise detection model output in the last round is used as the target noise detection model, wherein the following operations are performed during one round of iteration: Inputting the image frames including the compressed noise in the training sample pair set into the noise detection model to be trained to obtain a predicted noise intensity matrix corresponding to the image frames including the compressed noise; Based on the predicted noise intensity matrix, filtering the image frame including the compression noise to obtain a predicted denoised image frame; Determining a loss value based on the predicted denoised image frame and the image frame that does not include compression noise in the training sample pair set; Based on the loss value, model parameters of the noise detection model to be trained are adjusted.

6. The method according to claim 5, wherein The loss value is determined as follows: Determining an absolute difference between a pixel value of a second pixel and a pixel value of a third pixel, wherein the second pixel is any one of a plurality of pixels included in the predicted denoised image frame, and the third pixel is a pixel corresponding to the second pixel in the image frame not including compression noise, and the image frame including compression noise used to generate the predicted denoised image frame and the image frame not including compression noise belong to a sample training pair; The weighted sum of all determined absolute differences is used as the loss value.

7. An image compression noise removal device, characterized in that: include: An acquisition module, used for acquiring a compressed image frame to be processed; a noise detection module, configured to detect the compressed image frame to be processed based on a target noise detection model to obtain a target noise intensity matrix, wherein the target noise intensity matrix includes a plurality of elements corresponding to a plurality of pixels included in the compressed image frame to be processed, and the elements are used to represent the compression noise intensity of the corresponding pixels; a filtering construction module, configured to construct a target filtering template corresponding to a first pixel point based on an element corresponding to the first pixel point and a plurality of target pixels including the first pixel point, wherein the first pixel point is any pixel point among the plurality of pixels points, and the plurality of target pixels points are selected from the pixels included in the compressed image frame to be processed according to a preset window size, and the preset window size is used to represent the scope of action of the target filtering template; A processing module is used to apply the target filtering template to perform filtering processing on the first pixel point to obtain a target compressed image frame.

8. The device according to claim 7, wherein The target noise detection model includes an encoding network layer, a decoding network layer and an output layer; The noise detection module is specifically used for: Performing feature extraction on the compressed image frame to be processed through the coding network layer to obtain feature vectors corresponding to a plurality of pixel points included in the compressed image frame to be processed; Parsing the feature vectors corresponding to the plurality of pixel points respectively through the decoding network layer to obtain the compression noise intensities corresponding to the plurality of pixel points respectively; Processing the compressed noise intensities corresponding to the plurality of pixel points respectively through the output layer to obtain the target noise intensity matrix; Among them, for any pixel point among the multiple pixel points, the compression noise intensity corresponding to the pixel point is used as an element in the target noise intensity matrix, and the position information corresponding to the pixel point in the compressed image frame to be processed is used as the position information corresponding to the element in the target noise intensity matrix.

9. An electronic device, characterized in that: include: a memory for storing executable instructions; A processor is configured to read and execute the executable instructions stored in the memory to implement the steps of the image compression noise removal method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to execute the steps of the method for removing image compression noise according to any one of claims 1 to 6.

Images