An image processing method, system, storage medium, and electronic device

Abstract

The present application provides an image processing method, system, storage medium, and electronic device for denoising an image, which solves the problem of losing image details when reducing noise during image denoising. The image processing method includes: dividing the image to be processed into a plurality of first image blocks; for each first image block, determining a corresponding second image block set; performing a transformation from the spatial domain to the frequency domain on the second image block set to obtain a combination of transformation coefficients; filtering the combination of transformation coefficients based on the frequency domain characteristics of the combination of transformation coefficients; performing a transformation from the frequency domain to the spatial domain on the filtered combination of transformation coefficients to obtain a third image block set; and using an inertia coefficient to weight and aggregate the third image block set to obtain a denoised image corresponding to the image to be processed. The solution of the present application solves the problem that some detail textures are lost when obtaining a better denoising effect during image denoising, and protects the detail textures while ensuring a smooth filtering effect in flat areas.

Description

Technical Field

[0001] The present invention relates to the field of image denoising, and in particular, to an image processing method, system, storage medium, and electronic device. Background Art

[0002] The Block Matching 3D (BM3D) is a traditional video or image denoising method that uses block matching technology and the transformation from the spatial domain to the frequency domain to filter the video or image in the frequency domain to achieve the purpose of denoising.

[0003] Filtering the image in the frequency domain can eliminate most of the noise energy, but at the same time, some detail information will be lost, making the filtered image blurred; and if the filtering threshold is reduced to retain the detail information, the overall denoising effect of the image will become worse. Summary of the Invention

[0004] In view of this, the embodiments of the present application are committed to providing an image processing method, system, storage medium, and electronic device to solve the problem of losing image details when reducing noise during image denoising.

[0005] In a first aspect, the present application provides an image processing method, including: dividing an image to be processed into a plurality of first image blocks, where the union of the plurality of first image blocks completely covers the image to be processed; for each first image block, determining a corresponding second image block set, where the second image block set includes a second image block with the smallest pixel value difference from the first image block, and the size of the second image block is the same as that of the first image block; performing a transformation from the spatial domain to the frequency domain on the second image block set to obtain a transformation coefficient combination corresponding to the second image block set; filtering the transformation coefficient combination based on the frequency domain characteristics of the transformation coefficient combination; performing a transformation from the frequency domain to the spatial domain on the filtered transformation coefficient combination to obtain a third image block set; and using inertial coefficient weighted aggregation on the third image block set to obtain a denoised image corresponding to the image to be processed.

[0006] In combination with the first aspect, in some implementation manners of the first aspect, filtering the transformation coefficient combination based on the frequency domain characteristics of the transformation coefficient combination includes: calculating a gradient value corresponding to the transformation coefficient combination, where the gradient value is used to represent the texture richness of the second image block set; determining a filtering threshold according to the gradient value; and filtering the transformation coefficient combination according to the filtering threshold.

[0007] In combination with the first aspect, in some implementations of the first aspect, calculating the gradient value corresponding to the combination of transform coefficients includes: determining the spectral components of the first-layer transform coefficients in the combination of transform coefficients and the first parameter value corresponding to the spectral components, where the first parameter value is used to reflect the contribution degree of the spectral components to the texture richness; calculating the gradient value corresponding to the combination of transform coefficients based on the spectral components and the first parameter value corresponding to the spectral components.

[0008] In combination with the first aspect, in some implementations of the first aspect, calculating the gradient value corresponding to the combination of transform coefficients based on the spectral components and the first parameter value corresponding to the spectral components includes: determining the second parameter value for adjusting the gradient value range; calculating the gradient value corresponding to the combination of transform coefficients by combining the second parameter value, the spectral components, and the first parameter value.

[0009] In combination with the first aspect, in some implementations of the first aspect, filtering the combination of transform coefficients based on the frequency-domain characteristics of the combination of transform coefficients further includes: determining the first adjustment coefficient; re-filtering the upper-left A×A region of the first layer in the filtered combination of transform coefficients according to the first adjustment coefficient, where A is less than or equal to 4.

[0010] In combination with the first aspect, in some implementations of the first aspect, obtaining the noise-reduced image corresponding to the image to be processed by weighted aggregation of the third set of image blocks using the inertia coefficient includes: determining the inertia coefficient; determining the aggregation weight of the third set of image blocks; performing weighted aggregation on the third set of image blocks according to the inertia coefficient and the aggregation weight to obtain the noise-reduced image corresponding to the image to be processed.

[0011] In combination with the first aspect, in some implementations of the first aspect, determining the second set of image blocks corresponding to the first image block includes: obtaining the video image sequence where the image to be processed is located; determining the search window of the first image block in the video image sequence and searching for the second set of image blocks corresponding to the first image block within the search window.

[0012] Second aspect, an embodiment of the present application provides an image processing system, which includes: a division module for dividing an image to be processed into a plurality of first image blocks, where the union of the plurality of first image blocks completely covers the image to be processed; a determination module for determining, for each first image block, a corresponding second image block set, where the second image block set includes a second image block with the smallest pixel value difference from the first image block, and the size of the second image block is the same as that of the first image block; a transformation module for performing a transformation from the spatial domain to the frequency domain on the second image block set to obtain a combination of transformation coefficients corresponding to the second image block set; a filtering module for filtering the combination of transformation coefficients based on the frequency domain characteristics of the combination of transformation coefficients; an inverse transformation module for performing a transformation from the frequency domain to the spatial domain on the filtered combination of transformation coefficients to obtain a third image block set; an aggregation module for aggregating the third image block set with a weighted inertial coefficient to obtain a denoised image corresponding to the image to be processed.

[0013] Third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program for executing the image processing method mentioned in the first aspect above.

[0014] Fourth aspect, an embodiment of the present application provides an electronic device, which includes: a processor for executing the image processing method mentioned in the first aspect above; a memory for storing instructions executable by the processor.

[0015] The image processing method provided in this embodiment solves the problem that some detailed textures are lost when obtaining a better denoising effect during image denoising, and protects the detailed textures while ensuring a smooth filtering effect in flat areas. Description of the Drawings

[0016] Figure 1 The figure shows a flow diagram of BM3D in the prior art.

[0017] Figure 2 The figure shows a flow diagram of the image processing method provided by an embodiment of the present application.

[0018] Figure 3 The figure shows a flow diagram of the image processing method provided by another embodiment of the present application.

[0019] Figure 4 The figure shows a flow diagram of the image processing method provided by yet another embodiment of the present application.

[0020] Figure 5 The figure shows a schematic diagram of the spectral distribution of the first-layer coefficients of the combination of transformation coefficients provided by an embodiment of the present application.

[0021] Figure 6The figure shows a schematic flowchart of an image processing method provided by another embodiment of the present application.

[0022] Figure 7 The figure shows a schematic flowchart of an image processing method provided by another embodiment of the present application.

[0023] Figure 8 The figure shows a schematic flowchart of an image processing method provided by another embodiment of the present application.

[0024] Figure 9 The figure shows a schematic flowchart of an image processing method provided by another embodiment of the present application.

[0025] Figure 10 The figure shows a schematic structural diagram of an image processing system provided by an embodiment of the present application.

[0026] Figure 11 The figure shows a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0027] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0028] Overview of the Application

[0029] BM3D is a method for image or video noise reduction and performs excellently in traditional noise reduction. Its general process is as Figure 1 shown. First, the image or video to be denoised is matched and grouped, that is, block matching is performed to obtain a similar block combination; then the obtained similar block combination is transformed from the spatial domain to the frequency domain; hard-thresholding filtering is performed in the frequency domain. Most of the energy of the noise can be eliminated through hard-thresholding filtering; after filtering, an inverse transformation from the frequency domain to the spatial domain is performed, and the inverse-transformed similar block combination is re-superimposed in the original superposition manner, that is, aggregation. During the aggregation process, two buffers are required. One buffers the weighted sum of pixel values, such as Figure 1 the white block pointed by the aggregation arrow in Figure 1 ; one buffers the sum of pixel weights, such as

[0030] the gray block pointed by the aggregation arrow in. Through this method, an image or video with most of the noise eliminated can be obtained.However, when performing hard threshold filtering, high-frequency information representing detailed textures will also be filtered out. The loss of this part of the information will cause the denoised image or video to be blurred and the clarity to decrease. In addition, when combining and superimposing similar blocks after inverse transformation in traditional technologies, the computational complexity is relatively high, reducing the denoising efficiency. The two required cache spaces consume a large amount of computing resources, increasing the chip cost and limiting their application in actual products.

[0031] To solve the problem that some detailed textures are lost when obtaining a better denoising effect during image denoising in the above traditional technologies, an embodiment of this application provides an image processing method.

[0032] Figure 2 It is a schematic flowchart of the image processing method provided by an embodiment of this application. The following will be described in detail Figure 2 The schematic flowchart of the image processing method provided by an embodiment of this application shown below, and this method includes the following steps.

[0033] Step S10: Divide the image to be processed into multiple first image blocks.

[0034] The union of the multiple first image blocks completely covers the image to be processed. Specifically, the sizes of each first image block may not be equal, and adjacent first image blocks may overlap or not overlap. Here, no specific restrictions are imposed on the size and overlapping method of the first image blocks.

[0035] Exemplarily, the sizes of each first image block are the same, all being N×N, and there will be a certain number of overlapping pixels between adjacent first image blocks.

[0036] Step S12: For each first image block, determine the second image block set corresponding to the first image block.

[0037] Among them, the second image block set includes the second image block with the smallest pixel value difference from the first image block, and the size of the second image block is the same as that of the first image block.

[0038] Specifically, determine the search window of the first image block and search for the second image block set corresponding to the first image block within the search window. The shape of the search window can be rectangular, and the sizes and relative positions of the search windows of different first image blocks can be the same. Embodiments of this application do not impose restrictions on the shape of the search window, the size and relative position of the search window.

[0039] Exemplarily, the search window can be selected in the surrounding area of the first image block.

[0040] Specifically, for each first image block, through the method of block matching, search for a certain number of second image blocks with the same size as the first image block within the search window of the first image block.

[0041] Exemplarily, each pixel in the search window can be used as the upper left corner of a candidate second image block. If there are M pixels in the search window, there are M candidate second image blocks, and the first image block is also a candidate second image block. Among the candidate second image blocks, multiple second image blocks with the smallest pixel value difference from the first image block are searched by means of block matching.

[0042] Exemplarily, the absolute value of the average difference MAD between the current first image block and each candidate second image block is calculated, and the calculation formula is as follows:

[0043]

[0044] The MAD between the first image block and the M candidate second image blocks is obtained. The obtained M candidate second image blocks are sorted from small to large according to the corresponding MAD values; H candidate second image blocks with smaller MAD values are selected, where H can be a power of 2, such as 16; a three-dimensional N×N×H combination, that is, the second image block set, is obtained, and each pixel value therein can be represented by P(i,j,k). Since the first image block itself is among the candidate blocks (i.e., the second image blocks), and its MAD value is 0, the first image block will also be in the second image block set.

[0045] Step S14: Perform a transformation from the spatial domain to the frequency domain on the second image block set to obtain a transformation coefficient combination corresponding to the second image block set.

[0046] Exemplarily, a two-dimensional discrete cosine transform or discrete wavelet transform can be first performed on each N×N second image block in the second image block set, and then a one-dimensional discrete cosine transform or discrete wavelet transform is performed in the third dimension, that is, the second image block set is transformed from the spatial domain to the frequency domain to obtain a transformation coefficient combination of N×N×H. Each coefficient component therein can be represented by T(i,j,k).

[0047] Step S16: Filter the transformation coefficient combination based on the frequency domain characteristics of the transformation coefficient combination.

[0048] Specifically, the first layer component of the transformation coefficient combination reflects the texture spectrum information of the first image block. A filtering threshold is set in combination with the spectrum information of the first layer component, and the transformation coefficient combination is filtered.

[0049] Step S18: Perform a transformation from the frequency domain to the spatial domain on the filtered transformation coefficient combination to obtain a third image block set.

[0050] Specifically, an inverse transformation of a three-dimensional transformation is performed on the filtered transformation coefficient combination to obtain a third image block set, and the method of the inverse transformation is not limited in the embodiments of the present application.

[0051] Exemplarily, the filtered N×N×H transform coefficient combination is inversely transformed to obtain a filtered third image block set of N×N×H, where each pixel value in the third image block set can be represented by P’(i,j,k).

[0052] Step S20: Use the inertia coefficient to weight and aggregate the third image block set to obtain a denoised image corresponding to the image to be processed.

[0053] Specifically, each third image block in the third image block set is weighted and aggregated according to the superposition manner of its corresponding second graphic block to obtain a denoised image corresponding to the image to be processed. Wherein, when performing weighted aggregation, an inertia coefficient is introduced to scale the aggregation weight, thereby realizing normalized weighted average.

[0054] The image processing method provided by the embodiment of the present application combines the frequency domain characteristics of the first image block. When filtering the transform coefficient combination, the filtering value can be adjusted according to the texture information of the image, so as to retain the detailed texture while obtaining a better denoising effect.

[0055] Figure 3 It is a schematic flowchart of an image processing method provided by another embodiment of the present application, which extends from the Figure 2 embodiment shown, and Figure 3 extends to the Figure 3 embodiment shown. Next, the differences between the Figure 2 embodiment shown and the

[0056] embodiment shown will be emphasized, and the same parts will not be elaborated. Figure 3 As shown in

[0057] Step S160: Calculate the gradient value (Grad) corresponding to the transform coefficient combination.

[0058] The gradient value is used to represent the texture richness of the second image block set.

[0059] Specifically, each coefficient component of the transform coefficient combination can reflect various frequency characteristics of the corresponding first image block. For example, in the first-layer coefficient component of the transform coefficient combination, the upper left corner represents the component with the lowest frequency. The farther it extends in the lower right corner direction, the higher the corresponding frequency will be. Correspondingly, the gradient value is also larger. The gradient value corresponding to the transform coefficient combination can be reasonably estimated using a formula, and the embodiment of the present application does not limit the calculation method.

[0060] Step S162: Determine the filtering threshold according to the gradient value.

[0061] Specifically, since the gradient value can reflect the texture richness of the combination of transform coefficients, a filtering threshold is set in combination with the gradient value to filter the coefficient components in the combination of transform coefficients.

[0062] Exemplarily, the filtering threshold can be calculated by the following formula:

[0063] Thr = p·exp(-q·Grad)+r,

[0064] p,q,r>0

[0065] Where r is used to limit the minimum filtering threshold, p is used to limit the maximum filtering threshold, and q is used to control the speed at which the filtering threshold changes with Grad. The specific values need to be determined according to the specific shooting scenario.

[0066] Step S164: Filter the combination of transform coefficients according to the filtering threshold.

[0067] Exemplarily, if the filtering threshold is Thr, the combination of transform coefficients can be filtered by the following formula:

[0068]

[0069] The image processing method provided by the embodiments of the present application can perform different degrees of filtering on the first image block with different texture complexities by calculating the gradient value and determining the filtering threshold according to the gradient value, so as to adapt to each region of the image with different amounts of detail information, and more specifically retain the detail information of the image during the noise reduction process and ensure the image noise reduction effect in the flat area.

[0070] Figure 4 It is a schematic flowchart of the image processing method provided by another embodiment of the present application, which extends from the Figure 3 shown embodiment and Figure 4 the shown embodiment will be described below in combination with Figure 5 and the differences between the Figure 4 shown embodiment and the Figure 3 shown embodiment will be emphasized. The same parts will not be repeated.

[0071] As Figure 4 shown, in the embodiments of the present application, calculating the gradient value corresponding to the combination of transform coefficients includes the following steps.

[0072] Step S1602: Determine the spectral components of the first-layer transform coefficients in the combination of transform coefficients, and the first parameter values corresponding to the spectral components.

[0073] Where the first parameter value is used to reflect the contribution degree of the spectral component to the texture richness.

[0074] Exemplarily, in the first-layer coefficient components of the N×N×H combination of transform coefficients, according to the frequency from low to high, these N×N components can be divided into four categories: direct current (DC), low frequency (L), edge (E), and high frequency (H). DC reflects the average brightness level of the first image block, L reflects the speed of brightness change of the first image block, E reflects the edge information with obvious directionality existing in the first image block, and H reflects whether there is grid-like detailed texture in the first image block.

[0075] Specifically, the category to which each component belongs needs to be determined according to the size of N and the type of transformation algorithm, and the embodiments of the present application do not limit this.

[0076] Exemplarily, when N = 8 and the two-dimensional spatial-frequency domain transformation type is discrete cosine transform, a common division method is as Figure 5 shown. Calculate the sum of the absolute values of each component, and the calculation formula is as follows:

[0077] S DC = ∑ (i,j)∈{DC} |T(i,j,0)|, S L = ∑ (i,j)∈{L} |T(i,j,0)|,

[0078] S E = ∑ (i,j)∈{E} |T(i,j,0)|, S H = ∑ (i,j)∈{H} |T(i,j,0)|.

[0079] Among them, when a certain category in DC / L / E / H is missing, it is default that the sum of the absolute values of the corresponding components is 0.

[0080] Determine a preset first parameter value corresponding to the spectral component, where the first parameter value is used to control the contribution degree of the four categories to the gradient value. For example, W DC , W L , W E , W H respectively represent the first parameter values corresponding to the four categories of DC, L, E, and H. For different shooting environments, the preset values are different. The more commonly used values are: W DC = 0.2, W L = 0.5, W E = 0.7, W H = 0.4. The DC category and the L category are less affected by noise. The components of the E category are more easily affected by noise, and the components of the H category are extremely easily affected by noise. When the noise is weak, the first parameter value corresponding to the H category takes a smaller value, the first parameter values corresponding to the L and E categories take larger values, and the first parameter value corresponding to the DC category also takes a smaller value. For example, W DC = 0.1, WL = 0.5, W E = 0.5, W H = 0.5; If greater protection for detailed information is required, the value of the first parameter corresponding to the H category can be appropriately increased. When the noise is strong, reduce the value of the first parameter corresponding to the H category, increase the value of the first parameter corresponding to the L category, keep the value of the first parameter corresponding to the E category moderate, and appropriately increase the value of the first parameter corresponding to the DC category. For example, W DC = 0.3, W L = 0.8, W E = 0.5, W H = 0.1. If the denoised image is too blurred, the value of the first parameter corresponding to the E category can be appropriately increased. For example, W E = 0.6; If the denoised image has a large amount of noise, the value of the first parameter corresponding to the E category can be appropriately decreased. For example, W E = 0.3.

[0081] Step S1604: Calculate the gradient value corresponding to the combination of transform coefficients based on the spectral components and the first parameter values corresponding to the spectral components.

[0082] Specifically, calculate the gradient value corresponding to the combination of transform coefficients according to the mapping from the spectral components and the first parameter values to the gradient value.

[0083] Exemplarily, the gradient value can be calculated according to the following formula:

[0084] Grad = W DC * S DC + W L * S L + W E * S E + W H * S H

[0085] The image processing method provided by the embodiments of the present application can adaptively allocate a suitable filtering threshold according to the environmental information and spectral components of each first image block. When the texture of the first image block is relatively rich, more details can be retained during denoising; when the texture of the first image block is less, more noise can be removed during denoising.

[0086] Figure 6 It is a schematic flowchart of the image processing method provided by another embodiment of the present application, which is extended based on the embodiment shown in Figure 4 The embodiment shown in Figure 6 The differences between the embodiment shown in Figure 6 and the embodiment shown in Figure 4 will be mainly described below, and the same parts will not be repeated.

[0087] As Figure 6 shown, in the embodiment of the present application, based on the spectral components and the corresponding first parameter values of the spectral components, calculating the gradient value corresponding to the combination of transform coefficients includes the following steps.

[0088] Step S16040: Determine the second parameter value for adjusting the gradient value range.

[0089] Specifically, the selection of different first image block sizes will directly affect the range of gradient values. Different second parameter values are preset according to the size of the first image block to adjust the value range of the gradient value.

[0090] Exemplarily, to keep the output range of the gradient value Grad consistent for different data bit widths and the sizes of the first image blocks, two second parameter values a and b are added to increase the flexibility of adjustment. The values of a and b are not restricted. Before the method steps of the embodiment of the present application are executed, appropriate values of a and b are preset for each combination of data bit width and first image block size. For example, when the pixel value range of the image is 0 to 255 and the first image block size N = 8, a = 16, and b is flexibly selected in [4, 16].[[]END]]

[0091] Step S16042: Combine the second parameter value, the spectral component, and the first parameter value to calculate the gradient value corresponding to the combination of transform coefficients.

[0092] Exemplarily, based on the second parameter value, the spectral component, and the first parameter value, a reasonable estimation of the gradient value corresponding to the combination of transform coefficients can be made, and the gradient value can be calculated according to the following formula:

[0093] Grad = W DC *S DC / a + (W L *S L + W E *S E + W H *S H ) / b

[0094] The image processing method provided by the embodiment of the present application supports images to be processed with multiple data bit widths by configuring appropriate second parameter values, which is convenient for chip design and improves the practicability of the image processing method provided by the embodiment of the present application.

[0095] Figure 7 For the flowchart of the image processing method provided by another embodiment of the present application, on the basis of the embodiment shown in Figure 6 the embodiment shown is extended to Figure 7 the embodiment shown. The following will focus on describing Figure 7 the embodiment shown and Figure 6The differences of the illustrated embodiments are not described again, and the same parts are not elaborated.

[0096] As Figure 7 shown, in the embodiments of the present application, filtering is performed on the transform coefficient combination based on the frequency domain characteristics of the transform coefficient combination, and the following steps are further included.

[0097] Step S166: Determine the first adjustment coefficient.

[0098] Specifically, for the upper left 4×4 region of the first layer coefficient components in the transform coefficient combination, that is, the region {T(i,j,k)|i < 4, j < 4, k = 0}, most of the energy of the transform coefficient combination is contained in this part of the coefficient components. Only adjusting the filtering threshold of this part of the coefficients can obtain a relatively obvious improvement effect. The filtering threshold of this part of the coefficients can be scaled by the first adjustment coefficient.

[0099] Exemplarily, let the scaling coefficient be Scale(i,j), i < 4, j < 4, and the value range is [0, 5], where Scale(0,0) = 0, and the rest are greater than 1.

[0100] Step S168: Re-filter the upper left A×A region of the first layer in the filtered transform coefficient combination according to the first adjustment coefficient.

[0101] Among them, A is less than or equal to 4.

[0102] Exemplarily, re-filtering the upper left A×A region of the first layer in the filtered transform coefficient combination can be calculated through the following formula:

[0103] If i < 4 and j < 4 and k = 0:

[0104]

[0105] else:

[0106]

[0107] The scaling coefficient is applicable to all the first image blocks and their corresponding second image block sets in the image to be processed. When the size of the first image block is less than 4, A can directly take the value of the size of the first image block.

[0108] The image processing method provided by the embodiments of the present application significantly improves the filtering effect without increasing the computational complexity, and achieves a better noise reduction effect.

[0109] Figure 8 For the flowchart of the image processing method provided by another embodiment of the present application, based on the Figure 2 illustrated embodiment, it extendsFigure 8 In the embodiments shown below, the following will be emphasized Figure 8 the differences between the embodiments shown and Figure 2 the embodiments shown. The same parts will not be described again.

[0110] As Figure 8 shown, in the embodiments of the present application, a denoised image corresponding to the image to be processed is obtained by weighted aggregation of the third image block set using an inertia coefficient, including the following steps.

[0111] Step S200: Determine the inertia coefficient.

[0112] Specifically, in the traditional BM3D denoising method, when weighted aggregation is performed on the combined similar blocks after inverse transformation, two buffers are required, including a numerator buffer and a denominator buffer. The numerator buffer is used to cache the numerical weighted sum of the filtering results of different second image blocks corresponding to each pixel, and the denominator buffer is used to cache the weight sum of the filtering results of different second image blocks for each pixel. To avoid the use of the denominator buffer, an inertia coefficient can be preset.

[0113] Exemplarily, let the inertia coefficient be m, and the value range is [0,1]. Usually, m = 0.2. The inertia coefficient can take equal values for all the first image blocks of the image to be processed and their corresponding second image block sets, that is, the inertia coefficient m can be equal for the same image to be processed.

[0114] Step S202: Determine the aggregation weight of the third image block set.

[0115] Exemplarily, w(i,j,0) is the weight of each pixel P’(i,j,0) in the first second image block in the second image block set, which can be configured by the user or automatically adjusted by the algorithm, and the value range is [0,1]. The weights of the pixels located at the boundary of the second image block are lower than those of the pixels at the block center, which can suppress the possible block effect.

[0116] Step S204: Perform weighted aggregation on the third image block set according to the inertia coefficient and the aggregation weight to obtain a denoised image corresponding to the image to be processed.

[0117] Specifically, when the numerator buffer is zero, the filtered pixel value is directly stored. When a new filtered pixel value needs to be aggregated at a non-zero pixel position in the numerator buffer, the aggregation weight is scaled and weighted averaged in combination with the inertia coefficient.

[0118] Exemplarily, let a non-zero pixel position in the numerator buffer be Numer(x,y). The weighted aggregation of the third image block set according to the aggregation weight can be calculated by the following formula:

[0119] If Numer(x+i,y+j)=0:

[0120] Numer(x+i, y+j) = P'(i, j, 0),

[0121] else:

[0122] Numer(x+i, y+j) = P'(i, j, 0) * w(i, j, 0) * m +

[0123] Numer(x+i, y+j) * (1 - w(i, j, 0) * m).

[0124] Using an inertia coefficient m to scale the aggregation weight w(i, j, k), and then performing a normalized weighted average of Numer(x, y) and P'(i, j, k), the resulting value is still a valid pixel value, and there is no longer a need to use a denominator buffer for normalization.

[0125] The image processing method provided by the embodiments of the present application saves the storage space of the denominator buffer required for normalization and reduces the production cost of the chip. In addition, the result of global normalization adopted by the original aggregation method is approximately obtained with a small error, greatly reducing the computational complexity.

[0126] Figure 9 It is a schematic flowchart of the image processing method provided by another embodiment of the present application, extending from the embodiment shown in Figure 2 On the basis of the embodiment shown, the embodiment shown in Figure 9 will be emphasized below. The differences between the embodiment shown in Figure 9 and the embodiment shown in Figure 2 will be described, and the same parts will not be elaborated.

[0127] As shown in Figure 9 , in the embodiments of the present application, determining the set of second image blocks corresponding to the first image block includes the following steps.

[0128] Step S120: Obtain the video image sequence where the image to be processed is located.

[0129] Specifically, for noise reduction of the video image sequence, the second image block can be searched on other frames of the video image sequence. Therefore, it is necessary to first obtain the video image sequence where the image to be processed is located.

[0130] Step S122: Determine the search window of the first image block in the video image sequence and search for the set of second image blocks corresponding to the first image block within the search window.

[0131] Specifically, the search window of the video image sequence can be a three-dimensional cuboid, which is the image sequences of multiple frames before and after the position corresponding to the first image block.

[0132] The image processing method provided by the embodiments of the present application can perform noise reduction on a video image sequence on the basis of performing noise reduction on an image.

[0133] In an embodiment of the present application, the image to be processed may be a single-frame multi-channel image.

[0134] Exemplarily, for a multi-channel image in YUV format, in step S12: for each first image block, determining the second image block set corresponding to the first image block, a block matching operation may be performed on the Y component to determine the second image block corresponding to the first image block, and the positions of the second image blocks of the U / V components reuse those of the second image block of the Y component. However, the U component and the V component are each stacked into a 3D second image block set from 2D second image blocks in the manner of a grayscale image, and then subsequent filtering aggregation steps are performed.

[0135] Through the image processing method provided by the embodiments of the present application, the present image processing method can be applied to noise reduction of images in different formats, improving the generality of the image processing method.

[0136] Figure 10 It is a schematic structural diagram of an image processing system provided by an embodiment of the present application. As shown, the video processing system 1000 provided by the embodiments of the present application includes: a partitioning module 100, a determination module 200, a transformation module 300, a filtering module 400, an inverse transformation module 500, and an aggregation module 600.

[0137] Specifically, the partitioning module 100 is configured to partition the image to be processed into a plurality of first image blocks, where the union of the plurality of first image blocks completely covers the image to be processed; the determination module 200 is configured to, for each first image block, determine the second image block set corresponding to the first image block, where the second image block set includes the second image block with the smallest pixel value difference from the first image block, and the size of the second image block is the same as that of the first image block; the transformation module 300 is configured to perform a transformation of the second image block set from the spatial domain to the frequency domain to obtain a combination of transformation coefficients corresponding to the second image block set; the filtering module 400 is configured to filter the combination of transformation coefficients based on the frequency domain characteristics of the combination of transformation coefficients; the inverse transformation module 500 is configured to perform a transformation of the filtered combination of transformation coefficients from the frequency domain to the spatial domain to obtain a third image block set; the aggregation module 600 is configured to perform weighted aggregation on the third image block set using an inertia coefficient to obtain a noise-reduced image corresponding to the image to be processed.

[0138] In an embodiment of the present application, when the filtering module 400 executes the step of filtering the combination of transformation coefficients based on the frequency domain characteristics of the combination of transformation coefficients, it executes the following steps: calculating a gradient value corresponding to the combination of transformation coefficients, where the gradient value is used to represent the texture richness of the second image block set; determining a filtering threshold according to the gradient value; and filtering the combination of transformation coefficients according to the filtering threshold.

[0139] In an embodiment of the present application, when the filtering module 400 executes the step of calculating the gradient value corresponding to the combination of transform coefficients, the following steps are executed: determining the spectral components of the first-layer transform coefficients in the combination of transform coefficients and the first parameter values corresponding to the spectral components, where the first parameter values are used to reflect the contribution degree of the spectral components to the texture richness; calculating the gradient value corresponding to the combination of transform coefficients based on the spectral components and the first parameter values corresponding to the spectral components.

[0140] In an embodiment of the present application, when the filtering module 400 executes the step of calculating the gradient value corresponding to the combination of transform coefficients based on the spectral components and the first parameter values corresponding to the spectral components, the following steps are executed: determining the second parameter value for adjusting the gradient value range; calculating the gradient value corresponding to the combination of transform coefficients by combining the second parameter value, the spectral components, and the first parameter values.

[0141] In an embodiment of the present application, when the filtering module 400 executes the step of filtering the combination of transform coefficients based on the frequency domain characteristics of the combination of transform coefficients, the following steps are also executed: determining the first adjustment coefficient; performing secondary filtering on the upper left A×A region of the first layer in the filtered combination of transform coefficients according to the first adjustment coefficient, where A is less than or equal to 4.

[0142] In an embodiment of the present application, when the aggregation module 600 executes the step of aggregating the third image block set with an inertia coefficient weighting to obtain the noise-reduced image corresponding to the image to be processed, the following steps are executed: determining the inertia coefficient; determining the aggregation weight of the third image block set; performing weighted aggregation on the third image block set according to the inertia coefficient and the aggregation weight to obtain the noise-reduced image corresponding to the image to be processed.

[0143] In an embodiment of the present application, when the determination module 200 executes the step of determining the second image block set corresponding to the first image block, the following steps are executed: obtaining the video image sequence where the image to be processed is located; determining the search window of the first image block in the video image sequence and searching for the second image block set corresponding to the first image block within the search window.

[0144] Figure 10 The following shows the structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown, the electronic device 1100 includes one or more processors 1110 and a memory 1120.

[0145] The processor 1110 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1100 to execute desired functions.

[0146] The memory 1120 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1110 may run the program instructions to implement the image processing methods of the various embodiments of the present application above and / or other desired functions. Various contents such as the aggregation weights of the third image block set may also be stored in the computer-readable storage medium.

[0147] In one example, the electronic device 1100 may further include: an input device 1130 and an output device 1140, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0148] The input device 1130 may include, for example, a keyboard, a mouse, and so on.

[0149] The output device 1140 may output various information to the outside, including reminder information, etc. The output device 1140 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, and so on.

[0150] Of course, for simplicity, Figure 11 Figure 11 Figure 11 only some of the components related to the present application in the electronic device 1100 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1100 may further include any other appropriate components.

[0151] In addition to the above methods and devices, the embodiments of the present application may also be a computer program product, which includes computer program instructions that, when run by a processor, cause the processor to execute the steps in the image processing methods according to the various embodiments of the present application described above in this specification.

[0152] The computer program product may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0153] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An image processing method, characterized in that, Including: Dividing the image to be processed into a plurality of first image blocks, wherein the union of the plurality of first image blocks completely covers the image to be processed; For each of the first image blocks, determining a corresponding second image block set, wherein the second image block set includes a second image block with the smallest pixel value difference from the first image block, and the size of the second image block is the same as the size of the first image block; Performing a transformation on the second image block set from the spatial domain to the frequency domain to obtain a combination of transformation coefficients corresponding to the second image block set; Filtering the combination of transformation coefficients based on the frequency domain characteristics of the combination of transformation coefficients; Performing a transformation on the filtered combination of transformation coefficients from the frequency domain to the spatial domain to obtain a third image block set; Using an inertia coefficient to perform weighted aggregation on the third image block set to obtain a denoised image corresponding to the image to be processed.

2. The image processing method according to claim 1, wherein The filtering the combination of transformation coefficients based on the frequency domain characteristics of the combination of transformation coefficients includes: Calculating a gradient value corresponding to the combination of transformation coefficients, where the gradient value is used to represent the texture richness degree of the second image block set; Determining a filtering threshold according to the gradient value; Filtering the combination of transformation coefficients according to the filtering threshold.

3. The image processing method according to claim 2, wherein The calculating the gradient value corresponding to the combination of transformation coefficients includes: Determining a spectral component of the first layer of transformation coefficients in the combination of transformation coefficients, and a first parameter value corresponding to the spectral component, where the first parameter value is used to reflect the contribution degree of the spectral component to the texture richness degree; Based on the spectral component and the first parameter value corresponding to the spectral component, calculating the gradient value corresponding to the combination of transformation coefficients.

4. The image processing method according to claim 3, wherein The based on the spectral component and the first parameter value corresponding to the spectral component, calculating the gradient value corresponding to the combination of transformation coefficients includes: Determining a second parameter value for adjusting the gradient value range; Combining the second parameter value, the spectral component, and the first parameter value to calculate the gradient value corresponding to the combination of transformation coefficients.

5. The image processing method according to claim 2, characterized in that, The filtering the combination of transformation coefficients based on the frequency domain characteristics of the combination of transformation coefficients further includes: Determining a first adjustment coefficient; According to the first adjustment coefficient, performing secondary filtering on the upper left A×A area of the first layer in the filtered combination of transformation coefficients, where A is less than or equal to 4.

6. The image processing method according to claim 1, wherein The using an inertia coefficient to perform weighted aggregation on the third image block set to obtain a denoised image corresponding to the image to be processed includes: Determining the inertia coefficient; Determining the aggregation weight of the third image block set; According to the inertia coefficient and the aggregation weight, performing weighted aggregation on the third image block set to obtain a denoised image corresponding to the image to be processed.

7. The image processing method according to claim 1, wherein The determining the second image block set corresponding to the first image block includes: Obtaining the video image sequence where the image to be processed is located; Determining a search window of the first image block in the video image sequence and searching for the second image block set corresponding to the first image block within the search window.

8. An image processing system, characterized in that, Including: A partitioning module, configured to partition an image to be processed into a plurality of first image blocks, wherein the union of the plurality of first image blocks completely covers the image to be processed; A determination module, configured to determine, for each of the first image blocks, a corresponding second image block set, wherein the second image block set includes a second image block with the smallest pixel value difference from the first image block, and the size of the second image block is the same as that of the first image block; A transformation module, configured to transform the second image block set from the spatial domain to the frequency domain to obtain a transformation coefficient combination corresponding to the second image block set; A filtering module, configured to filter the transformation coefficient combination based on the frequency domain characteristics of the transformation coefficient combination; An inverse transformation module, configured to transform the filtered transformation coefficient combination from the frequency domain to the spatial domain to obtain a third image block set; An aggregation module, configured to aggregate the third image block set with weighted aggregation using an inertia coefficient to obtain a denoised image corresponding to the image to be processed.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is used to execute the image processing method according to any one of claims 1 to 7 above.

10. An electronic device, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; The processor is configured to execute the image processing method according to any one of claims 1 to 7 above.

Images