Image processing method, device and computer readable storage medium

Abstract

The present application discloses an image processing method, device and computer-readable storage medium, wherein the image processing method comprises: determining bad pixel in the image to be processed; determining multiple similar reference image blocks in the image to be processed; for the target bad pixel of the target image block in the multiple reference image blocks, adjusting the characteristic value of the target bad pixel by using the characteristic value of the non-bad pixel in the target image block or the characteristic value of the non-bad pixel in the same position in the remaining reference image blocks except the target image block, so as to obtain multiple adjusted reference image blocks; adjusting the characteristic value of the image to be processed by using the characteristic value of the adjusted multiple reference image blocks. By the above method, the quality of the image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, and computer-readable storage medium. Background Art

[0002] Images have a wide range of applications, encompassing biomedicine, machine vision, intelligent driving, security, and other fields. Image quality directly impacts the effectiveness of these applications. However, images are inevitably subject to noise during acquisition, processing, and transmission. Therefore, filtering out noise from images is extremely important.

[0003] The existing mainstream image noise reduction technologies are:

[0004] 1. Spatial domain image denoising:

[0005] Simple algorithms include mean filtering, Gaussian filtering, and median filtering. These noise reduction algorithms do not distinguish between boundary areas and flat areas. Therefore, while reducing noise, they can also lose a lot of useful image information. Slightly more complex algorithms include bilateral filtering and non-local means. However, spatial domain algorithms have the disadvantage of not preserving texture well.

[0006] 2. Frequency domain image denoising:

[0007] Major denoising algorithms include wavelet and discrete cosine transform (DCT). These algorithms transform the image into the frequency domain and perform noise reduction in the frequency domain. The disadvantage of frequency-domain image denoising is that it doesn't suppress large noise as effectively as spatial-domain denoising.

[0008] 3. Image denoising by mixing spatial domain and frequency domain:

[0009] These algorithms primarily include BM3D (Block-Matching and 3D filtering) and WNNM (Weighted Nuclear Norm Minimization). These algorithms combine spatial and frequency domain noise reduction, significantly improving image noise reduction capabilities.

[0010] 4. Learning-based image denoising:

[0011] Denoising algorithms primarily rely on deep learning and dictionary learning. These algorithms use pre-collected samples to train the denoising algorithm, achieving relatively good results. However, they require significant computational effort, making them difficult to implement in real-world applications.

[0012] The aforementioned noise reduction methods are all general and do not specifically address noise caused by image sensor failures. Most image sensors have a certain number of faulty photosensitive elements. These faulty photosensitive elements can affect image quality and reduce the processing power of image processing algorithms. Summary of the Invention

[0013] To solve the above problems, the present application provides an image processing method, device and computer-readable storage medium, which can improve the quality of images.

[0014] A technical solution provided by the present application is: to provide an image processing method, which includes: determining bad pixels in an image to be processed; wherein the bad pixels are caused by a failure of a photosensitive element at a corresponding position in an image sensor; determining multiple similar reference image blocks in the image to be processed; for a target bad pixel in a target image block among the multiple reference image blocks, adjusting the characteristic value of the target bad pixel using the characteristic value of a non-bad pixel in the target image block, or the characteristic value of a non-bad pixel at the same position in the remaining reference image blocks except the target image block, to obtain multiple adjusted reference image blocks; and adjusting the characteristic value of the image to be processed using the characteristic values ​​of the adjusted multiple reference image blocks.

[0015] In one embodiment, determining a bad pixel in an image to be processed includes: determining a maximum eigenvalue and a minimum eigenvalue of a plurality of pixels in a neighborhood centered on a target pixel; determining that the target pixel is a bad pixel in response to the eigenvalue of the target pixel being greater than the maximum eigenvalue and the difference from the maximum eigenvalue being greater than a set limit threshold; or determining that the target pixel is a bad pixel in response to the eigenvalue of the target pixel being less than the minimum eigenvalue and the difference from the minimum eigenvalue being greater than the limit threshold.

[0016] In one embodiment, determining the maximum eigenvalue and the minimum eigenvalue of a plurality of pixels in a neighborhood centered on a target pixel includes: determining a 3×3 pixel array centered on the target pixel; and determining the maximum eigenvalue and the minimum eigenvalue of the remaining pixels in the 3×3 pixel array except the target pixel.

[0017] In one embodiment, determining a plurality of similar reference image blocks in an image to be processed includes: determining a first image block in the image to be processed; determining candidate image blocks having the same size as the first image block in a set search area in the image to be processed; determining a similarity between each candidate image block and the first image block; and determining a set number of candidate image blocks with the highest similarity rankings as second image blocks, wherein the first image block and the plurality of second image blocks constitute a plurality of reference image blocks.

[0018] In one embodiment, for a target bad pixel of a target image block among multiple reference image blocks, characteristic values ​​of the target bad pixel are adjusted using characteristic values ​​of non-bad pixels in the target image block, or characteristic values ​​of non-bad pixels at the same position in reference image blocks other than the target image block. The method includes: stacking multiple reference image blocks to form a first three-dimensional image block; determining multiple reference pixels corresponding to the target bad pixel of the target image block among the multiple reference image blocks in a stacking direction of the first three-dimensional image block; and, in response to the multiple reference pixels including at least one good pixel, adjusting the characteristic value of the target bad pixel to: an average characteristic value of the at least one good pixel; or, in response to the multiple reference pixels being all bad pixels, adjusting the characteristic value of the target bad pixel to: an average characteristic value of multiple good pixels in a neighborhood centered on the target bad pixel in the target image block.

[0019] In one embodiment, adjusting the characteristic value of a target bad pixel to be the average characteristic value of a plurality of non-bad pixels in a neighborhood centered on the target bad pixel in a target image block includes: determining a 3×3 pixel array centered on the target bad pixel in the target image block; and adjusting the characteristic value of the target bad pixel to be the average characteristic value of a plurality of non-bad pixels in the remaining pixels in the 3×3 pixel array excluding the target bad pixel.

[0020] In one embodiment, the image processing method further includes: expanding the first three-dimensional image block to form a first two-dimensional image block; wherein each column of pixels in the first two-dimensional image block corresponds to multiple pixels of a reference image block in the first three-dimensional image block; determining a first reference matrix corresponding to the first two-dimensional image block; performing shrinkage processing on the singular values ​​of the first reference matrix to obtain a second reference matrix; determining a second two-dimensional image block corresponding to the second reference matrix; and restoring the second two-dimensional image block to form a second three-dimensional image block; wherein each column of pixels in the second two-dimensional image block corresponds to multiple pixels of a reference image block in the first three-dimensional image block.

[0021] In one embodiment, the characteristic values ​​of the image to be processed are adjusted using the adjusted characteristic values ​​of the multiple reference image blocks, including: replacing the characteristic value of the bad pixel in the image to be processed with the average characteristic value of the characteristic values ​​of at least one pixel in the same position in the multiple adjusted reference image blocks with the same position.

[0022] Another technical solution provided by the present application is to provide an image processing device, which includes a processor and a memory, the memory is used to store program data, and the processor is used to execute the program data to implement the image processing method as described above.

[0023] Another technical solution provided by the present application is to provide a computer-readable storage medium, which stores program data. When the program data is executed by a processor, it is used to implement the image processing method as described above.

[0024] The image processing method provided by the present application includes: determining bad pixel in the image to be processed; determining multiple similar reference image blocks in the image to be processed; for the target bad pixel of the target image block in the multiple reference image blocks, adjusting the characteristic value of the target bad pixel using the characteristic value of the non-bad pixel in the target image block, or the characteristic value of the non-bad pixel at the same position in the remaining reference image blocks except the target image block, to obtain multiple adjusted reference image blocks; adjusting the characteristic value of the image to be processed using the characteristic value of the adjusted multiple reference image blocks. Through the above method, the method of determining bad pixel, determining similar blocks, and adjusting bad pixel using similar blocks can filter out the impact noise in the image to be processed. Since the impact noise is generally caused by the failure of the photosensitive element in the image sensor, the above image processing method can remove the noise caused by the damage of the photosensitive element in the image sensor, thereby improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0026] Figure 1 This is a flow chart of an embodiment of the image processing method of the present application;

[0027] Figure 2 is a schematic diagram of a 3×3 pixel array in one embodiment;

[0028] Figure 3 This is an embodiment Figure 1 Flow diagram of step 12 in FIG.

[0029] Figure 4 is a schematic diagram of multiple reference image blocks in one embodiment;

[0030] Figure 5 This is an embodiment Figure 1 Flow chart of step 13 in FIG.

[0031] Figure 6 is a schematic diagram of stacking three-dimensional image blocks in one embodiment;

[0032] Figure 7This is a flow chart of another embodiment of the image processing method of the present application;

[0033] Figure 8 This is an embodiment Figure 7 Flow diagram of step 74;

[0034] Figure 9 is a schematic diagram of unfolding a first three-dimensional image block in one embodiment;

[0035] Figure 10 is a schematic diagram of stacking of a second two-dimensional image block to form a three-dimensional image block in one embodiment;

[0036] Figure 11 is a structural diagram of an embodiment of an image processing device provided by the present application;

[0037] Figure 12 It is a structural diagram of an embodiment of a computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some, rather than all, structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] The terms "first," "second," and the like in this application are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0040] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0041] See Figure 1 , Figure 1It is a schematic flowchart of an embodiment of the image processing method of this application. The image processing method includes:

[0042] Step 11: Determine the defective pixel in the image to be processed.

[0043] Among them, the defective pixel is caused by the failure of the photosensitive element at the corresponding position in the image sensor. Understandably, most image sensors (sensors) have more or less failures of a certain number of photosensitive elements. These failed photosensitive elements will affect the imaging quality of the image sensor and at the same time reduce the processing ability of the image processing algorithm.

[0044] Understandably, in the field of image ISP (Image Signal Processing), the defective pixels of the image sensor are often divided into two categories. One category is dynamic defective pixels, and the imaging values of this type of defective pixels are unstable and have a certain range of fluctuations. The other category is called static defective pixels, and the imaging of this type of defective pixels is similar to impulse noise. Moreover, generally, the pixel values (or gray scale values, brightness values) of adjacent pixels are equal, so the pixel values of adjacent pixels can be used to determine whether the central pixel is a defective pixel.

[0045] Optionally, step 11 can be specifically: Determine the maximum eigenvalue and the minimum eigenvalue of multiple pixels in the neighborhood centered on the target pixel; in response to the eigenvalue of the target pixel being greater than the maximum eigenvalue and the difference from the maximum eigenvalue being greater than the set limit threshold, determine the target pixel as a defective pixel; or in response to the eigenvalue of the target pixel being less than the minimum eigenvalue and the difference from the minimum eigenvalue being greater than the limit threshold, determine the target pixel as a defective pixel.

[0046] Among them, the eigenvalue here can be pixel value, gray scale value or brightness value, etc.

[0047] For example, the maximum eigenvalue of multiple pixels in the neighborhood is Vmax, the minimum eigenvalue of multiple pixels in the neighborhood is Vmin, and the set limit threshold is defThr. Then, if the eigenvalue V0 of the central pixel to be judged is V0 > Vmax + defThr, or V0 < Vmin - defThr, then it can be considered that the central pixel is a defective pixel.

[0048] In a specific embodiment, the above neighborhood can be a 3×3 pixel area. Specifically, determine a 3×3 pixel array centered on the target pixel; determine the maximum eigenvalue and the minimum eigenvalue of the remaining pixels in the 3×3 pixel array except the target pixel.

[0049] As Figure 2 shown, Figure 23×3 pixel array in one embodiment. To detect whether the central pixel E is a bad pixel, the feature values ​​of the 8 pixels in the 3×3 area are sorted, excluding the pixel E. , compare the characteristic value of pixel E with V1 and V8. If it is smaller than V1, or larger than V8, and the difference exceeds a certain range, then E is considered a bad pixel. Specifically, the following formula can be used:

[0050]

[0051] Where defThr is the threshold value, which represents the maximum difference between the feature value of pixel E and V1 or V8. A mask of 0 indicates that the pixel is a bad pixel, and a mask of 1 indicates that the pixel is a normal pixel.

[0052] It can be understood that for each pixel, the above-mentioned method can be used to determine whether it is a bad pixel, so that a mask image can be obtained.

[0053] Step 12: Determine multiple similar reference image blocks in the image to be processed.

[0054] The size of the image block can be set as needed and is not limited here. For example, a 3×3 pixel array similar to that in step 11 can be selected as the image block, or a 4×4, 5×5, etc. image block can be selected. It is understood that the larger the image block size, the better the effect of adjusting bad pixels in subsequent image processing.

[0055] The multiple reference image blocks are similar blocks. The similar blocks here refer to reference image blocks having similar feature values. The similarity of the feature values ​​can be determined by calculating the difference between the two image blocks.

[0056] Alternatively, as Figure 3 As shown, Figure 3 This is an embodiment Figure 1 Schematic diagram of the process of step 12 in the embodiment, step 12 may include:

[0057] Step 121: Determine a first image block in the image to be processed.

[0058] The selection of the first image block may be arbitrary, and the size of the first image block needs to be defined as BR, that is, a BR×BR pixel array.

[0059] Step 122: Determine a candidate image block having the same size as the first image block in a set search area in the image to be processed.

[0060] Here, the search radius needs to be determined as SR.

[0061] like Figure 4As shown, Figure 4 FIG. 1 is a schematic diagram of multiple reference image blocks in one embodiment. Here, BR=3 and SR=4 are taken as an example, that is, a 3×3 pixel array is an image block and the search radius is 4.

[0062] First, determine the first image block, that is, the red image block centered on pixel C, and then determine the search range centered on the first image block as follows: Figure 4 As shown, there are a total of candidate blocks, including the red image block itself, and the size of each candidate block is 3×3.

[0063] Step 123: Determine the similarity between each candidate image block and the first image block.

[0064] The similarity between the candidate image block and the first image block is defined by the distance between the blocks, that is, the average value of the absolute value of the distance between the pixels of the blocks, as shown in the following formula:

[0065]

[0066] Among them, diff is the difference between any image block in the search range and the first image block, diff0 is Figure 4 The difference between the B0 image block and the first image block (C image block), and is the mask block at the corresponding position, the value of mask is 0 or 1, and i is the coordinate within the block.

[0067] Step 124 : Determine a set number of candidate image blocks with the highest similarity rankings as second image blocks, and the first image block and the plurality of second image blocks form a plurality of reference image blocks.

[0068] After calculating the diff of all candidate image blocks within the search range, sort them, select the first M blocks with the smallest diff, and add the original block. multiple reference image blocks.

[0069] Step 13: For target bad pixel of a target image block in a plurality of reference image blocks, characteristic values ​​of the target bad pixel are adjusted using characteristic values ​​of non-bad pixel in the target image block or characteristic values ​​of non-bad pixel at the same position in the remaining reference image blocks except the target image block, to obtain a plurality of adjusted reference image blocks.

[0070] There are multiple similar reference image blocks. Therefore, for any image block, each defective pixel can be adjusted using the feature values ​​of healthy pixels at the same position in other similar image blocks. If all pixels at the same position in other similar image blocks are defective, the feature values ​​of the neighboring pixels in the pixel block are used for adjustment.

[0071] Alternatively, as Figure 5 As shown, Figure 5 This is an embodiment Figure 1 Schematic diagram of the process of step 13 in the embodiment, the step 13 may include:

[0072] Step 131: stack multiple reference image blocks to form a first 3D image block.

[0073] like Figure 6 As shown, Figure 6 3D image blocks are stacked in a schematic diagram in one embodiment. The multiple reference images determined in the above embodiment are stacked in sequence to form a first 3D image block, i.e. A three-dimensional pixel array, Figure 6 In the figure, it is a 3×3×4 three-dimensional pixel array.

[0074] Step 132: Determine a target bad pixel of a target image block in a plurality of reference image blocks, and a plurality of reference pixels corresponding in position in the stacking direction of the first 3D image block.

[0075] Specifically, the above three-dimensional image block is represented by Z, Where N represents the layer of the two-dimensional image block (reference image block), and I and J represent the coordinates within the 2D image block.

[0076] Since it has been determined in the above step 11 whether each pixel is a bad pixel, represented by 0 and 1 respectively, 0 represents a bad pixel and 1 represents a normal pixel, then, the mask in step 11 can be used to determine whether each pixel in the first three-dimensional image block is a bad pixel.

[0077] Step 133: In response to the plurality of reference pixels including at least one non-defective pixel, adjust the characteristic value of the target defective pixel to: an average characteristic value of the at least one non-defective pixel.

[0078] Continue reading Figure 6 , if a pixel in Z is a bad pixel, the position information is , n represents the number of layers of the first three-dimensional image block, i and j represent the coordinates in the two-dimensional image block (reference image block), if in addition to the two-dimensional image block of the nth layer, The other two-dimensional image blocks of this coordinate are normal pixels, so the characteristic values ​​of these normal pixels can be averaged and replaced with this average value. The pixel value is expressed as follows:

[0079]

[0080] Here ZM represents a three-dimensional mask stack built with mask. The two-dimensional mask blocks contained in this three-dimensional mask stack and the two-dimensional image blocks in Z have the same position in the original image.

[0081] This uses Adv i，j Replace coordinates In other words, if n layers of two-dimensional image, at the corresponding coordinates If there are m bad pixels and nm normal pixels at the position, then the average of the eigenvalues ​​of these nm normal pixels is used to replace the eigenvalues ​​of the m bad pixels.

[0082] Step 134: In response to the fact that the plurality of reference pixels are all bad pixels, the characteristic value of the target bad pixel is adjusted to be the average characteristic value of a plurality of non-bad pixels in a neighborhood centered on the target bad pixel in the target image block.

[0083] Continue reading Figure 6 , if Z If the values ​​of all the two-dimensional image blocks (reference image blocks) at the position are 0 (all are bad pixels), then the calculation method in step 133 cannot be used. Here, the average value of the characteristic values ​​of the normal pixels in the neighborhood of the two-dimensional image block is used instead.

[0084] Specifically, a 3×3 pixel array centered on a target bad pixel is determined in a target image block; and a characteristic value of the target bad pixel is adjusted to be: an average characteristic value of a plurality of non-bad pixel pixels in the remaining pixels in the 3×3 pixel array except the target bad pixel.

[0085] For example, If the position is a bad pixel, then use the nth two-dimensional image block, The bad pixel is replaced by the average value of the characteristic values ​​of the normal pixels in a 3×3 neighborhood, which can be expressed as follows:

[0086]

[0087] Among them, Adv is used here n，i，j Replace coordinates The value of , the bad pixels in other layers need to be calculated independently.

[0088] Step 14: Use the adjusted eigenvalues ​​of the multiple reference image blocks to adjust the eigenvalues ​​of the image to be processed.

[0089] The characteristic value of the bad pixel in the image to be processed is replaced by the average characteristic value of the characteristic values ​​of at least one pixel at the same position in the multiple reference image blocks at the same position after adjustment.

[0090] It can be understood that the adjusted multiple reference image blocks are tiled to the original position of the reference image block in the image to be processed. It can be understood that because the process of steps 11 to 13 above is adopted for each pixel, many of the above-mentioned reference image blocks can be tiled on the same pixel position, so the average eigenvalue of the eigenvalues ​​of multiple pixels at the same position in the multiple reference image blocks corresponding to the pixel can be used as the latest eigenvalue of the pixel.

[0091] Through the above-mentioned process of step 11 to step 14, the processing of the image to be processed is completed, that is, the noise removal processing is performed on the image to be processed.

[0092] The image processing method provided by this embodiment includes: determining bad pixel in the image to be processed; determining multiple similar reference image blocks in the image to be processed; for the target bad pixel of the target image block in the multiple reference image blocks, adjusting the characteristic value of the target bad pixel using the characteristic value of the non-bad pixel in the target image block, or the characteristic value of the non-bad pixel at the same position in the remaining reference image blocks other than the target image block, to obtain multiple adjusted reference image blocks; and adjusting the characteristic value of the image to be processed using the characteristic values ​​of the adjusted multiple reference image blocks. Through the above method, the method of determining bad pixel, determining similar blocks, and adjusting bad pixel using similar blocks can filter out the impact noise in the image to be processed. Since impact noise is generally caused by the failure of the photosensitive element in the image sensor, the above image processing method can remove the noise caused by the damage of the photosensitive element in the image sensor, thereby improving the image quality.

[0093] See Figure 7 , Figure 7 1 is a flow chart of another embodiment of the image processing method of the present application, the image processing method comprising:

[0094] Step 71: Determine the bad pixels in the image to be processed.

[0095] Among them, bad pixels are caused by failure of the photosensitive element at the corresponding position in the image sensor.

[0096] Optionally, step 11 can be specifically as follows: determining the maximum eigenvalue and the minimum eigenvalue of multiple pixels in a neighborhood centered on the target pixel; in response to the eigenvalue of the target pixel being greater than the maximum eigenvalue and the difference from the maximum eigenvalue being greater than a set limit threshold, determining that the target pixel is a bad pixel; or in response to the eigenvalue of the target pixel being less than the minimum eigenvalue and the difference from the minimum eigenvalue being greater than the limit threshold, determining that the target pixel is a bad pixel.

[0097] The characteristic value here can be a pixel value, a grayscale value, a brightness value, etc.

[0098] In a specific embodiment, the neighborhood may be a 3×3 pixel area. Specifically, a 3×3 pixel array centered on the target pixel is determined; and the maximum eigenvalue and the minimum eigenvalue of the remaining pixels in the 3×3 pixel array except the target pixel are determined.

[0099] like Figure 2 As shown, Figure 2 3×3 pixel array in one embodiment. To detect whether the central pixel E is a bad pixel, the feature values ​​of the 8 pixels in the 3×3 area are sorted, excluding the pixel E. , compare the characteristic value of pixel E with V1 and V8. If it is smaller than V1, or larger than V8, and the difference exceeds a certain range, then E is considered a bad pixel. Specifically, the following formula can be used:

[0100]

[0101] Where defThr is the threshold value, which represents the maximum difference between the feature value of pixel E and V1 or V8. A mask of 0 indicates that the pixel is a bad pixel, and a mask of 1 indicates that the pixel is a normal pixel.

[0102] It can be understood that for each pixel, the above-mentioned method can be used to determine whether it is a bad pixel, so that a mask image can be obtained.

[0103] Step 72: Determine a plurality of similar reference image blocks in the image to be processed.

[0104] Optionally, in one embodiment, step 72 may include: determining a first image block in the image to be processed; determining candidate image blocks with the same size as the first image block in a set search area in the image to be processed; determining the similarity between each candidate image block and the first image block; determining a set number of candidate image blocks with the highest similarity rankings as second image blocks, and the first image block and the multiple second image blocks forming multiple reference image blocks.

[0105] The size of the first image block is defined as BR, that is, a BR×BR pixel array, and the search radius is SR.

[0106] like Figure 4 As shown, Figure 4 FIG. 1 is a schematic diagram of multiple reference image blocks in one embodiment. Here, BR=3 and SR=4 are taken as an example, that is, a 3×3 pixel array is an image block and the search radius is 4.

[0107] First, determine the first image block, that is, the red image block centered on pixel C, and then determine the search range centered on the first image block as follows: Figure 4 As shown, there are a total of candidate blocks, including the red image block itself, and the size of each candidate block is 3×3.

[0108] The similarity between the candidate image block and the first image block is defined by the distance between the blocks, that is, the average value of the absolute value of the distance between the pixels of the blocks, as shown in the following formula:

[0109]

[0110] Among them, diff is the difference between any image block in the search range and the first image block, diff0 is Figure 4 The difference between the B0 image block and the first image block (C image block), and is the mask block at the corresponding position, the value of mask is 0 or 1, and i is the coordinate within the block.

[0111] After calculating the diff of all candidate image blocks within the search range, sort them, select the first M blocks with the smallest diff, and add the original block. multiple reference image blocks.

[0112] Step 73: For target bad pixel of a target image block in a plurality of reference image blocks, characteristic values ​​of the target bad pixel are adjusted using characteristic values ​​of non-bad pixel in the target image block or characteristic values ​​of non-bad pixel at the same position in the remaining reference image blocks except the target image block, to obtain a plurality of adjusted reference image blocks.

[0113] Optionally, in one embodiment, step 73 may include: stacking multiple reference image blocks to form a first three-dimensional image block; determining a target bad pixel of a target image block from the multiple reference image blocks, and multiple reference pixels corresponding to each other in a stacking direction of the first three-dimensional image block; in response to the multiple reference pixels including at least one non-bad pixel, adjusting the characteristic value of the target bad pixel to: an average characteristic value of the at least one non-bad pixel; or in response to the multiple reference pixels being all bad pixels, adjusting the characteristic value of the target bad pixel to: an average characteristic value of multiple non-bad pixels in a neighborhood centered on the target bad pixel in the target image block.

[0114] like Figure 6 As shown, Figure 6 : is a schematic diagram of stacking multiple reference image blocks to form a three-dimensional image block in one embodiment. In which, the multiple reference images determined in the above embodiment are stacked in sequence to form a first three-dimensional image block, that is, A three-dimensional pixel array, Figure 6 In the figure, it is a 3×3×4 three-dimensional pixel array.

[0115] Specifically, the above three-dimensional image block is represented by Z, Where N represents the layer of the two-dimensional image block (reference image block), and I and J represent the coordinates within the 2D image block.

[0116] Since it has been determined in the above step 11 whether each pixel is a bad pixel, represented by 0 and 1 respectively, 0 represents a bad pixel and 1 represents a normal pixel, then, the mask in step 11 can be used to determine whether each pixel in the first three-dimensional image block is a bad pixel.

[0117] Continue reading Figure 6 , if a pixel in Z is a bad pixel, the position information is , n represents the number of layers of the first three-dimensional image block, i and j represent the coordinates in the two-dimensional image block (reference image block), if in addition to the two-dimensional image block of the nth layer, The other two-dimensional image blocks of this coordinate are normal pixels, so the characteristic values ​​of these normal pixels can be averaged and replaced with this average value. The pixel value is expressed as follows:

[0118]

[0119] Here ZM represents a three-dimensional mask stack built with mask. The two-dimensional mask blocks contained in this three-dimensional mask stack and the two-dimensional image blocks in Z have the same position in the original image.

[0120] This uses Adv i，j Replace coordinates In other words, if n layers of two-dimensional image, at the corresponding coordinates If there are m bad pixels and nm normal pixels at the position, then the average of the eigenvalues ​​of these nm normal pixels is used to replace the eigenvalues ​​of the m bad pixels.

[0121] Continue reading Figure 6 , if Z If the values ​​of all the two-dimensional image blocks (reference image blocks) at the position are 0 (all are bad pixels), then the calculation method in step 133 cannot be used. Here, the average value of the characteristic values ​​of the normal pixels in the neighborhood of the two-dimensional image block is used instead.

[0122] Specifically, a 3×3 pixel array centered on a target bad pixel is determined in a target image block; and a characteristic value of the target bad pixel is adjusted to be: an average characteristic value of a plurality of non-bad pixel pixels in the remaining pixels in the 3×3 pixel array except the target bad pixel.

[0123] For example, If the position is a bad pixel, then use the nth two-dimensional image block, The bad pixel is replaced by the average value of the characteristic values ​​of the normal pixels in a 3×3 neighborhood, which can be expressed as follows:

[0124]

[0125] Among them, Adv is used here n，i，j Replace coordinates The value of , the bad pixels in other layers need to be calculated independently.

[0126] Step 74: Perform singular value decomposition and noise reduction processing on the first three-dimensional image block formed by the adjusted multiple reference image blocks to obtain a second three-dimensional image block.

[0127] Alternatively, as Figure 8 As shown, Figure 8 This is an embodiment Figure 7 74, step 74 may include:

[0128] Step 741: Expand the first 3D image block to form a first 2D image block.

[0129] Each column of pixels in the first two-dimensional image block corresponds to a plurality of pixels in a reference image block in the first three-dimensional image block.

[0130] See also Figure 6 and Figure 9 , Figure 9is a schematic diagram of the expansion of the first three-dimensional image block in one embodiment. Each reference image block (two-dimensional image block) in the first two-dimensional image block is expanded into a one-dimensional column vector, and these column vectors are arranged to form a new two-dimensional image block, namely the first two-dimensional image block. Taking BR=3 and N=3 as an example, a The 3D image stack is converted to The two-dimensional image block is represented by H here.

[0131] Step 742: Determine a first reference matrix corresponding to the first two-dimensional image block.

[0132] The first reference matrix here adopts the singular value decomposition method, and the singular value decomposition H can be expressed as:

[0133]

[0134] The steps for calculating singular values ​​are as follows:

[0135] a: Calculation The eigenvalues ​​and eigenvectors of are expressed as follows:

[0136]

[0137] Where x is the eigenvector, is the eigenvalue, H is the first reference matrix, H T is the transposed matrix of the first reference matrix. Arrange from large to small, Diagonal elements are placed from the upper left corner to the lower right corner. The eigenvector x is The order is put into the V matrix from left to right.

[0138] b: Calculation The eigenvector of .

[0139] H is After the singular value decomposition of the matrix, V is Matrix, U is matrix, for matrix.

[0140] Step 743: Perform shrinkage processing on the singular values ​​of the first reference matrix to obtain a second reference matrix.

[0141] It can be understood that this embodiment uses singular value contraction to perform matrix low-rank approximation, thereby achieving the purpose of noise reduction. Specifically, in The diagonal elements are the singular values. Just shrink The diagonal elements can achieve the purpose of low-rank approximation of the matrix.

[0142] in, is a The matrix can be expressed as follows:

[0143]

[0144] in, for matrix, . for Matrix, all zeros.

[0145] Low-rank approximation requires the following processing:

[0146]

[0147] It can be understood that the reference threshold deThr is set here, when When it is less than the threshold deThr, it shrinks directly to 0.

[0148] After processing all singular values, replace the original singular values ​​to get a new matrix , and replace it with the original Using the U and V matrices obtained by the previous singular value decomposition, the new matrix can be recalculated The calculation formula is as follows.

[0149]

[0150] Step 744: Determine a second two-dimensional image block corresponding to the second reference matrix.

[0151] Step 744 is an inverse process to step 742 and will not be described in detail here.

[0152] Step 745: Restore the second two-dimensional image block to form a second three-dimensional image block.

[0153] Each column of pixels in the second two-dimensional image block corresponds to a plurality of pixels in a reference image block in the first three-dimensional image block.

[0154] like Figure 10 As shown, Figure 10 FIG. 7 is a schematic diagram of stacking the second two-dimensional image block to form a three-dimensional image block in an embodiment. It can be understood that step 745 is an inverse process to step 741 and will not be described in detail here.

[0155] It is worth noting that in the above process, the three-dimensional image block is converted into a two-dimensional image block. The arrangement order of the multiple pixels in each reference image block in the column direction is not limited. For example, Figure 2The order can be ABCDEFGHI from top to bottom, or any other set order. However, the order in which each reference image block is converted to a one-dimensional vector remains the same. Similarly, when restoring a two-dimensional image block to a second three-dimensional image block, the sorting order is determined by expanding the first two-dimensional image block into the second two-dimensional image block, and must be consistent.

[0156] Step 75: Use the adjusted feature values ​​of the plurality of reference image blocks in the second three-dimensional image block to adjust the feature values ​​of the image to be processed.

[0157] The characteristic value of the bad pixel in the image to be processed is replaced by the average characteristic value of the characteristic values ​​of at least one pixel at the same position in the multiple reference image blocks at the same position after adjustment.

[0158] It can be understood that the 3D image stack before denoising is represented as , the 3D image stack after denoising is represented as According to this embodiment, the 3D image heap size is . There are a total of 4 two-dimensional image blocks, all of which are denoised image blocks. Repeat the above steps 71-74 to obtain the three-dimensional image blocks after all pixels are denoised. And the two-dimensional image blocks in these denoised three-dimensional image blocks are flattened according to their positions in the original image. There will inevitably be many positions where multiple two-dimensional image blocks overlap. Averaging these overlaps can get a denoised image. For example, a certain pixel point has 3 overlapping blocks, and the corresponding pixel eigenvalues ​​are V1, V2 and V3 respectively. Then average them, as expressed by the following formula:

[0159]

[0160] V is the final eigenvalue of the point after noise reduction.

[0161] The image processing method provided by this embodiment includes: determining bad pixel pixels in a to-be-processed image; determining multiple similar reference image blocks in the to-be-processed image; adjusting the characteristic values ​​of target bad pixel pixels in a target image block among the multiple reference image blocks using the characteristic values ​​of non-bad pixel pixels in the target image block or the characteristic values ​​of non-bad pixel pixels at the same position in reference image blocks other than the target image block, thereby obtaining multiple adjusted reference image blocks; performing singular value decomposition and noise reduction on a first three-dimensional image block formed by the adjusted multiple reference image blocks to obtain a second three-dimensional image block; and adjusting the characteristic values ​​of the to-be-processed image using the characteristic values ​​of the multiple reference image blocks in the adjusted second three-dimensional image block. By adjusting the pixel values ​​of bad pixel pixels and combining singular value decomposition and noise reduction, the above-described method can, on the one hand, filter out impulse noise in the to-be-processed image, as impulse noise is generally caused by a faulty photosensitive element in an image sensor; and, on the other hand, can further correct previously corrected bad pixels in an image stack. This correction is performed on the entire image stack, resulting in higher accuracy and improved image quality.

[0162] See Figure 11 , Figure 11 1 is a schematic structural diagram of an embodiment of an image processing device provided by the present application. The image processing device 110 includes a processor 81 and a memory 82. The memory 82 is used to store program data, and the processor 81 is used to execute the program data to achieve:

[0163] Determine bad pixel in the image to be processed; determine multiple similar reference image blocks in the image to be processed; for a target bad pixel of a target image block in the multiple reference image blocks, adjust the characteristic value of the target bad pixel by using the characteristic value of a non-bad pixel in the target image block or the characteristic value of a non-bad pixel at the same position in the remaining reference image blocks except the target image block to obtain multiple adjusted reference image blocks; and adjust the characteristic value of the image to be processed by using the characteristic values ​​of the multiple adjusted reference image blocks.

[0164] See Figure 12 , Figure 12 1 is a schematic diagram of the structure of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 120 stores program data 121. When the program data 121 is executed by a processor, it is used to implement:

[0165] Determine bad pixel in the image to be processed; determine multiple similar reference image blocks in the image to be processed; for a target bad pixel of a target image block in the multiple reference image blocks, adjust the characteristic value of the target bad pixel by using the characteristic value of a non-bad pixel in the target image block or the characteristic value of a non-bad pixel at the same position in the remaining reference image blocks except the target image block to obtain multiple adjusted reference image blocks; and adjust the characteristic value of the image to be processed by using the characteristic values ​​of the multiple adjusted reference image blocks.

[0166] In the embodiments of the above-mentioned image processing device 110 and computer-readable storage medium 120, when the program data is executed by the processor, it is also used to implement: determining the maximum eigenvalue and the minimum eigenvalue of multiple pixels in a neighborhood centered on the target pixel; in response to the eigenvalue of the target pixel being greater than the maximum eigenvalue and the difference from the maximum eigenvalue being greater than a set limit threshold, determining that the target pixel is a bad pixel; or in response to the eigenvalue of the target pixel being less than the minimum eigenvalue and the difference from the minimum eigenvalue being greater than the limit threshold, determining that the target pixel is a bad pixel.

[0167] In the embodiments of the above-mentioned image processing device 110 and computer-readable storage medium 120, when the program data is executed by the processor, it is also used to implement: determining a first image block in the image to be processed; determining candidate image blocks with the same size as the first image block in a set search area in the image to be processed; determining the similarity between each candidate image block and the first image block; determining a set number of candidate image blocks with the highest similarity rankings as second image blocks, and the first image block and the multiple second image blocks forming multiple reference image blocks.

[0168] In the embodiments of the above-mentioned image processing device 110 and computer-readable storage medium 120, the program data, when executed by the processor, is further used to implement: stacking multiple reference image blocks to form a first three-dimensional image block; determining a target bad pixel of a target image block among the multiple reference image blocks, and multiple reference pixels corresponding to each other in the stacking direction of the first three-dimensional image block; in response to the multiple reference pixels including at least one non-bad pixel, adjusting the characteristic value of the target bad pixel to: the average characteristic value of the at least one non-bad pixel; or in response to the multiple reference pixels being all bad pixels, adjusting the characteristic value of the target bad pixel to: the average characteristic value of multiple non-bad pixels in a neighborhood centered on the target bad pixel in the target image block.

[0169] In the embodiments of the above-mentioned image processing device 110 and computer-readable storage medium 120, when the program data is executed by the processor, it is further used to implement: expanding the first three-dimensional image block to form a first two-dimensional image block; wherein each column of pixels of the first two-dimensional image block corresponds to multiple pixels of a reference image block in the first three-dimensional image block; determining a first reference matrix corresponding to the first two-dimensional image block; performing shrinkage processing on the singular values ​​of the first reference matrix to obtain a second reference matrix; determining a second two-dimensional image block corresponding to the second reference matrix; and restoring the second two-dimensional image block to form a second three-dimensional image block; wherein each column of pixels of the second two-dimensional image block corresponds to multiple pixels of a reference image block in the first three-dimensional image block.

[0170] In the embodiments of the above-mentioned image processing device 110 and computer-readable storage medium 120, when the program data is executed by the processor, it is also used to implement: replacing the characteristic value of the bad pixel in the image to be processed with the average characteristic value of the characteristic values ​​of at least one pixel at the same position in multiple reference image blocks with the same adjusted position.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0172] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0173] In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0174] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Determine the maximum eigenvalue and the minimum eigenvalue of multiple pixels in a neighborhood centered on the target pixel; In response to the characteristic value of the target pixel being greater than the maximum characteristic value and the difference between the characteristic value and the maximum characteristic value being greater than a set limit threshold, the target pixel is determined to be a bad pixel; or, in response to the characteristic value of the target pixel being less than the minimum characteristic value and the difference between the characteristic value and the minimum characteristic value being greater than the limit threshold, the target pixel is determined to be a bad pixel; wherein the bad pixel is caused by a failure of a photosensitive element at a corresponding position in the image sensor; Determine a plurality of similar reference image blocks in the image to be processed; Stacking a plurality of the reference image blocks to form a first three-dimensional image block; Determine a target bad pixel of a target image block in a plurality of the reference image blocks, and a plurality of reference pixels corresponding to positions in a stacking direction of the first three-dimensional image block; In response to the plurality of reference pixels including at least one non-bad pixel, adjusting the characteristic value of the target bad pixel by using the characteristic value of the at least one non-bad pixel to obtain a plurality of adjusted reference image blocks; The characteristic values ​​of the image to be processed are adjusted using the adjusted characteristic values ​​of the plurality of reference image blocks.

2. The image processing method according to claim 1, characterized in that: The step of determining the maximum eigenvalue and the minimum eigenvalue of a plurality of pixels in a neighborhood centered on the target pixel comprises: Determine a 3×3 pixel array centered on the target pixel; The maximum eigenvalue and the minimum eigenvalue of the remaining pixels in the 3×3 pixel array except the target pixel are determined.

3. The image processing method according to claim 1, characterized in that: The determining of a plurality of similar reference image blocks in the image to be processed comprises: Determining a first image block in the image to be processed; Determine, in a set search area in the image to be processed, a candidate image block having the same size as the first image block; Determining a similarity between each of the candidate image blocks and the first image block; A set number of candidate image blocks ranked top in similarity are determined as second image blocks, and the first image block and a plurality of the second image blocks form a plurality of reference image blocks.

4. The image processing method according to claim 1, characterized in that: The adjusting the characteristic value of the target bad pixel by using the characteristic value of the at least one non-bad pixel includes: adjusting the characteristic value of the target bad pixel to: the average characteristic value of the at least one non-bad pixel.

5. The method according to claim 1, characterized in that The method further comprises: In response to the fact that the plurality of reference pixels are all bad pixel pixels, the characteristic value of the target bad pixel is adjusted to be: an average characteristic value of a plurality of non-bad pixel pixels in a neighborhood centered on the target bad pixel in the target image block.

6. The image processing method according to claim 5, characterized in that: The step of adjusting the characteristic value of the target bad pixel to the average characteristic value of a plurality of non-bad pixel pixels in a neighborhood centered on the target bad pixel in the target image block includes: Determine a 3×3 pixel array centered on the target bad pixel in the target image block; The characteristic value of the target bad pixel is adjusted to: an average characteristic value of a plurality of non-bad pixel pixels in the remaining pixels except the target bad pixel in the 3×3 pixel array.

7. The image processing method according to claim 5, characterized in that: The image processing method further includes: Expanding the first three-dimensional image block to form a first two-dimensional image block; wherein each column of pixels of the first two-dimensional image block corresponds to a plurality of pixels of one of the reference image blocks in the first three-dimensional image block; Determining a first reference matrix corresponding to the first two-dimensional image block; Performing shrinkage processing on the singular values ​​of the first reference matrix to obtain a second reference matrix; Determine a second two-dimensional image block corresponding to the second reference matrix; The second two-dimensional image block is restored to form a second three-dimensional image block; wherein each column of pixels in the second two-dimensional image block corresponds to a plurality of pixels of one of the reference image blocks in the first three-dimensional image block.

8. The image processing method according to claim 1, characterized in that: The step of adjusting the characteristic value of the image to be processed by using the adjusted characteristic values ​​of the plurality of reference image blocks comprises: The characteristic value of the bad pixel in the to-be-processed image is replaced by the average characteristic value of the characteristic values ​​of at least one pixel at the same position in the multiple reference image blocks at the same position after adjustment.

9. An image processing device, characterized in that: The image processing device comprises a processor and a memory, the memory is used to store program data, and the processor is used to execute the program data to implement the image processing method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and when the program data is executed by a processor, it is used to implement the image processing method according to any one of claims 1 to 8.

Images