Image processing method, image processing device, terminal and readable storage medium

Abstract

The application discloses an image processing method, an image processing device, a terminal and a nonvolatile computer readable storage medium. The image processing method comprises the following steps: performing defect detection on pixel units in a first image to obtain defect units of the first image, each pixel unit comprising a plurality of adjacent same-color pixels; obtaining a first pixel sum and a second pixel sum, the first pixel sum being a sum of pixel values of each pixel in the defect units of the first image, and the second pixel sum being a sum of pixel values of each pixel in each same-color unit in a preset neighborhood of the defect units of the first image which is of the same color as the defect units of the first image; and correcting pixel values of each pixel in the defect units of the first image according to the first pixel sum and the second pixel sum. The image processing method, the image processing device, the terminal and the nonvolatile computer readable storage medium can obtain light and dark interlaced texture defects in the first image and eliminate the defects.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, in particular to an image processing method, an image processing device, a terminal and a nonvolatile computer readable storage medium. BACKGROUND

[0002] In order to take into account high sensitivity and high resolution, the industry has developed a multi-in-one pixel arrangement form, that is, four adjacent pixels are the same color filter, and share a micro lens. Since four pixels share a micro lens, there is a difference in the light received by the four pixels, resulting in a phenomenon of alternating light and dark on the edges of the image of the object being photographed. SUMMARY

[0003] The present application provides an image processing method, an image processing device, a terminal and a nonvolatile computer readable storage medium.

[0004] The image processing method of the present application includes: defect detection on a pixel unit in a first image to obtain a defect unit of the first image, each pixel unit containing a plurality of adjacent same color pixels; obtaining a first pixel sum and a second pixel sum, the first pixel sum being the sum of pixel values of each pixel in the defect unit of the first image, and the second pixel sum being the sum of pixel values of each pixel in a same color unit in a preset neighborhood of the defect unit of the first image; and correcting the pixel values of each pixel in the defect unit of the first image according to the first pixel sum and the second pixel sum.

[0005] Another image processing method of the present application includes: defect detection on a pixel unit in a first image to obtain a defect unit of the first image, each pixel unit containing a plurality of adjacent same color pixels; and inputting the first image into a trained image processing network, and correcting the pixel values of each pixel in the defect unit of the first image using the image processing network.

[0006] The image processing apparatus of the embodiment of the present application comprises a first detection module, a first acquisition module and a first correction module. The first detection module is configured to detect defects of pixel units in a first image to obtain defect units of the first image, each of the pixel units comprising a plurality of adjacent same-color pixels. The first acquisition module is configured to acquire a first pixel sum and a second pixel sum, the first pixel sum being a sum of pixel values of each pixel in the defect units of the first image, and the second pixel sum being a sum of pixel values of each pixel in each same-color unit in a preset neighborhood of the defect units of the first image, the same-color unit being same in color as the defect units of the first image. The first correction module is configured to correct the pixel values of each pixel in the defect units of the first image according to the first pixel sum and the second pixel sum.

[0007] Another image processing apparatus of the embodiment of the present application comprises a second detection module and a second correction module. The second detection module is configured to detect defects of pixel units in a first image to obtain defect units of the first image, each of the pixel units comprising a plurality of adjacent same-color pixels. The second correction module is configured to input the first image into a trained image processing network, and correct the pixel values of each pixel in the defect units of the first image by using the image processing network.

[0008] A terminal of the embodiment of the present application comprises one or more processors, a memory and one or more programs. The one or more programs are stored in the memory and executed by the one or more processors, and the program comprises instructions for executing an image processing method. The image processing method comprises: detecting defects of pixel units in a first image to obtain defect units of the first image, each of the pixel units comprising a plurality of adjacent same-color pixels; acquiring a first pixel sum and a second pixel sum, the first pixel sum being a sum of pixel values of each pixel in the defect units of the first image, and the second pixel sum being a sum of pixel values of each pixel in each same-color unit in a preset neighborhood of the defect units of the first image, the same-color unit being same in color as the defect units of the first image; and correcting the pixel values of each pixel in the defect units of the first image according to the first pixel sum and the second pixel sum.

[0009] A terminal of the embodiment of the present application comprises one or more processors, a memory and one or more programs. The one or more programs are stored in the memory and executed by the one or more processors, and the program comprises instructions for executing an image processing method. The image processing method comprises: detecting defects of pixel units in a first image to obtain defect units of the first image, each of the pixel units comprising a plurality of adjacent same-color pixels; and inputting the first image into a trained image processing network, and correcting the pixel values of each pixel in the defect units of the first image by using the image processing network.

[0010] The non-transitory computer-readable storage medium containing the computer program of the embodiment of the present application, when executed by one or more processors, causes the processors to implement an image processing method. The image processing method comprises: performing defect detection on pixel units in a first image to obtain defect units of the first image, each of the pixel units comprising a plurality of adjacent same-color pixels; obtaining a first pixel sum and a second pixel sum, the first pixel sum being a sum of pixel values of each pixel in the defect units of the first image, and the second pixel sum being a sum of pixel values of each pixel in a preset neighborhood of each same-color unit of the defect units of the first image which is same in color as the defect units of the first image; and correcting pixel values of each pixel in the defect units of the first image according to the first pixel sum and the second pixel sum.

[0011] The non-transitory computer-readable storage medium containing the computer program of the embodiment of the present application, when executed by one or more processors, causes the processors to implement an image processing method. The image processing method comprises: performing defect detection on pixel units in a first image to obtain defect units of the first image, each of the pixel units comprising a plurality of adjacent same-color pixels; and inputting the first image into a trained image processing network, and correcting pixel values of each pixel in the defect units of the first image by using the image processing network.

[0012] The image processing method, the image processing device, the terminal and the non-transitory computer-readable storage medium of the embodiment of the present application can obtain and eliminate the alternating light and dark texture defects in the first image.

[0013] Additional aspects and advantages of the embodiments of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0014] The above and / or additional aspects and advantages of the present application can become apparent and be easily understood from the description of the embodiments, given below, in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a schematic diagram of a defocused scene;

[0016] Figure 2 is a flowchart of an image processing method of some embodiments of the present application;

[0017] Figure 3 is a structural diagram of an image processing device of some embodiments of the present application;

[0018] Figure 4is a structural schematic diagram of a terminal of some embodiments of the present application;

[0019] Figure 5 is a schematic diagram of a pixel array of some embodiments of the present application;

[0020] Figure 6 is a flowchart of an image processing method of some embodiments of the present application;

[0021] Figure 7 is a schematic diagram of a scenario of generating pixel encoding values of some embodiments of the present application;

[0022] Figure 8 is a schematic diagram of a scenario of generating pixel encoding values of some embodiments of the present application;

[0023] Figure 9 is a flowchart of an image processing method of some embodiments of the present application;

[0024] Figure 10 is a flowchart of an image processing method of some embodiments of the present application;

[0025] Figure 11 is a flowchart of an image processing method of some embodiments of the present application;

[0026] Figure 12 is a flowchart of an image processing method of some embodiments of the present application;

[0027] Figure 13 is a structural schematic diagram of yet another image processing apparatus of some embodiments of the present application;

[0028] Figure 14 is a structural schematic diagram of yet another image processing apparatus of some embodiments of the present application;

[0029] Figure 15 is a flowchart of an image processing method of some embodiments of the present application;

[0030] Figure 16 is a flowchart of an image processing method of some embodiments of the present application;

[0031] Figure 17 is a flowchart of an image processing method of some embodiments of the present application;

[0032] Figure 18 is a schematic diagram of a scenario of training an image processing network of some embodiments of the present application;

[0033] Figure 19 is a flowchart of an image processing method of some embodiments of the present application;

[0034] Figure 20 is a schematic diagram of a scene for obtaining a final input image according to some embodiments of the present application;

[0035] Figure 21 is a schematic diagram of a connection relationship between a computer readable storage medium and a processor according to some embodiments of the present application. DETAILED DESCRIPTION

[0036] Embodiments of the present application are described in detail below with reference to the attached drawing figures, wherein the same or like reference numerals are used throughout the drawing figures to refer to the same or like elements or elements having the same or similar functionality. The embodiments described below are exemplary and are merely intended to explain the present application, and are not to be understood as limiting the present application.

[0037] In order to balance high sensitivity and high resolution, the industry has developed a multi-in-one pixel arrangement form. A typical multi-in-one pixel arrangement form is Quad Bayer Coding. Four adjacent pixels are the same color filter, and share a microlens. When shooting in a low-illumination environment such as night scenes, the signals of the four adjacent same-color pixels are combined to output one signal, which can reduce noise. When shooting in bright scenes such as daytime outdoors, the signals of the four adjacent same-color pixels are output respectively, achieving high resolution.

[0038] Referring to Figure 1 In the high-resolution mode of the multi-in-one pixel, Remosaic algorithm and Demosaic algorithm are usually used to analyze the single-channel raw image (i.e. Raw image) collected by the camera to obtain an RGB image. However, in the collected single-channel raw image, adjacent pixels often appear to be alternating bright and dark, resulting in the RGB image obtained by Remosaic algorithm and Demosaic algorithm having special noise in the form of grid lines. One of the reasons is that there is a defocus area when shooting. Since the four pixels share a microlens, the light received by the four pixels may be misaligned, resulting in alternating bright and dark textures at the edge of the image of the object being photographed, affecting the clarity of the image. As shown in Figure 1 L1 layer is the real object, L2 layer is a schematic diagram of misalignment of light received by the left pixel due to defocus, L3 layer is a schematic diagram of the right pixel normally receiving light, and L4 layer is the original image obtained by combining the left and right pixels. It can be seen that due to defocus, the light originally at the dark position penetrates into the bright position, resulting in alternating bright and dark textures in the original image.

[0039] With the evolution of image processing technology, especially the application of deep learning, the ability of Remosaic algorithm to restore details is getting stronger and stronger, and the stronger the Remosaic algorithm is, the more likely it is to analyze the aforementioned chiaroscuro phenomenon as real details, resulting in the appearance of special grid-shaped noise in the RGB image obtained through the Remosaic algorithm, which is difficult to be removed by the subsequent noise reduction module.

[0040] The present application provides an image processing method for correcting the defects of chiaroscuro of multi-in-one pixels, avoiding the generation of grid-shaped noise in the RGB image, and improving the image quality.

[0041] Please refer to Figure 2 The image processing method of the embodiment of the present application comprises the following steps:

[0042] 01: Defect detection is performed on the pixel units in the first image to obtain the defect units of the first image, and each pixel unit contains a plurality of adjacent same-color pixels;

[0043] 02: Obtain the first pixel sum and the second pixel sum, the first pixel sum is the sum of the pixel values of each pixel in the defect unit of the first image, and the second pixel sum is the sum of the pixel values of each pixel in each same-color unit in the preset neighborhood of the defect unit of the first image which is the same color as the defect unit of the first image; and

[0044] 03: Correct the pixel values of each pixel in the defect unit of the first image according to the first pixel sum and the second pixel sum.

[0045] Please refer to Figure 3 The embodiment of the present application also provides an image processing device 10, and the image processing method of the embodiment of the present application can be applied to the image processing device 10. The image processing device 10 comprises a first detection module 11, a first acquisition module 12 and a first correction module 13. The first detection module 11 is used to perform the method in step 01, the first acquisition module 12 is used to perform the method in step 02, and the first correction module 13 is used to perform the method in step 03. That is, the first detection module 11 is used to perform defect detection on the pixel units in the first image to obtain the defect units of the first image; the first acquisition module 12 is used to obtain the first pixel sum and the second pixel sum; and the first correction module 13 is used to correct the pixel values of each pixel in the defect unit of the first image according to the first pixel sum and the second pixel sum.

[0046] Please refer to Figure 4The embodiment of the present application further provides a terminal 100. The terminal 100 comprises one or more processors 30, a memory 20 and one or more programs. The one or more programs are stored in the memory 20 and executed by the one or more processors 30. The programs comprise instructions for performing the image processing method in steps 01, 02 and 03. That is, the processor 30 is configured to perform the image processing method in steps 01, 02 and 03.

[0047] In some embodiments, the terminal 100 can be a mobile phone, a desktop computer, a notebook computer, a camera, a camcorder, a smart watch or other electronic device with an image acquisition function, which is not limited herein. For example, the terminal 100 further comprises an image sensor 40, and the first image in step 01 is a single-channel image acquired by the image sensor 40.

[0048] Please refer to Figure 5 , the pixel unit is a multi-in-one pixel, each pixel unit comprises a plurality of adjacent same-color pixels, and the adjacent same-color pixels share a microlens. Figure 5 In an illustrative embodiment, the pixel array of the image sensor 40 is a Quad Bayer array, and there are 4*4 pixel units, each pixel unit comprises 4 adjacent same-color pixels (four-in-one), and the 4 adjacent same-color pixels share a microlens. For example, G0 is a pixel unit, and the pixel unit G0 comprises 4 adjacent green pixels, which are g1, g2, g3 and g4. As shown in Figure 1 , since the 4 adjacent same-color pixels share a microlens, the light received by the 4 adjacent same-color pixels can be misaligned, thereby causing a phenomenon of light and shade interlacing at the image edge of the object. Such pixel unit with the phenomenon of light and shade interlacing due to misalignment of light is a defective unit.

[0049] The first pixel sum is the sum of the pixel values of each pixel in the defective unit of the first image. For example, in Figure 5 an illustrative embodiment, the pixel unit G0 is a defective unit, and the defective unit G0 comprises four green pixels g1, g2, g3 and g4. If the pixel values of the four green pixels g1, g2, g3 and g4 are 1, 7, 8 and 9 respectively, then the first pixel sum S0 corresponding to the defective unit G0 is 1+7+8+9=25. In other embodiments, the defective unit can also be a blue large pixel composed of a plurality of blue pixels, or a red large pixel composed of a plurality of red pixels, or a large pixel of other colors, which is not limited herein.

[0050] The second pixel sum is a sum of pixel values of each pixel in each same-color unit in a preset neighborhood of the defective unit of the first image, which is the same color as the defective unit of the first image. The preset neighborhood can be a neighborhood of preset N*N pixel units, where N>1. For example, a neighborhood of 2*2, 3*3, or 4*4 pixel units, which are not listed one by one. The preset neighborhood can also be a neighborhood of preset N*M pixel units, where N>1, M>1, and N≠M. For example, a neighborhood of 2*3, 3*2, 4*3, or 3*4 pixel units, which are not listed one by one. For example, in Figure 5 In an illustrative embodiment, the preset neighborhood is a neighborhood of 3*3 pixel units, and the preset neighborhood of the defective unit G0 includes nine pixel units, namely G1, R1, G2, B1, G0, B2, G3, R2, and G4. Among them, the pixel units G1, G2, G3, and G4 are the same color as the defective unit G0. The second pixel sum S1 corresponding to the pixel unit G1 is a sum of pixel values of each pixel in the pixel unit G1, and the calculation method is the same as that of the first pixel sum, which is not described here. Similarly, the second pixel sums S2, S3, and S4 corresponding to the pixel units G2, G3, and G4 can be calculated respectively.

[0051] The image processing method of the present application corrects the pixel values of each pixel in the defective unit of the first image according to the first pixel sum and the second pixel sum, so as to eliminate the phenomenon of light and shade interlacing presented by the defective unit by correcting the pixel values of each pixel in the defective unit.

[0052] Please refer to Figure 5For example, assuming that a real object changes from left to right, the brightness should change from bright to dark. If pixel unit G0 (containing four green pixels) is a defective unit, the brightness of pixel units G1, B1, G3 in the neighborhood of the defective unit G0 is actually higher, the brightness of pixel units R1, G0, R2 should be moderate, and the brightness of pixel units G2, B2, G4 is actually lower, and the overall should show a trend of left bright and right dark. However, the overall brightness trend of the four green pixels g1, g2, g3, g4 in the defective unit G0 is not distributed from left bright to right dark, for example, the brightness of green pixel g1 and green pixel g3 is lower, and the brightness of green pixel g2 and green pixel g4 is higher, which is left dark and right bright, which is inconsistent with the overall brightness trend of the neighborhood of the defective unit G0, so that when the defective unit G0 is observed in the neighborhood, the abnormal bright and dark interlaced texture is easily observed. After obtaining the first pixel sum S0 corresponding to the defective unit G0 and obtaining the second pixel sums S1, S2, S3, S4 corresponding to the same color units G1, G2, G3, G4 respectively, the pixel values of the pixels g1, g2, g3, g4 can be corrected in combination with the first pixel sum S0 and the second pixel sums S1, S2, S3, S4, so that in the corrected pixels g1, g2, g3, g4: the pixels g1 and g3 are high-brightness pixels, and the pixels g2 and g4 are low-brightness pixels. That is, the overall brightness trend of the corrected pixel unit G0 is consistent with the overall brightness trend of the neighborhood of the pixel unit G0, thereby eliminating the bright and dark interlaced texture defect of the defective unit.

[0053] In summary, the image processing method of the embodiments of the present application can obtain the first pixel sum corresponding to the defective unit and the second pixel sum corresponding to the neighborhood same color unit of the defective unit, so as to correct the pixel values of each pixel in the defective unit of the first image according to the first pixel sum and the second pixel sum, thereby eliminating the bright and dark interlaced texture defect of the defective unit.

[0054] The above is further illustrated in combination with the accompanying drawings.

[0055] Please refer to Figure 6 In some embodiments, 01: detecting defects in the pixel units in the first image to obtain the defective units of the first image, comprising:

[0056] 011: taking a pixel unit to be detected in the first image as a center unit, determining a preset neighborhood of the center unit, and the preset neighborhood containing same color units of the same color as the center unit;

[0057] 012: obtaining a first encoding value of the center unit and a second encoding value of the same color units in the preset neighborhood; and

[0058] 013: In a case where the number of the same-color units having the second encoding value same as the first encoding value is greater than or equal to the preset number, the same-color units having the second encoding value same as the first encoding value and the center unit are determined as the defective unit of the first image.

[0059] In combination with Figure 3 In some embodiments, the first detection module 11 is further configured to perform the method in steps 011, 012, and 013. That is, the first detection module 11 is further configured to take a pixel unit to be detected in the first image as a center unit, determine a preset neighborhood of the center unit, the preset neighborhood including same-color units same as the center unit; obtain a first encoding value of the center unit and second encoding values of the same-color units in the preset neighborhood; and in a case where the number of the same-color units having the second encoding value same as the first encoding value is greater than or equal to the preset number, determine the same-color units having the second encoding value same as the first encoding value and the center unit as the defective unit of the first image.

[0060] In combination with Figure 4 In some embodiments, the processor 30 is further configured to perform the image processing method in steps 011, 012, and 013.

[0061] In combination with Figure 5 In some embodiments, all the pixel units in the first image that have not been subjected to defect detection are the pixel units to be detected. In one embodiment, the preset neighborhood of the center unit is determined after taking the pixel unit to be detected in the first image as the center unit. For example, the pixel unit G0 to be detected is taken as the center unit, and the preset neighborhood range is 3*3 pixel units. The units same as the pixel unit G0 in the neighborhood of the 3*3 pixel units centered on the pixel unit G0 are taken as the neighborhood same-color units. In another embodiment, the range of the neighborhood can be determined first, and then the pixel unit at the middle position of the neighborhood is taken as the center unit. For example, the range of the preset neighborhood is 3*3 pixel units. First, the 3*3 pixel units are selected, and then it is determined whether the 2nd row and 2nd column pixel units in the 3*3 pixel units are pixel units that have not been subjected to defect detection. If yes, the 2nd row and 2nd column pixel units in the neighborhood of the 3*3 pixel units are taken as the center unit. The selection of the preset neighborhood is as explained above, and thus will not be repeated here. Figure 5 In the above embodiment, if the center unit is the pixel unit G0 and the preset neighborhood is 3*3 pixel units, the units same as the center unit G0 are G1, G2, G3, and G4.

[0062] In combination with Figure 7From left to right, the first figure is an array of pixel units, the second figure is a binary image showing a plurality of pixels in each pixel unit in the first figure, the third figure is a binary coding image of the pixel units, and the fourth figure is a final coding value image of the pixel units. The first coding value represents the brightness of the center unit as a whole, and the second coding value represents the brightness of the same-color unit as a whole in the preset neighborhood. In an embodiment, if the number of same-color units in the preset neighborhood whose second coding value is the same as the first coding value is greater than or equal to a preset number K, the center unit and the same-color units whose second coding value is the same as the first coding value of the center unit are considered to be defective units. For example, the preset neighborhood range is a neighborhood of N*N pixel units, and N>1. The preset number K=N-1. In combination with Figure 5 and Figure 7 In the case of N=3, K=2, and the preset neighborhood range is 3*3 pixel units. If the pixel unit G0 is selected as the center unit, the first coding value of the center unit G0 is 1, and the second coding values of the same-color units G1, G2, G3, and G4 in the neighborhood are 2, 1, 1, and 8, respectively. There are 2 same-color units whose second coding value is the same as the first coding value 1, which satisfies the judgment condition of defective units, so the center unit G0 and the neighborhood same-color units G2 and G3 are determined to be defective units.

[0063] In yet another embodiment, if the number of same-color units whose second coding value is the same as the first coding value is greater than or equal to a preset number K in J adjacent same-color units in the horizontal direction, vertical direction, or diagonal direction, the center unit and the same-color units whose second coding value is the same as the first coding value of the center unit are considered to be defective units. For example, as shown in Figure 8 J=7, K=2, the pixel unit G4 is the center unit, there are 3 adjacent same-color units on the left and right sides of the center unit G4 in the horizontal direction, and the coding values of the pixel units G1, G2, G3, G4, G5, G6, and G7 are 2, 2, 1, 1, 1, 1, and 5, respectively. The first coding value corresponding to the center unit G4 is 1, and there are 3 same-color units G3, G5, and G6 in the horizontal direction whose second coding value is the same as the first coding value, which satisfies the judgment condition of defective units, so the center unit G0 and the same-color units G3, G5, and G6 are determined to be defective units.

[0064] Referring to Figure 9 In some embodiments, 012: obtaining the first coding value of the center unit and the second coding value of the same-color unit in the preset neighborhood, comprises:

[0065] 0121: obtaining the pixel value of each pixel in the pixel unit and the brightness threshold of the pixel unit;

[0066] 0122: Binarize the pixel values ​​of each pixel in the pixel unit based on the pixel values ​​of each pixel and the brightness threshold of the pixel unit to obtain the binarized value corresponding to each pixel value;

[0067] 0123: Perform binary encoding on the binarized values ​​of each pixel in the pixel unit to obtain the pixel code value of the pixel unit; and

[0068] 0124: Use the pixel encoding value of the central unit as the first encoding value, and use the pixel encoding value of the same color unit in the preset neighborhood of the central unit as the second encoding value.

[0069] Please combine Figure 3 In some embodiments, the first detection module 11 is further configured to perform the methods in steps 0121, 0122, 0123, and 0124. Specifically, the first detection module 11 is further configured to acquire the pixel value of each pixel in the pixel unit and the brightness threshold of the pixel unit; perform binarization processing on the pixel value of each pixel in the pixel unit according to the pixel value of each pixel in the pixel unit and the brightness threshold of the pixel unit to acquire the binarized value corresponding to each pixel value; perform binary encoding on the binarized value of each pixel in the pixel unit to acquire the pixel encoding value of the pixel unit; and use the pixel encoding value of the central unit as the first encoding value, and use the pixel encoding values ​​of units of the same color in the preset neighborhood of the central unit as the second encoding value.

[0070] Please combine Figure 4 In some embodiments, the processor 30 is also used to perform the image processing methods in steps 0121, 0122, 0123, and 0124 described above.

[0071] Please combine Figure 7 The brightness threshold is used to evaluate whether a pixel's brightness value is bright or dark. If a pixel's value is greater than the brightness threshold, the binarized value of that pixel is 1; if a pixel's value is less than the brightness threshold, the binarized value of that pixel is 0. Thus, each pixel in a pixel unit is binarized according to its brightness.

[0072] Binary encoding is performed on the binarized values ​​of each pixel in a pixel unit. That is, the binarized values ​​of each pixel in each pixel unit are sorted in a predetermined order to obtain a string of binary characters. This binary character is then converted into a decimal value, which is the encoded value of that pixel unit. Specifically, if the pixel unit is the center unit, it corresponds to the first encoded value; if the pixel unit is a unit of the same color, it corresponds to the second encoded value. For example... Figure 7As shown, each pixel unit includes 4 pixels, wherein the binarization values of the four pixels g1, g2, g3, g4 of the pixel unit G0 are 0, 0, 0, 1 respectively, the binary character obtained after sorting is "0001", the binary character "0001" is converted into the decimal value "1", and the encoding value of the pixel unit G0 is 1. Since the pixel unit G0 is the center unit, the first encoding value corresponding to the center unit G0 is 1. The second encoding values corresponding to the same color units G1, G2, G3, G4 are also obtained in the same way, which will not be described in detail.

[0073] Referring to Figure 10 In some embodiments, the step 0121 of obtaining the pixel value of each pixel in the pixel unit and the brightness threshold of the pixel unit includes:

[0074] 01211: taking the mean value of the pixel values of each pixel in the pixel unit as the brightness threshold; or

[0075] 01212: sorting the pixel values of each pixel in the pixel unit according to the size, calculating the difference value between the two adjacent pixel values in size, and taking the mean value of the two pixel values with the maximum absolute value of the difference value as the brightness threshold.

[0076] Referring to Figure 3 In some embodiments, the first detection module 11 is further configured to perform the method in the step 01211 or 01212. That is, the first detection module 11 is further configured to take the mean value of the pixel values of each pixel in the pixel unit as the brightness threshold; or sort the pixel values of each pixel in the pixel unit according to the size, calculate the difference value between the two adjacent pixel values in size, and take the mean value of the two pixel values with the maximum absolute value of the difference value as the brightness threshold.

[0077] Referring to Figure 4 In some embodiments, the processor 30 is further configured to perform the image processing method in the above step 01211 or 01212.

[0078] Referring to Figure 7For example, the pixel values of the four pixels g1, g2, g3, g4 in the pixel unit G0 are 1, 2, 2, and 7 respectively. In an embodiment, the luminance threshold L is the average of the pixel values of the four pixels g1, g2, g3, g4, i.e. L = (1 + 2 + 2 + 7) / 4 = 3. Then, the pixel values of the pixels g1, g2, g3 are less than the luminance threshold L, and the corresponding binary values are 0; the pixel value of the pixel g4 is greater than the luminance threshold L, and the corresponding binary value is 1. In another embodiment, for the pixel values 1, 2, 2, and 7 of the pixels g1, g2, g3, g4, 2 and 7 are adjacent and the difference between them is the largest, then the average of 2 and 7, i.e. 4.5, is taken as the luminance threshold L. At this time, the pixel values of the pixels g1, g2, g3 are less than the luminance threshold L, and the corresponding binary values are 0; the pixel value of the pixel g4 is greater than the luminance threshold L, and the corresponding binary value is 1.

[0079] Please refer to Figure 11 In some embodiments, 03: correcting the pixel values of each pixel in the defective unit of the first image according to the first pixel sum and the second pixel sum, comprising:

[0080] 031: obtaining the Euclidean distance between each pixel in the defective unit of the first image and the same color unit in the preset neighborhood;

[0081] 032: obtaining the estimated value of each pixel in the defective unit according to the second pixel sum and the Euclidean distance;

[0082] 033: obtaining the correction value of each pixel in the defective unit of the first image according to the first pixel sum and the estimated value; and

[0083] 034: correcting the pixel values of each pixel in the defective unit according to the correction value of each pixel in the defective unit of the first image.

[0084] Please refer to Figure 3 In some embodiments, the first correction module 13 is further configured to perform the methods in steps 031, 032, 033, and 034. That is, the first correction module is further configured to obtain the Euclidean distance between each pixel in the defective unit of the first image and the same color unit in the preset neighborhood; obtain the estimated value of each pixel in the defective unit according to the second pixel sum and the Euclidean distance; obtain the correction value of each pixel in the defective unit of the first image according to the first pixel sum and the estimated value; and correct the pixel values of each pixel in the defective unit according to the correction value of each pixel in the defective unit of the first image.

[0085] Please refer to Figure 4 In some embodiments, the processor 30 is further configured to perform the image processing methods in steps 031, 032, 033, and 034.

[0086] Please refer toFigure 5 Taking repairing the defective unit G0 as an example, the defective unit G0 includes four pixels g1, g2, g3, g4. The first pixel sum of the defective unit G0 is S0, and the second pixel sums of the neighboring same-color units G1, G2, G3, G4 of the defective unit G0 are S1, S2, S3, S4 respectively. Taking the pixel g1 as an example, the Euclidean distance between the pixel g1 and the same-color unit G1 is The Euclidean distance between the pixel g1 and the same-color unit G2 is The Euclidean distance between the pixel g1 and the same-color unit G3 is The Euclidean distance between the pixel g1 and the same-color unit G4 is The estimated value t1 of the pixel g1 is d1*S1+d2*S2+d3*S3+d4*S4. Similarly, the estimated values t2, t3, t4 of the pixels g2, g3, g4 can be calculated respectively. The estimated value of a pixel is a value determined according to the luminance of the neighboring same-color units of the defective unit and the distance between the neighboring same-color units and the pixel in the defective unit, and the luminance corresponding to the estimated value of the pixel is closer to the luminance of the neighboring same-color unit close to the pixel.

[0087] According to the estimated values t1, t2, t3, t4 of the pixels g1, g2, g3, g4 and the first pixel sum S0 of the defective unit G0, the correction values x1, x2, x3, x4 of the pixels g1, g2, g3, g4 can be obtained respectively. It can be seen that the sum of the correction values x1, x2, x3, x4 is still equal to S0, so that the pixel values of each pixel in the defective unit can be changed towards the estimated value to adapt to the luminance of the neighboring same-color units, and the first pixel sum S0 of the defective unit remains unchanged before and after correction, that is, the luminance sum of the multi-in-one pixel remains unchanged before and after correction, without the need for additional resources to compensate for the luminance.

[0088] After the correction values of each pixel in the defective unit are calculated respectively, the pixel values of each pixel in the defective unit are replaced by the corresponding correction values, so that the luminance of each pixel in the corrected defective unit is close to the luminance distribution of the neighboring same-color units of the defective unit, achieving the effect of eliminating the light and dark interlaced defects. Each defective unit in the first image can be corrected according to the above method to correct all the light and dark interlaced defects in the first image. Of course, in other embodiments, after the correction values of each pixel in the defective unit are calculated respectively, a weight can be attached to each correction value, and then the pixel values of each pixel in the defective unit are replaced by the values after the weight is attached. Similarly, the luminance of each pixel in the corrected defective unit can be close to the luminance distribution of the neighboring same-color units of the defective unit, achieving the effect of eliminating the light and dark interlaced defects, and the setting of the weight also gives the adaptive adjustment function for different scenes.

[0089] Referring to Figure 12 The embodiments of the present application provide still another image processing method, which comprises the following steps:

[0090] 04: performing defect detection on the pixel units in the first image to obtain defect units of the first image, each pixel unit comprising a plurality of adjacent same-color pixels; and

[0091] 05: inputting the first image into the trained image processing network to correct the pixel values of the pixels in the defect units of the first image by using the image processing network.

[0092] Referring to Figure 13 The embodiments of the present application also provide an image processing device 10, and the image processing method described above can be applied to the image processing device 10. The image processing device 10 can comprise a second detection module 14 and a second correction module 15. The second detection module 14 is configured to perform the method in step 04, and the second correction module 15 is configured to perform the method in step 05. That is, the second detection module 14 is configured to perform defect detection on the pixel units in the first image to obtain defect units of the first image, each pixel unit comprising a plurality of adjacent same-color pixels; and the second correction module 15 is configured to input the first image into the trained image processing network to correct the pixel values of the pixels in the defect units of the first image by using the image processing network.

[0093] As Figure 3 and Figure 13 shown, in some embodiments, the image processing device 10 can only comprise the first detection module 11, the first acquisition module 12 and the first correction module 13, or the image processing device 10 can only comprise the second detection module 14 and the second correction module 15.

[0094] Referring to Figure 14 In still another embodiment, the image processing device 10 can comprise the first detection module 11, the first acquisition module 12, the first correction module 13, the second detection module 14 and the second correction module 15. That is, the image processing device 10 can be configured to perform any one or more of the image processing methods in steps 01, 02, 03, 04 and 05.

[0095] Referring to Figure 4 In some embodiments, the processor 30 is further configured to perform the image processing methods in steps 04 and 05.

[0096] Referring to Figure 2The defect detection method in step 04 is consistent with the defect detection method in step 01, which will not be repeated here. The image processing network is used to repair the alternating light and dark defects in the image, so that the light and dark junctions in the image can transition naturally. The first image with alternating light and dark defects is input into the trained image processing network, and the second image after repair can be obtained, which eliminates the defects.

[0097] Please refer to Figure 15 In some embodiments, step 04: performing defect detection on the pixel units in the first image, comprising:

[0098] 041: taking a pixel unit to be detected in the first image as a center unit, determining a preset neighborhood of the center unit, and the preset neighborhood containing homochromatic units of the same color as the center unit;

[0099] 042: obtaining a first encoding value of the center unit and a second encoding value of the homochromatic units in the preset neighborhood; and

[0100] 043: in the case where the number of homochromatic units with the same second encoding value as the first encoding value is greater than or equal to a preset number, determining the homochromatic units with the same second encoding value as the first encoding value and the center unit as defect units of the first image.

[0101] Please refer to Figure 13 In some embodiments, the second detection module 14 is also used to perform the method in step 04. That is, the second detection module 14 is also used to take a pixel unit to be detected in the first image as a center unit, determine a preset neighborhood of the center unit, and the preset neighborhood containing homochromatic units of the same color as the center unit; obtain a first encoding value of the center unit and a second encoding value of the homochromatic units in the preset neighborhood; and in the case where the number of homochromatic units with the same second encoding value as the first encoding value is greater than or equal to a preset number, determine the homochromatic units with the same second encoding value as the first encoding value and the center unit as defect units of the first image.

[0102] Please refer to Figure 4 In some embodiments, the processor 30 is also used to perform the image processing method in steps 041, 042, and 043.

[0103] Please refer to Figure 5 and Figure 6 The image processing method in steps 041, 042, and 043 is similar to the image processing method in steps 011, 012, and 013, which will not be repeated here.

[0104] Please refer to Figure 16 In some embodiments, the image processing method further comprises:

[0105] 06: The image processing network is trained based on the pre-acquired output reference image and the original image collected by the sensor. The original image and the output reference image are in one-to-one correspondence. The original image contains only single color channel information, while the output reference image contains multi-channel color information.

[0106] Please combine Figure 13 and 14 In some embodiments, the image processing apparatus 10 further includes a training module 16, which is used to execute the method in step 06. That is, the training module 16 is used to train an image processing network based on a pre-acquired output reference image and the original image acquired by the sensor. Figure 1 In a one-to-one correspondence, the original image contains only single-color channel information, while the output reference image contains multi-channel color information.

[0107] Please combine Figure 4 In some embodiments, the processor 30 is also used to perform the image processing method in step 06 above.

[0108] In one embodiment, the output reference image is a color image containing R, G, and B channels. In other embodiments, the output reference image may also be a multi-channel color image containing other color channels; this is not a limitation. The aforementioned interlacing of light and dark areas is absent in the output reference image. However, the raw images acquired by the sensor are prone to interlacing of light and dark areas due to the photosensitive principle of multi-pixel binning. The purpose of training the image processing network is to enable it to generate images with interlacing of light and dark areas that have been eliminated, based on images with this defect.

[0109] Please see Figure 17 In some implementations, 06: An image processing network is trained based on a pre-acquired output reference image and the original images acquired by the sensor, including:

[0110] 061: Extract information from a single color channel from the output reference image to generate the initial input image;

[0111] 062: Perform defect detection on pixel units in the original image to obtain defect units in the original image;

[0112] 063: Perform simulated defect processing on the initial input image based on the defect units in the original image to obtain the final input image; and

[0113] 064: The final input image and the corresponding output reference image are used as the corresponding input image and label image in the training set, respectively, to form an image processing network.

[0114] Please combine Figure 13 and 14In some embodiments, the training module is configured to perform the method in steps 061, 062, 063, and 064. That is, the training module is configured to extract the information of a single color channel from the output reference image to generate an initial input image; perform defect detection on the pixel units in the original image to obtain defect units in the original image; perform simulated defect processing on the initial input image according to the defect units in the original image to obtain a final input image; and input the final input image and the corresponding output reference image into the image processing network as a corresponding set of input images and a set of label images in the training set, respectively, to form the image processing network.

[0115] Please refer to Figure 4 In some embodiments, the processor 30 is further configured to perform the image processing method in steps 061, 062, 063, and 064.

[0116] Please refer to Figure 2 and Figure 12 In step 062, the method of detecting defect units is consistent with the method of detecting defect units in steps 01 and 04, which is not limited here. In the original image, the defect units are obtained, and the final input image is obtained by performing simulated defect processing on the initial input image according to the defect units in the original image, so that the final input image appears consistent defects with the original image.

[0117] Please refer to Figure 18 As shown in the upper half of Figure 18 In the original image processing network, the set of input images is the initial input image, and the set of label images is the output reference image. The image processing network can restore a color image according to the input single-channel initial input image, and verify the restoration degree of the restored color image by the output reference image in the set of label images to train the image processing network, and the training is completed when the restoration degree reaches a preset threshold. For example, the image processing network restores a three-channel color image according to the input R-channel initial input image, G-channel initial input image, and B-channel initial input image, wherein the R-channel initial input image, G-channel initial input image, and B-channel initial input image are all single-color channel images extracted from the output reference image in the set of label images, so the restoration degree of the restored three-channel color image can be verified by the output reference image in the set of label images.

[0118] As shown in the upper half of Figure 18the lower half of FIG. 6, the final input image is obtained according to the single-channel input image extracted from the output reference image, so that the image processing network can restore the color image according to the single-channel initial input image and the color image according to the final input image corresponding to the initial input image. In the case where the final input image replaces the initial input image as the set input image, defects are introduced in the initial input image, and the output reference image is still used as the set label image, so that the image processing network can restore the color image close to the output reference image from the final input image with defects, so that the image processing network can eliminate the defects in the input image. Therefore, in the case where the input image is the first image obtained by the sensor, the trained image processing network can eliminate the defects in the first image.

[0119] Please refer to Figure 19 In some embodiments, 063: performing simulation defect processing on the initial input image according to the defect unit in the original image to obtain a final input image, comprising:

[0120] 0631: cropping the initial input image and the original image to the same size;

[0121] 0632: obtaining the coordinate value and the brightness value of each pixel in the defect unit of the original image;

[0122] 0633: determining the to-be-modified unit corresponding to the position of the defect unit in the original image in the initial input image according to the coordinate value, wherein the to-be-modified unit includes a plurality of to-be-modified pixels corresponding to the position of each pixel in the defect unit of the original image;

[0123] 0634: obtaining the third pixel sum of the to-be-modified unit; and

[0124] 0635: modifying the pixel value of the to-be-modified pixel according to the brightness value and the third pixel sum to obtain the final input image.

[0125] Please refer to Figure 13 and 14 In some embodiments, the training module is configured to perform the methods in steps 0631, 0632, 0633, 0634, and 0635. That is, the training module is configured to crop the initial input image and the original image to the same size; obtain the coordinate value and the brightness value of each pixel in the defect unit of the original image; determine the to-be-modified unit corresponding to the position of the defect unit in the original image in the initial input image according to the coordinate value, wherein the to-be-modified unit includes a plurality of to-be-modified pixels corresponding to the position of each pixel in the defect unit of the original image; obtain the third pixel sum of the to-be-modified unit; and modify the pixel value of the to-be-modified pixel according to the brightness value and the third pixel sum to obtain the final input image.

[0126] Please refer to Figure 4In some embodiments, the processor 30 is further configured to perform the image processing method in steps 0631, 0632, 0633, 0634, and 0635 described above.

[0127] In one embodiment, the initial input images are all generated from the single color channel information extracted from the output reference images in the set of label images, and the size of the initial input images are all larger than the maximum size of the raw images that the image sensor 40 is capable of generating, so that the initial input images are all cropped to the same size according to the size of the raw images.

[0128] Please refer to Figure 20 For example, the coordinates of the pixels g1, g2, g3, g4 in the defective unit G0 of the raw image are Q1(i1, j1), Q2(i2, j2), Q3(i3, j3), Q4(i4, j4), respectively, and the corresponding luminance values are P1, P2, P3, P4, respectively. Then, the to-be-modified unit is determined according to the coordinates Q1(i1, j1), Q2(i2, j2), Q3(i3, j3), Q4(i4, j4) in the initial input image. Since the size of the initial input image is the same as that of the raw image, the positions of the to-be-modified pixels in the initial input image are consistent with the positions of the pixels g1, g2, g3, g4 in the raw image. For example, the position of the pixel unit A0 in the initial input image is consistent with the position of the defective unit G0 in the raw image, the pixel unit A0 is the to-be-modified unit, and the positions of the to-be-modified pixels a1, a2, a3, a4 in the to-be-modified unit A0 correspond to the positions of the pixels g1, g2, g3, g4, respectively, i.e., the to-be-modified pixel a1 is the i1th j1th pixel, the to-be-modified pixel a2 is the i2th j2th pixel, the to-be-modified pixel a3 is the i3th j3th pixel, and the to-be-modified pixel a4 is the i4th j4th pixel.

[0129] Let the third pixel sum of the to-be-modified pixels a1, a2, a3, a4 be P0, and the modification values corresponding to the to-be-modified pixels a1, a2, a3, a4 be y1, y2, y3, y4, respectively, then The modification values y1, y2, y3, y4 are set as the pixel values of the to-be-modified pixels a1, a2, a3, a4, respectively, to obtain a modified unit. In this way, the sum of the third modification values y1, y2, y3, y4 is still P0, and the luminance of the to-be-modified pixels a1, a2, a3, a4 changes towards the luminance trend of the pixels g1, g2, g3, g4 in the defective unit G0, so as to simulate the defect of the defective unit G0. After modifying the pixel values of all the to-be-modified units corresponding to the defective units in the initial input image according to the above method, the final input image is obtained.

[0130] In summary, the image processing method of the embodiments of the present application can correct the light-and-dark interlaced texture defects in the first image obtained by the sensor by using the image processing network. Wherein, the light-and-dark interlaced texture defects of the original image can be simulated in the initial input image to obtain the final input image, and the image processing network is used to restore the color image without defects from the final input image to train the image processing network, so that the image processing network has the ability to correct the defects of the first image.

[0131] Please refer to Figure 21 The embodiments of the present application also provide a non-volatile computer readable storage medium 300 containing a computer program 301. One or more non-volatile computer readable storage media 300 containing a computer program 301 of the embodiments of the present application, when the computer program 301 is executed by one or more processors 30, enable the processor 30 to execute the image processing method of any of the above embodiments.

[0132] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0133] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and that the various embodiments of the present application can include additional or fewer steps or codes, and that the method steps, codes or portions thereof can be combined or reordered in any suitable manner, and that the method steps, codes or portions thereof can be implemented in any suitable manner, as would be understood by one of ordinary skill in the art, including as described in connection with the other embodiments of the present application.

[0134] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. An image processing method characterized by, The method comprises the following steps: acquiring a first image; each pixel unit in the first image comprises a plurality of adjacent same-color pixels; the plurality of adjacent same-color pixels share a microlens; and inputting the first image into a trained image processing network, and correcting the pixel values of each pixel in the defective unit of the first image by using the image processing network; the image processing network is used for repairing the light and shade defects in the first image, so that the light and shade junctions in the first image can transition naturally; wherein the training process of the image processing network comprises: extracting the information of a single color channel from a pre-acquired output reference image to generate an initial input image, detecting the defects of the pixel units in the original image collected by the sensor to acquire the defective units in the original image, cropping the initial input image and the original image to the same size to acquire the coordinate values and brightness values of each pixel in the defective units of the original image, determining the to-be-modified units corresponding to the positions of the defective units in the original image in the initial input image according to the coordinate values, the to-be-modified units comprising a plurality of to-be-modified pixels corresponding to the positions of each pixel in the defective units of the original image; acquiring a third pixel sum of the to-be-modified units; and modifying the pixel values of the to-be-modified pixels according to the brightness values and the third pixel sum to acquire a final input image; and inputting the final input image and the corresponding output reference image into the training set as the corresponding set input image and set label image respectively to form the image processing network; the original image and the output reference image are in one-to-one correspondence, the original image only contains single-color channel information, and the output reference image contains multi-channel color information.

2. The image processing method of claim 1, wherein, The method comprises the following steps: detecting the defects of the pixel units in the original image collected by the sensor to acquire the defective units in the original image, comprising: taking a pixel unit to be detected in the original image as a center unit, determining a preset neighborhood of the center unit, and the preset neighborhood comprising same-color units of the same color as the center unit; acquiring a first encoding value of the center unit and a second encoding value of the same-color units in the preset neighborhood; and 3. The image processing method of claim 1, wherein, in the case where the number of same-color units with the same second encoding value as the first encoding value is greater than or equal to a preset number, determining the same-color units with the same second encoding value as the first encoding value and the center unit as the defective units of the original image. The method comprises the following steps: taking the first image as the final input image in the image processing network; 4. An image processing apparatus characterized by comprising: finding a set input image identical to the final input image and a set label image corresponding to the set input image in the training set, and correcting the pixel values of each pixel at the corresponding positions in the defective units of the first image according to the pixel values of each pixel in the set label image. The method comprises the following steps: a second detection module is configured to acquire a first image; each pixel unit in the first image comprises a plurality of adjacent same-color pixels; the plurality of adjacent same-color pixels share a microlens; The second correction module is configured to input the first image into a trained image processing network, and correct pixel values of each pixel in a defective unit of the first image by using the image processing network. The image processing network is configured to repair a light-and-dark interlaced defect in the first image, and enable a light-and-dark junction in the first image to transition naturally; wherein a training process of the image processing network includes: extracting information of a single color channel from an output reference image acquired in advance to generate an initial input image; detecting a defect in a pixel unit of an original image collected by a sensor to obtain a defective unit in the original image; cropping the initial input image and the original image to the same size; obtaining coordinate values and brightness values of each pixel in the defective unit of the original image; determining a to-be-modified unit corresponding to a position of the defective unit in the original image in the initial input image according to the coordinate values, wherein the to-be-modified unit includes a plurality of to-be-modified pixels corresponding to the position of each pixel in the defective unit of the original image; obtaining a third pixel sum of the to-be-modified unit; modifying pixel values of the to-be-modified pixels according to the brightness values and the third pixel sum to obtain a final input image; and inputting the final input image and the corresponding output reference image as a corresponding set input image and a set label image in a training set respectively to form the image processing network; the original image and the output reference image are in one-to-one correspondence, the original image only includes single color channel information, and the output reference image includes multi-channel color information.

5. A terminal, characterized by comprising: The terminal includes: one or more processors, memories; and one or more programs, wherein the one or more programs are stored in the memories and executed by the one or more processors, and the programs include instructions for executing the image processing method of any one of claims 1 to 3.

6. A non-volatile computer-readable storage medium containing a computer program, when the computer program is executed by one or more processors, causes the processors to implement the image processing method of any one of claims 1 to 3.

Images