Image processing method, device, electronic device and computer readable storage medium

Abstract

The present application relates to an image processing method and device, an electronic device, and a computer-readable storage medium. First, the original image obtained by the image sensor is interpolated to generate a full-size full-color channel image. The full-size full-color channel image obtained by interpolation can greatly improve the brightness of the image, making the picture layer clearer and the picture more transparent. Therefore, the color pixels of different colors in the original image can be filtered through the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors. Finally, based on each specific color channel image, a target image is generated, thereby improving the clarity of the target image finally obtained and making the resolution higher.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the development of electronic devices, more and more users are taking pictures through electronic devices. Among them, the camera in the electronic device includes an image sensor, and the image sensor is usually provided with a filter array arranged in the form of a Bayer array. During the shooting process, the pixel array in the image sensor can receive the light passing through the filter, thereby generating pixel signals of different color channels. Then, the pixel signals of different color channels are interpolated to generate the target image.

[0003] However, in the process of interpolating pixel signals of different color channels to generate a target image, the accuracy of the interpolation result is low, which leads to low clarity of the target image finally generated. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can improve the clarity of a target image ultimately obtained.

[0005] On the one hand, an image processing method is provided, which is applied to an electronic device, wherein the electronic device includes an image sensor, the image sensor includes a filter array and a pixel array, the filter array includes a minimum repeating unit, the minimum repeating unit includes a plurality of filter groups, the filter group includes a color filter and a panchromatic filter; in the filter group, the color filter is arranged in a first diagonal direction, the panchromatic filter is arranged in a second diagonal direction, and the first diagonal direction is different from the second diagonal direction; the pixel array includes a plurality of panchromatic pixels and a plurality of color pixels, each of the panchromatic pixels corresponds to the panchromatic filter, and each of the color pixels corresponds to the color filter;

[0006] The method comprises:

[0007] Performing interpolation processing on the original image acquired by the image sensor to generate a full-size full-color channel image; wherein the original image includes color pixels and full-color pixels;

[0008] Performing filtering processing on color pixels of different colors in the original image through the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors;

[0009] A target image is generated based on each of the specific color channel images.

[0010] On the other hand, an image processing device is also provided, which is applied to an electronic device, wherein the electronic device includes an image sensor, the image sensor includes a filter array and a pixel array, the filter array includes a minimum repeating unit, the minimum repeating unit includes a plurality of filter groups, the filter group includes a color filter and a panchromatic filter; in the filter group, the color filter is arranged in a first diagonal direction, and the panchromatic filter is arranged in a second diagonal direction, and the first diagonal direction is different from the second diagonal direction; the pixel point array includes a plurality of panchromatic pixels and a plurality of color pixels, each of the panchromatic pixels corresponds to the panchromatic filter, and each of the color pixels corresponds to the color filter;

[0011] The device comprises:

[0012] A first generating module, configured to perform interpolation processing on an original image acquired by the image sensor to generate a full-size full-color channel image; wherein the original image includes color pixels and full-color pixels;

[0013] A second generating module is used to perform filtering processing on the color pixels of different colors in the original image of the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors respectively;

[0014] The third generating module is used to generate a target image based on each of the specific color channel images.

[0015] On the other hand, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and wherein when the computer program is executed by the processor, the processor executes the steps of the image processing method described above.

[0016] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described above are implemented.

[0017] The above-mentioned image processing method and device, electronic device, and computer-readable storage medium first interpolate the original image acquired by the image sensor to generate a full-size full-color channel image. The full-size full-color channel image obtained by interpolation can greatly improve the brightness of the image, making the picture layer clearer and the picture more transparent. Therefore, the color pixels of different colors in the original image can be filtered through the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors. Finally, based on each specific color channel image, a target image is generated, thereby improving the clarity of the target image finally obtained and making the resolution higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0020] Figure 2 is a flowchart of an image processing method in one embodiment;

[0021] Figure 3 It is a schematic diagram of an output image in a Bayer format after three channels are integrated and outputted through bilateral filtering calculation in one embodiment;

[0022] Figure 4 It is a schematic diagram of filtering W pixels in a W channel image and color pixels in an original image in one embodiment;

[0023] Figure 5 is a schematic diagram of an image processing flow in one embodiment;

[0024] Figure 6 for Figure 2 A flowchart of interpolating the original image obtained by the image sensor to generate a full-size full-color channel image;

[0025] Figure 7 for Figure 6 A flowchart for generating a new pixel value of W pixels;

[0026] Figure 8 A schematic diagram of generating a new pixel value of W pixels in one embodiment;

[0027] Fig. 9 It is a schematic diagram of generating pixel values ​​of new W pixels for replacing color pixel a for a flat area in one embodiment;

[0028] Fig.10 It is a schematic diagram of generating pixel values ​​of new W pixels for replacing color pixel a for a non-flat area in one embodiment;

[0029] Fig.11 is a schematic diagram of W pixel pairs in each texture direction in one embodiment;

[0030] Fig.12 is a schematic diagram of W pixel pairs in each texture direction in another embodiment;

[0031] Fig.13 Schematic diagram of a weight matrix W_weight of W pixels, a weight matrix RGB_weight of color pixels, and a distribution matrix B_mask of B pixels in one embodiment;

[0032] Fig.14 is a flowchart of an image processing method in a specific embodiment;

[0033] Fig.15 for Fig.14 Flowchart of a method for calculating texture direction of an input pixel in;

[0034] Fig.16 is a schematic structural diagram of an image sensor in one embodiment;

[0035] Fig.17 is a connection diagram of a pixel array and a readout circuit in one embodiment;

[0036] Fig.18 is a structural block diagram of an image processing device in one embodiment;

[0037] Fig.19 FIG. 1 is a schematic diagram of the internal structure of an electronic device in an embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first preset area may be referred to as a second preset area, and similarly, a second preset area may be referred to as a first preset area. Both the first preset area and the second preset area are preset areas, but they are not the same preset area.

[0040] Figure 1 FIG. 1 is a schematic diagram of an application environment of an image processing method in an embodiment. Figure 1As shown, the application environment includes an electronic device 120, the electronic device 120 includes an image sensor, the image sensor includes a filter array and a pixel array, the filter array includes a minimum repeating unit, the minimum repeating unit includes multiple filter groups, the filter group includes a color filter and a full-color filter; in the filter group, the color filter is arranged in a first diagonal direction, the full-color filter is arranged in a second diagonal direction, and the first diagonal direction is different from the second diagonal direction; the pixel array includes multiple full-color pixels and multiple color pixels, each full-color pixel corresponds to a full-color filter, and each color pixel corresponds to a color filter.

[0041] The electronic device 120 performs interpolation processing on the original image acquired by the image sensor to generate a full-size full-color channel image; wherein the original image includes color pixels and full-color pixels; the color pixels of different colors in the original image are filtered through the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors; based on each specific color channel image, a target image is generated. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a wearable device (smart bracelet, smart watch, smart glasses, smart gloves, smart socks, smart belt, etc.), a VR (virtual reality) device, a smart home, and an unmanned vehicle.

[0042] Figure 2 FIG. 1 is a flowchart of an image processing method in an embodiment. The image processing method in this embodiment is executed on Figure 1 The electronic device in the example is used to describe. Figure 2 As shown, the image processing method includes steps 220 to 260. Among them,

[0043] Step 220, interpolation processing is performed on the original image acquired by the image sensor to generate a full-size full-color channel image; wherein the original image includes color pixels and full-color pixels.

[0044] The electronic device includes an image sensor, and the image sensor includes a filter array and a pixel array. The original image can be obtained through the filter array and the pixel array. Among them, the pixel array can be called an RGBW pixel array. Among them, the original image can also be called an RGBW image, including images of four channels: R channel (red channel), G channel (green channel), B channel (blue channel), and W channel (white channel), that is, the original image includes red pixels (R pixels), green pixels (G pixels), blue pixels (B pixels) and full-color pixels (or called W pixels). Among them, red pixels (R pixels), green pixels (G pixels) and blue pixels (B pixels) can also be called color pixels.

[0045] After the original image is acquired through the filter array and the RGBW pixel array, the original image acquired through the image sensor can be interpolated to generate a full-size full-color channel image. Among them, the full-color channel image refers to the W channel image, and the full-size full-color channel image refers to the W channel image with the same image size as the original image, that is, the color pixels in the original image are replaced with full-color pixels, and the full-size full-color channel image is obtained. Specifically, interpolation processing is performed based on the full-color pixels in the original image to generate new full-color pixels. The new full-color pixels are used to replace the color pixels in the original image, so that a full-size full-color channel image is generated.

[0046] Step 240 , filtering the color pixels of different colors in the original image through the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors.

[0047] The full-size full-color channel image obtained by interpolation can greatly improve the brightness of the image, making the picture layer clearer and the picture more transparent. Therefore, when generating the target image, the color pixels of different colors in the original image can be filtered through the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors. The specific color channel image includes a specific R channel image, a specific G channel image, and a specific B channel image. The specific R channel image includes the R pixel generated at a specific position after filtering, the specific G channel image includes the G pixel generated at a specific position after filtering, and the specific B channel image includes the B pixel generated at a specific position after filtering. Here, the specific position is not limited. Specifically, the color pixels (R pixels, G pixels, B pixels) of different colors in the original image can be filtered through the full-size full-color channel image to generate R pixels, G pixels, and B pixels at specific positions.

[0048] Optionally, a bilateral filtering method can be used to generate a color channel image corresponding to each color pixel. The bilateral filtering is mainly to perform a smoothing process on the flat area. The processing process is as follows:

[0049]

[0050] Among them, Ω represents a local window, which can be 7 by 7 or other sizes. q represents the coordinate position of the pixel, and Iq represents the pixel value in the window before filtering. f represents the weight of each coordinate point in the 9×9 window, which is fixed. The closer to the center, the greater the weight. g represents the weight of the difference between the pixel at other positions and the central pixel. The greater the difference, the smaller the weight. p is the position to be calculated, and Jp is the pixel value to be calculated in a certain channel.

[0051] In the local window represented by Ω with p as the center, find the coordinate q of the original value of the channel to be found, Iq is its intensity value, kp is the number of original values ​​of the channel to be found, and Jp is equal to the weighted average of all Iq in the local window Ω. The weight distance corresponding to Iq can be calculated by the f function, and the intensity difference weight corresponding to each Iq can be calculated by the g function. The f function is a distance function, and the closer to the center, the greater the weight; g is an intensity difference function, and the greater the intensity difference, the smaller the weight.

[0052] like Figure 3 , which is a schematic diagram of an output image in a bayer format after three channels are integrated and outputted after bilateral filtering calculation in an embodiment. Based on the full-size full-color channel image 302, bilateral filtering is used to interpolate the color pixels of different colors in the original image 304 to obtain an R channel image 306, a G channel image 308, and a B channel image 310. The R channel image 306, the G channel image 308, and the B channel image 310 are fused to obtain a target image 312.

[0053] like Figure 4 As shown, taking the B pixel as an example, the pixel position of the B pixel to be calculated is (i, j), and in the original image, the n×n window 402 with the pixel position (i, j) as the center is used. Based on the above distance weight function f, the distance weight F of the B pixel in the n×n window 402 is calculated, and F is an n×n matrix. In the full-size full-color channel image, the n×n window 404 with the pixel position (i, j) as the center is used. Based on the above intensity difference function g, the intensity difference weight G of the panchromatic pixel in the n×n window is calculated. J is an n×n panchromatic pixel window, and I is an n×n B pixel window. Where there is no B pixel, the matrix value is 0.

[0054] For each B pixel in each n×n window 402, the distance weight F between each B pixel and the B pixel at (i, j) is calculated (which can be regarded as a fixed weight template), the panchromatic pixel corresponding to each B pixel in the window 404 is determined, and the intensity difference weight G between each pixel position and the panchromatic pixel at (i, j) is calculated. HF is the product weight of G and F at the B pixel position, mosaicB is the position matrix of the original B pixels, and the pixel value B(i, j) of the B pixel at the pixel position (i, j) in the target image 406 can be calculated according to the following formula:

[0055] HF=G.*F.*mosaicB

[0056] meanW = sum(sum(HF.*J))

[0057] meanB = sum(sum(HF.*I))

[0058] B(i,j)=W(i,j)*meanB / meanW.

[0059] In the above, the bilateral filtering method is used to interpolate the panchromatic pixels in the W channel image and the color pixels in the original image. In the process of generating the color channel images corresponding to each color pixel, for each pixel position (i, j) to be interpolated, the intensity difference weight between the target panchromatic pixel in the full-size panchromatic channel image and other panchromatic pixels in the full-size panchromatic channel image corresponding to the pixel position (i, j) is considered. The distance weight between the pixels of the same color around the pixel position (i, j) in the original image is also considered. The pixels of the same color refer to the pixels of the same color as the pixels to be interpolated at the pixel position (i, j). Therefore, the pixel difference weight here can be used to adjust the pixel value of the pixel at the pixel position (i, j) to be interpolated from the perspective of the full-size panchromatic channel image, and then combined with the distance weight, the accuracy of the pixel value of the pixel at the pixel position (i, j) to be interpolated can be improved.

[0060] Step 260: Generate a target image based on each specific color channel image.

[0061] After filtering the color pixels of different colors in the original image through the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors, the specific R channel image, the specific G channel image, and the specific B channel image are fused to obtain a target image. The target image may be a Bayer array image.

[0062] Specifically, a specific R channel image includes R pixels generated at a specific position after filtering, a specific G channel image includes G pixels generated at a specific position after filtering, and a specific B channel image includes B pixels generated at a specific position after filtering. Therefore, by fusing a specific R channel image, a specific G channel image, and a specific B channel image, a target image in which R pixels, G pixels, and B pixels are arranged in a specific Bayer array can be obtained. For example, a target image in which a Bayer array such as RGGB / GRBG / GBRG / BGGR is arranged can be obtained. In the present application, the arrangement type of the Bayer array is not limited, and other pixel arrays other than the Bayer array can also be included.

[0063] Combination Figure 5 As shown, it is a schematic diagram of the image processing flow in one embodiment. First, the original image 502 is obtained by the image sensor Sensor; secondly, the original image obtained by the image sensor is interpolated (interpolation) to generate a full-size full-color channel image 504. Then, the color pixels of different colors in the original image can be filtered through the full-size full-color channel image to generate a specific R channel image 506, a specific G channel image 508, and a specific B channel image 510. Finally, based on the specific R channel image 506, the specific G channel image 508, and the specific B channel image 510, fusion (Fusion) is performed to generate a target image 512.

[0064] The image processing method in this embodiment first performs interpolation processing on the original image acquired by the image sensor to generate a full-size full-color channel image. The full-size full-color channel image obtained by interpolation can greatly improve the brightness of the image, making the picture layer clearer and the picture more transparent. Therefore, the color pixels of different colors in the original image can be filtered through the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors. Finally, based on each specific color channel image, a target image is generated, thereby improving the clarity of the target image finally obtained and making the resolution higher.

[0065] In one embodiment, Figure 6 As shown, step 220, interpolation processing is performed on the original image acquired by the image sensor to generate a full-size full-color channel image, including:

[0066] Step 222, acquiring the original image through the image sensor, and traversing each pixel in the original image;

[0067] Step 224, when it is determined that the current pixel of the original image is a color pixel, interpolate the current pixel into a new full-color pixel based on each full-color pixel in the first preset area including the color pixel;

[0068] Step 226, until all color pixels in the original image are interpolated into new full-color pixels, a full-size full-color channel image is generated.

[0069] Specifically, an original image is acquired through an image sensor, and then, when converting the original image into a full-size full-color channel image, first, each pixel in the original image is traversed to determine whether the current pixel is a color pixel; when it is determined that the current pixel of the original image is a color pixel, a first preset area containing color pixels is acquired around the color pixel, for example, an area of ​​a preset size centered on the color pixel is used as the first preset area. Figure 5 As shown, in the calculation of the color pixel a ( Figure 5 When the pixel value of the full-color pixel at the position of the color pixel a (not shown in the figure) is obtained, an 11*11 area centered on the color pixel a is obtained from the original image as the first preset area. The 11*11 area represents an area where 11 pixels are arranged in sequence in the horizontal direction and 11 pixels are also arranged in sequence in the vertical direction. Of course, the present application does not limit the preset size, and other sizes can also be selected, for example, 5*5, 9*9, 15*15, etc.

[0070] Secondly, the pixel values ​​of the full-color pixels are obtained from the first preset area. Here, the pixel values ​​of all full-color pixels in the first preset area may be obtained, or the pixel values ​​of some full-color pixels may be obtained. This application does not limit this. This application does not limit the number of full-color pixels obtained, which may be one or more.

[0071] Again, the current pixel is interpolated into a new full-color pixel based on each full-color pixel in the first preset area including color pixels. If the pixel values ​​of all full-color pixels are obtained from the first preset area, the weighted average of the pixel values ​​of all full-color pixels in the first preset area is calculated to generate the pixel value of the new full-color pixel, and the new full-color pixel is the pixel value of the current pixel. If the pixel values ​​of some full-color pixels are obtained from the first preset area, the weighted average of the pixel values ​​of the some full-color pixels in the first preset area is calculated to generate the pixel value of the new full-color pixel, and the new full-color pixel is the pixel value of the current pixel. This application does not limit this.

[0072] Finally, after calculating the pixel value of the current pixel in the current traversal, the next traversal is performed until all color pixels in the original image are interpolated into new full-color pixels to generate a full-size full-color channel image.

[0073] By traversing each pixel in the original image, when it is determined that the current pixel of the original image is a color pixel, for each color pixel in the original image, the current pixel is interpolated into a new full-color pixel based on each full-color pixel in the first preset area containing the color pixel. In this way, the original image 502 is converted into a full-size full-color channel image 504.

[0074] The image processing method in this embodiment, for each color pixel in the original image, when calculating the pixel value of the full-color pixel at the position where the color pixel is located, first, the original image is acquired through the image sensor, and each pixel in the original image is traversed. Secondly, when it is determined that the current pixel of the original image is a color pixel, the current pixel is interpolated as a new full-color pixel based on each full-color pixel in the first preset area containing the color pixel. Finally, the loop is traversed until all the color pixels in the original image are interpolated as new full-color pixels to generate a full-size full-color channel image, so that the original image is converted into a full-size full-color channel image. Based on one or more full-color pixels in the first preset area containing the color pixels, new full-color pixels can be accurately interpolated. Thereby, the accuracy of the generated full-size full-color channel image is improved.

[0075] In one embodiment, Figure 7 As shown, step 224, for each color pixel in the original image, interpolation processing is performed on the pixel values ​​of the panchromatic pixels in the first preset area containing the color pixels to generate a new panchromatic pixel value for replacing the color pixel, including:

[0076] Step 224a, for each color pixel in the original image, obtaining a region type of a second preset region containing the color pixel; the region type includes a flat region and a non-flat region;

[0077] Step 224b, according to the region type, interpolate the color pixels into new panchromatic pixels based on the panchromatic pixels in the first preset region.

[0078] Specifically, for each color pixel in the original image, when calculating the pixel value of the full-color pixel at the position where the color pixel is located, first, a second preset area containing color pixels is obtained around the color pixel, and the pixels in the second preset area can reflect the area type of the area, that is, the area type of the second preset area can be determined according to the pixels in the second preset area. Among them, the position and size of the second preset area can be the same as the first preset area, or different from the first preset area, and this application does not limit this. For example, a region of a preset size centered on the color pixel is used as the second preset area. When calculating the pixel value of the full-color pixel at the position where the color pixel a is located in the original image 502, a 5*5 region centered on the color pixel a is obtained from the original image as the first preset area. Among them, the 5*5 region represents an area where 5 pixels are arranged in sequence in the horizontal direction and 5 pixels are also arranged in sequence in the vertical direction. Of course, this application does not limit the preset size, and other sizes can also be selected, such as 4*4, 5*7, etc.

[0079] The region types here are mainly reflected in different texture directions, where the region types include flat regions and non-flat regions. Flat regions have no specific texture direction, while non-flat regions have specific texture directions. Different types of regions have different texture directions, and the pixel arrangements in images with different texture directions have different characteristics. For example, the pixel values ​​of pixels in an image with horizontal textures are more similar in the same horizontal direction, while on the contrary, the pixel values ​​of pixels in an image with vertical textures are more similar in the same vertical direction. Therefore, when generating pixel values ​​of new full-color pixels used to replace color pixels, different region types correspond to different calculation methods.

[0080] Then, after the calculation method of the pixel value of the new full-color pixel is determined according to the region type, the color pixel can be interpolated into a new full-color pixel based on the full-color pixels in the first preset region. Specifically, the pixel value of each full-color pixel in the first preset region including the color pixel is interpolated to generate the pixel value of the new full-color pixel for replacing the color pixel. For example, according to the region type, the target full-color pixel for calculating the pixel value of the new full-color pixel is determined from the first preset region, and the interpolation weight of the target full-color pixel is determined. Then, based on the target full-color pixel and the interpolation weight of the target full-color pixel, the pixel value of the new full-color pixel is calculated.

[0081] The image processing method in this embodiment obtains the region type of the second preset region containing the color pixel for each color pixel in the original image. Among them, different types of regions have different texture directions, and the pixel arrangement in images with different texture directions has different characteristics. Therefore, different calculation methods are used according to the region type, and the color pixels are interpolated into new full-color pixels based on the full-color pixels in the first preset region. It is achieved that new full-color pixels can be accurately interpolated for different types of regions, thereby improving the accuracy of the full-size full-color channel image finally obtained.

[0082] In one embodiment, for each color pixel in the original image, obtaining the region type of the second preset region containing the color pixel includes:

[0083] For each color pixel in the original image, obtaining a second preset area containing the color pixel;

[0084] Calculating the standard deviation of each panchromatic pixel according to the pixel values ​​of the panchromatic pixels in the second preset area, and using the standard deviation as the area type characteristic value of the second preset area;

[0085] If the area type characteristic value is less than the first preset threshold, the area type of the second preset area containing color pixels is determined to be a flat area; if the area type characteristic value is greater than or equal to the first preset threshold, the area type of the second preset area containing color pixels is determined to be a non-flat area.

[0086] Specifically, for each color pixel in the original image, first, a second preset area containing color pixels is obtained around the color pixel. Secondly, the region type feature value is calculated according to the pixel values ​​of the panchromatic pixels in the second preset area. For example, the pixel mean of the panchromatic pixels in the second preset area is calculated, and the standard deviation of each panchromatic pixel is calculated according to the pixel values ​​of each panchromatic pixel in the second preset area and the pixel mean of the panchromatic pixels, and the standard deviation is used as the region type feature value of the second preset area. The region type feature value is used to characterize whether the second preset area is a flat area or a non-flat area.

[0087] Thirdly, it is determined whether the region type characteristic value is less than the first preset threshold value. If the region type characteristic value is less than the first preset threshold value, the region type of the second preset region containing the color pixels is determined to be a flat region. Finally, if the region type characteristic value is greater than or equal to the first preset threshold value, the region type of the second preset region containing the color pixels is determined to be a non-flat region. The first preset threshold value is a threshold value set according to an empirical value, which is used to distinguish between a flat region and a non-flat region, and this application does not limit this.

[0088] The image processing method in this embodiment obtains a second preset area containing color pixels for each color pixel in the original image. The standard deviation of each panchromatic pixel is calculated based on the pixel value of the panchromatic pixel in the second preset area, and the standard deviation is used as the area type characteristic value of the second preset area. Then, based on the size relationship between the area type characteristic value and the first preset threshold, it is possible to accurately determine whether the area type of the second preset area containing color pixels is a flat area or a non-flat area. Furthermore, it is convenient to interpolate the color pixels into new panchromatic pixels based on the panchromatic pixels in the first preset area according to the area type. It is achieved that new panchromatic pixels can be accurately generated for different types of areas, thereby improving the accuracy of the full-size panchromatic channel image finally obtained.

[0089] In one embodiment, calculating the standard deviation of each panchromatic pixel according to the pixel value of the panchromatic pixel in the second preset area, and taking the standard deviation as the area type characteristic value of the second preset area includes:

[0090] Calculating a pixel mean of panchromatic pixels within a second preset area including color pixels;

[0091] The standard deviation of each panchromatic pixel is calculated according to the pixel value of each panchromatic pixel in the second preset area including the color pixels and the pixel mean of the panchromatic pixels, and the standard deviation is used as the area type characteristic value of the second preset area.

[0092] Among them, calculating the pixel mean of the full-color pixels within the second preset area including the color pixels may be calculating the pixel mean of all the full-color pixels within the second preset area, or may be calculating the pixel mean Wmean of some of the full-color pixels within the second preset area, and this application does not impose any limitation on this.

[0093] like Figure 8As shown, when calculating the pixel value of the panchromatic pixel at the position of the color pixel a in the original image 802, a 3*5 area with the color pixel a as the center and along the horizontal and vertical directions is obtained from the original image as the second preset area. Among them, the 3*5 area means an area where 5 pixels are arranged in sequence in the horizontal direction and 3 pixels are also arranged in sequence in the vertical direction. It is also possible to calculate the pixel value of the panchromatic pixel at the position of the color pixel a in the original image 804, and then obtain the 5*3 area with the color pixel a as the center and along the horizontal and vertical directions from the original image as the second preset area. Among them, the 5*3 area means an area where 3 pixels are arranged in sequence in the horizontal direction and 5 pixels are also arranged in sequence in the vertical direction. It is also possible to calculate the pixel value of the panchromatic pixel at the position of the color pixel a in the original image 606, and then obtain the 2*4 area with the color pixel a as the center and along the anti-diagonal direction from the original image as the second preset area. Among them, a 2*4 area represents an area where 2 pixels are arranged in sequence along the diagonal direction and 4 pixels are arranged in sequence along the anti-diagonal direction. Of course, the present application does not limit the preset size, and other sizes can also be selected, for example, 4*4, 5*7, etc.

[0094] Then, the quotient W / Wmean between the pixel value W of each full-color pixel in the second preset area and the pixel mean Wmean is calculated, and then the standard deviation of all W / Wmean is calculated. The standard deviation is used as the area type characteristic value of the second preset area, and the area type characteristic value of the second preset area is obtained.

[0095] The image processing method in this embodiment calculates the region type characteristic value of the second preset region based on the pixel values ​​of the panchromatic pixels in the second preset region. Specifically, the pixel mean of the panchromatic pixels in the second preset region including the color pixels is calculated, and then the standard deviation of each panchromatic pixel is calculated based on the pixel values ​​of each panchromatic pixel in the second preset region including the color pixels and the pixel mean of the panchromatic pixels, and the standard deviation is used as the region type characteristic value of the second preset region of the second preset region. In this way, based on the degree of difference between the pixel values ​​of each panchromatic pixel and the pixel mean of the panchromatic pixel, the region type characteristic value of the second preset region can be accurately calculated, and then the second preset region can be accurately determined to be a flat region or a non-flat region.

[0096] In one embodiment, according to the region type, interpolating color pixels into new panchromatic pixels based on panchromatic pixels in the first preset region includes:

[0097] If the area type is a flat area, determining a target full-color pixel in a preset area adjacent to the color pixel in the horizontal direction and the vertical direction from the full-color pixels in the first preset area;

[0098] The pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel.

[0099] Specifically, for each color pixel in the original image, the standard deviation of each panchromatic pixel is calculated according to the pixel value of the panchromatic pixel in the second preset area, and the standard deviation is used as the area type characteristic value of the second preset area; if the area type characteristic value is less than the first preset threshold, it is determined that the area type of the second preset area containing the color pixels is a flat area.

[0100] If the area type is a flat area, a target panchromatic pixel in a preset area adjacent to the color pixel in the horizontal direction and the vertical direction is determined from the panchromatic pixels in the first preset area including the color pixel. Fig. 9 As shown, when calculating the pixel value of the panchromatic pixel at the position of the color pixel a in the original image 902, the region type feature value calculated based on the pixel value of the panchromatic pixel in the second preset region including the color pixel a is less than the first preset threshold value, and the region type of the second preset region including the color pixel is determined to be a flat region. Since the flat region has no specific texture direction, that is, the pixel values ​​of the pixels in the flat region are relatively similar, therefore, from the panchromatic pixels in the first preset region including the color pixel a, the panchromatic pixels (W1 pixels, W2 pixels, W3 pixels, and W4 pixels) in the preset region adjacent to the color pixel in the horizontal direction and the vertical direction are determined as the target panchromatic pixels.

[0101] Then, the pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel. For example, the pixel value of the new panchromatic pixel is calculated using the following formula:

[0102] out(i,j)=(W1+W2+W3+W4) / 4;

[0103] Wherein, (i, j) is the pixel position of the B pixel, out(i, j) is the pixel value of the new full-color pixel, W1 is the pixel value of the full-color pixel above the B pixel, W2 is the pixel value of the full-color pixel below the B pixel, W3 is the pixel value of the full-color pixel to the left of the B pixel, and W4 is the pixel value of the full-color pixel to the right of the B pixel. Here, the interpolation weight of each target full-color pixel is 1 / 4. Of course, in other embodiments, the interpolation weight may not be 1 / 4, and this application does not limit this.

[0104] In the image processing method of this embodiment, for the case where the area type of the second preset area is a flat area, since the flat area has no specific texture direction, the pixel values ​​of the pixels in the flat area are relatively similar. Therefore, the target panchromatic pixel of the preset area adjacent to the color pixel in the horizontal direction and the vertical direction is determined from the panchromatic pixels in the first preset area. The pixel value of the new panchromatic pixel is calculated based on the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel. In this way, the pixel value of the new panchromatic pixel is calculated based on the target panchromatic pixel of the preset area adjacent to the color pixel in the horizontal direction and the vertical direction, thereby improving the accuracy of the pixel value of the new panchromatic pixel generated in the flat area.

[0105] In one embodiment, step 224b, based on the region type, interpolating the pixel values ​​of the panchromatic pixels in the first preset region including the color pixels to generate the pixel values ​​of the new panchromatic pixels for replacing the color pixels, includes:

[0106] If the region type is a non-flat region, the texture direction of the color pixel is calculated;

[0107] Along the texture direction of the color pixel, determine, from the panchromatic pixels in the first preset area including the color pixel, a target panchromatic pixel in a preset area adjacent to the color pixel in the texture direction;

[0108] The pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel.

[0109] Specifically, for each color pixel in the original image, the region type feature value is calculated according to the pixel value of the panchromatic pixel in the second preset area containing the color pixel. The region type feature value can also be calculated according to the pixel value of the color pixel in the second preset area containing the color pixel, which is not limited in the present application. If the region type feature value is greater than or equal to the first preset threshold value, the region type of the second preset area containing the color pixel is determined to be a non-flat area. The non-flat area has a specific texture direction, wherein different types of non-flat areas have different texture directions, and the pixel arrangement in images with different texture directions has different characteristics. For example, the pixel values ​​of pixels in an image with horizontal texture in the same horizontal direction are more similar, and conversely, the pixel values ​​of pixels in an image with vertical texture in the same vertical direction are more similar. Therefore, if the region type is a non-flat area, the texture direction of the color pixel needs to be calculated.

[0110] After calculating the texture direction of the color pixel, along the texture direction of the color pixel, a target full-color pixel in a preset area adjacent to the color pixel in the texture direction is determined from the full-color pixels in a first preset area containing the color pixel. Fig.10As shown in (a), the texture direction of the B (blue) pixel (shown at position a in the figure) on the original image 1002 is horizontal. Then, along the horizontal direction, the full-color pixels in the first preset area including the B pixel and the full-color pixels in the preset area adjacent to the B pixel in the horizontal direction (W3 pixel, W4 pixel) are determined as the target full-color pixels.

[0111] Then, the pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel. For example, the pixel value of the new panchromatic pixel is calculated using the following formula:

[0112] out(i,j)=(W3+W4) / 2;

[0113] Wherein, (i, j) is the pixel position of the B pixel, out(i, j) is the pixel value of the new full-color pixel, W3 is the pixel value of the full-color pixel to the left of the B pixel, and W4 is the pixel value of the full-color pixel to the right of the B pixel. Here, the interpolation weight of each target full-color pixel is 1 / 2. Of course, in other embodiments, the interpolation weight may not be 1 / 2, and this application does not limit this.

[0114] Combination Fig.10 As shown in (b), the texture direction of the B (blue) pixel (shown at position a in the figure) on the original image 1002 is the vertical direction. Then, along the vertical direction, the full-color pixels in the first preset area including the B pixel and the full-color pixels in the preset area adjacent to the B pixel in the horizontal direction (W1 pixel, W2 pixel) are determined as the target full-color pixels.

[0115] Then, the pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel. For example, the pixel value of the new panchromatic pixel is calculated using the following formula:

[0116] out(i,j)=(W1+W2) / 2;

[0117] Wherein, (i, j) is the pixel position where the B pixel is located, out(i, j) is the pixel value of the new full-color pixel, W1 is the pixel value of the full-color pixel above the B pixel, and W2 is the pixel value of the full-color pixel below the B pixel. Here, the interpolation weight of each target full-color pixel is 1 / 2. Of course, in other embodiments, the interpolation weight may not be 1 / 2, and this application does not limit this.

[0118] Combination Fig.10As shown in (c), the texture direction of the B (blue) pixel (shown in position a in the figure) on the original image 1002 is the anti-diagonal direction. Then, along the anti-diagonal direction, the full-color pixels in the first preset area including the B pixel and the adjacent preset area full-color pixels in the anti-diagonal direction (W1 pixel, W2 pixel, W3 pixel, W4 pixel, W5 pixel, W6 pixel, W7 pixel, W8 pixel) are determined as the target full-color pixels.

[0119] Then, the pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel. For example, the pixel value of the new panchromatic pixel is calculated using the following formula:

[0120] out(i,j)=(W1+2*W2+2*W3+W4+W5+2*W6+2*W7+W8) / 12;

[0121] Wherein, (i, j) is the pixel position of the B pixel, out(i, j) is the pixel value of the new full-color pixel, W1, W2, W3, W4 are the pixel values ​​of the full-color pixel at the upper left of the B pixel, and W5, W6, W7, W8 are the pixel values ​​of the full-color pixel at the lower right of the B pixel. Here, the interpolation value of each target full-color pixel is different. Of course, in other embodiments, the interpolation weight can take other values, which is not limited in this application.

[0122] Combination Fig.10 As shown in (d), the texture direction of the B (blue) pixel (shown at position a in the figure) on the original image 1002 is the main diagonal direction. Then, along the main diagonal direction, the full-color pixels in the preset area adjacent to the B pixel on the main diagonal (W1 pixel, W2 pixel, W3 pixel, W4 pixel, W5 pixel, W6 pixel, W7 pixel, W8 pixel) are determined from the full-color pixels in the first preset area including the B pixel as the target full-color pixels.

[0123] Then, the pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel. For example, the pixel value of the new panchromatic pixel is calculated using the following formula:

[0124] out(i,j)=(W1+2*W2+2*W3+W4+W5+2*W6+2*W7+W8) / 12;

[0125] Wherein, (i, j) is the pixel position of the B pixel, out(i, j) is the pixel value of the new full-color pixel, W1, W2, W3, W4 are the pixel values ​​of the full-color pixel located at the upper right of the B pixel, and W5, W6, W7, W8 are the pixel values ​​of the full-color pixel located at the lower left of the B pixel. Here, the interpolation value of each target full-color pixel is different. Of course, in other embodiments, the interpolation weight can take other values, which is not limited in this application.

[0126] The image processing method in this embodiment calculates the region type feature value according to the pixel value of the panchromatic pixel in the second preset region containing the color pixel for each color pixel in the original image. If the region type feature value is greater than or equal to the first preset threshold value, the region type of the second preset region containing the color pixel is determined to be a non-flat region. Since the non-flat region has a specific texture direction, the texture direction of the color pixel is calculated, and along the texture direction of the color pixel, the target panchromatic pixel of the preset region adjacent to the color pixel in the texture direction is determined from the panchromatic pixels in the first preset region containing the color pixel. Finally, the pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel. It is achieved that the pixel value of the new panchromatic pixel that replaces the color pixel is calculated by using different calculation methods for the color pixels with different texture directions, thereby improving the accuracy of the pixel value of the new panchromatic pixel generated in the non-flat region.

[0127] In one embodiment, calculating the texture direction of a color pixel includes:

[0128] Taking the color pixel as the center, acquiring at least one first full-color pixel pair in different preset directions;

[0129] Calculating first pixel value gradients of color pixels in different preset directions according to the difference of pixel values ​​of at least one first panchromatic pixel pair; the first pixel value gradients are used to characterize the texture direction of the color pixels;

[0130] The texture direction of the color pixel is determined according to the magnitude relationship between the first pixel value gradients in different preset directions.

[0131] Fig.11 FIG. 1 is a schematic diagram of a full-color pixel pair in each texture direction in an embodiment. Fig.11 As shown in the figure, taking an 11*11 pixel window as an example, it shows the full-color pixel pairs of color pixels (pixels at the location of the black dots in the figure) in the horizontal direction, vertical direction, anti-diagonal direction, and main diagonal direction. The full-color pixel pair is Fig.11 The arrow in the figure points to the full-color pixel.

[0132] If the preset direction is the horizontal direction, for the three panchromatic pixel pairs in the horizontal direction, the absolute value of the difference between the two panchromatic pixel pairs pointed to by the same arrow is calculated to obtain three absolute values. The weighted average value of the three absolute values ​​in the horizontal direction is calculated to obtain the first pixel value gradient DirH of the color pixel in the horizontal direction. In the same processing manner, the first pixel value gradient DirV of the color pixel in the vertical direction, the first pixel value gradient DirA of the color pixel in the anti-diagonal direction, and the first pixel value gradient DirD of the color pixel in the main diagonal direction can be obtained.

[0133] The texture direction of the color pixel is determined according to the size relationship between the first pixel value gradients in different preset directions. Specifically, the first pixel value gradients in different preset directions are sorted in order of size, and the sorting result is DirH>DirV>DirA>DirD. Then, the minimum first pixel value gradient DirD and the second smallest first pixel value gradient DirA can be obtained from the sorting result. The texture direction of the color pixel is determined according to the size relationship between the minimum first pixel value gradient DirD and the second smallest first pixel value gradient DirA. That is, only when the difference between the minimum first pixel value gradient DirD and the second smallest first pixel value gradient DirA meets a certain threshold, the minimum first pixel value gradient DirD can be used as the texture direction of the color pixel.

[0134] In the image processing method of this embodiment, when determining the texture direction of a color pixel, at least one first full-color pixel pair is obtained in a preset direction with the color pixel as the center. Then, a weighted average value of the difference between the pixel values ​​of the at least one first full-color pixel pair is calculated, and the weighted average value is used as the first pixel value gradient of the color pixel in the preset direction. At this time, the preset direction corresponding to the smallest first pixel value gradient is not directly determined as the texture direction of the color pixel, but the texture direction of the color pixel is determined by combining the size relationship between the smallest first pixel value gradient and the second smallest first pixel value gradient. Thereby, the accuracy of the determined texture direction of the color pixel is further improved.

[0135] In one embodiment, determining the texture direction of the color pixel according to the magnitude relationship between the smallest first pixel value gradient and the second smallest first pixel value gradient includes:

[0136] Determine whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets a preset condition;

[0137] If so, the texture direction of the color pixel is determined to be a preset direction corresponding to the minimum first pixel value gradient.

[0138] Specifically, when judging whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets the preset condition, first, judge whether the brightness of the area where the color pixel is located is greater than the first preset brightness threshold. If the brightness of the area where the color pixel is located is greater than the first preset brightness threshold, judge whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets the first preset condition. Among them, the first preset brightness threshold can be set to 200 candelas / square meter (cd / m2), and this application is not limited to this. If the brightness of the area where the color pixel is located is greater than 200 candelas / square meter (cd / m2), at this time, it is determined that the area where the color pixel is located is a bright area.

[0139] If the brightness of the area where the color pixel is located is less than or equal to the first preset brightness threshold, it is determined whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets the second preset condition. If the brightness of the area where the color pixel is located is less than or equal to 200 candelas per square meter (cd / m2), then it is determined that the area where the color pixel is located is a dark area.

[0140] Furthermore, for color pixels in the bright area, the difference between the calculated minimum first pixel value gradient and the second smallest first pixel value gradient can be set to be greater than the difference between the calculated minimum first pixel value gradient and the second smallest first pixel value gradient for color pixels in the dark area. Because the pixel values ​​of pixels in the bright area vary greatly, only when the difference between the calculated minimum first pixel value gradient and the second smallest first pixel value gradient is large, can the texture direction of the color pixel be accurately determined to be the preset direction corresponding to the minimum first pixel value gradient.

[0141] For example, for color pixels in the bright area, the first preset condition is DirA>a*DirD; for color pixels in the dark area, the second preset condition is DirA>DirD+b. Wherein, a and b can be set according to actual conditions, and this application does not limit this.

[0142] The image processing method in this embodiment, when determining the texture direction of the color pixel according to the size relationship between the smallest first pixel value gradient and the second smallest first pixel value gradient, first divides the color pixel into color pixels in the bright area and color pixels in the dark area. Then, for the two types of color pixels, it is judged whether they meet the corresponding different preset conditions. If so, the texture direction of the color pixel is determined to be the preset direction corresponding to the smallest first pixel value gradient. The color pixels are divided into color pixels in the bright area and color pixels in the dark area based on brightness, and then it is specifically judged whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets different preset conditions. Thus, the texture direction of the color pixel can be obtained more accurately.

[0143] In one embodiment, an image processing method is provided, further comprising:

[0144] If the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient does not meet the preset condition, taking the color pixel as the center, obtaining at least one second full-color pixel pair in a preset direction; the number of the second full-color pixel pairs is greater than the number of the first full-color pixel pairs, and the distance between the second full-color pixel pairs is greater than the distance between the first full-color pixel pairs;

[0145] Calculating a second pixel value gradient of a color pixel in a preset direction based on a difference in pixel values ​​of at least one second panchromatic pixel pair;

[0146] The texture direction of the color pixel is determined according to the magnitude relationship between the second pixel value gradients in different preset directions.

[0147] Specifically, if the brightness of the area where the color pixel is located is greater than the first preset brightness threshold, the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient does not meet the first preset condition. If the brightness of the area where the color pixel is located is less than or equal to the first preset brightness threshold, the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient does not meet the second preset condition. Both of the above situations are situations where the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient does not meet the preset condition.

[0148] For the above two situations, at least one second full-color pixel pair is obtained in a preset direction with the color pixel as the center; the number of second full-color pixel pairs is greater than the number of first full-color pixel pairs, and the distance between the second full-color pixel pairs is greater than the distance between the first full-color pixel pairs.

[0149] Fig.12 FIG. 1 is a schematic diagram of a full-color pixel pair in each texture direction in another embodiment. Fig.12 As shown in the figure, taking a 11*11 pixel window as an example, it shows the full-color pixel pairs of the color pixel (the pixel at position a in the figure) in the horizontal direction, vertical direction, anti-diagonal direction, and main diagonal direction. The full-color pixel pair is Fig.12 The arrow in the figure points to the full-color pixel.

[0150] If the preset direction is the horizontal direction, then for the five full-color pixel pairs in the horizontal direction, the absolute value of the difference between the two full-color pixel pairs pointed to by the same arrow is calculated to obtain five absolute values. The weighted average of the five absolute values ​​in the horizontal direction is calculated to obtain the first pixel value gradient DirH of the color pixel in the horizontal direction. According to the same processing method, the first pixel value gradient DirV of the color pixel in the vertical direction, the first pixel value gradient DirA of the color pixel in the anti-diagonal direction, and the first pixel value gradient DirD of the color pixel in the main diagonal direction can be obtained. Because Fig.12 The number of full-color pixel pairs collected by this method is greater than that in Fig.11 This method collects a large number of full-color pixel pairs, so Fig.12 This method is called determining the texture direction of color pixels using high frequency. Fig.11 This method is called determining the texture direction of color pixels in a low-frequency manner.

[0151] Then, according to the size relationship between the first pixel value gradients in different preset directions, the texture direction of the color pixel is determined. Specifically, the first pixel value gradients in different preset directions are sorted in order of size, and the sorting result is DirH>DirV>DirA>DirD. Then, the minimum first pixel value gradient DirD and the second smallest first pixel value gradient DirA can be obtained from the sorting result. According to the size relationship between the minimum first pixel value gradient DirD and the second smallest first pixel value gradient DirA, the texture direction of the color pixel is determined. That is, only when the difference between the minimum first pixel value gradient DirD and the second smallest first pixel value gradient DirA meets a certain threshold, the minimum first pixel value gradient DirD can be used as the texture direction of the color pixel.

[0152] Specifically, when judging whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets the preset condition, first, judge whether the brightness of the area where the color pixel is located is greater than the first preset brightness threshold. If the brightness of the area where the color pixel is located is greater than the first preset brightness threshold, judge whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets the first preset condition. Among them, the first preset brightness threshold can be set to 200 candelas / square meter (cd / m2), and this application is not limited to this. If the brightness of the area where the color pixel is located is greater than 200 candelas / square meter (cd / m2), at this time, it is determined that the area where the color pixel is located is a bright area.

[0153] If the brightness of the area where the color pixel is located is less than or equal to the first preset brightness threshold, it is determined whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets the second preset condition. If the brightness of the area where the color pixel is located is less than or equal to 200 candelas per square meter (cd / m2), then it is determined that the area where the color pixel is located is a dark area.

[0154] Furthermore, for color pixels in the bright area, the difference between the calculated minimum first pixel value gradient and the second smallest first pixel value gradient can be set to be greater than the difference between the calculated minimum first pixel value gradient and the second smallest first pixel value gradient for color pixels in the dark area. Because the pixel values ​​of pixels in the bright area vary greatly, only when the difference between the calculated minimum first pixel value gradient and the second smallest first pixel value gradient is large, can the texture direction of the color pixel be accurately determined to be the preset direction corresponding to the minimum first pixel value gradient.

[0155] For example, for color pixels in the bright area, the first preset condition is DirA>a*DirD; for color pixels in the dark area, the second preset condition is DirA>DirD+b. Wherein, a and b can be set according to actual conditions, and this application does not limit this.

[0156] In the image processing method of this embodiment, when it is determined that the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient does not meet the preset condition, the texture direction of the color pixel is determined by the following method: taking the color pixel as the center, at least one second full-color pixel pair is obtained in the preset direction. Then, the weighted average value of the difference between the pixel values ​​of the at least one second full-color pixel pair is calculated, and the weighted average value is used as the second pixel value gradient of the color pixel in the preset direction. At this time, the preset direction corresponding to the smallest second pixel value gradient is not directly determined as the texture direction of the color pixel, but the texture direction of the color pixel is determined by combining the size relationship between the smallest second pixel value gradient and the second smallest second pixel value gradient. Thereby, the accuracy of the determined texture direction of the color pixel is further improved.

[0157] Continuing from the previous embodiment, an image processing method is provided, further comprising:

[0158] If the difference between the smallest second pixel value gradient and the second smallest second pixel value gradient does not meet the preset condition, the pixel value of each new panchromatic pixel is calculated based on the pixel mean of each color pixel and the pixel mean of the panchromatic pixel in the first preset area including the color pixels.

[0159] Specifically, if the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient does not meet the preset condition, at least one second full-color pixel pair is acquired in a preset direction with the color pixel as the center. Based on the difference in pixel values ​​of the at least one second full-color pixel pair, the second pixel value gradient of the color pixel in the preset direction is calculated. At this time, if the difference between the smallest second pixel value gradient and the second smallest second pixel value gradient still does not meet the preset condition, it means that the texture direction of the RGBW image is difficult to determine and is a special texture direction.

[0160] At this time, the pixel value of each new panchromatic pixel can be calculated based on the pixel mean of each color pixel in the first preset area containing the color pixels and the pixel mean of the panchromatic pixel. First, for each color pixel, based on the pixel value of the color pixel in the first preset area containing the color pixel and the weight matrix of the color pixel, the pixel mean of the color pixel in the first preset area containing the color pixel is calculated. Among them, for the R pixel in the RGBW image, the weight matrix of the R pixel is a matrix obtained by assigning weights to the positions of the R pixels in the RGBW image. Similarly, the weight matrix of the G pixel, the weight matrix of the B pixel and the weight matrix of the panchromatic pixel can be obtained.

[0161] Secondly, based on the pixel values ​​of the panchromatic pixels in the first preset area including the color pixels and the weight matrix of the panchromatic pixels, the pixel mean of the panchromatic pixels in the first preset area including the color pixels is calculated. Among them, for the panchromatic pixels in the RGBW image, the weight matrix of the panchromatic pixels is a matrix obtained by assigning weights to the positions of the panchromatic pixels in the RGBW image.

[0162] Finally, the pixel value of each new panchromatic pixel is calculated according to the pixel value of the color pixel, the pixel mean of the color pixel and the pixel mean of the panchromatic pixel.

[0163] In the image processing method of this embodiment, in view of the fact that the texture direction of the RGBW image is difficult to determine and is a special texture direction, for each color pixel, the target panchromatic pixel is not determined according to the texture direction, but the pixel value of the new panchromatic pixel is directly calculated according to the pixel average of the panchromatic pixels in the first preset area containing the color pixels and the pixel average of the color pixels in the first preset area containing the color pixels. Because the number of panchromatic pixels and color pixels involved in the calculation process is large, the accuracy of the pixel value of the calculated new panchromatic pixel can also be improved.

[0164] In one embodiment, the pixel value of each new panchromatic pixel is calculated based on the pixel average value of each color pixel and the pixel average value of the panchromatic pixel in the first preset area including the color pixels, including:

[0165] For each color pixel, based on the pixel values ​​of the color pixels in the first preset area including the color pixels and the weight matrix of the color pixels, calculate the pixel mean value of the color pixels in the first preset area including the color pixels;

[0166] Calculating a pixel mean value of the panchromatic pixels in the first preset area including the color pixels based on the pixel values ​​of the panchromatic pixels in the first preset area including the color pixels and a weight matrix of the panchromatic pixels;

[0167] The pixel value of each new panchromatic pixel is calculated according to the pixel value of the color pixel, the pixel mean of the color pixel and the pixel mean of the panchromatic pixel.

[0168] Specific, combined Fig.13 As shown, taking a 11*11 pixel window as an example, Fig.13 (a) shows a schematic diagram of the weight matrix W_weight of the full-color pixel. Fig.13 (b) shows a schematic diagram of the weight matrix RGB_weight of color pixels (including RGB pixels). Fig.13 (c) shows a schematic diagram of the distribution matrix B_mask of B pixels. Wherein W_weight can be a Gaussian weight matrix, and the closer to the center, the greater the weight.

[0169] Taking the B pixel at the center of the 11*11 pixel window as an example, the formula for calculating the pixel mean G_mean of the B pixels in the first preset area including the B pixels is as follows:

[0170] G_mean=I.*RGB_weight.*B_mask / sum(sum(RGB_weight.*B_mask));

[0171] The formula for calculating the pixel mean W_mean of the panchromatic pixels in the first preset area including the color pixels is as follows:

[0172] W_mean=I.*W_weight / sum(sum(W_weight));

[0173] The formula for calculating the pixel value of each new panchromatic pixel is as follows:

[0174] out(i,j)=I(i,j)*W_weight / G_weight;

[0175] Among them, (i, j) is the pixel position of the B pixel, and I represents the pixel value matrix of the 11*11 data window in the RGBW image.

[0176] In the image processing method of this embodiment, in view of the fact that the texture direction of the RGBW image is difficult to determine and is a special texture direction, for each color pixel, the target panchromatic pixel is not determined according to the texture direction, but the pixel value of the new panchromatic pixel is directly calculated according to the pixel average of the panchromatic pixels in the first preset area containing the color pixels and the pixel average of the color pixels in the first preset area containing the color pixels. Because the number of panchromatic pixels and color pixels involved in the calculation process is large, the accuracy of the pixel value of the calculated new panchromatic pixel can also be improved.

[0177] In a specific embodiment, Fig.14 As shown, an image processing method is provided, comprising the following steps:

[0178] Step 1410, obtaining an original image through an image sensor Sensor;

[0179] Step 1420, determine whether the input pixel (i) is a full-color pixel (also called a W pixel); if so, proceed to step 1421; if not, proceed to step 1422;

[0180] Step 1430, determine whether the preset area including the input pixel is a flat area; if so, proceed; if not, proceed;

[0181] Step 1440, using a calculation method corresponding to a flat area to calculate the pixel value of a new W pixel, specifically: determining a target panchromatic pixel in a preset area adjacent to the input pixel in a horizontal direction and a vertical direction from the panchromatic pixels in a preset area including the input pixel; calculating the pixel value of a new panchromatic pixel according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel;

[0182] Step 1450, calculating the texture direction of the input pixel;

[0183] like Fig.15 As shown, step 1450 includes: step 1451, taking the color pixel as the center, acquiring at least one first full-color pixel pair (also referred to as the first W pixel pair) in a preset direction;

[0184] Step 1452, calculating a weighted average of differences in pixel values ​​of at least one first panchromatic pixel pair, and using the weighted average as a first pixel value gradient of the color pixel in a preset direction;

[0185] Step 1453, obtaining the minimum first pixel value gradient and the second minimum first pixel value gradient from the first pixel value gradients in different preset directions;

[0186] Step 1454, determining whether the brightness of the area where the color pixel is located is greater than a first preset brightness threshold;

[0187] Step 1455, if the brightness of the area where the color pixel is located is greater than the first preset brightness threshold, determine whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets the first preset condition;

[0188] Step 1456, if the brightness of the area where the color pixel is located is less than or equal to the first preset brightness threshold, determine whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets the second preset condition;

[0189] Step 1457, if yes, determine that the texture direction of the color pixel is a preset direction corresponding to the minimum first pixel value gradient;

[0190] Step 1458: if the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient does not meet the preset condition, obtaining at least one second full-color pixel pair in a preset direction with the color pixel as the center; the number of the second full-color pixel pairs is greater than the number of the first full-color pixel pairs, and the distance between the second full-color pixel pairs is greater than the distance between the first full-color pixel pairs;

[0191] Step 1459, calculating a second pixel value gradient of the color pixel in a preset direction based on a difference in pixel values ​​of at least one second panchromatic pixel pair;

[0192] Step 1460: Determine the texture direction of the color pixel according to the magnitude relationship between the second pixel value gradients in different preset directions.

[0193] Step 1470, along the texture direction of the color pixel, determine, from the panchromatic pixels in the first preset area including the color pixel, a target panchromatic pixel in a preset area adjacent to the color pixel in the texture direction;

[0194] Step 1480, calculating the pixel value of the new panchromatic pixel according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel;

[0195] Step 1490, outputting the pixel value of each new full-color pixel, and using the new full-color pixel to replace the color pixel in the original image to generate a full-size full-color channel image;

[0196] Step 1492, performing bilateral filtering on the panchromatic pixels in the W channel image and the color pixels in the original image to generate a specific R channel image, a specific G channel image, and a specific B channel image corresponding to each color pixel;

[0197] Step 1494, the specific R channel image, the specific G channel image, and the specific B channel image are fused to obtain a target image.

[0198] In the embodiment of the present application, for each color pixel in the original image, when calculating the pixel value of the panchromatic pixel at the position of the color pixel, first, the original image is obtained through the image sensor, and each pixel in the original image is traversed; secondly, when it is determined that the current pixel of the original image is a color pixel, the current pixel is interpolated as a new panchromatic pixel based on each panchromatic pixel in the first preset area containing the color pixel. Finally, according to the position of the color pixel, the pixel value of the color pixel in the original image is replaced with the pixel value of the new panchromatic pixel, so that the original image is converted into a full-size full-color channel image. Based on one or more panchromatic pixels in the first preset area containing the color pixel, the pixel value of the new panchromatic pixel used to replace the color pixel can be accurately calculated. Thus, the accuracy of the generated full-size full-color channel image is improved. Therefore, filtering processing can be performed based on the panchromatic pixels in the full-size full-color channel image and the color pixels in the original image to generate the color channel image corresponding to each color pixel. Finally, based on each color channel image, the target image is generated, thereby improving the clarity of the target image finally obtained, so that the resolution is higher.

[0199] In one embodiment, an image sensor is provided, including a filter array and a pixel array, wherein the filter array includes a minimum repeating unit, the minimum repeating unit includes a plurality of filter groups, the filter groups include color filters and panchromatic filters, each filter group includes 8 panchromatic filters and 8 color filters of the same color, and each filter group includes 4 filter units, the filter units include 2 panchromatic filters and 2 color filters; in the filter units, every 2 color filters are arranged in a first diagonal direction, and every 2 panchromatic filters are arranged in a second diagonal direction, and the first diagonal direction is different from the second diagonal direction. Wherein, each color pixel corresponds to a color filter.

[0200] Specifically, Fig.16, which is a schematic diagram of the structure of an image sensor in an embodiment. The image sensor includes a filter array 1620 and a pixel array 1630, wherein the filter array 1620 includes a plurality of minimum repeating units 1621. The minimum repeating unit 1621 includes a plurality of filter groups 1622. In this embodiment, the minimum repeating unit 1621 includes four filter groups 1622, and the four filter groups 1622 are arranged in a matrix. Each filter group 1622 includes a plurality of full-color filters 1623 and a plurality of color filters 1624. For example, each filter group 1622 may include 8 full-color filters 1623 and 8 color filters 1624, and every two full-color filters 1623 are arranged diagonally, and every two color filters 1624 are also arranged diagonally. Different color filters 1624 are included in different filter groups. Among them, in the minimum repeating unit 1621, full-color pixels account for 50%, G pixels account for 25%, and R pixels and B pixels account for 12.5%.

[0201] Similarly, the pixel array 1630 includes a plurality of minimum repeating units 1631, and the minimum repeating unit 1631 includes a plurality of pixel groups 1632, corresponding to the filter group 1622 in the minimum repeating unit 1621. In the present embodiment, the minimum repeating unit 1631 includes four pixel groups 1632, and the four pixel groups 1632 are arranged in a matrix, and each pixel group 1632 corresponds to one filter group 1622. The light transmitted through the full-color filter 1623 is projected onto the full-color pixel point 1633 to obtain a full-color pixel; the light transmitted through the color filter 1624 is projected onto the color pixel point 1634 to obtain a color pixel. The pixel array 1630 includes a plurality of full-color pixels and a plurality of color pixels, each full-color pixel corresponds to the full-color filter 1623, and each color pixel corresponds to the color filter 1624.

[0202] The minimum repeating unit is an array of 8 rows and 8 columns generated by arranging 32 full-color filters and 32 color filters in a preset arrangement. The preset arrangement is:

[0203]

[0204]

[0205] Among them, w represents a full-color filter, and r, g, and b all represent color filters.

[0206] like Fig.17As shown, the readout circuit 1740 is electrically connected to the pixel array 1730, and is used to control the exposure of the pixel array 1730 and the reading and output of the pixel value of the pixel point. The readout circuit 1740 includes a vertical driving unit 1741, a control unit 1742, a column processing unit 1743, and a horizontal driving unit 1744. The vertical driving unit 1741 includes a shift register and an address decoder. The vertical driving unit 1741 includes a readout scanning and a reset scanning function. The control unit 1742 configures the timing signal according to the operation mode, and uses a variety of timing signals to control the vertical driving unit 1741, the column processing unit 1743, and the horizontal driving unit 1744 to work together. The column processing unit 1743 may have an analog-to-digital (A / D) conversion function for converting an analog pixel signal into a digital format. The horizontal driving unit 1744 includes a shift register and an address decoder. The horizontal driving unit 1744 sequentially scans the pixel array 1730 column by column.

[0207] In the image sensor of this embodiment, each filter group contains the same color pixels. Therefore, when the generated pixel array is merged at the second level, the pixels of the same channel will not be merged across rows, so there will be no information interference. At the same time, the full-color pixels are evenly arranged in each filter group, and the brightness information can be evenly collected so that the texture direction can be accurately judged based on the full-color pixels. Furthermore, the accuracy of interpolating the original image based on the texture direction to generate a full-size full-color channel image is improved. Finally, the clarity of the obtained target image is improved, making the resolution higher.

[0208] In one embodiment, an imaging device includes a microlens, a filter, and an image sensor, wherein the microlens, the filter, and the image sensor are sequentially located on an incident light path;

[0209] The image sensor includes a filter array and a pixel array. The filter array includes a minimum repeating unit. The minimum repeating unit includes multiple filter groups. The filter groups include color filters and panchromatic filters. Each filter group includes 8 panchromatic filters and 8 color filters. Every 2 panchromatic filters are arranged diagonally, and every 2 color filters are arranged diagonally. The pixel array includes multiple panchromatic pixels and multiple color pixels. Each panchromatic pixel corresponds to a panchromatic filter, and each color pixel corresponds to a color filter.

[0210] Combination Fig.16As shown, the microlens array 1610 includes a plurality of microlenses 1611. The microlenses 1611, the sub-filters in the filter array 1620, and the pixels in the pixel array 1630 are arranged in a one-to-one correspondence. The microlenses 1611 are used to gather the incident light. The gathered light passes through the corresponding sub-filters and is then projected onto the pixels and received by the corresponding pixels. The pixels then convert the received light into electrical signals.

[0211] In the imaging device of this embodiment, each filter group contains the same color pixels. Therefore, when the generated pixel array is merged at the second level, the pixels of the same channel will not be merged across rows, so no information interference will be caused. At the same time, the full-color pixels are evenly arranged in each filter group, and the brightness information can be evenly collected so that the texture direction can be accurately judged based on the full-color pixels. Furthermore, the accuracy of interpolating the original image based on the texture direction to generate a full-size full-color channel image is improved. Finally, the clarity of the obtained target image is improved, making the resolution higher.

[0212] In one embodiment, Fig.18 Provided is an image processing device 1800, which is applied to an electronic device. The electronic device includes an image sensor, the image sensor includes a filter array and a pixel array, the filter array includes a minimum repeating unit, the minimum repeating unit includes a plurality of filter groups, the filter group includes a color filter and a panchromatic filter; in the filter group, the color filter is arranged in a first diagonal direction, the panchromatic filter is arranged in a second diagonal direction, and the first diagonal direction is different from the second diagonal direction; the pixel array includes a plurality of panchromatic pixels and a plurality of color pixels, each panchromatic pixel corresponds to a panchromatic filter, and each color pixel corresponds to a color filter;

[0213] The image processing apparatus 1800 comprises:

[0214] The first generating module 1820 is used to perform interpolation processing on the original image acquired by the image sensor to generate a full-size full-color channel image; wherein the original image includes color pixels and full-color pixels;

[0215] The second generating module 1840 is used to perform filtering processing on the color pixels of different colors in the original image of the full-size full-color channel image to generate specific color channel images corresponding to the color pixels of different colors respectively;

[0216] The third generating module 1860 is used to generate a target image based on each specific color channel image.

[0217] In one embodiment, the first generating module 1820 includes:

[0218] An original image acquisition unit, used to acquire the original image through an image sensor and traverse each pixel in the original image;

[0219] a full-color pixel generation unit, configured to interpolate the current pixel into a new full-color pixel based on the full-color pixels in the first preset area including the color pixels, when determining that the current pixel of the original image is a color pixel;

[0220] The full-size full-color channel image generating unit is used to generate a full-size full-color channel image until all color pixels in the original image are interpolated into new full-color pixels.

[0221] In one embodiment, the full-color pixel generation unit is also used to obtain the region type of a second preset region containing color pixels for each color pixel in the original image; the region type includes a flat region and a non-flat region; according to the region type, the color pixels are interpolated into new full-color pixels based on the full-color pixels in the first preset region.

[0222] In one embodiment, the full-color pixel generation unit is also used to obtain a second preset area containing color pixels for each color pixel in the original image; calculate the standard deviation of each full-color pixel based on the pixel value of the full-color pixels in the second preset area, and use the standard deviation as the area type characteristic value of the second preset area; if the area type characteristic value is less than a first preset threshold, determine that the area type of the second preset area containing color pixels is a flat area; if the area type characteristic value is greater than or equal to the first preset threshold, determine that the area type of the second preset area containing color pixels is a non-flat area.

[0223] In one embodiment, the full-color pixel generation unit is also used to determine, if the area type is a flat area, a target full-color pixel in a preset area adjacent to the color pixel in the horizontal and vertical directions from the full-color pixels in the first preset area; and calculate the pixel value of a new full-color pixel based on the pixel value of the target full-color pixel and the interpolation weight of the target full-color pixel.

[0224] In one embodiment, the full-color pixel generation unit is also used to calculate the texture direction of the color pixel if the area type is a non-flat area; along the texture direction of the color pixel, determine the target full-color pixel in the preset area adjacent to the color pixel in the texture direction from the full-color pixels in the first preset area containing the color pixel; calculate the pixel value of the new full-color pixel based on the pixel value of the target full-color pixel and the interpolation weight of the target full-color pixel.

[0225] In one embodiment, the full-color pixel generation unit is also used to obtain at least one first full-color pixel pair in different preset directions with the color pixel as the center; calculate the first pixel value gradient of the color pixel in different preset directions based on the difference in pixel values ​​of at least one first full-color pixel pair; the first pixel value gradient is used to characterize the texture direction of the color pixel; and determine the texture direction of the color pixel based on the size relationship between the first pixel value gradients in different preset directions.

[0226] In one embodiment, the preset direction includes a horizontal direction, a vertical direction, a diagonal direction, and an anti-diagonal direction; the full-color pixel generation unit is further used to calculate a weighted average of the difference in pixel values ​​of at least one first full-color pixel pair, and use the weighted average as the first pixel value gradient of the color pixel in the preset direction.

[0227] In one embodiment, the full-color pixel generation unit is also used to obtain the minimum first pixel value gradient and the second smallest first pixel value gradient from the first pixel value gradients in different preset directions; and determine the texture direction of the color pixel based on the size relationship between the minimum first pixel value gradient and the second smallest first pixel value gradient.

[0228] In one embodiment, the full-color pixel generation unit is also used to determine whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets a preset condition; if so, the texture direction of the color pixel is determined to be the preset direction corresponding to the smallest first pixel value gradient.

[0229] In one embodiment, the full-color pixel generation unit is further used to determine whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets a first preset condition if the brightness of the area where the color pixel is located is greater than a first preset brightness threshold.

[0230] In one embodiment, the preset conditions include a second preset condition, and the full-color pixel generation unit is also used to determine whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets the second preset condition if the brightness of the area where the color pixel is located is less than or equal to the first preset brightness threshold.

[0231] In one embodiment, the full-color pixel generation unit is further used to obtain at least one second full-color pixel pair in a preset direction with the color pixel as the center if the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient does not meet a preset condition; the number of second full-color pixel pairs is greater than the number of first full-color pixel pairs, and the distance between the second full-color pixel pairs is greater than the distance between the first full-color pixel pairs; based on the difference in pixel values ​​of at least one second full-color pixel pair, calculate the second pixel value gradient of the color pixel in the preset direction; and determine the texture direction of the color pixel according to the size relationship between the second pixel value gradients in different preset directions.

[0232] In one embodiment, the full-color pixel generation unit is also used to obtain the minimum second pixel value gradient and the second minimum second pixel value gradient from the second pixel value gradients in different preset directions; and determine the texture direction of the color pixel based on the size relationship between the minimum second pixel value gradient and the second minimum second pixel value gradient.

[0233] In one embodiment, the full-color pixel generation unit is also used to determine whether the difference between the smallest second pixel value gradient and the second smallest second pixel value gradient meets a preset condition; if so, the texture direction of the color pixel is determined to be the preset direction corresponding to the smallest second pixel value gradient.

[0234] In one embodiment, the full-color pixel generation unit is also used to calculate the pixel value of each new full-color pixel based on the pixel mean of each color pixel and the pixel mean of the full-color pixel in the first preset area including the color pixels if the difference between the smallest second pixel value gradient and the second smallest second pixel value gradient does not meet the preset condition.

[0235] In one embodiment, the full-color pixel generation unit is also used to calculate, for each color pixel, a pixel mean of the color pixels in the first preset area including the color pixels based on the pixel values ​​of the color pixels in the first preset area including the color pixels and a weight matrix of the color pixels; calculate a pixel mean of the full-color pixels in the first preset area including the color pixels based on the pixel values ​​of the full-color pixels in the first preset area including the color pixels and a weight matrix of the full-color pixels; and calculate the pixel value of each new full-color pixel based on the pixel value of the color pixel, the pixel mean of the color pixel and the pixel mean of the full-color pixel.

[0236] In one embodiment, each filter group includes 8 full-color filters and 8 color filters of the same color, and each filter group includes 4 filter units, and the filter unit includes 2 full-color filters and 2 color filters; in the filter unit, every 2 color filters are arranged in a first diagonal direction, and every 2 full-color filters are arranged in a second diagonal direction, and the first diagonal direction is different from the second diagonal direction.

[0237] In one embodiment, the minimum repeating unit is an array of 8 rows and 8 columns generated by arranging 32 full-color filters and 32 color filters in a preset arrangement. The preset arrangement is:

[0238]

[0239] Among them, w represents a full-color filter, and r, g, and b all represent color filters.

[0240] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0241] The division of the modules in the above-mentioned image processing device is only for illustration. In other embodiments, the image processing device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned image processing device.

[0242] For the specific definition of the image processing device, please refer to the definition of the image processing method above, which will not be repeated here. Each module in the above image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0243] Fig.19 Schematic diagram of the internal structure of an electronic device in an embodiment. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a car computer, a wearable device, etc. The electronic device includes a processor and a memory connected via a system bus. Among them, the processor may include one or more processing units. The processor may be a CPU (Central Processing Unit) or a DSP (Digital Signal Processing), etc. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement an image processing method provided in each of the following embodiments. The internal memory provides a cache operating environment for the operating system computer program in the non-volatile storage medium.

[0244] The implementation of each module in the image processing device provided in the embodiment of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. The program module constituted by the computer program may be stored in a memory of an electronic device. When the computer program is executed by a processor, the steps of the method described in the embodiment of the present application are implemented.

[0245] The embodiment of the present application also provides a computer-readable storage medium, which is one or more non-volatile computer-readable storage media containing computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the image processing method.

[0246] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0247] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include ROM (Read-Only Memory), PROM (Programmable Read-only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-only Memory) or flash memory. Volatile memory may include RAM (Random Access Memory), which is used as an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDRSDRAM (Double Data Rate Synchronous Dynamic Random Access memory), ESDRAM (Enhanced Synchronous Dynamic Random Access memory), SLDRAM (Sync Link Dynamic Random Access Memory), RDRAM (Rambus Dynamic Random Access Memory), and DRDRAM (Direct Rambus Dynamic Random Access Memory).

[0248] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device comprises an image sensor, the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter groups, the filter group comprises a color filter and a panchromatic filter; in the filter group, the color filter is arranged in a first diagonal direction, the panchromatic filter is arranged in a second diagonal direction, the first diagonal direction is different from the second diagonal direction; the pixel array comprises a plurality of panchromatic pixels and a plurality of color pixels, each of the panchromatic pixels corresponds to the panchromatic filter, and each of the color pixels corresponds to the color filter; The method comprises: Acquire an original image through the image sensor, traverse each pixel in the original image, and when it is determined that the current pixel of the original image is a color pixel, acquire a second preset area containing the color pixel for each color pixel in the original image, calculate the standard deviation of each panchromatic pixel according to the pixel value of the panchromatic pixel in the second preset area, and use the standard deviation as the characteristic value of the area type of the second preset area, if the characteristic value of the area type is less than a first preset threshold, determine that the area type of the second preset area containing the color pixel is a flat area, and if the characteristic value of the area type is greater than or equal to the first preset threshold, determine that the area type of the second preset area containing the color pixel is a non-flat area; interpolate the color pixel into a new panchromatic pixel based on the panchromatic pixels in the first preset area containing the color pixel according to the area type, until all color pixels in the original image are interpolated into new panchromatic pixels, and generate a full-size panchromatic channel image; wherein the original image includes color pixels and panchromatic pixels; Performing filtering processing on color pixels of different colors in the original image through the full-size full-color channel image to generate color channel images corresponding to the color pixels of different colors; A target image is generated based on each of the color channel images.

2. The method according to claim 1, characterized in that The interpolating the color pixel to the new panchromatic pixel based on the panchromatic pixels in the first preset area according to the area type includes: If the area type is a flat area, determining a target full-color pixel in a preset area adjacent to the color pixel in a horizontal direction and a vertical direction from the full-color pixels in the first preset area; The pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel.

3. The method according to claim 1, characterized in that The interpolating the color pixel to the new panchromatic pixel based on the panchromatic pixels in the first preset area according to the area type includes: If the region type is a non-flat region, calculating the texture direction of the color pixel; Along the texture direction of the color pixel, determining, from the panchromatic pixels in a first preset area including the color pixel, a target panchromatic pixel in a preset area adjacent to the color pixel in the texture direction; The pixel value of the new panchromatic pixel is calculated according to the pixel value of the target panchromatic pixel and the interpolation weight of the target panchromatic pixel.

4. The method according to claim 3, characterized in that The calculating the texture direction of the color pixel comprises: Taking the color pixel as the center, acquiring at least one first full-color pixel pair in different preset directions; Calculating a first pixel value gradient of the color pixel in different preset directions according to a difference in pixel values ​​of the at least one first panchromatic pixel pair; the first pixel value gradient is used to characterize a texture direction of the color pixel; The texture direction of the color pixel is determined according to the magnitude relationship between the first pixel value gradients in different preset directions.

5. The method according to claim 4, characterized in that The preset directions include a horizontal direction, a vertical direction, a diagonal direction, and an anti-diagonal direction; and calculating the first pixel value gradient of the color pixel in different preset directions according to the difference of the pixel values ​​of the at least one first panchromatic pixel pair includes: A weighted average value of the difference values ​​of the pixel values ​​of the at least one first panchromatic pixel pair is calculated, and the weighted average value is used as the first pixel value gradient of the color pixel in the preset direction.

6. The method according to claim 5, characterized in that The step of determining the texture direction of the color pixel according to the magnitude relationship between the first pixel value gradients in different preset directions includes: Obtaining the minimum first pixel value gradient and the second minimum first pixel value gradient from the first pixel value gradients in the different preset directions; The texture direction of the color pixel is determined according to the magnitude relationship between the minimum first pixel value gradient and the second minimum first pixel value gradient.

7. The method according to claim 6, characterized in that The step of determining the texture direction of the color pixel according to the magnitude relationship between the minimum first pixel value gradient and the second minimum first pixel value gradient comprises: Determine whether a difference between the minimum first pixel value gradient and the second minimum first pixel value gradient satisfies a preset condition; If so, the texture direction of the color pixel is determined to be the preset direction corresponding to the minimum first pixel value gradient.

8. The method according to claim 7, characterized in that The preset condition includes a first preset condition, and the determining whether the difference between the minimum first pixel value gradient and the second minimum first pixel value gradient satisfies the preset condition includes: If the brightness of the area where the color pixel is located is greater than a first preset brightness threshold, it is determined whether the difference between the smallest first pixel value gradient and the second smallest first pixel value gradient meets a first preset condition.

9. The method according to claim 7, characterized in that: The preset condition includes a second preset condition, and the determining whether the difference between the minimum first pixel value gradient and the second minimum first pixel value gradient satisfies the preset condition includes: If the brightness of the area where the color pixel is located is less than or equal to the first preset brightness threshold, it is determined whether the difference between the minimum first pixel value gradient and the second minimum first pixel value gradient meets a second preset condition.

10. The method according to claim 7, characterized in that The method further comprises: If the difference between the minimum first pixel value gradient and the second minimum first pixel value gradient does not meet a preset condition, taking the color pixel as the center, acquiring at least one second full-color pixel pair in the preset direction; the number of the second full-color pixel pairs is greater than the number of the first full-color pixel pairs, and the distance between the second full-color pixel pairs is greater than the distance between the first full-color pixel pairs; Calculating a second pixel value gradient of the color pixel in the preset direction based on a difference in pixel values ​​of the at least one second panchromatic pixel pair; The texture direction of the color pixel is determined according to the magnitude relationship between the second pixel value gradients in different preset directions.

11. The method according to claim 10, characterized in that The step of determining the texture direction of the color pixel according to the magnitude relationship between the second pixel value gradients in different preset directions includes: Obtaining the minimum second pixel value gradient and the second minimum second pixel value gradient from the second pixel value gradients in different preset directions; The texture direction of the color pixel is determined according to the magnitude relationship between the minimum second pixel value gradient and the second minimum second pixel value gradient.

12. The method according to claim 11, characterized in that The determining the texture direction of the color pixel according to the magnitude relationship between the minimum second pixel value gradient and the second minimum second pixel value gradient comprises: Determine whether the difference between the minimum second pixel value gradient and the second minimum second pixel value gradient satisfies the preset condition; If so, the texture direction of the color pixel is determined to be the preset direction corresponding to the minimum second pixel value gradient.

13. The method according to claim 12, characterized in that The method further comprises: If the difference between the minimum second pixel value gradient and the second smallest second pixel value gradient does not meet the preset condition, the pixel value of each new panchromatic pixel is calculated based on the pixel mean of each color pixel and the pixel mean of the panchromatic pixel in the first preset area including the color pixel.

14. The method according to claim 13, characterized in that The step of calculating the pixel value of each of the new panchromatic pixels based on the pixel mean value of each of the color pixels and the pixel mean value of the panchromatic pixels in the first preset area including the color pixels comprises: For each color pixel, based on the pixel value of the color pixel in the first preset area including the color pixel and the weight matrix of the color pixel, calculate the pixel mean value of the color pixel in the first preset area including the color pixel; Calculating a pixel mean value of the panchromatic pixels in the first preset area including the color pixels based on the pixel values ​​of the panchromatic pixels in the first preset area including the color pixels and a weight matrix of the panchromatic pixels; The pixel value of each of the new panchromatic pixels is calculated according to the pixel value of the color pixel, the pixel mean value of the color pixel and the pixel mean value of the panchromatic pixel.

15. The method according to any one of claims 1 to 14, characterized in that Each of the filter groups includes 8 full-color filters and 8 color filters of the same color, and each of the filter groups includes 4 filter units, and the filter units include 2 full-color filters and 2 color filters; in the filter units, every 2 color filters are arranged in a first diagonal direction, and every 2 full-color filters are arranged in a second diagonal direction, and the first diagonal direction is different from the second diagonal direction.

16. The method according to claim 15, characterized in that The minimum repeating unit is an array of 8 rows and 8 columns generated by arranging 32 of the full-color filters and 32 of the color filters in a preset arrangement, and the preset arrangement is: Among them, w represents the full-color filter, and r, g, and b all represent the color filters.

17. An image processing device, characterized in that: Applied to an electronic device, the electronic device comprises an image sensor, the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter groups, the filter group comprises a color filter and a panchromatic filter; in the filter group, the color filter is arranged in a first diagonal direction, the panchromatic filter is arranged in a second diagonal direction, the first diagonal direction is different from the second diagonal direction; the pixel array comprises a plurality of panchromatic pixels and a plurality of color pixels, each of the panchromatic pixels corresponds to the panchromatic filter, and each of the color pixels corresponds to the color filter; The device comprises: A first generating module is used to obtain an original image through the image sensor, traverse each pixel in the original image, and when it is determined that the current pixel of the original image is a color pixel, obtain a second preset area containing the color pixel for each color pixel in the original image, calculate the standard deviation of each panchromatic pixel according to the pixel value of the panchromatic pixel in the second preset area, and use the standard deviation as a characteristic value of the area type of the second preset area; if the characteristic value of the area type is less than a first preset threshold, determine that the area type of the second preset area containing the color pixel is a flat area; if the characteristic value of the area type is greater than or equal to the first preset threshold, determine that the area type of the second preset area containing the color pixel is a non-flat area; according to the area type, interpolate the color pixel into a new panchromatic pixel based on the panchromatic pixels in the first preset area containing the color pixel, until all the color pixels in the original image are interpolated into new panchromatic pixels, and generate a full-size panchromatic channel image; wherein the original image includes color pixels and panchromatic pixels; A second generating module is used to perform filtering processing on color pixels of different colors in the original image through the full-size full-color channel image to generate color channel images corresponding to the color pixels of different colors respectively; The third generating module is used to generate a target image based on each of the color channel images.

18. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image processing method according to any one of claims 1 to 16.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images