Image processing method, electronic device, readable medium and program product

Abstract

The present invention relates to the field of image processing technology, and more particularly to an image processing method, electronic device, readable medium, and program product. The method corrects the pixel values ​​of abnormal pixels (hereinafter referred to as pixels to be corrected) among edge pixels in Remosaic Raw using the pixel values ​​of non-abnormal, same-color pixels adjacent to the pixels (i.e., correction pixels). In this way, the pixels to be corrected after Remosaic processing of the raw image data output by a 2×2 OCL sensor can be corrected while preserving the image's details and texture and ensuring the clarity of the output image, thereby resolving the problem of false colors or black edges in images obtained based on the image data processed after Remosaic.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method, electronic equipment, readable medium and program product. Background Art

[0002] Image sensors are commonly used in electronic devices with camera functions, such as mobile phones, tablets, digital cameras, watches, laptops, and other devices that run applications that capture images. Typically, an image sensor has an array of cells (e.g., pixels) arranged in rows and columns. Each image sensor contains a photosensitive element (also called a sensor element, such as a photodiode) that generates an electric charge in response to incident light. An on-chip lens (OCL) can be provided for each pixel to efficiently direct the incident light onto the pixel's photosensitive area.

[0003] Currently, more and more electronic devices are adopting quadrant photodiode (QPD) image sensors with a 2×2 OCL structure, which can achieve higher resolution and lower cost. A 2×2 OCL refers to a sensor structure in which four pixels in a color channel share a single OCL (also known as a microlens). However, this 2×2 OCL sensor structure is prone to receiving signal imbalance due to reflection and refraction. This can cause the output sensor raw data to be prone to false color, black edges, and other issues after pixel remosaicing algorithms. These issues are particularly prone to high-contrast edges in certain images. Summary of the Invention

[0004] The present application provides an image processing method, electronic device, readable medium, and program product, which can correct the pixels to be corrected after the original image data output by a 2×2 OCL type sensor is processed by a remosaic algorithm, while preserving the details and texture of the image and ensuring the clarity of the output image, thereby solving the problems of false color or black edges in the image obtained based on the image data processed after remosaic.

[0005] In a first aspect, the present application provides an image processing method for application to an electronic device, the method comprising: acquiring first image data captured by an image sensor, wherein a single microlens in the image sensor corresponds to multiple pixels; performing pixel re-arrangement (Remosaic) on the first image data to obtain second image data; determining pixels to be corrected among edge pixels of the second image data; and correcting the pixel values ​​of the pixels to be corrected using pixel values ​​of correction pixels that meet a distance condition with the pixels to be corrected to obtain third image data.

[0006] Specifically, the pixel values ​​of abnormal pixels (hereinafter referred to as "to-be-corrected pixels") within edge pixels in the raw image data (remosaic raw, for example, the aforementioned second image data) after pixel remosaic processing are corrected using the pixel values ​​of adjacent, non-abnormal, same-color pixels (i.e., correction pixels). The edge pixels in the remosaic raw can constitute edge image data. When implementing the image processing method provided herein, an electronic device can extract edge image data from the remosaic raw to obtain edge pixels.

[0007] It can be understood that the original image data collected and output by the image sensor, such as the above-mentioned first image data, can be image data in Quad Bayer format. After the first image data is subjected to pixel rearrangement (Remosaic) processing, the second image data obtained can be image data of an image in Bayer format. The above-mentioned pixels to be corrected can be pixels with abnormal pixel values ​​in Remosaic Raw or pixels that constitute a pixel block, which may be referred to as abnormal pixels hereinafter. After the pixel values ​​of the pixels to be corrected are corrected by the above-mentioned image processing method provided in the present application, the third image data obtained can be processed by demosaicing (Demosaic) and other image processing algorithms to generate a captured image displayed on the screen of an electronic device. The above-mentioned electronic device can be, for example, a mobile phone, a tablet computer, etc., which is not limited here.

[0008] It will be appreciated that the aforementioned distance condition is used to restrict the acquisition of each correction pixel to a range adjacent to the pixel to be corrected, thereby accurately correcting the pixel value of the pixel to be corrected. For example, the corresponding adjacent range may be [-3, 3], meaning that the distance between each correction pixel and the pixel to be corrected can be limited to a distance range corresponding to three pixels.

[0009] In this way, the error caused by abnormal pixel values ​​on the correction process can be avoided, so that the pixel values ​​of abnormal pixels can be corrected while retaining the details and texture of the image and ensuring the clarity of the output image, thereby eliminating the pseudo color or black edge problems in the image obtained by processing the original image data.

[0010] In a possible implementation of the first aspect, the second image data includes image data in a Bayer format, the second image data includes multiple pixel units, a first pixel unit among the multiple pixel units includes multiple pixels corresponding to a single microlens; and the first pixel unit includes multi-color channel data, the multi-color channel data including pixel values ​​of the multiple pixels after being processed by each color channel filter.

[0011] For example, the multiple pixel units comprising the second image data may be pixel blocks or pixel points comprising Remosaic Raw, where a pixel block may include four or more pixels. For example, a pixel block (i.e., a pixel unit) of raw image data captured by a 2×2 OCL-type image sensor may include four pixels, with the four pixels in each pixel block sharing a single microlens to capture pixel values. Correspondingly, the Remosaic Raw obtained after Remosaic processing, i.e., the second image data, may also have each pixel block (i.e., a pixel unit) comprised of four pixels. The first pixel unit may, for example, be any pixel block in the Remosaic Raw obtained after Remosaic processing.

[0012] In a possible implementation of the first aspect, determining pixels to be corrected among edge pixels of the second image data includes detecting that a similarity between a first pixel value in a first color channel and a second pixel value in a second color channel is less than or equal to a first threshold, and determining that a plurality of pixels constituting a first pixel unit are pixels to be corrected, wherein the first color channel and the second color channel use the same color filter.

[0013] In a possible implementation of the first aspect above, the multi-color channel data includes four color channel data of R, Gr, Gb, and B; and the first color channel is Gr color channel data, and the second color channel is Gb color channel data; or, the first color channel is Gb color channel data, and the second color channel is Gr color channel.

[0014] For example, the first color channel is Gr color channel data, and the second color channel is Gb color channel data, that is, both the first color channel and the second color channel use a green filter to obtain pixel values ​​of corresponding pixels.

[0015] In a possible implementation of the first aspect above, the method for determining the first threshold includes: determining the first threshold based on the average similarity between the pixel value of each pixel unit of the fourth image data in the first color channel and the pixel value in the second color channel, wherein there is no pixel to be corrected in the fourth image data.

[0016] In one possible implementation of the first aspect, correcting the pixel value of the pixel to be corrected using the pixel value of a correction pixel that satisfies a distance condition with the pixel to be corrected includes: obtaining a plurality of same-color pixels in the same color channel as the first pixel to be corrected from edge pixels of the first image data, where the plurality of same-color pixels includes the pixel to be corrected; determining a plurality of correction pixels from the plurality of same-color pixels that satisfy the distance condition with the first pixel to be corrected; and correcting the pixel value of the first pixel to be corrected using the pixel values ​​of the plurality of correction pixels.

[0017] That is, after eliminating other pixels to be corrected with abnormal pixel values ​​from multiple same-color pixels adjacent to the pixel to be corrected, the multiple corrected pixels determined can serve as the basis for correcting the pixel value of the pixel to be corrected. As the name suggests, same-color pixels are pixels in the same color channel as the pixel to be corrected. In actual applications, the same-color pixels in the edge pixels of the first image data that are in the same color channel as the first pixel to be corrected can be used to construct a same-color pixel correction map for the color channel to which the pixel to be corrected belongs. Taking the R color channel as an example, the same-color pixels in the R color channel of the edge pixels of the first image data can construct an R-Map.

[0018] In a possible implementation of the first aspect above, the distance condition includes: a distance corresponding to a first number of pixels less than or equal to the distance between the pixels to be corrected, or a number of pixels spaced between the pixels to be corrected less than or equal to a second number.

[0019] If the first number and the second number correspond to the same correction pixel acquisition range, the difference between the first number and the second number may be 1. For example, if the first number is 3, the corresponding correction pixel acquisition range may correspond to the adjacent range [-3, 3] of the pixel to be corrected. Correspondingly, the second number may be 2, i.e., the number of pixels between the pixel to be corrected and the pixel to be corrected is 2.

[0020] In a possible implementation of the first aspect described above, the pixel value of the first pixel to be corrected is corrected using the pixel values ​​of multiple correction pixels, including: performing statistical calculations on the pixel values ​​of the multiple correction pixels to obtain a first statistical value, wherein the first statistical value is used as the pixel value correction value of the first pixel to be corrected.

[0021] In a possible implementation of the first aspect, the first statistical value includes any one of a median, an average, or a mode in a statistical sequence composed of pixel values ​​of a plurality of corrected pixels.

[0022] In a possible implementation of the first aspect, the image sensor is a 2×2 OCL sensor.

[0023] In a second aspect, the present application provides an electronic device comprising one or more processors; one or more memories; and one or more memories storing one or more programs. When one or more programs are executed by one or more processors, the electronic device executes the image processing method provided by the above-mentioned first aspect and various possible implementations of the first aspect.

[0024] In a third aspect, the present application provides a computer-readable medium having instructions stored thereon. When the instructions are executed on a computer, the computer executes the image processing method provided by the first aspect and various possible implementations of the first aspect.

[0025] In a fourth aspect, the present application provides a computer program product, comprising: computer instructions, which, when executed on an electronic device, enable the electronic device to execute the image processing method provided in the above-mentioned first aspect and various possible implementations.

[0026] The beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions of the first aspect and various possible implementations of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Shown are schematic diagrams of the sensor structures of 1×1 OCL type and 2×2 OCL type.

[0028] Figure 2 The figure shows the processing principle of the pixel rearrangement (Remosaic) algorithm and the pixel distribution diagram of Quad Bayer format and Bayer format images.

[0029] Figure 3 The figure shows a schematic diagram of an image processing scenario.

[0030] Figure 4a The figure shows a schematic diagram of a captured image with a black edge problem at the edge of a building.

[0031] Figure 4b The figure shows the structural changes of the original image data before and after the pixel remosaic algorithm is used.

[0032] Figure 5a The figure shows a schematic diagram of an image processing scenario in which an electronic device performs abnormal pixel correction, provided by an embodiment of the present application.

[0033] Figure 5b The figure shows a schematic diagram of an image processing scenario in which an ISP performs abnormal pixel correction, provided by an embodiment of the present application.

[0034] Figure 6 The figure shows a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application.

[0035] Figure 7a FIG2 is a schematic diagram showing pixel blocks and component pixel distribution of edge image data extracted from Remosaic Raw provided by an embodiment of the present application.

[0036] Figure 7b Shown is a schematic diagram of abnormal pixel distribution of edge image data provided by an embodiment of the present application.

[0037] Figure 7cShown is a schematic diagram of pixel distribution of the R color channel in each pixel block of edge image data provided by an embodiment of the present application.

[0038] Figure 7d Shown is a schematic diagram of a same-color pixel correction map of the R color channel provided in an embodiment of the present application.

[0039] Figure 8 The figure shows a schematic diagram comparing the image processing effects before and after correction provided by an embodiment of the present application.

[0040] Figure 9 Shown is a schematic diagram of another implementation flow of an image processing method provided in an embodiment of the present application.

[0041] Figure 10 Shown is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0042] Figure 11 The figure shows a schematic diagram of the software structure of an operating system of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0044] (1) On-chip lens (OCL, also known as microlens). In the composition structure of camera lens (i.e. camera), the role of microlens is to gather light for photodiode, that is, to receive light signals. In the field of cameras, the number of image sensor (Sensor) pixels corresponds to the number of microlenses. Based on this, the commonly used sensor structure types may include 1×1 OCL, 2×2 OCL and super 2×2 OCL. Among them, 1×1 OCL means that 1 pixel uses 1 microlens independently, for example Figure 1 The microlens 01a is shown in (a). 2×2 OCL means that 4 pixels share one microlens, for example Figure 1 The microlens 01b shown in (b) is shown in FIG. Figure 1 The microlens 01a and the microlens 01b shown are both front views of the microlens structure of the sensor.

[0045] refer to Figure 1 As shown in (a), in the 1×1 OCL type sensor structure, the intensity of the light signal received by each microlens is relatively uniform, so the pixel value of each pixel is also uniform, that is, there is no obvious difference in the pixel value of each pixel.

[0046] refer to Figure 1In (b), in a 2×2 OCL sensor structure, the intensity of the light signal received at different positions on the same microlens is affected by the incident angle of the incident light, which can easily lead to uneven received signals due to reflection, refraction, etc. Therefore, the pixel values ​​of the four pixels sharing one microlens will be uneven, that is, the pixel values ​​of each pixel may be quite different. For example, the pixel values ​​of the four pixels can be Figure 1 The distribution of high, middle, middle, and low is shown in (b).

[0047] (2) A quadrant photodiode (QPD) is a special photodiode that can divide the incident light into four quadrants and detect the light intensity of each quadrant independently. This structure enables the QPD to measure the position and intensity of the light spot simultaneously.

[0048] (3) The pixel rearrangement (Remosaic) algorithm, also known as the Bayer image regeneration algorithm, can be used to rearrange pixels into a Bayer format image before output. For example, when using the Remosaic algorithm to output an image, a Quad Bayer format with four pixels of the same color arranged together can be converted into a Bayer format image; or an RGBCMYRaw image, i.e., a multi-channel Raw image, can be converted into a Bayer format image.

[0049] refer to Figure 2 As shown in (a) of the figure, the four pixel blocks in the raw image data output by the 2×2 OCL type image sensor can form a classic Bayer array (RGGB) structure, including four color channels R, G, G, and B. This data format can be called the Quad Bayer format. The pixel block describes the combination result of multiple pixels. Figure 2 As shown in (b), each pixel block in the Quad Bayer format image data can be merged by 4 pixels of the same color. After processing by the Remosaic algorithm, refer to Figure 2 As shown in (c), each pixel block can be converted from a same-color pixel array to a Bayer array (RGGB), and correspondingly, a Quad Bayer format image can be converted to a Bayer format image.

[0050] (4) Demosaic algorithm, which is used to convert Bayer format data, such as RGBRaw images, into RGB format data. It is understood that the output of the photosensitive element only responds to light intensity and does not provide color information. Therefore, when it is necessary to capture a color image, a color filter array (CFA) can be provided. The CFA includes color filter elements above the pixels of the pixel array. The color filter elements can include red, green, and blue filter elements arranged in a so-called Bayer pattern, but other colors and / or other arrangement patterns can also be used. However, since pixels covered by filter elements of one color cannot respond to other colors, the missing color values ​​must be determined by interpolation. The process of obtaining the color value of each color for all pixels in the pixel array by interpolation is called demosaicing.

[0051] The Demosaic algorithm can be understood as a color interpolation processing algorithm, which is a processing algorithm that restores the real-world color that conforms to the color display device from the Bayer format image data obtained from the sensor. For example, the Demosaic algorithm can be used to Figure 2 The Bayer format image shown in (c) is converted into RGB format data.

[0052] Figure 3 According to an embodiment of the present application, a schematic diagram of an image processing scenario is shown.

[0053] like Figure 3 As shown, the lens 101 of an electronic device 100 with image processing capabilities can capture images of people / scenery, etc. and generate optical signals, which are transmitted to the photosensitive area on the surface of the image sensor 102. After being photoelectrically converted by the image sensor 102, raw image data is formed. As previously mentioned, this raw image data can be, for example, QuadBayer format image data. After the Quad Bayer format image output by the image sensor 102 is processed by a pixel remosaic algorithm, the resulting Bayer format image can be transmitted to the image signal processor (ISP) 103 for image processing. ISP 103 can process the raw image data using various preset image processing algorithms and encode the processed image data into an image that the electronic device 100 can support display. In some shooting scenarios, this is the captured image.

[0054] The image sensor 102 may be a metal oxide semiconductor (CMOS) or a charge-coupled device (CCD). The various image processing algorithms used by the ISP 103 to process raw image data may include a black level compensation (BLC) algorithm, a lens shading correction (LSC) algorithm, an automatic white balance (AWB) algorithm, a demosaicing algorithm, a color correction matrix (CCM), and a gamma correction algorithm. The image data obtained by processing the above-mentioned image processing algorithms may be RGB format image data. After Joint Photographic Experts Group (JPEG) encoding, an image in a format such as JPG or JPEG may be generated, i.e., an image that the electronic device 100 supports displaying.

[0055] However, as mentioned above, the 2×2 OCL sensor structure is prone to receiving signal imbalance due to reflection, refraction, etc., which in turn causes the output sensor raw data (Sensor Raw) to be prone to problems such as false color and black edges after the pixel remosaic algorithm. Figure 4a As shown, the black edge problem that appears on the edge of the building image captured by the electronic device 100 is caused by the fact that after the original image data output by the 2×2 OCL type sensor is processed by Remosaic, there is still a large difference in the pixel values ​​of adjacent pixels.

[0056] The current solution to the above problem is usually to globally calculate the average pixel value of the four pixels obtained by the 2×2 OCL type sensor receiving the light signal. In some embodiments, this global average calculation method is also called global filtering. Figure 4b As shown, the above global filtering method calculates the average value of each pixel value of the image data after the original image data is processed by the Remosaic algorithm, including Figure 4bThe "high" pixel value of the R channel in the "Remosaiced Image Data" shown is averaged with the lower pixel values ​​of the Gr, Gb, and B channels under the same microlens. This average is then used as the replacement value for the pixel values ​​in each channel (R, Gr, Gb, and B). This corrects artifacts such as false colors or black edges in images processed based on the remosaiced image data (hereinafter referred to as Remosaic Raw).

[0057] However, the above global filtering method will cause the loss of some image details, textures, etc., and some higher or lower pixel values ​​will cause the average value obtained by global calculation to be too high or too low, which cannot guarantee the clarity of the output image.

[0058] It is understood that the details and texture of an image are related to the edge pixels in the image. For example, the edge pixels in an image can depict the details of the edge of a building or the texture of a scene such as leaves.

[0059] To address the aforementioned issues, this application provides an image processing method that corrects the pixel values ​​of abnormal pixels (hereinafter referred to as "corrected pixels") among edge pixels in RemosaicRaw using the pixel values ​​of adjacent, non-abnormal, same-color pixels (hereinafter referred to as "corrected pixels"). For example, the statistical median of the pixel values ​​of multiple corrected pixels is used as the corrected pixel value of the corrected pixel, or the statistical mean of the pixel values ​​of multiple corrected pixels is used as the corrected pixel value of the corrected pixel.

[0060] In this way, based on the above-mentioned image processing method provided by this application, it is possible to correct the pixels to be corrected after the original image data output by the 2×2 OCL type sensor is processed by the Remosaic algorithm while retaining the details and texture of the image and ensuring the clarity of the output image, thereby solving the problem of pseudo color or black edges in the image obtained based on the image data processed after Remosaic.

[0061] The aforementioned edge pixels can be understood as pixels that constitute edge image data. This edge image data can be obtained by performing edge detection on the original image data, for example, detecting the edges of buildings in the original image data and extracting edge image data, or detecting the edges of people in the original image data and extracting edge image data. Whether there are pixels to be corrected in the aforementioned edge image data can be determined by comparing the pixel value similarity of the color channels Gr and Gb with similar pixel values. For example, when the pixel value similarity of the Gr and Gb channel pixels exceeds a certain similarity threshold, it can be determined that the four pixels in the corresponding pixel block are all normal pixels; conversely, when the pixel value similarity of the Gr and Gb channel pixels is less than or equal to the aforementioned similarity threshold, it can be determined that the four pixels in the corresponding pixel block are all pixels to be corrected. The aforementioned similarity threshold can be determined based on a statistical value (e.g., a mean) of the pixel value similarity of normal Gr and Gb channel pixels.

[0062] refer to Figure 5a The image processing algorithm provided in this application can be integrated into an image processing module, such as the abnormal pixel correction module 104. The abnormal pixel correction module 104 can be integrated into a system service or application of the operating system installed in the electronic device 100 and executed by a processor configured in the electronic device 100. The processor can be another processor different from the ISP 103, such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0063] refer to Figure 5b The abnormal pixel correction 104 may also be a component of the image processing algorithm module executed by the ISP 103, that is, the image processing method provided by the present application is executed by the ISP 103. This is not limited here.

[0064] It is understood that in some embodiments of the present application, the electronic device 100 or ISP 103 for executing the image processing method provided by the present application may have Figure 5a or Figure 5b The abnormal pixel correction 104 shown in the figure and the black level compensation (BLC), lens correction (LSC), automatic white balance (AWB), demosaic, color correction matrix (CCM) and gamma correction module structure. In other embodiments, the electronic device 100 or ISP 103 may have a larger Figure 5a or Figure 5b The structures shown may be more or less, but this is not limiting.

[0065] Specifically, Figure 6 According to an embodiment of the present application, a schematic diagram of an implementation flow of an image processing method is shown.

[0066] I understand. Figure 6 The execution body of each step of the implementation process shown can be the electronic device 100 or the processor such as ISP103, and there is no limitation here. Figure 6 The execution entities of each step will not be repeatedly introduced when explaining the content of each step in the implementation process shown.

[0067] like Figure 6 As shown, the implementation process of the image processing method provided in the embodiment of the present application may include the following steps:

[0068] 601: Obtaining original image data after pixel rearrangement processing.

[0069] Exemplarily, the raw image data may be image data acquired based on a 2×2 OCL sensor and processed by a Remosaic algorithm, namely, the Remosaic Raw data.

[0070] 602: Extract edge image data from the original image data, including obtaining pixel values ​​of pixels constituting each pixel block at the edge of the original image.

[0071] For example, the electronic device 100 or the ISP 103 may extract edge image data from the original image data based on the gradient information of the original image data, etc. The extracted edge image data includes multiple pixel blocks constituting the edge of the original image. Each pixel block may include multiple pixels, namely, the aforementioned constituent pixels, such as the four pixels of the four color channels R, Gr, Gb, and B that constitute a pixel block.

[0072] It will be appreciated that the aforementioned gradient information essentially describes the rate and direction of change in pixel values ​​(or color, brightness, etc.) in the original image data. In a digital image (such as the aforementioned original image data), each pixel has one or more associated gradient values, typically obtained by calculating the difference or rate of change of surrounding pixels. These gradient values ​​can form a gradient map, which represents the magnitude and direction of changes in color or brightness, etc. Gradient information can help algorithms understand edges, textures, and other important structural features in an image. By utilizing this information, algorithms can more effectively restore high-resolution images from low-resolution images while maintaining the clarity and detail of these key features. In embodiments of the present application, gradient information can be used to obtain edge information in an image and extract edge image data. In other embodiments, the electronic device 100 or ISP 103 executing the image processing algorithm provided herein can also obtain texture information, etc. in the image, and correspondingly extract texture image data to correct for problems such as pseudo-color in some textures within the image. This is not a limitation here.

[0073] It can be understood that in the embodiments of this application, reference is made to the above Figure 5a or Figure 5b As shown, the gradient information in Remosaic Raw refers to the gradient information used when directly acquiring image data from the image sensor 102. After pixel remosaic processing, this gradient information can still be used to accurately extract edge image data even in low light, high noise, or other shooting conditions that are not conducive to capturing clear images.

[0074] 603: Calculate the pixel value similarity between the Gr and Gb color channels in each pixel block.

[0075] Exemplarily, the electronic device 100 or the ISP 103 can obtain the pixel values ​​of the constituent pixels of the Gr and Gb color channels in each pixel block from the extracted edge image data, and calculate the similarity between the two, that is, calculate the similarity of the pixel values ​​of the Gr and Gb color channels. It can be understood that in normal image data, the Gr / Gb channel pixel values ​​of each pixel block are relatively close or even the same, while the pixel value difference between the R and B channels is usually large. Therefore, the embodiment of the present application can determine whether the constituent pixels of each pixel block are abnormal by calculating the similarity of the pixel values ​​of the constituent pixels of the Gr and Gb color channels. The specific judgment process can be referred to the relevant description in the following steps 604 to 605, which will not be repeated here.

[0076] refer to Figure 7aAs shown in FIG. 1 , in the edge image data extracted from Remosaic Raw, each pixel block may include pixel values ​​of four color channels: R, Gr, Gb, and B. Before the pixels are rearranged, each pixel block composed of four pixels is also a pixel block of four color channels: R, Gr, Gb, and B. When the electronic device 100 or the ISP 103 executes the image processing method provided by the present application, the pixel values ​​of the pixels of the Gr and Gb color channels in each pixel block may be calculated separately, for example, Figure 7a The pixel value similarity between the Gr color channel pixel 711 and the Gb color channel pixel 712 in the R color channel pixel block, or the pixel value similarity between the Gr color channel pixel 711 and the Gb color channel pixel 712 in the Gb color channel pixel block, etc.

[0077] 604: Determine whether the similarity of the pixel values ​​corresponding to each pixel block is less than or equal to a similarity threshold.

[0078] If the judgment result is yes, then continue to execute the following steps 605 to 610 to correct the pixels determined to be abnormal.

[0079] If the judgment result is no, the process ends and there is no need to correct abnormal pixels.

[0080] Exemplarily, the similarity threshold value can be determined based on the similarity statistics of the normal image data. For example, the similarity threshold value can be the statistical mean of the similarity of the pixel values ​​of the Gr and Gb color channel pixels in the normal image data, or it can be the statistical median or mode of the similarity of the pixel values ​​of the Gr and Gb color channel pixels in the normal image data, etc., without limitation here. In some embodiments, the similarity threshold value can be different based on images captured in different scenes or different lighting environments, different noise environments, etc. For example, a group or one similarity threshold value corresponding to different scenes or different environments can be set in the electronic device 100 or the ISP 103, without limitation here.

[0081] Based on this, if the similarity of the pixel values ​​corresponding to each pixel block calculated in the above step 603 is less than or equal to the above similarity threshold, it indicates that the pixel values ​​of each component pixel or part of the component pixels in the corresponding pixel block may be abnormal. At this time, the electronic device 100 or the ISP 103 can continue to execute the following steps 605 to 610, and use the 4 pixels in the corresponding pixel block as abnormal pixels to be corrected, that is, pixels to be corrected, and perform pixel value correction processing. On the contrary, if the similarity of the pixel values ​​corresponding to each pixel block calculated in the above step 603 is greater than the above similarity threshold, it indicates that the pixel values ​​of each component pixel or part of the component pixels in the corresponding pixel block are normal and do not need to be corrected.

[0082] 605: Determine each pixel in the pixel block whose pixel value similarity is less than or equal to the similarity threshold as an abnormal pixel.

[0083] For example, based on the judgment result of comparing the similarity of the pixel values ​​corresponding to each pixel block with the similarity threshold in step 604, the electronic device 100 or the ISP 103 can determine that if the judgment result is yes, that is, if the pixel value similarity is less than or equal to the similarity threshold, each component pixel in the corresponding pixel block is determined to be an abnormal pixel, that is, a pixel to be corrected. Figure 7b As shown, the light-colored area, such as the pixels in each pixel block in the frame 721 (not all pixels are selected), can be determined as normal pixels, given that the pixel value similarity of the Gr and Gb color channel pixels in the corresponding pixel block is greater than the similarity threshold. Figure 7b As shown, the constituent pixels of each pixel block in the dark area, such as box 722 (not all pixels are selected), can be determined as abnormal pixels because the similarity of the pixel values ​​of the Gr and Gb color channel pixels in the corresponding pixel blocks is less than or equal to the similarity threshold.

[0084] It can be understood that the abnormal pixels determined in step 605 are also pixels to be corrected during the execution of steps 606 to 609 described below.

[0085] 606: Based on the extracted edge image data, construct a correction map of the same-color pixels corresponding to each color channel.

[0086] For example, the same-color pixel correction map corresponding to each color channel may include the pixel values ​​of the same-color pixels in each color channel, and the arrangement of the same-color pixels may be the same as the arrangement of the corresponding pixel blocks in the edge image data. Because the same-color pixel correction map is essentially a distribution array of the components of the same color channel, in some embodiments, the same-color pixel correction map can also be described as a pixel array of the corresponding color channel.

[0087] refer to Figure 7c As shown, in the edge image data extracted from Remosaic Raw, each pixel block can include pixel values ​​of four color channels: R, Gr, Gb, and B. Taking the pixels of the R color channel as an example, the pixel values ​​of each component pixel of the R color channel of each pixel block in the edge image data are extracted, and after rearranging the pixel blocks to which each component pixel belongs in the edge image data, a pixel value can be formed. Figure 7d The same color pixel correction map of the R color channel shown can be recorded as R-Map. The pixel blocks to which each component pixel belongs can be referred to Figure 7c The pixel block 730 shown is another pixel block.

[0088] Continue to refer Figure 7bAs shown, the same-color pixel correction map of the R color channel may include the constituent pixels of the R color channel, and the constituent pixels may include the abnormal pixels determined by the above steps 604 to 605, for example Figure 7d The abnormal pixel 740 of the R channel shown may also be a normal pixel determined in step 604 .

[0089] Similarly, the same color pixel correction maps of other color channels such as Gr, Gb, B, etc. can also refer to the above Figures 7c to 7d The construction process of the R-Map illustrated in the example is completed, and this application will not go into details here.

[0090] 607: Obtain pixel values ​​of multiple same-color pixels in the same-color pixel correction map that meet a distance condition with the pixel to be corrected.

[0091] Illustratively, the above-mentioned same-color pixels that meet the distance condition with the pixel to be corrected may include multiple non-abnormal same-color pixels within a certain distance range adjacent to the pixel to be corrected in the same-color pixel correction map, that is, the above-mentioned correction pixels, and may also include abnormal pixels within the above-mentioned certain distance range, that is, other pixels to be corrected.

[0092] It is understood that in order to improve the accuracy of correction, the above-mentioned adjacent may include adjacent within a certain distance range, and the distance range may also be referred to as an adjacent range. Figure 7d As shown, for the currently updated pixel to be corrected 741, its adjacent same-color pixels may include multiple normal same-color pixels within the range [-3, 3] centered around the pixel to be corrected 741, i.e., corrected pixels, and may also include one or more abnormal pixels. Correspondingly, the distance condition may, for example, be that the distance from the pixel to be corrected is within a distance range corresponding to three pixels, or that the number of pixels between the pixel to be corrected and the pixel to be corrected is within two, i.e., the number of pixels between the pixel to be corrected and the pixel to be corrected is within a preset range.

[0093] refer to Figure 7d As shown, the same-color pixels adjacent to the currently updated pixel to be corrected 741 may include normal pixels or abnormal pixels in the same color channel, wherein the normal pixels in the same color channel serve as the correction pixels for correcting the pixel value of the pixel to be corrected. To improve the accuracy of pixel value correction, the electronic device 100 or ISP 103 may proceed to step 608 below, remove the pixel values ​​of abnormal pixels in the same color channel adjacent to the pixel to be corrected, and then proceed to the correction process in step 609 below. For details, please refer to the description of the relevant steps below and will not be repeated here.

[0094] It can be understood that the larger the adjacent range is, the more pixel values ​​of the same-color pixels within the adjacent range obtained by the electronic device 100 or the ISP 103 when executing this step, and the more accurate the correction result of the pixel to be corrected when executing the following steps 608 to 609. In other embodiments, the adjacent range may be different from the above. Figure 7d The adjacent range [-3, 3] exemplified in the example may also be the adjacent range [-4, 4], the adjacent range [-5, 5], etc., which is not limited here.

[0095] 608: After deleting abnormal pixels from a plurality of pixels of the same color adjacent to the pixel to be corrected, pixel value statistics of each non-abnormal correction pixel are calculated.

[0096] For example, given that the pixel values ​​of abnormal pixels may be abnormally large or small, which may ultimately lead to unsatisfactory correction results, the electronic device 100 or ISP 103 in the embodiment of the present application can eliminate the pixel values ​​of abnormal pixels from the acquired multiple same-color pixels. Specifically, the abnormal pixels from the multiple same-color pixels adjacent to the pixel to be corrected are removed before performing the pixel correction calculation. This pixel correction calculation may, for example, include the process of calculating the pixel value statistics of each correction pixel within the adjacent range of the pixel to be corrected (i.e., satisfying the distance condition) in step 608.

[0097] refer to Figure 7d As shown, the electronic device 100 or the ISP 103 deletes abnormal pixels among a plurality of pixels of the same color adjacent to the pixel to be corrected 741, for example Figure 7d The dark areas within the adjacent range [-3,3] are shown to mark the pixels.

[0098] It can be understood that the pixel value statistical value calculated above may include any one of the statistical values ​​such as average value (abbreviated as mean), median value and mode value, and is not limited here.

[0099] 609: Correct the pixel value of the corresponding pixel to be corrected based on the pixel value statistics.

[0100] Exemplarily, the electronic device 100 or ISP 103 corrects the pixel value of the corresponding pixel to be corrected based on the pixel value statistics. For example, the pixel value statistics may be used to replace the pixel value of the corresponding pixel to be corrected, that is, the pixel value statistics calculated in the above step 608 are used as the new pixel value of the currently updated pixel to be corrected.

[0101] In other embodiments, the electronic device 100 or ISP 103 corrects the pixel value of the corresponding pixel to be corrected based on the pixel value statistics, and may also include correcting the pixel value of the currently updated pixel to be corrected according to a certain weight, etc., which is not limited here.

[0102] 610: Using edge image data with abnormal pixel correction completed, the edge image data with abnormal pixels in the original image data is replaced to obtain corrected original image data.

[0103] For example, after completing pixel value correction for all pixels to be corrected in the extracted edge image data based on steps 605 to 609, the pixel values ​​of all pixels to be corrected are updated to corrected pixel values, such as the pixel value statistics calculated in step 608. The electronic device 100 or ISP 103 can then add the edge image data with the abnormal pixel correction completed to the original image data as replacement data for the corresponding edge image data in the original image data. The corresponding edge image data in the original image data can be the edge image data extracted from the original image data in step 602, or data corresponding to that portion of the edge image data.

[0104] It is understood that when the electronic device 100 or ISP 103 extracts the edge image data in step 602, it may delete the corresponding portion of data from the original image data. Furthermore, when the electronic device 100 or ISP 103 executes step 610, it may use the edge image data after abnormal pixel correction as supplemental data for the extracted and deleted edge image data from the original image data, and fuse it with the remaining data from the original image data to obtain the original image data with the abnormal pixel correction.

[0105] In other embodiments, when the electronic device 100 or ISP 103 extracts the edge image data in step 602, it may also preserve the integrity of the original image data and not delete the extracted edge image data. Furthermore, when the electronic device 100 or ISP 103 executes step 610, it may use the edge image data after abnormal pixel correction as replacement data or update data for the corresponding edge image data in the original image data to correct and update the original image data.

[0106] It is understandable that based on the above Figure 6 In the implementation process of the image processing method from step 601 to step 610 shown in FIG, the electronic device 100 or the ISP 103 can perform abnormal pixel correction on the raw image data (Remosaic Raw) output by the 2×2 OCL type image sensor and processed by the Remosaic algorithm to eliminate problems such as false color or black edges caused by unbalanced received signals. This allows the electronic device 100 using the 2×2 OCL type image sensor to adapt to different reflection or refraction environments and capture clear images without problems such as false color or black edges.

[0107] As an example, Figure 8 According to an embodiment of the present application, a schematic diagram of image processing effects before and after correction is shown.

[0108] refer to Figure 8 As shown, before using the image processing method provided by this application to correct abnormal pixels, that is, Figure 8 As shown in the “before correction” figure, the image data output by the 2×2 OCL sensor is processed by the pixel remosaic algorithm. The edge image data corresponding to the edge of the building in the image can be presented as follows: Figure 8 The Bayer format image (a) before correction is shown. After further processing by other image processing algorithms and demosaicing algorithms, the final output image is, for example, Figure 8 The captured image (b) shows a black border.

[0109] Continue to refer Figure 8 As shown, after using the image processing method provided by this application to correct abnormal pixels on the original image data (Remosaic Raw) processed by the Remosaic algorithm, that is, Figure 8 After the “correction” shown, the edge image data corresponding to the edge of the building in the image can be presented as Figure 8 The "rectified" Bayer format image (a') is shown. After further processing by other image processing algorithms and demosaicing algorithms, the final output captured image is, for example, Figure 8 The captured image (b') does not have any black edges.

[0110] Similarly, in other embodiments, after performing abnormal pixel correction on the original image data (Remosaic Raw) processed by the Remosaic algorithm using the image processing method provided in the present application, it is also possible to eliminate the pseudo-color of the captured image that was originally output with pseudo-color problems, or eliminate the pseudo-color and black edges of the captured image that was originally output with pseudo-color and black edges, etc., so that the image captured by the electronic device 100 is clearer with realistic colors and clear edges.

[0111] In other embodiments, the implementation process of an image processing method provided in the embodiment of the present application can also refer to Figure 9 shown. Figure 9 According to the embodiment of the present application, another image processing method is shown in the flowchart. It can be understood that Figure 9 The execution entities of each step of the illustrated process may also be the electronic device 100 or the ISP 103, which is not limited here.

[0112] Specifically, if Figure 9As shown, the implementation process includes the following steps ① to ⑨:

[0113] ① Get the problematic Remosaic Raw. The problematic Remosaic Raw may correspond to the above Figure 6 The original image data after the pixel rearrangement process in the steps shown. It is understood that whether the original image data (e.g., RemosaicRaw) has a problem can be determined based on whether there are pixels to be corrected (i.e., abnormal pixels) in the original image data. If there is a problem, i.e., there are abnormal pixels in the original image data, the electronic device 100 or the ISP 103 can implement Figure 9 Steps ① to ⑨ shown, or implement the above Figure 6 Steps 601 to 610 are shown.

[0114] Specifically, the detailed execution process of obtaining the original image data in this step ① can refer to the relevant description in the above step 601, and will not be repeated here.

[0115] ② Extract edge image data based on gradient and other information.

[0116] Specifically, the detailed extraction process of this step ② can refer to the relevant description in the above step 602, and the relevant description of information such as gradient can also refer to the relevant description in the above step 602, which will not be repeated here.

[0117] ③ Obtain the pixel values ​​of the four pixels that make up each pixel block in the edge image data.

[0118] Specifically, the detailed process of obtaining the pixel values ​​of the constituent pixels of each pixel block in step ③ can be referred to the relevant description in the above step 602, and will not be repeated here.

[0119] ④ Calculate the pixel value similarity S of the constituent pixels of the Gr and Gb color channels in each pixel block.

[0120] Specifically, the detailed execution process of calculating the similarity of the pixel values ​​of the Gr and Gb color channels corresponding to each pixel block in step ④ can refer to the relevant description in the above step 603, which will not be repeated here.

[0121] ⑤ Based on the judgment result that the similarity S is less than or equal to the similarity threshold, the four pixels in the corresponding pixel block are determined to be abnormal pixels.

[0122] Specifically, the detailed judgment process of step ⑤ can refer to the relevant description in the above step 604, and the process of determining abnormal pixels in step ⑤ can refer to the relevant description in the above step 605, which will not be repeated here.

[0123] ⑥ Extract the pixel values ​​of the four color channels of each pixel block in the edge image data and construct a correction map of the same-color pixels in each color channel.

[0124] Specifically, the detailed execution process of constructing the same-color pixel correction map of each color channel in step ⑥ can refer to the relevant description in the above step 606 and Figures 7c to 7d The above and related descriptions will not be repeated here.

[0125] ⑦ Based on the same-color pixel correction map, after removing the pixel values ​​of multiple same-color pixels adjacent to the abnormal pixel, the pixel value statistics are calculated.

[0126] Specifically, the detailed execution process of calculating the pixel value statistics of the same-color pixels adjacent to the abnormal pixel based on the same-color pixel correction map in step ⑦ can be referred to the relevant description in the above steps 607 to 608, which will not be repeated here.

[0127] ⑧Use the calculated pixel value statistics to correct the pixel values ​​of the corresponding abnormal pixels and perform abnormal pixel correction.

[0128] Specifically, the execution process of correcting the corresponding abnormal pixel using the pixel value statistics of adjacent pixels of the same color in step ⑧ can refer to the relevant description in the above step 609 and will not be repeated here.

[0129] ⑨ Fuse the edge image data after abnormal pixel correction with the rest of the Remosaic Raw image data to obtain the corrected Remosaic Raw.

[0130] Specifically, the image processing execution process of obtaining the original image data (Remosaic Raw) after pixel correction in step ⑨ can refer to the relevant description in the above step 610, and will not be repeated here.

[0131] Figure 10 According to an embodiment of the present application, a hardware structure diagram of an electronic device is shown. The electronic device may be the electronic device 100 described above, or any electronic device including the ISP 103 described above.

[0132] like Figure 10As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a Subscriber Identity Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0133] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0134] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (Ap), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0135] In some embodiments of the present application, the electronic device 100 can implement the image processing method provided by the present application through the processor 110, for example, Figure 6 or Figure 9In other embodiments of the present application, the electronic device 100 may also implement the image processing method provided by the present application through a processing unit of the processor 110, such as the ISP 103, for example, Figure 6 or Figure 9 The steps of the implementation process of the image processing method shown are not limited here.

[0136] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0137] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0138] In the embodiment of the present application, the processor 110 of the electronic device 100 can control the above Figure 6 or Figure 9 The execution process of each step in the implementation flow of the image processing method shown.

[0139] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM card interface, and / or a Universal Serial Bus (USB) interface.

[0140] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0141] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0142] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0143] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0144] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G applied on the electronic device 100.

[0145] The wireless communication module 160 can provide wireless communication solutions applied to the electronic device 100, including Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) network), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), infrared technology (IR), etc.

[0146] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with a network and other devices via wireless communication technologies. Such wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The above-mentioned GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the Beidou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS) and / or the Satellite Based Augmentation System (SBAS).

[0147] The electronic device 100 implements a display function through the GPU, display screen 194, and application processor. For example, it displays an image captured by the electronic device 100 using the camera function. It will be understood that based on the image processing method provided in this application, the images captured by the camera function of the electronic device 100 can no longer have problems such as false colors or black edges.

[0148] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0149] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0150] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted to the camera's photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193. In the embodiment of the present application, the ISP can be, for example, the above-mentioned Figure 5a or Figure 5b The ISP 103 in the image processing scenario shown can execute various image processing algorithms to perform image processing on the original image data output by the image sensor 102 after being processed by the Remosaic algorithm.

[0151] Camera 193 is used to capture still images or videos. The lens generates an optical image of an object and projects it onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In some embodiments, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than one.

[0152] Digital signal processors (DSPs) are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0153] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0154] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0155] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0156] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device comprising at least two parallel plates with conductive material. When force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0157] Gyro sensor 180B can be used to determine the motion of electronic device 100. In some embodiments, gyro sensor 180B can be used to determine the angular velocity of electronic device 100 around three axes (i.e., x, y, and z). Gyro sensor 180B can also be used for image stabilization. For example, when the shutter button is pressed, gyro sensor 180B detects the angle of vibration of electronic device 100 and calculates the distance the lens module needs to compensate based on the angle. This allows the lens to counteract the vibration of electronic device 100 through reverse motion, achieving image stabilization.

[0158] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0159] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some shooting scenarios, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0160] The ambient light sensor 180L is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos.

[0161] The touch sensor 180K, also known as a "touch device," can be mounted on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen." The touch sensor 180K detects touch operations applied to or near the touch sensor. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194.

[0162] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0163] Motor 191 can generate vibration alerts. This can be used for incoming call vibration alerts and touch vibration feedback. For example, touch operations applied to different applications (such as taking photos, playing audio, etc.) can correspond to different vibration feedback effects. Motor 191 can also generate different vibration feedback effects for touch operations applied to different areas of the display 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. Touch vibration feedback effects can also be customized.

[0164] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0165] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting or removing the SIM card into or from the SIM card interface 195 .

[0166] Figure 11 According to an embodiment of the present application, a schematic diagram of the software structure of an operating system of an electronic device is shown.

[0167] It is understood that the operating system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the system software structure of the electronic device 100.

[0168] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android™ system is divided into four layers: from top to bottom: the application layer, the application framework layer, the Android™ runtime and system libraries, and the kernel layer.

[0169] like Figure 11 As shown, the application layer may include a series of application packages, including camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0170] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0171] The application framework layer may include a window manager, content provider, view system, telephony manager, resource manager, notification manager, etc.

[0172] In the embodiment of the present application, the application framework layer may also include a camera service. The camera service may integrate the above Figure 5a or Figure 5b The related algorithms or functional modules of abnormal pixel correction 104 in the illustrated scene are used to implement the image processing method provided in the embodiment of the present application.

[0173] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0174] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0175] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0176] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0177] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0178] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0179] The Android™ Runtime consists of core libraries and a virtual machine. The Android™ Runtime is responsible for scheduling and management of the Android™ system.

[0180] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0181] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0182] The system library can include multiple functional modules, such as the Surface Manager, Media Libraries, 3D graphics processing library (such as OpenGL ES), and 2D graphics engine (such as SGL).

[0183] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0184] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0185] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0186] A 2D graphics engine is a drawing engine for 2D drawings.

[0187] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0188] The following describes the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0189] When touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the application framework layer interface to start the camera application, which then calls the kernel layer to start the camera driver and capture still images or video through camera 193.

[0190] The embodiments of the present application also provide a computer program product for implementing the image processing methods provided in the above embodiments.

[0191] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as computer program modules or module codes executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0192] A computer program module or module code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0193] Module code can be implemented with high-level modular language or object-oriented programming language to communicate with the processing system. When necessary, module code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0194] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, the instructions may be distributed over a network or via other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to floppy disks, optical disks, optical discs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or tangible machine-readable storage for transmitting information via the Internet using electrical, optical, acoustic, or other propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Accordingly, machine-readable media includes any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0195] References in the specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one exemplary implementation or technique disclosed according to the embodiment of the present application. The appearances of the phrase "in one embodiment" in various places in the specification do not necessarily all refer to the same embodiment.

[0196] The disclosure of the embodiments of the present application also relates to a device for performing the operations described in the text. The device may be constructed specifically for the required purpose or it may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer-readable medium, such as, but not limited to, any type of disk, including a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application-specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions, and each may be coupled to a computer system bus. In addition, the computer mentioned in the specification may include a single processor or may be an architecture involving multiple processors for increased computing power.

[0197] In addition, the language used in this specification has been primarily selected for readability and instructional purposes and may not be selected to describe or limit the disclosed subject matter. Therefore, the present disclosure of embodiments is intended to illustrate, not to limit, the scope of the concepts discussed herein.

Claims

1. An image processing method, applied to electronic equipment, characterized in that: include: Acquiring first image data collected by an image sensor, wherein a single microlens in the image sensor corresponds to a plurality of pixels; Rearranging pixels of the first image data to obtain second image data, where the second image data includes a plurality of pixel units, the plurality of pixel units include a first pixel unit, the first pixel unit includes multi-color channel data, and the multi-color channel data includes pixel values ​​of the plurality of pixels after being processed by each color channel filter; determining pixels to be corrected among edge pixels of the second image data; Correcting the pixel value of the pixel to be corrected by using the pixel value of a correction pixel that meets a distance condition with the pixel to be corrected to obtain third image data; The determining of pixels to be corrected among edge pixels of the second image data includes: detecting that a similarity between a first pixel value of the first pixel unit in the first color channel and a second pixel value in the second color channel is less than or equal to a first threshold, and determining that a plurality of pixels constituting the first pixel unit are pixels to be corrected, wherein the first pixel unit is a pixel unit in the second image data, the first pixel unit includes a plurality of pixels corresponding to the single microlens, and the first color channel and the second color channel use the same color filter; The first color channel is Gr color channel data, and the second color channel is Gb color channel data; or The first color channel is Gb color channel data, and the second color channel is Gr color channel data.

2. The method according to claim 1, characterized in that The second image data includes image data in a Bayer format.

3. The method according to claim 2, characterized in that The multi-color channel data includes four color channel data of R, Gr, Gb, and B.

4. The method according to claim 3, characterized in that The first threshold is determined by: The first threshold is determined based on an average similarity between the pixel value of each pixel unit of the fourth image data in the first color channel and the pixel value in the second color channel, wherein: The pixel to be corrected does not exist in the fourth image data.

5. The method according to claim 1, wherein Correcting the pixel value of the pixel to be corrected by using the pixel value of the corrected pixel that meets the distance condition with the pixel to be corrected includes: Acquire a plurality of same-color pixels in the same color channel as the first pixel to be corrected from edge pixels of the first image data, wherein the plurality of same-color pixels include the pixel to be corrected; Determining, from the plurality of same-color pixels, a plurality of correction pixels that meet a distance condition with the first pixel to be corrected; The pixel value of the first pixel to be corrected is corrected by using the pixel values ​​of the plurality of corrected pixels.

6. The method according to claim 5, characterized in that The distance conditions include: The distance between the pixels to be corrected is less than or equal to the distance corresponding to the first number of pixels, or, The number of pixels spaced from the pixel to be corrected is less than or equal to a second number.

7. The method according to claim 5 or 6, characterized in that Correcting the pixel value of the first pixel to be corrected by using the pixel values ​​of the plurality of correction pixels includes: Performing statistical calculation on the pixel values ​​of the plurality of corrected pixels to obtain a first statistical value, wherein: The first statistical value is used as a pixel value correction value of the first pixel to be corrected.

8. The method according to claim 7, characterized in that The first statistical value includes any one of a median, an average, or a mode in a statistical sequence consisting of pixel values ​​of the plurality of corrected pixels.

9. The method according to any one of claims 1 to 6 or 8, characterized in that The image sensor is a 2×2 OCL type sensor.

10. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the image processing method according to any one of claims 1 to 9.

11. A computer-readable medium, characterized in that The readable medium stores instructions, which, when executed on a computer, cause the computer to execute the image processing method according to any one of claims 1 to 9.

12. A computer program product, characterized in that include: Computer instructions, when the computer instructions are executed on an electronic device, enable the electronic device to perform the method according to any one of claims 1 to 9.

Images