Image processing method, electronic device, readable medium and program product

Abstract

The invention relates to the technical field of image processing, in particular to an image processing method, electronic equipment, a readable medium and a program product. According to the method, the pixel value of an abnormal pixel (hereinafter referred to as a to-be-corrected pixel) in edge pixels in Remote Raw is corrected by using the pixel value of a non-abnormal same-color pixel (namely a corrected pixel) adjacent to the pixel. Thus, on the premise that details and textures of the image are reserved and the definition of the output image is guaranteed, the to-be-corrected pixels obtained after processing of the Remosic algorithm is carried out on the original image data output by the 2 * 2OCL type sensor can be corrected, and the problem of false colors or black edges existing in the image obtained through processing of the Remosic-based image data is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method, electronic equipment, readable medium and program product. Background Art

[0002] Image sensors are usually used in electronic devices with shooting functions, such as mobile phones, tablets, digital cameras, watches, notebooks, and other devices that involve running applications that capture images. Typically, image sensors have an array of units (such as pixels) arranged in rows and columns. Each image sensor contains a photosensitive element (also called a sensor element, such as a photodiode), which generates an electric charge in response to incident light, and an on-chip lens (On Chip Lens, OCL) can be provided for each pixel to effectively guide the incident light to the photosensitive area of ​​the pixel.

[0003] Currently, more and more electronic devices are using quadrant photodiode (QPD) image sensors of the super 2×2 OCL type, which can achieve higher resolution and lower cost. Among them, 2×2 OCL refers to a sensor structure in which 4 pixels of a color channel share one OCL (also known as a microlens). However, the 2×2 OCL type sensor structure is prone to uneven receiving signals due to reflection, refraction and other reasons, which in turn causes the output sensor raw data (Sensor Raw) to be prone to false color (False Color), black edges and other problems after the pixel remosaic algorithm, especially in some images with high contrast edges. Summary of the invention

[0004] The present application provides an image processing method, an electronic device, a readable medium and a program product, which can correct the pixels to be corrected after the original image data output by a 2×2 OCL type sensor is processed by a Remosaic algorithm while retaining the details and texture of the image and ensuring the clarity of the output image, and solve the problem of pseudo color or black edges in the image obtained by processing the image data after Remosaic.

[0005] In a first aspect, the present application provides an image processing method for application to an electronic device, the method comprising: acquiring first image data acquired by an image sensor, wherein a single microlens in the image sensor corresponds to multiple pixels; performing pixel rearrangement (Remosaic) on the first image data to obtain second image data; determining pixels to be corrected among edge pixels of the second image data; and correcting the pixel values ​​of the pixels to be corrected using pixel values ​​of correction pixels that meet distance conditions with the pixels to be corrected to obtain third image data.

[0006] That is, the pixel values ​​of abnormal pixels (hereinafter referred to as pixels to be corrected) in edge pixels in the original image data (referred to as Remosaic Raw, such as the second image data) after pixel rearrangement (Remosaic) processing are corrected using the pixel values ​​of non-abnormal pixels of the same color (i.e., corrected pixels) adjacent to the pixels. The edge pixels in Remosaic Raw can constitute edge image data, and when the electronic device implements the above-mentioned image processing method provided by the present application, the edge image data can be extracted from Remosaic Raw to obtain edge pixels.

[0007] It can be understood that the original image data collected and output by the image sensor, such as the first image data mentioned above, can be image data in Quad Bayer format. After the first image data is subjected to pixel rearrangement (Remosaic) processing, the second image data obtained can be image data of an image in Bayer format. The above-mentioned pixels to be corrected can be pixels with abnormal pixel values ​​in Remosaic Raw or pixels that constitute a pixel block, which may be referred to as abnormal pixels hereinafter. After the pixel values ​​of the pixels to be corrected are corrected by the above-mentioned image processing method provided by the present application, the third image data obtained can be processed by demosaicing (Demosaic) and other image processing algorithms to generate a captured image displayed on the screen of an electronic device. The above-mentioned electronic device can be, for example, a mobile phone, a tablet computer, etc., which is not limited here.

[0008] It can be understood that the above-mentioned distance condition is used to limit the acquisition of each correction pixel within the adjacent range of the pixel to be corrected, so as to accurately correct the pixel value of the pixel to be corrected. The corresponding adjacent range can be, for example, [-3, 3], that is, the distance between each correction pixel and the pixel to be corrected can be limited to a distance range corresponding to 3 pixels.

[0009] In this way, the error influence caused by abnormal pixel values ​​on the correction process can be avoided, so that the pixel values ​​of abnormal pixels can be corrected while retaining the details and texture of the image and ensuring the clarity of the output image, thereby eliminating the pseudo color or black edge problems existing in the image obtained by processing the original image data.

[0010] In a possible implementation of the first aspect above, the second image data includes image data in a Bayer format, the second image data includes multiple pixel units, a first pixel unit among the multiple pixel units includes multiple pixels corresponding to a single microlens; and the first pixel unit includes multi-color channel data, the multi-color channel data includes pixel values ​​of the multiple pixels after being processed by each color channel filter.

[0011] For example, the multiple pixel units constituting the second image data may be pixel blocks or pixel points constituting Remosaic Raw, wherein a pixel block may include 4 or more pixels. For example, a pixel block (i.e., a pixel unit) of the original image data collected by a 2×2 OCL type image sensor may include 4 pixels, and the 4 pixels of each pixel block share 1 microlens to collect pixel values. Correspondingly, the Remosaic Raw obtained after Remosaic processing, i.e., the second image data, each pixel block (i.e., a pixel unit) may also include 4 pixels. The first pixel unit may, for example, be any pixel block in the Remosaic Raw obtained after Remosaic processing.

[0012] In a possible implementation of the first aspect above, determining pixels to be corrected in edge pixels of the second image data includes: detecting that a similarity between a first pixel value in a first color channel and a second pixel value in a second color channel is greater than a first threshold, and determining that a plurality of pixels constituting a first pixel unit are pixels to be corrected, wherein the color filter used by the first color channel and the second color channel is the same.

[0013] In a possible implementation of the first aspect above, the multi-color channel data includes four color channel data of R, Gr, Gb, and B; and the first color channel is Gr color channel data, and the second color channel is Gb color channel data; or, the first color channel is Gb color channel data, and the second color channel is Gr color channel.

[0014] For example, the first color channel is Gr color channel data, and the second color channel is Gb color channel data, that is, both the first color channel and the second color channel use a green filter to obtain pixel values ​​of corresponding pixels.

[0015] In a possible implementation of the first aspect above, a method for determining the first threshold includes: determining the first threshold based on an average value of similarity between a pixel value of each pixel unit of the fourth image data in a first color channel and a pixel value in a second color channel, wherein there is no pixel to be corrected in the fourth image data.

[0016] In a possible implementation of the first aspect, the pixel value of the pixel to be corrected is corrected by using the pixel value of the correction pixel that satisfies the distance condition with the pixel to be corrected, including: obtaining a plurality of same-color pixels in the same color channel as the first pixel to be corrected from edge pixels of the first image data, wherein the plurality of same-color pixels include the pixel to be corrected; determining a plurality of correction pixels that satisfy the distance condition with the first pixel to be corrected from the plurality of same-color pixels; and correcting the pixel value of the first pixel to be corrected by using the pixel values ​​of the plurality of correction pixels.

[0017] That is, after eliminating other pixels to be corrected with abnormal pixel values ​​from multiple same-color pixels adjacent to the pixel to be corrected, the determined multiple correction pixels can be used as the correction basis for correcting the pixel value of the pixel to be corrected. Among them, the same-color pixels, as the name implies, are pixels in the same color channel as the pixel to be corrected. In practical applications, the multiple same-color pixels in the edge pixels of the above-mentioned first image data that are in the same color channel as the first pixel to be corrected can correspond to construct a same-color pixel correction map of the color channel to which the pixel to be corrected belongs. Taking the R color channel as an example, the same-color pixels in the edge pixels of the first image data that are in the R color channel can construct an R-Map.

[0018] In a possible implementation of the first aspect above, the distance condition includes: the distance between the pixels to be corrected is less than or equal to the distance corresponding to a first number of pixels, or the number of pixels spaced between the pixels to be corrected is less than or equal to a second number.

[0019] In the case where the correction pixel acquisition ranges determined corresponding to the first number and the second number are the same, the difference between the first number and the second number may be 1. For example, the first number is 3, and the corresponding correction pixel acquisition range may correspond to the adjacent range [-3, 3] of the pixel to be corrected. Correspondingly, the second number may be 2, that is, the number of pixels between the pixel to be corrected and the pixel to be corrected is 2.

[0020] In a possible implementation of the first aspect described above, the pixel values ​​of a plurality of correction pixels are used to correct the pixel value of a first pixel to be corrected, including: performing statistical calculations on the pixel values ​​of the plurality of correction pixels to obtain a first statistical value, wherein the first statistical value is used as the pixel value correction value of the first pixel to be corrected.

[0021] In a possible implementation of the first aspect, the first statistical value includes any one of a median, an average, or a mode in a statistical series composed of pixel values ​​of a plurality of corrected pixels.

[0022] In a possible implementation of the first aspect, the image sensor is a 2×2 OCL type sensor.

[0023] In a second aspect, the present application provides an electronic device comprising one or more processors; one or more memories; one or more memories storing one or more programs, and when one or more programs are executed by one or more processors, the electronic device executes the image processing method provided by the above-mentioned first aspect and various possible implementations of the first aspect.

[0024] In a third aspect, the present application provides a computer-readable medium having instructions stored thereon, which, when executed on a computer, causes the computer to execute the image processing method provided by the first aspect and various possible implementations of the first aspect.

[0025] In a fourth aspect, the present application provides a computer program product, comprising: computer instructions, which, when executed on an electronic device, enable the electronic device to execute the image processing method provided in the first aspect and various possible implementations.

[0026] The beneficial effects of the second to fourth aspects mentioned above can be referred to the relevant descriptions in the first aspect mentioned above and various possible implementations of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Shown are schematic diagrams of the sensor structures of 1×1 OCL type and 2×2 OCL type.

[0028] Figure 2 Shown is the processing principle of the pixel rearrangement (Remosaic) algorithm and the pixel distribution diagram of Quad Bayer format and Bayer format images.

[0029] Figure 3 The figure shows a schematic diagram of an image processing scenario.

[0030] Figure 4a The figure is a schematic diagram of a photographed image with a black edge problem at the edge of a building.

[0031] Figure 4b The figure shows the structural changes of the original image data before and after being processed by the pixel rearrangement (Remosaic) algorithm.

[0032] Figure 5a The figure is a schematic diagram of an image processing scenario in which an electronic device performs abnormal pixel correction provided by an embodiment of the present application.

[0033] Figure 5b The figure is a schematic diagram of an image processing scenario in which an ISP performs abnormal pixel correction, provided in an embodiment of the present application.

[0034] Figure 6 The figure is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application.

[0035] Figure 7a The figure shows a schematic diagram of pixel blocks and component pixel distribution of edge image data extracted from Remosaic Raw provided by an embodiment of the present application.

[0036] Figure 7b Shown is a schematic diagram of abnormal pixel distribution of edge image data provided by an embodiment of the present application.

[0037] Figure 7cShown is a schematic diagram of pixel distribution of the R color channel in each pixel block of edge image data provided by an embodiment of the present application.

[0038] Figure 7d Shown is a schematic diagram of a same-color pixel correction map of the R color channel provided in an embodiment of the present application.

[0039] Figure 8 Shown is a schematic diagram comparing image processing effects before and after correction provided by an embodiment of the present application.

[0040] Fig. 9 The figure shows another implementation flow diagram of an image processing method provided in an embodiment of the present application.

[0041] Fig.10 Shown is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0042] Fig.11 The figure shows a schematic diagram of the software structure of an operating system of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0044] (1) On-chip lens (OCL, also known as microlens). In the composition structure of the camera lens (i.e., camera), the function of the microlens is to gather light for the photodiode, that is, to receive the light signal. In the field of cameras, the number of pixels of the image sensor (Sensor) corresponds to the number of microlenses. Based on this, the commonly used sensor structure types can include 1×1 OCL, 2×2 OCL, and super 2×2 OCL. Among them, 1×1 OCL means that 1 pixel uses 1 microlens independently, for example Figure 1 The microlens 01a shown in (a) is shown in FIG. 2×2 OCL means that 4 pixels share one microlens, for example Figure 1 The microlens 01b shown in (b) is shown in FIG. Figure 1 The microlens 01a and the microlens 01b shown are both front views of the microlens structure of the sensor.

[0045] refer to Figure 1 As shown in (a), in the 1×1 OCL type sensor structure, the intensity of the light signal received by each microlens is relatively uniform, so the pixel value of each pixel is also uniform, that is, there is no obvious difference in the pixel value of each pixel.

[0046] refer to Figure 1In (b), in a 2×2 OCL sensor structure, the intensity of the light signal received at different positions on the same microlens will be affected by the incident angle of the incident light, which can easily lead to uneven received signals due to reflection, refraction, etc. Therefore, the pixel values ​​of the four pixels sharing one microlens will be uneven, that is, the pixel values ​​of each pixel may be quite different. For example, the pixel values ​​of the four pixels can be Figure 1 The distribution of high (High), middle (Middle), middle (Middle), and low (Low) shown in (b).

[0047] (2) A quadrant photodiode (QPD) is a special photodiode that can divide the incident light into four quadrants and detect the light intensity of each quadrant independently. This structure enables the QPD to measure the position and intensity of the light spot simultaneously.

[0048] (3) The pixel rearrangement (Remosaic) algorithm, or Bayer image regeneration algorithm, can be used to rearrange pixels into a Bayer format image for output. For example, when using the Remosaic algorithm to output an image, a Quad Bayer format in which four pixels of the same color are arranged together can be converted into a Bayer format image; or an RGBCMYRaw image, i.e. a multi-channel Raw image, can be converted into a Bayer format image.

[0049] refer to Figure 2 As shown in (a), the 4 pixel blocks in the raw image data output by the above 2×2 OCL type image sensor can form a classic Bayer array (RGGB) structure, including four color channels of R, G, G, and B. This data format can be called the Quad Bayer format. The pixel block is a description of the combination result of multiple pixels. Figure 2 As shown in (b) in the figure, each pixel block in the Quad Bayer format image data can be composed of four pixels of the same color. After being processed by the Remosaic algorithm, Figure 2 As shown in (c), each pixel block can be converted from a same-color pixel array to a Bayer array (RGGB), and correspondingly, a Quad Bayer format image can be converted to a Bayer format image.

[0050] (4) Demosaic algorithm, which is used to convert Bayer format data, such as RGBRaw images, into RGB format data. It can be understood that the output of the photosensitive element only responds to light intensity and does not provide color information. Therefore, when it is necessary to capture a color image, a color filter array (CFA) can be provided. The CFA includes color filter elements above the pixels of the pixel array. The color filter elements may include red, green, and blue filter elements arranged in a so-called Bayer pattern, but other colors and / or other arrangement patterns may also be used. However, since pixels covered by filter elements of one color cannot respond to other colors, those missing color values ​​must be determined by interpolation. The process of obtaining the color value of each color for all pixels in the pixel array by interpolation is called demosaicing.

[0051] The Demosaic algorithm can be understood as a color interpolation processing algorithm, which is a processing algorithm that restores the Bayer format image data obtained from the sensor to the real-world color that conforms to the color display device. Figure 2 The Bayer format image shown in (c) is converted into RGB format data.

[0052] Figure 3 According to an embodiment of the present application, a schematic diagram of an image processing scenario is shown.

[0053] like Figure 3 As shown, the lens 101 of the electronic device 100 with image processing function can collect images of people / scenery and generate optical signals, which are transmitted to the photosensitive area on the surface of the image sensor 102; then the image sensor 102 performs photoelectric conversion to form raw image data. As mentioned above, the raw image data can be, for example, QuadBayer format image data. After the Quad Bayer format image output by the image sensor 102 is processed by the pixel re-arrangement algorithm, the obtained Bayer format image can be further transmitted to the image signal processor (Image Signal Processor, ISP) 103 for image processing. The ISP 103 can use various preset image processing algorithms to process the raw image data, and encode the processed image data into an image that the electronic device 100 can support display, which is the captured image in some shooting scenes.

[0054] The image sensor 102 may be a complementary metal-oxide semiconductor (CMOS) or a charge-coupled device (CCD). The various image processing algorithms used by the ISP 103 to process the raw image data may include a black level compensation (BLC) algorithm, a lens shading correction (LSC) algorithm, an automatic white balance (AWB) algorithm, a demosaic algorithm, a color correction matrix (CCM) and a gamma correction algorithm. The image data obtained by the above-mentioned image processing algorithms may be RGB format image data, and after being processed by the Joint Photographic Experts Group (JPEG) encoding, an image in a format such as JPG or JPEG may be generated, i.e., an image supported for display by the electronic device 100.

[0055] However, as mentioned above, the 2×2 OCL sensor structure is prone to receiving signal imbalance due to reflection, refraction, etc., which in turn causes the output sensor raw data (Sensor Raw) to have problems such as pseudo color and black edges after the pixel remosaic algorithm. Figure 4a As shown, the black edge problem that appears on the edge of the building image captured by the electronic device 100 is caused by the fact that after the original image data output by the 2×2 OCL type sensor is processed by Remosaic, there is still a large difference in the pixel values ​​of adjacent pixels.

[0056] At present, the solution to the above problem is usually to globally calculate the average of the pixel values ​​of the four pixels obtained by receiving the light signal of the 2×2 OCL type sensor. In some embodiments, this method of globally calculating the average is also called global filtering. Figure 4b As shown, the above global filtering method, for example, calculates the average value of each pixel value of the image data after the original image data is processed by the Remosaic algorithm, including Figure 4bThe "high" pixel value of the R channel in the "Remosaiced image data" shown is averaged with the lower pixel values ​​of the pixels of the Gr, Gb and B channels under the same microlens, and the average value calculated is then used as the replacement value of the pixel values ​​of the R, Gr, Gb and B channels under each channel. In this way, the pseudo-color or black edge problem existing in the image processed based on the Remosaiced image data (hereinafter also referred to as Remosaic Raw) can be corrected.

[0057] However, the above global filtering method will cause the loss of some image details, textures, etc., and some higher or lower pixel values ​​will cause the average value obtained by global calculation to be too high or too low, and the clarity of the output image cannot be guaranteed.

[0058] It can be understood that the details and texture of an image are related to edge pixels in the image. For example, edge pixels in an image can depict details of the edge of a building, or depict textures of objects such as leaves.

[0059] In order to solve the above problems, the present application provides an image processing method, which corrects the pixel values ​​of abnormal pixels (hereinafter referred to as pixels to be corrected) in edge pixels in RemosaicRaw using the pixel values ​​of non-abnormal pixels of the same color (hereinafter referred to as corrected pixels) adjacent to the pixels. For example, the statistical median of the pixel values ​​of multiple corrected pixels is used as the pixel value correction value of the pixel to be corrected, or the statistical mean of the pixel values ​​of multiple corrected pixels is used as the pixel value correction value of the pixel to be corrected.

[0060] In this way, based on the above-mentioned image processing method provided by the present application, it is possible to correct the pixels to be corrected after the original image data output by the 2×2OCL type sensor is processed by the Remosaic algorithm while retaining the details and texture of the image and ensuring the clarity of the output image, thereby solving the problem of pseudo color or black edges in the image obtained based on the image data processing after Remosaic.

[0061] Among them, the above-mentioned edge pixels can be understood as the constituent pixels of edge image data. The edge image data can be obtained by edge detection and extraction of the original image data, for example, detecting the edge of the building in the original image data and extracting the edge image data, or detecting the edge of the person in the original image data and extracting the edge image data. Whether there are pixels to be corrected in the above-mentioned edge image data can be judged by the similarity of the pixel values ​​of the color channels Gr and Gb with similar pixel values. For example, when the pixel value similarity of the Gr and Gb channel pixels is less than or equal to a certain similarity threshold, it can be judged that the 4 pixels of the corresponding pixel block are all normal pixels; conversely, when the pixel value similarity of the Gr and Gb channel pixels is greater than the above-mentioned similarity threshold, it can be judged that the 4 pixels of the corresponding pixel block are all pixels to be corrected. The above-mentioned similarity threshold can be determined based on the statistical value (such as the mean) of the pixel value similarity of the normal Gr and Gb channel pixels.

[0062] refer to Figure 5a The image processing algorithm provided in the present application can be integrated into an image processing module such as an abnormal pixel correction 104. The abnormal pixel correction 104 can be integrated into a system service or application of an operating system carried by the electronic device 100, and executed by a processor configured by the electronic device 100. The processor can be other processors different from the ISP 103, such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0063] refer to Figure 5b The abnormal pixel correction 104 may also be used as one of the components of the image processing algorithm module executed by the ISP 103, that is, the image processing method provided by the present application is executed by the ISP 103. No limitation is made here.

[0064] It can be understood that in some embodiments of the present application, the electronic device 100 or the ISP 103 for executing the image processing method provided by the present application may have Figure 5a or Figure 5b The abnormal pixel correction 104 shown in the figure and the module structures such as black level compensation (BLC), lens correction (LSC), automatic white balance (AWB), demosaic, color correction matrix (CCM) and gamma correction are shown in the figure. In other embodiments, the electronic device 100 or the ISP 103 may have a structure of Figure 5a or Figure 5b The structures shown may be more or less, but are not limiting.

[0065] Specifically, Figure 6 According to an embodiment of the present application, a schematic diagram of an implementation flow of an image processing method is shown.

[0066] Understandably, Figure 6 The execution body of each step of the implementation process shown can be the electronic device 100 or the processor such as ISP103, and there is no limitation here. Figure 6 The execution entities of each step will not be repeatedly introduced when explaining the contents of each step in the implementation process shown.

[0067] like Figure 6 As shown, the implementation process of the image processing method provided in the embodiment of the present application may include the following steps:

[0068] 601: Obtaining original image data after pixel rearrangement processing.

[0069] Exemplarily, the raw image data may be image data acquired based on a 2×2 OCL type sensor and processed by a Remosaic algorithm, namely, the Remosaic Raw.

[0070] 602: Extract edge image data from the original image data, including obtaining pixel values ​​of constituent pixels of each pixel block at the edge of the original image.

[0071] Exemplarily, the electronic device 100 or the ISP 103 may extract edge image data from the original image data based on the gradient information of the original image data, etc. The extracted edge image data includes a plurality of pixel blocks constituting the edge of the original image, each pixel block may include a plurality of pixels, namely the above-mentioned constituent pixels, for example, the four color channels constituting one pixel block, namely, the four pixels of R, Gr, Gb, and B.

[0072] It can be understood that the above-mentioned gradient information basically describes the speed and direction of change of the pixel value (or color, brightness, etc.) in the original image data. In a digital image (such as the above-mentioned original image data), each pixel has one or more gradient values ​​associated with it, which are usually obtained by calculating the difference or change rate of the pixels around the pixel. These gradient values ​​can form a gradient map to represent the size and direction of changes in color or brightness. Gradient information can help the algorithm understand the edges, textures and other important structural features in the image. By using this information, the algorithm can more effectively restore a high-resolution image from a low-resolution image while maintaining the clarity and details of these key features. In an embodiment of the present application, the edge information in the image can be obtained using the gradient information, and the edge image data can be extracted. In other embodiments, the electronic device 100 or the ISP 103 executes the image processing algorithm provided by the present application, and can also obtain texture information in the image, etc., and correspondingly extract texture image data to correct problems such as pseudo-color of some textures in the image, which is not limited here.

[0073] It can be understood that in the embodiments of the present application, referring to the above Figure 5a or Figure 5b As shown, the gradient information of Remosaic Raw refers to the gradient information used when directly acquiring image data from the image sensor 102. After the pixel re-arrangement (Remosaic) process, the gradient information can still be used to accurately extract edge image data even in low light, high noise or other shooting environments that are not conducive to taking clear images.

[0074] 603: Calculate the pixel value similarity between the Gr and Gb color channels in each pixel block.

[0075] Exemplarily, the electronic device 100 or the ISP 103 can obtain the pixel values ​​of the constituent pixels of the Gr and Gb color channels in each pixel block from the extracted edge image data, and calculate the similarity between the two, that is, calculate the similarity of the pixel values ​​of the Gr and Gb color channels. It can be understood that in normal image data, the Gr / Gb channel pixel values ​​of each pixel block are relatively close or even the same, while the pixel value gap between the R and B channels is usually large. Therefore, the embodiment of the present application can determine whether the constituent pixels of each pixel block are abnormal by calculating the similarity of the pixel values ​​of the constituent pixels of the Gr and Gb color channels. The specific judgment process can refer to the relevant description in the following steps 604 to 605, which will not be repeated here.

[0076] refer to Figure 7aAs shown, in the edge image data extracted from Remosaic Raw, each pixel block may include pixel values ​​of four color channels: R, Gr, Gb, and B. Before the pixels are rearranged, each pixel block composed of four pixels is also a pixel block of four color channels: R, Gr, Gb, and B. When the electronic device 100 or the ISP 103 executes the image processing method provided by the present application, the pixel values ​​of the pixels of the Gr and Gb color channels in each pixel block may be calculated respectively, for example, Figure 7a The similarity of pixel values ​​between the Gr color channel pixel 711 and the Gb color channel pixel 712 in the R color channel pixel block, or the similarity of pixel values ​​between the Gr color channel pixel 711 and the Gb color channel pixel 712 in the Gb color channel pixel block, etc.

[0077] 604: Determine whether the similarity of the pixel values ​​corresponding to each pixel block is greater than a similarity threshold.

[0078] If the judgment result is yes, then continue to execute the following steps 605 to 610 to correct the pixels determined to be abnormal.

[0079] If the judgment result is no, the process ends, that is, there is no need to correct abnormal pixels.

[0080] Exemplarily, the similarity threshold mentioned above can be determined based on the similarity statistics of the normal image data. For example, the similarity threshold mentioned above can be the statistical mean of the similarity of the pixel values ​​of the Gr and Gb color channel pixels in the normal image data, or can be the statistical median or mode of the similarity of the pixel values ​​of the Gr and Gb color channel pixels in the normal image data, etc., without limitation here. In some embodiments, the similarity threshold mentioned above can be different based on the captured images of different scenes or different light environments, different noise environments, etc. For example, a group or one similarity threshold corresponding to different scenes or different environments can be set in the electronic device 100 or the ISP 103, without limitation here.

[0081] Based on this, if the similarity of the pixel values ​​corresponding to each pixel block calculated in the above step 603 is greater than the above similarity threshold, it indicates that the pixel values ​​of each component pixel or part of the component pixels in the corresponding pixel block may be abnormal. At this time, the electronic device 100 or the ISP 103 can continue to execute the following steps 605 to 610, and use the 4 pixels in the corresponding pixel block as abnormal pixels to be corrected, that is, pixels to be corrected, and perform pixel value correction processing. On the contrary, if the similarity of the pixel values ​​corresponding to each pixel block calculated in the above step 603 is less than or equal to the above similarity threshold, it indicates that the pixel values ​​of each component pixel or part of the component pixels in the corresponding pixel block are normal and do not need to be corrected.

[0082] 605: Determine that each pixel in the pixel block whose pixel value similarity is greater than a similarity threshold is an abnormal pixel.

[0083] Exemplarily, based on the judgment result of comparing the similarity of the pixel values ​​corresponding to each pixel block with the similarity threshold in step 604, the electronic device 100 or the ISP 103 can determine that if the judgment result is yes, that is, when the pixel value similarity is greater than the similarity threshold, each component pixel in the corresponding pixel block is determined to be an abnormal pixel, that is, a pixel to be corrected. Figure 7b As shown, the light-colored area, such as the pixels of each pixel block in the frame 721 (not all pixels are selected), can be determined as normal pixels, considering that the pixel value similarity of the Gr and Gb color channel pixels in the corresponding pixel block is less than or equal to the similarity threshold. Figure 7b As shown, the constituent pixels of each pixel block in the dark area, such as box 722 (not all are selected), can be determined as abnormal pixels, considering that the pixel value similarity of the Gr and Gb color channel pixels in the corresponding pixel blocks is greater than the similarity threshold.

[0084] It can be understood that the abnormal pixels determined in step 605 are also pixels to be corrected during the execution of steps 606 to 609 described below.

[0085] 606: Based on the extracted edge image data, construct a same-color pixel correction map corresponding to each color channel.

[0086] Exemplarily, the same-color pixel correction map corresponding to each color channel may include pixel values ​​of the same-color pixels of each color channel, and the arrangement of each same-color pixel may be the same as the arrangement of the corresponding pixel block in the edge image data. Based on the essence of the above-mentioned same-color pixel correction map being the distribution array of each component pixel of the same color channel, in some embodiments, the same-color pixel correction map may also be described as a pixel array of the corresponding color channel.

[0087] refer to Figure 7c As shown in FIG. 1 , in the edge image data extracted from Remosaic Raw, each pixel block may include pixel values ​​of four color channels: R, Gr, Gb, and B. Taking the pixels of the R color channel as an example, the pixel values ​​of each component pixel of the R color channel of each pixel block in the edge image data are extracted, and the pixel blocks to which each component pixel belongs are rearranged according to the arrangement of the pixel blocks in the edge image data to form Figure 7d The same color pixel correction map of the R color channel shown in FIG. 1 can be recorded as R-Map. The pixel block to which each component pixel belongs can be referred to as Figure 7c The pixel block 730 shown is another pixel block.

[0088] Continue to refer Figure 7bAs shown, the same-color pixel correction map of the R color channel may include each component pixel of the R color channel, and the component pixel may include the abnormal pixels determined by the above steps 604 to 605, for example Figure 7d The abnormal pixel 740 of the R channel shown may also be a normal pixel determined by the above step 604 .

[0089] Similarly, the same color pixel correction maps of other color channels such as Gr, Gb, B, etc. can also refer to the above Figure 7c to Figure 7d The construction process of the R-Map illustrated in the example has been completed, and this application will not go into details here.

[0090] 607: Obtain pixel values ​​of multiple same-color pixels in the same-color pixel correction map that meet a distance condition with the pixel to be corrected.

[0091] Exemplarily, the above-mentioned same-color pixels that meet the distance condition with the pixels to be corrected may include multiple non-abnormal same-color pixels within a certain distance range adjacent to the pixels to be corrected in the same-color pixel correction map, that is, the above-mentioned correction pixels, and may also include abnormal pixels within the above-mentioned certain distance range, that is, other pixels to be corrected.

[0092] It can be understood that in order to improve the accuracy of correction, the above-mentioned adjacent may include adjacent within a certain distance range, and the distance range may also be referred to as an adjacent range. Figure 7d As shown, for the currently updated pixel to be corrected 741, its adjacent same-color pixels may include a plurality of normal same-color pixels within the range of [-3,3] with the pixel to be corrected 741 as the center, i.e., correction pixels, and may also include one or more abnormal pixels. Correspondingly, the above distance condition may be, for example, that the distance between the pixel to be corrected is within a distance range corresponding to 3 pixels, or that the number of pixels spaced between the pixel to be corrected is within 2, i.e., the number of pixels spaced between the pixel to be corrected is within a preset number range.

[0093] refer to Figure 7d As shown, the same-color pixels adjacent to the pixel to be corrected 741 currently being updated may include normal pixels or abnormal pixels of the same color channel, wherein the normal pixels of the same color channel serve as the correction pixels for correcting the pixel values ​​of the pixel to be corrected. In order to improve the accuracy of pixel value correction, the electronic device 100 or the ISP 103 may continue to execute the following step 608, remove the pixel values ​​of the abnormal pixels of the same color channel adjacent to the pixel to be corrected, and then continue to execute the correction process of the following step 609. For details, please refer to the description of the relevant steps below, which will not be repeated here.

[0094] It can be understood that the larger the adjacent range is, the more pixel values ​​of the same-color pixels in the adjacent range obtained by the electronic device 100 or the ISP 103 when executing this step are greater, and the corresponding correction result of the pixel to be corrected when executing the following steps 608 to 609 can be more accurate. Figure 7d The adjacent range [-3, 3] in the example may also be, for example, an adjacent range [-4, 4], an adjacent range [-5, 5], etc., and is not limited here.

[0095] 608: After deleting abnormal pixels among a plurality of pixels of the same color adjacent to the pixel to be corrected, calculate pixel value statistics of each non-abnormal corrected pixel.

[0096] For example, considering that the pixel value of an abnormal pixel may be abnormally large or small, which may eventually lead to an unsatisfactory correction result, the electronic device 100 or ISP 103 in the embodiment of the present application may remove the pixel values ​​of the abnormal pixels among the acquired multiple same-color pixels, that is, after removing the abnormal pixels among the multiple same-color pixels adjacent to the pixel to be corrected, the pixel correction calculation is performed. The pixel correction calculation may, for example, include the process of calculating the pixel value statistics of each correction pixel within the adjacent range of the pixel to be corrected (i.e., satisfying the distance condition) in step 608.

[0097] refer to Figure 7d As shown, the electronic device 100 or the ISP 103 deletes abnormal pixels among a plurality of pixels of the same color adjacent to the pixel to be corrected 741, for example Figure 7d The dark areas within the adjacent range [-3,3] are shown to mark each pixel.

[0098] It can be understood that the pixel value statistical value calculated above may include any one of the statistical values ​​such as the average value (abbreviated as mean), the median value and the mode value, and no limitation is made here.

[0099] 609: Correct the pixel value of the corresponding pixel to be corrected based on the pixel value statistics.

[0100] Exemplarily, the electronic device 100 or ISP 103 corrects the pixel value of the corresponding pixel to be corrected based on the pixel value statistics. For example, the pixel value statistics may be used to replace the pixel value of the corresponding pixel to be corrected, that is, the pixel value statistics calculated in the above step 608 are used as the new pixel value of the currently updated pixel to be corrected.

[0101] In other embodiments, the electronic device 100 or ISP 103 corrects the pixel value of the corresponding pixel to be corrected based on the pixel value statistics, which may also include correcting the pixel value of the currently updated pixel to be corrected according to a certain weight, etc., which is not limited here.

[0102] 610: Using edge image data after abnormal pixel correction to replace edge image data with abnormal pixels in original image data, to obtain corrected original image data.

[0103] Exemplarily, after the pixel values ​​of all pixels to be corrected in the extracted edge image data are corrected based on the above steps 605 to 609, the pixel values ​​of all pixels to be corrected are updated to the corrected pixel values, such as the pixel value statistics calculated in the above step 608, and then the electronic device 100 or ISP 103 can add the edge image data after the abnormal pixel correction to the original image data as replacement data for the corresponding edge image data in the original image data. The corresponding edge image data in the original image data can be the edge image data extracted from the original image data in the above step 602, or the corresponding data of the part of the edge image data.

[0104] It can be understood that when the electronic device 100 or ISP 103 extracts the edge image data in step 602, the part of the data in the original image data can be deleted accordingly. Furthermore, when the electronic device 100 or ISP 103 executes step 610, the edge image data after the abnormal pixel correction is completed can be used as the supplementary data of the edge image data extracted and deleted in the original image data, and merged with the rest of the data in the original image data to obtain the original image data with the abnormal pixel corrected.

[0105] In other embodiments, the electronic device 100 or ISP 103 may also retain the integrity of the original image data and not delete the extracted edge image data when performing the above step 602 to extract the edge image data. Furthermore, when the electronic device 100 or ISP 103 performs this step 610, the edge image data after the abnormal pixel correction is completed may be used as replacement data or update data for the corresponding edge image data in the original image data to correct and update the original image data.

[0106] It is understandable that based on the above Figure 6 In the implementation process of the image processing method from step 601 to step 610 shown in the figure, the electronic device 100 or the ISP 103 can perform abnormal pixel correction on the raw image data (Remosaic Raw) output by the 2×2 OCL type image sensor and processed by the Remosaic algorithm to eliminate problems such as pseudo color or black edges caused by unbalanced received signals, so that the electronic device 100 using the 2×2 OCL type image sensor can adapt to different reflection or refraction environments and capture clear images without problems such as pseudo color or black edges.

[0107] As an example, Figure 8 According to an embodiment of the present application, a schematic diagram of image processing effects before and after correction is shown.

[0108] refer to Figure 8 As shown, before using the image processing method provided by the present application to correct abnormal pixels, that is, Figure 8 As shown in the “before correction”, the image data output by the 2×2 OCL type sensor is processed by the pixel remosaic algorithm, and the edge image data corresponding to the edge of the building in the image can be presented as follows: Figure 8 The Bayer format image (a) before correction is shown. After being processed by other image processing algorithms and demosaicing algorithms, the final output captured image is, for example, Figure 8 The captured image (b) shows a black border.

[0109] Continue to refer Figure 8 As shown, after using the image processing method provided by the present application to correct abnormal pixels on the original image data (Remosaic Raw) processed by the Remosaic algorithm, that is, Figure 8 After the “correction” shown, the edge image data corresponding to the edge of the building in the image can be presented as Figure 8 The "corrected" Bayer format image (a') is shown. After being processed by other image processing algorithms and demosaicing algorithms, the final output captured image is, for example, Figure 8 The captured image (b') does not have any black edges.

[0110] Similarly, in other embodiments, after performing abnormal pixel correction on the original image data (Remosaic Raw) processed by the Remosaic algorithm using the image processing method provided by the present application, it is also possible to eliminate the pseudo color of the captured image that was originally output with pseudo color problems, or eliminate the pseudo color and black edges of the captured image that was originally output with problems such as pseudo color and black edges, etc., so that the image captured by the electronic device 100 is clearer with true colors and clear edges.

[0111] In other embodiments, the implementation process of an image processing method provided in the embodiment of the present application can also refer to Fig. 9 shown. Fig. 9 According to the embodiment of the present application, another implementation flow diagram of an image processing method is shown. It can be understood that Fig. 9 The execution entities of each step of the illustrated process may also be the electronic device 100 or the ISP 103, which is not limited here.

[0112] Specifically, Fig. 9As shown, the implementation process includes the following steps ① to ⑨:

[0113] ① Obtain the problematic Remosaic Raw. The problematic Remosaic Raw may correspond to the above Figure 6 The original image data after the pixel rearrangement processing in the steps shown. It can be understood that whether the original image data (such as RemosaicRaw) has a problem can be determined based on whether there are pixels to be corrected (i.e., abnormal pixels) in the original image data. If there is a problem, that is, there are abnormal pixels in the original image data, the electronic device 100 or the ISP 103 can implement Fig. 9 Steps ① to ⑨ shown in the figure, or implement the above Figure 6 Steps 601 to 610 are shown.

[0114] Specifically, the detailed execution process of obtaining the original image data in this step ① can refer to the relevant description in the above step 601, which will not be repeated here.

[0115] ② Extract edge image data based on gradient and other information.

[0116] Specifically, the detailed extraction process of step ② can refer to the relevant description in the above step 602, and the relevant description of information such as gradient can also refer to the relevant description in the above step 602, which will not be repeated here.

[0117] ③ Obtain the pixel values ​​of the four pixels that make up each pixel block in the edge image data.

[0118] Specifically, the detailed process of obtaining the pixel values ​​of the constituent pixels of each pixel block in step ③ can refer to the relevant description in the above step 602, which will not be repeated here.

[0119] ④ Calculate the pixel value similarity S of the constituent pixels of the Gr and Gb color channels in each pixel block.

[0120] Specifically, the detailed execution process of calculating the similarity of the pixel values ​​of the Gr and Gb color channels corresponding to each pixel block in step ④ can refer to the relevant description in the above step 603, which will not be repeated here.

[0121] ⑤ Based on the judgment result that the similarity S is greater than the similarity threshold, it is determined that the four pixels in the corresponding pixel block are abnormal pixels.

[0122] Specifically, the detailed judgment process of step ⑤ can refer to the relevant description in the above step 604, and the process of determining abnormal pixels in step ⑤ can refer to the relevant description in the above step 605, which will not be repeated here.

[0123] ⑥ Extract the pixel values ​​of the four color channels of each pixel block in the edge image data, and construct a correction map of the same-color pixels in each color channel.

[0124] Specifically, the detailed execution process of constructing the same-color pixel correction map of each color channel in step ⑥ can refer to the relevant description in the above step 606 and Figure 7c to Figure 7d The above and related descriptions will not be repeated here.

[0125] ⑦ Based on the same-color pixel correction map, after removing the pixel values ​​of multiple same-color pixels adjacent to the abnormal pixel, the pixel value statistics are calculated.

[0126] Specifically, the detailed execution process of calculating the pixel value statistics of the same-color pixels adjacent to the abnormal pixel based on the same-color pixel correction map in step ⑦ can refer to the relevant description in the above steps 607 to 608, which will not be repeated here.

[0127] ⑧Use the calculated pixel value statistics to correct the pixel values ​​of the corresponding abnormal pixels and perform abnormal pixel correction.

[0128] Specifically, the execution process of correcting the corresponding abnormal pixels using the pixel value statistics of adjacent pixels of the same color in step ⑧ can refer to the relevant description in the above step 609, which will not be repeated here.

[0129] ⑨ The edge image data after abnormal pixel correction is merged with the rest of the image data of Remosaic Raw to obtain the corrected Remosaic Raw.

[0130] Specifically, the image processing execution process of obtaining the original image data (Remosaic Raw) after pixel correction in step ⑨ can refer to the relevant description in the above step 610, which will not be repeated here.

[0131] Fig.10 According to an embodiment of the present application, a hardware structure diagram of an electronic device is shown. The electronic device may be the electronic device 100 described above, or may be any electronic device including the ISP 103 described above.

[0132] like Fig.10As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identity Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0133] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0134] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (Application Processor, Ap), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (Digital Signal Processor, DSP), a baseband processor, and / or a neural network processor (Neural-Network Processing Unit, NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0135] In some embodiments of the present application, the electronic device 100 can implement the image processing method provided by the present application through the processor 110, for example, Figure 6 or Fig. 9In other embodiments of the present application, the electronic device 100 may also implement the image processing method provided by the present application through a processing unit of the processor 110, such as the ISP 103, for example, Figure 6 or Fig. 9 The steps of the implementation process of the image processing method shown are not limited here.

[0136] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0137] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0138] In the embodiment of the present application, the processor 110 of the electronic device 100 can control the above Figure 6 or Fig. 9 The execution process of each step in the implementation flow of the image processing method shown.

[0139] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM card interface, and / or a Universal Serial Bus (USB) interface, etc.

[0140] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0141] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0142] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0143] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0144] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 100 .

[0145] The wireless communication module 160 can provide wireless communication solutions for application in the electronic device 100, including Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) network), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), infrared technology (IR), etc.

[0146] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The above wireless communication technology may include Global System For Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The above-mentioned GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the Beidou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS) and / or the Satellite Based Augmentation System (SBAS).

[0147] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. For example, an image captured by the electronic device 100 based on a shooting function is displayed. It can be understood that based on the image processing method provided by the present application, the image captured by the shooting function of the electronic device 100 can no longer have problems such as pseudo color or black edges.

[0148] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-LED, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0149] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0150] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted to the camera sensor through the lens. The light signal is converted into an electrical signal, and the camera sensor transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193. In the embodiment of the present application, ISP can be, for example, the above-mentioned Figure 5a or Figure 5b The ISP 103 in the image processing scenario shown can execute various image processing algorithms to perform image processing on the original image data output by the image sensor 102 after being processed by the Remosaic algorithm.

[0151] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0152] A digital signal processor (DSP) is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals.

[0153] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0154] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0155] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0156] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0157] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake.

[0158] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0159] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some shooting scenarios, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0160] The ambient light sensor 180L is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed brightness of the ambient light. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.

[0161] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect a touch operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194.

[0162] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0163] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0164] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0165] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or removing the SIM card from the SIM card interface 195 .

[0166] Fig.11 According to an embodiment of the present application, a schematic diagram of the software structure of an operating system of an electronic device is shown.

[0167] It is understood that the operating system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes the Android system of the layered architecture as an example to exemplify the system software structure of the electronic device 100.

[0168] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Android TM The system is divided into four layers, from top to bottom: application layer, application framework layer, Android TM Runtime (Android TM runtime) and system libraries, as well as the kernel layer.

[0169] like Fig.11 As shown, the application layer may include a series of application packages, which may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0170] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0171] The application framework layer may include a window manager, content provider, view system, telephony manager, resource manager, notification manager, and the like.

[0172] In the embodiment of the present application, the application framework layer may also include a camera service. The camera service may integrate the above Figure 5a or Figure 5b The related algorithms or functional modules of abnormal pixel correction 104 in the illustrated scene are used to implement the image processing method provided in the embodiment of the present application.

[0173] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0174] Content providers are used to store and retrieve data and make it accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0175] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0176] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0177] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0178] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0179] Android TM Runtime includes core libraries and virtual machines. Android TM runtime is responsible for Android TM Scheduling and management of the system.

[0180] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0181] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0182] The system library may include multiple functional modules, such as: Surface Manager, Media Libraries, 3D graphics processing library (such as openGL ES), 2D graphics engine (such as SGL), etc.

[0183] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0184] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0185] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0186] A 2D graphics engine is a drawing engine for 2D drawings.

[0187] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0188] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0189] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0190] The embodiments of the present application also provide a computer program product for implementing the image processing methods provided in the above embodiments.

[0191] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program module or module code executed on a programmable system, and the programmable system includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0192] A computer program module or module code may be applied to input instructions to perform the functions described herein and generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0193] Module code can be implemented with high-level modular language or object-oriented programming language to communicate with the processing system. When necessary, module code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0194] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0195] References to "one embodiment" or "an embodiment" in the specification mean that the specific features, structures, or characteristics described in conjunction with the embodiment are included in at least one exemplary implementation or technology disclosed according to the embodiment of the present application. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily all refer to the same embodiment.

[0196] The disclosure of the embodiment of the present application also relates to an operating device for executing the text. The device can be specially constructed for the required purpose or it can include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer-readable medium, such as, but not limited to any type of disk, including a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application-specific integrated circuit (ASIC) or any type of medium suitable for storing electronic instructions, and each can be coupled to a computer system bus. In addition, the computer mentioned in the specification may include a single processor or may be an architecture involving multiple processors for increased computing power.

[0197] In addition, the language used in this specification has been primarily selected for readability and instructional purposes and may not be selected to describe or limit the disclosed subject matter. Therefore, the present application embodiment disclosure is intended to illustrate rather than limit the scope of the concepts discussed herein.

Claims

1. An image processing method, applied to an electronic device, characterized in that: include: Acquire first image data collected by an image sensor, wherein a single microlens in the image sensor corresponds to a plurality of pixels; Remosaicing the first image data to obtain second image data; Determining pixels to be corrected among edge pixels of the second image data; The pixel value of the pixel to be corrected is corrected by using the pixel value of the corrected pixel that meets the distance condition with the pixel to be corrected to obtain the third image data.

2. The method according to claim 1, characterized in that The second image data includes image data in a Bayer format, the second image data includes a plurality of pixel units, a first pixel unit among the plurality of pixel units includes a plurality of pixels corresponding to the single microlens; and The first pixel unit includes multi-color channel data, and the multi-color channel data includes pixel values ​​of the plurality of pixels processed by each color channel filter.

3. The method according to claim 2, characterized in that The step of determining pixels to be corrected among edge pixels of the second image data comprises: It is detected that the similarity between the first pixel value in the first color channel and the second pixel value in the second color channel is greater than a first threshold, and a plurality of pixels constituting the first pixel unit are determined to be pixels to be corrected, wherein: The first color channel and the second color channel use the same color filter.

4. The method according to claim 3, characterized in that: The multi-color channel data includes four color channel data of R, Gr, Gb, and B; and The first color channel is Gr color channel data, and the second color channel is Gb color channel data; or, The first color channel is Gb color channel data, and the second color channel is Gr color channel data.

5. The method according to claim 3 or 4, characterized in that: The method for determining the first threshold includes: The first threshold is determined based on the average similarity between the pixel value of each pixel unit of the fourth image data in the first color channel and the pixel value in the second color channel, wherein: The pixel to be corrected does not exist in the fourth image data.

6. The method according to claim 1, characterized in that The step of correcting the pixel value of the pixel to be corrected by using the pixel value of the corrected pixel that satisfies the distance condition with the pixel to be corrected comprises: Acquire a plurality of same-color pixels in the same color channel as the first pixel to be corrected from edge pixels of the first image data, wherein the plurality of same-color pixels include the pixel to be corrected; Determine, from the plurality of pixels of the same color, a plurality of correction pixels that satisfy a distance condition with the first pixel to be corrected; The pixel value of the first pixel to be corrected is corrected by using the pixel values ​​of the plurality of corrected pixels.

7. The method according to claim 6, characterized in that The distance conditions include: The distance between the pixels to be corrected is less than or equal to the distance corresponding to the first number of pixels, or, The number of pixels spaced from the pixel to be corrected is less than or equal to a second number.

8. The method according to claim 6 or 7, characterized in that: The step of correcting the pixel value of the first pixel to be corrected by using the pixel values ​​of the plurality of corrected pixels comprises: Performing statistical calculation on the pixel values ​​of the plurality of corrected pixels to obtain a first statistical value, wherein: The first statistical value is used as a pixel value correction value of the first pixel to be corrected.

9. The method according to claim 8, characterized in that The first statistical value includes any one of a median, an average or a mode in a statistical sequence composed of pixel values ​​of the plurality of corrected pixels.

10. The method according to any one of claims 1 to 9, characterized in that The image sensor is a 2×2 OCL type sensor.

11. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the image processing method described in any one of claims 1 to 10.

12. A computer readable medium, characterized in that The readable medium stores instructions, which, when executed on a computer, enable the computer to execute the image processing method according to any one of claims 1 to 10.

13. A computer program product, characterized in that include: Computer instructions, when the computer instructions are executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 10.

Images