Image sensors and electronics

Abstract

The present application provides an image sensor and an electronic device, which can make the final generated image closer to the real effect. The image sensor includes: a filter unit array, including multiple filter unit groups, each of the multiple filter unit groups includes a white filter unit and a color filter unit; a pixel unit array, located below the filter unit array, the pixel units in the pixel unit array correspond to the filter units in the multiple filter unit groups one by one, the white pixel units in the pixel unit array are used to receive the light signal passing through the corresponding white filter unit, the color pixel units in the pixel unit array are used to receive the light signal passing through the corresponding color filter unit, and the light intensity information of the light signal sensed by the pixel unit array is used to determine the pixel value of the color pixel unit and the pixel value of the white pixel unit to generate the target image of the photographed object.

Description

[0001] This application claims priority to a Chinese invention application filed with the China Patent Office on May 15, 2020, with application number 202010410639.2 and invention name “Image sensor and electronic device”, all contents of which are incorporated into this application by application. Technical Field

[0002] Embodiments of the present application relate to the field of images, and more specifically, to image sensors and electronic devices. Background Art

[0003] Imaging systems in electronic devices often rely on image sensors to create an electronic display of a visible image. Examples of such image sensors include charge-coupled device (CCD) image sensors and active pixel sensor (APS) devices, where APS devices are often also called CMOS sensors because they can be manufactured in a complementary metal oxide semiconductor (CMOS) process.

[0004] These image sensors include a plurality of light-sensitive pixels, often arranged in a regular pattern of rows and columns. In order to capture a color image, it is necessary to accumulate light signals of a specific wavelength on different pixels, that is, to receive signals of a specific color, so a color filter is installed in the image sensor. For example, a filter having a Bayer array configured to include each color in red, green, and blue (RGB) is generally used.

[0005] In order to make different pixels in the pixel array sensitive to only part of the visible spectrum, the color filters need to be set to different colors to pass the light signals of the corresponding colors, which reduces the amount of light reaching each photosensitive pixel, thereby reducing the light sensitivity of each photosensitive pixel. In addition, because the image sensor size is usually limited when used in mobile devices, the corresponding photosensitive area of ​​the pixel array is also limited, so the performance of taking pictures will be limited in low-light environments. Summary of the invention

[0006] The present application provides an image sensor and an electronic device, which can make the final generated image closer to the real effect.

[0007] In a first aspect, an image sensor is provided, which includes: a filter unit array, including a plurality of filter unit groups, each of the plurality of filter unit groups including a white filter unit and a color filter unit; a pixel unit array, located below the filter unit array, the pixel units in the pixel unit array corresponding one to one to the filter units in the plurality of filter unit groups, the white pixel units in the pixel unit array are used to receive light signals passing through the corresponding white filter units, the color pixel units in the pixel unit array are used to receive light signals passing through the corresponding color filter units, and the light intensity information of the light signals sensed by the pixel unit array is used to determine the pixel values ​​of the color pixel units and the pixel values ​​of the white pixel units to generate a target image of the photographed object.

[0008] Based on the above technical solution, the image sensor in the present application may include a white filter unit. Compared with a filter unit array that only has a monochrome filter unit, the amount of light entering can be greatly improved. Correspondingly, the amount of light entering the entire image sensor can be increased, so that the performance of the image sensor is not affected even in a low-light environment.

[0009] In addition, the image sensor in the present application can generate a target image using different fusion methods under different ambient light conditions, so that the target image finally generated is closer to the real effect.

[0010] In a possible implementation, each of the multiple filter unit groups includes 4×4 filter units, the ratio of white filter units to color filter units in each filter unit group is 1:1, and the white filter units and the color filter units are arranged alternately.

[0011] The 50% white filter unit can ensure that the white pixel unit has a higher spatial sampling rate, which is beneficial for the subsequent Remosaic algorithm to obtain better high-resolution grayscale images.

[0012] In addition, the white filter units and the color filter units can be arranged alternately, that is, in any row or any column, there will not be two adjacent white filter units or color filter units, so that the crosstalk can be relatively even. In addition, the white filter units and the color filter units are distributed relatively evenly, which can achieve a higher color space sampling rate, which is conducive to the restoration of subsequent images.

[0013] In a possible implementation, the color filter unit includes two red filter units, two blue filter units, and four green filter units.

[0014] In a possible implementation manner, the filter units on one diagonal line in each filter unit group are white filter units, and the filter units on another diagonal line are two red filter units and two blue filter units.

[0015] In a possible implementation manner, the two red filter units on the other diagonal line are arranged at a common vertex angle, and the two blue filter units are also arranged at a common vertex angle.

[0016] In a possible implementation, the white filter unit is located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the third row and second column, and the fourth row and first column of each filter unit group; the blue filter unit is located in the first row and fourth column, and the second row and third column of each filter unit group; the green filter unit is located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

[0017] In a possible implementation, the two red filter units on the other diagonal line are arranged at a common vertex angle, and the two blue filter units are arranged separately; or the two blue filter units on the other diagonal line are arranged at a common vertex angle, and the two red filter units are arranged separately.

[0018] In a possible implementation, the white filter unit is located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the first row and fourth column, and the fourth row and first column of each filter unit group; the blue filter unit is located in the second row and third column, and the third row and second column of each filter unit group; the green filter unit is located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

[0019] In a possible implementation, the white filter unit is located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the second row and third column, and the third row and second column of each filter unit group; the blue filter unit is located in the first row and fourth column, and the fourth row and first column of each filter unit group; the green filter unit is located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

[0020] In a possible implementation manner, the red filter units and the blue filter units on the other diagonal line are arranged alternately.

[0021] In a possible implementation, the white filter unit is located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the second row and third column, and the fourth row and first column of each filter unit group; the blue filter unit is located in the first row and fourth column, and the third row and second column of each filter unit group; the green filter unit is located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

[0022] In a possible implementation manner, the filter units on one diagonal line in each filter unit group are white filter units, and the filter units on another diagonal line are green filter units.

[0023] In a possible implementation, the white filter unit is located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the second row and first column, and the fourth row and third column of each filter unit group; the blue filter unit is located in the first row and second column, and the third row and fourth column of each filter unit group; the green filter unit is located in the first row and fourth column, the second row and third column, the third row and second column, and the fourth row and first column of each filter unit group.

[0024] In a possible implementation, it also includes a processor, which is used to: determine light intensity information based on the light signal sensed by the pixel unit array; determine the pixel value of the color pixel unit and / or the pixel value of the white pixel unit to generate a target image of the photographed object based on the light intensity information.

[0025] In a possible implementation, a processor is further included, which is used to: determine the light intensity based on the light signal sensed by the pixel unit array; when the light intensity is greater than or equal to a first preset threshold, use the pixel value of the color pixel unit to generate the target image.

[0026] In a possible implementation, a processor is further included, which is used to: when the light intensity is less than the first preset threshold and greater than or equal to the second preset threshold, determine the texture information according to the pixel value of the white pixel unit, determine the color information and pixel value of the position of the white pixel unit according to the pixel value of the white pixel unit and the color of the pixel units around the white pixel unit, and generate the target image according to the color information and pixel value of the position of the white pixel unit and the pixel value of the color pixel unit.

[0027] In a possible implementation, a processor is further included, which is used to: when the light intensity is less than the second preset threshold and greater than or equal to the third preset threshold, generate first image data according to the pixel value of the white pixel unit, generate second image data according to the pixel value of the color pixel unit, use a correction coefficient α to correct the first image data to obtain corrected first image data; and fuse the corrected first image data and the second image data to obtain the target image, wherein the correction coefficient α is determined according to the size of the light intensity, 0<α<1.

[0028] In a possible implementation, a processor is further included, which is used to: when the light intensity is less than the third preset threshold, generate first image data according to the pixel value of the white pixel unit, generate second image data according to the pixel value of the color pixel unit, and fuse the first image data and the second image data to obtain the target image.

[0029] The above fusion method can ensure that the generated target image can be close to the real effect regardless of whether it is in a strong light environment or a weak light environment.

[0030] In a possible implementation, it also includes: a microlens array, including multiple microlenses, located above the filter unit array, and used to converge the light signal returned by the photographed object to the filter unit array, wherein one microlens among the multiple microlenses corresponds to at least one filter unit in the filter unit array.

[0031] In a possible implementation manner, the microlenses in the microlens array correspond one-to-one to the filter units in the filter unit array.

[0032] In a second aspect, an electronic device is provided, comprising: the image sensor in the above-mentioned first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic block diagram of an image processing device provided in an embodiment of the present application.

[0034] Figure 2 Schematic diagram of color distribution of a traditional filter unit group.

[0035] Figure 3 is a schematic top view of another image sensor provided in an embodiment of the present application.

[0036] Figure 4 yes Figure 3 A schematic cross-sectional view of the image sensor along the AA' direction.

[0037] Figure 5 Schematic diagram of images taken under two different filter structures.

[0038] Figure 6-Figure 11 Schematic diagram of the arrangement of the filter units in the filter unit group provided in the embodiment of the present application.

[0039] Figure 12-13 Schematic diagram of the structure of the image sensor provided in the embodiment of the present application.

[0040] Figure 14-18 It is a schematic flow chart of the fusion process provided in the embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0042] The image processing device utilizes the photoelectric conversion function of the pixel array to convert the optical image of the photographed object into an electrical signal that is in a corresponding proportional relationship with the optical image, thereby obtaining an image of the photographed object. Figure 1 A schematic block diagram of an image processing device 100 is shown, and the image processing device 100 may refer to any electronic device, for example, the image processing device 100 may be a mobile phone; or, the image processing device 100 may also be a part of an electronic device, for example, it may be a camera module in the electronic device, but the embodiments of the present application are not limited to this.

[0043] like Figure 1As shown, the image processing device 100 generally includes a pixel array 101 (or may also be called a photoelectric conversion unit 101 or an image sensor 101), a signal reading circuit 102, a memory 103, a controller 104, an image processor 105, an output interface 106, and a power supply 107. Among them, the electrical signal output end of the pixel array 101 is connected to the input end of the signal reading circuit 102, the control end of the pixel array 101 is connected to the input end of the controller 104, the output end of the signal reading circuit 102 is connected to the input end of the memory 103 and the input end of the controller 104, the output end of the controller 104 is connected to the input end of the image processor 105, the output end of the image processor 105 is connected to the input end of the output interface 106, and the power supply 107 is used to provide power for each of the above modules.

[0044] The pixel array 101 can adopt two different semiconductor structures, CCD or CMOS, to achieve light capture and photoelectric conversion. The pixel array 101 can be used to collect the optical signal returned by the imaging object, convert the optical signal into an electrical signal, and reflect the optical image of the imaging object through the strength of the electrical signal. The signal reading circuit 102 is used to read the electrical signal output by each pixel. The signal reading circuit 102 can be an A / D converter for realizing analog-to-digital conversion. The memory 103 can be an internal memory for directly exchanging data. For example, the memory 103 can be a random access memory (RAM) for storing required data. The controller 104 can be a complex programmable logic device (CPLD) that can meet the logical operation and timing control of the sensor. The image processor 105 is used to pre-process the read data, and different algorithm processing can be performed for different filtering patterns. The output interface 106 is used as an external data interaction interface for transmitting image data to the outside. The controller 104 is used to output a control signal, and the control signal is used to control each pixel in the pixel array to work together.

[0045] The core component of the image processing device 100 is the pixel array 101. Each photosensitive structure in the pixel array 101 is similar. Generally, each pixel structure may include a lens (or microlens), a color filter, and a photosensitive element (or pixel). Among them, the lens is located above the color filter, and the color filter is located above the photosensitive element. After the light returned by the photographed object is focused by the lens, it is emitted from the lens exit area, and after being filtered by the color filter, it is emitted into a photosensitive element such as a photodiode (PD), and the photosensitive element converts the light signal into an electrical signal. Depending on the type of light that can be transmitted by different filters, the pixel may include a red pixel (hereinafter referred to as an R pixel), a green pixel (hereinafter referred to as a G pixel), and a blue pixel (hereinafter referred to as a B pixel). The R pixel refers to a pixel for receiving a red light signal filtered by a color filter. The principles of the G pixel and the B pixel are the same as those of the R pixel, and will not be repeated here.

[0046] The principle of the image processing device 100 to generate color image data is as follows: each pixel in the pixel array can only convert one type of light signal into an electrical signal, and then perform interpolation operations in combination with light signals collected by other types of pixels around it, so as to restore the image color of the area collected by the current pixel. This process is also called demosaicing, which is usually completed in the processor. For example, if the current pixel is an R pixel, and the R pixel can only convert the red light signal into an electrical signal, then the electrical signals collected by the surrounding B pixels or G pixels can be combined to restore the blue light and green light intensity of the current pixel and determine the image color of the current pixel.

[0047] Therefore, in order to capture color images, it is necessary to set a color filter with a specific color arrangement above the array of photosensitive elements included in the photosensitive array, or it can also be called a color filter array (CFA). At present, for most photosensitive arrays, such as CCD and CMOS image sensors, the CFA included therein adopts the Bayer format based on the RGB primary colors. The characteristic of the Bayer format is that its basic unit is a 2×2 four-pixel array, including 1 red pixel R, 1 blue pixel B and 2 green pixels G, where the two green pixels G are arranged adjacent to each other at the same vertex angle, such as Figure 2 As shown in the figure, since any pixel can only obtain the signal of one color in RGB, to restore the complete color information, it must be realized through a specific image processing algorithm.

[0048] This pure RGB bayer layout only allows light of a specific color to pass through each pixel, which means that most photons will be cut off. Therefore, in low-light environments, it may not be possible to accurately restore the image.

[0049] In addition, miniaturization and multi-pixel imaging devices have resulted in more and more pixels, and imaging equipment with high-density pixels is becoming increasingly important for capturing high-resolution images. As a result, as the number of pixels in image sensors continues to increase and the size of image sensors becomes smaller and smaller, the photosensitive area of ​​each pixel also decreases, which further affects the intensity of the light signal sensed by each pixel, making it impossible for the image sensor to accurately restore the image.

[0050] Optionally, the image processor 105 may include but is not limited to an image signal processor (ISP), which is used to perform linearization processing, bad pixel removal, noise removal, color correction, demosaicing, automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), etc. on the digital image.

[0051] For the above-mentioned image processing device 100 using the Bayer format CFA, the red pixel unit can only receive red light signals, the green pixel unit can only receive green light signals, and the blue pixel unit can only receive blue light signals. The intensity of the light signal received by each pixel unit is relatively small, resulting in a larger SNR of the image, thereby affecting the image quality.

[0052] Moreover, for the image sensor of the Bayer format CFA, the high-frequency information of brightness and the chromaticity information in the image are likely to overlap, resulting in color aliasing and color moire.

[0053] In addition, miniaturization and multi-pixel imaging devices have resulted in more and more pixels, and imaging equipment with high-density pixels is becoming increasingly important for capturing high-resolution images. As a result, as the number of pixels in image sensors continues to increase and the size of image sensors becomes smaller and smaller, the photosensitive area of ​​each pixel also decreases, which further affects the intensity of the light signal sensed by each pixel, making it impossible for the image sensor to accurately restore the image.

[0054] Based on this, an embodiment of the present application proposes an image sensor, which is conducive to the image sensor accurately restoring images in low-light environments.

[0055] Figure 3 is a schematic top view of an image sensor 200 provided in an embodiment of the present application, Figure 4 is a schematic cross-sectional view of the image sensor 200 along the AA′ direction.

[0056] like Figure 3 and Figure 4 As shown, the image sensor 200 includes a filter unit array 210 and a pixel unit array 220. The filter unit array 210 may include a plurality of filter unit groups 211, and each filter unit group 211 may include a white filter unit and a color filter unit. The pixel unit array 220 may be arranged below the filter unit array 210, and the white pixel units in the pixel unit array 220 are used to receive the light signal passing through the corresponding white filter unit, and the color pixel units are used to receive the light signal filtered by the corresponding color filter unit.

[0057] The image sensor in the embodiment of the present application adds a white (W) filter unit in the CFA, and the color pixel units in the pixel unit array receive color light signals, and the white pixel units receive white light signals, thereby increasing the overall light signal intensity received by the pixel unit array. For low-light environments, this solution can accurately restore the image of the object.

[0058] However, after adding a white filter unit in the CFA, if the pixel value of the white pixel unit is directly merged into the color pixel unit, when there is sufficient light, too much brightness information will cause the final image to be too bright, causing the final image to deviate from the real effect.

[0059] like Figure 5 As shown in the figure, when there is sufficient light, the image generated by the RGGB structure is shown in Figure (a), which is closer to the real effect; however, the image generated by the RGBW structure is shown in Figure (b). Since it integrates the brightness information of the W pixel unit, the brightness of the final generated image is too large and deviates from the real effect.

[0060] Based on this, an embodiment of the present application also provides an image sensor, which can generate a target image using different fusion methods under different ambient light conditions, so that the final generated target image is closer to the real effect.

[0061] The image sensor includes a filter unit array and a pixel unit array. The filter unit array includes a plurality of filter unit groups, each of which includes a white filter unit and a color filter unit. The pixel unit array is located below the filter unit array, the white pixel units in the pixel unit array are used to receive light signals passing through the white filter units corresponding thereto, and the color pixel units in the pixel unit array are used to receive light signals passing through the color filter units corresponding thereto, and the light intensity information of the light signals sensed by the pixel unit array is used to determine the pixel values ​​of the color pixel units and / or the pixel values ​​of the white pixel units to generate a target image of the photographed object.

[0062] The white pixel unit is a pixel unit corresponding to the white filter unit, and the color pixel unit is a pixel unit corresponding to the color filter unit.

[0063] It should be noted that in the present application, the white filter unit refers to a filter or filter material for transmitting white light. In some embodiments, the white filter unit may also be a transparent material or an air gap, which is used to transmit all light signals in the environment including white light. Specifically, the white light may be a mixture of multiple colored lights. For example, the three primary colors of light in the spectrum: blue, red and green, can be mixed in a certain proportion to obtain white light, or a mixture of all visible light in the spectrum is also white light.

[0064] The color filter unit in the embodiment of the present application may include at least one red filter unit, at least one blue filter unit and at least one green filter unit, which can ensure the integrity of the RGB space color.

[0065] The embodiment of the present application does not specifically limit the number of filter units included in the filter unit group. For example, a filter unit group may include 2×2 filter units, or a filter unit group may include 3×3 filter units, or a filter unit group may include 4×4 filter units, or a filter unit group may include 6×6 filter units, or a filter unit group may include 8×8 filter units, etc.

[0066] The proportion of white filter units in each filter unit group may be between 25% and 75%.

[0067] Preferably, the proportion of white filter units can be 50%, that is, the ratio of white filter units to color filter units in a filter unit group can be 1:1, which can improve the overall photosensitivity of the sensor. This can ensure that in a low-light environment, the white filter unit can provide sufficient brightness without causing excessive brightness, and can make the final image close to the real effect.

[0068] Secondly, the 50% white filter unit can ensure that the white pixel unit has a higher spatial sampling rate, which is beneficial for the subsequent mosaic reconstruction algorithm to obtain better high-resolution grayscale images.

[0069] In addition, the white filter units and the color filter units can be arranged alternately, that is, in any row or any column, there will not be two adjacent white filter units or color filter units, so that the crosstalk can be relatively even. In addition, the white filter units and the color filter units are distributed relatively evenly, which can achieve a higher color space sampling rate, which is conducive to the restoration of subsequent images.

[0070] The arrangement of the filter units in the embodiment of the present application is described below by taking an example in which each filter unit group includes 4×4 filter units.

[0071] A filter unit group may include 2 red filter units, 2 blue filter units and 4 green filter units. Since the human eye is more sensitive to green light than blue light and red light, the number of green filter units in the filter unit is greater than that of blue filter units and red filter units, which can better achieve color restoration and ensure that RGB has a relatively average spatial sampling rate, which is conducive to the subsequent Remosaic algorithm to obtain color images.

[0072] The filter units on one diagonal line in each filter unit group are white filter units, and the filter units on the other diagonal line are two red filter units and two blue filter units.

[0073] As an example, the two red filter units can be arranged at opposite corners, and the two blue filter units can also be arranged at the same corner, that is, two consecutive red filter units and two consecutive blue filter units are included on another diagonal line, such as Figure 6 As shown in Figure (a).

[0074] Figure 6 In the filter unit groups shown in Figure (a), the white filter unit can be located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the third row and second column, and the fourth row and first column of each filter unit group; the blue filter unit is located in the first row and fourth column, and the second row and third column of each filter unit group; the green filter unit is located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

[0075] In this structure, the filter units of the same color are all arranged at the same vertex angle, which can simplify the fusion process in the subsequent fusion process. Figure 4 For example, the pixel units corresponding to the two green filter units in the upper left corner can be directly merged into a green pixel unit, the pixel units corresponding to the two blue filter units in the upper right corner can be directly merged into a blue pixel unit, the pixel units corresponding to the two red filter units in the lower left corner can be directly merged into a red pixel unit, and the pixel units corresponding to the two green filter units in the lower right corner can be directly merged into a green pixel unit. The merged pixel units can directly form an RGGB pattern.

[0076] As another example, two red filter units on another diagonal line are arranged at a common vertex angle, and two blue filter units are arranged separately; or, two blue filter units on another diagonal line are arranged at a common vertex angle, and two red filter units are arranged separately.

[0077] by Figure 6 Taking Figure (b) in FIG. 1 as an example, the white filter unit is located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the first row and fourth column, and the fourth row and first column of each filter unit group; the blue filter unit is located in the second row and third column, and the third row and second column of each filter unit group; the green filter unit is located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

[0078] by Figure 6 Taking Figure (c) as an example, the white filter unit is located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the second row and third column, and the third row and second column of each filter unit group; the blue filter unit is located in the first row and fourth column, and the fourth row and first column of each filter unit group; the green filter unit is located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

[0079] Although Figure 6 The color filter units of the same color in the filter unit groups shown in (b) and (c) are not all arranged at opposite corners, but a plurality of the filter unit groups are arranged in an array so that the plurality of filter unit groups are tiled over the entire surface of the image sensor, as shown in FIG. Figure 8 As shown, it can be seen that the filter unit group after tiling can achieve the same Figure 6 The filter unit group 302 has the same effect as the filter unit group shown in FIG. Figure 6 The filter unit group shown in (a) of FIG. 1 is equivalent to the filter unit group 304. Figure 6 The filter unit group shown in (b) of FIG. 1 is equivalent to the filter unit group 306. Figure 6 The filter unit group shown in Figure (c) .

[0080] As yet another example, the red filter units and the blue filter units on another diagonal line are arranged alternately.

[0081] like Figure 6As shown in Figure (d), the white filter unit is located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the second row and third column, and the fourth row and first column of each filter unit group; the blue filter unit is located in the first row and fourth column, and the third row and second column of each filter unit group; the green filter unit is located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

[0082] Of course, the arrangement of the red filter unit and the blue filter unit on another diagonal line is not limited to the above description. For example, the red filter unit and the blue filter unit on another diagonal line may also be arranged as follows: Figure 7 Arrange in the manner shown.

[0083] Optionally, the filter units on one diagonal line in each filter unit group are white filter units, and the filter units on the other diagonal line are green filter units. The two red filter units can be separately arranged, and the two blue filter units are also separately arranged.

[0084] For example, Figure 6 As shown in Figure (e), the white filter unit is located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; the red filter unit is located in the second row and first column, and the fourth row and third column of each filter unit group; the blue filter unit is located in the first row and second column, and the third row and fourth column of each filter unit group; the green filter unit is located in the first row and fourth column, the second row and third column, the third row and second column, and the fourth row and first column of each filter unit group.

[0085] from Figure 8 It can be seen that Figure 8 The filter unit group 308 in the Figure 6 The filter unit group shown in Figure (e) in FIG. Figure 6 The filter unit group shown in Figure (e) can also achieve Figure 6 The effect of the filter unit group shown in (a), (b), and (c).

[0086] Apart from Figure 6 In addition to the method shown in Figure (e) in FIG. Fig. 9 Arrange in the manner shown.

[0087] In addition, from Figure 8It can be seen that the filter unit group 310 is also an arrangement in which all the white filter units are on a diagonal line, and three green filter units are arranged consecutively at the top corners, and another green filter unit is separated from the three green filter units.

[0088] In addition to the filter unit group 310, the filter unit can also be Fig.10 Arrange in the manner shown.

[0089] Apart from Figure 6-Figure 10 In addition to the filter unit group shown in FIG. 1 , the filter units in the filter unit group can also be Fig.11 Arrange in the manner shown.

[0090] like Fig.12 and Fig.13 As shown, the image sensor 200 in the embodiment of the present application may further include a microlens array 230, which may be disposed above the filter unit array 210, and is used to converge the light signal returned by the photographed object to the filter unit array 210. The microlens array 230 may include a plurality of microlenses, and one microlens among the plurality of microlenses corresponds to at least one filter unit in the filter unit array.

[0091] In the embodiment of the present application, the distribution of the microlens array 230 can be set corresponding to the filter unit array 210 located thereunder. For example, each microlens in the microlens array 230 can correspond to one or more of the filter unit arrays 210 located thereunder.

[0092] Optionally, as an embodiment, the microlenses in the microlens array 230 may correspond to the filter units in the filter unit array 210 one by one. Fig.12 As shown, the microlens array 230 includes a plurality of first microlenses 231 , and each first microlens 231 corresponds to a filter unit and also corresponds to a pixel unit.

[0093] Optionally, as another embodiment, at least one microlens in the microlens array 230 may correspond to a plurality of filter units in the filter unit array 230. For example, the microlens array 230 may include a plurality of second microlenses 232, each of which corresponds to a plurality of filter units, for example, each of which may correspond to 2×2 filter units, each of which corresponds to a pixel unit.

[0094] For another example, the microlens array 230 may also include at least one first microlens 231 and at least one second microlens 232, wherein each first microlens 231 corresponds to a filter unit in the filter unit array 210, and each second microlens 232 corresponds to a plurality of filter units in the filter unit array 210. Fig.13 As shown, the microlens array 230 includes a plurality of first microlenses 231 corresponding to the filter units one by one, and also includes at least one second microlens 232 corresponding to 2×2 filter units.

[0095] For the second microlens 232 corresponding to the plurality of filter units, the number of filter units corresponding to the second microlens 232 can be set according to the actual application and can be set to any value. For example, the second microlens 232 can correspond to 3×3 filter units or 1×2 filter units, but the embodiments of the present application are not limited thereto.

[0096] In addition, the plurality of filter units corresponding to the same second microlens 232 may have the same or different colors. Fig.13 As shown, the 2×2 filter units corresponding to the second microlens 232 may all be white filter units, wherein the 2×2 white filter units may belong to the same filter unit group, or may not belong to the same filter unit group. For example, the 2×2 filter units corresponding to the second microlens 232 may also belong to two or more adjacent filter unit groups, and the embodiments of the present application are not limited to this.

[0097] When multiple white filter units are arranged corresponding to the same second microlens, the phase difference of the electrical signal can be calculated using the electrical signal converted from the mixed light emitted from different emission areas of the second microlens, so as to adjust the focal length of the image sensor according to the phase difference.

[0098] The image sensor 200 in the embodiment of the present application may also include other parts. Fig.12 and Fig.13 As shown, a dielectric layer 240 may be further included between the filter unit array 210 and the pixel array 220 .

[0099] like Fig.12 and Fig.13 As shown, the filter unit array 210 may further include a dielectric 215 and a reflective grid 216 located around it; the pixel array 220 may include a semiconductor substrate 221 and a photosensitive element 222, wherein the photosensitive element 222 is located in the semiconductor substrate 221, and the photosensitive element 222 may be a PD. Optionally, the pixel array 220 may further include an isolation region 223 between two photosensitive elements 222.

[0100] Combine the following Figure 14-18 , describes the fusion method provided in the embodiment of the present application.

[0101] The embodiment of the present application can determine the method for generating the target image based on the light intensity information of the light signal sensed by the pixel unit array. That is, if the light intensity is different, the method for generating the target image is also different.

[0102] Since the signal-to-noise ratio of the light signal sensed by the white pixel unit is relatively high, the light signal sensed by the white pixel unit can be preferentially used as the basis for light intensity judgment.

[0103] The image sensor of the embodiment of the present application may further include a processor, which may be used to determine light intensity information based on the light signal sensed by the pixel unit array; and determine a method for generating a target image of the photographed object by using the pixel value of the color pixel unit and / or the pixel value of the white pixel unit based on the light intensity information.

[0104] The embodiment of the present application can divide the light intensity into different ranges, each range corresponds to a different fusion process, and the fused target image is sent to an image signal processor (ISP) for processing, such as Fig.14 shown.

[0105] When the light intensity is greater than or equal to the first preset threshold, it indicates that the light signal is saturated or close to saturation, and the processor can use process 1 to generate a target image.

[0106] When the light intensity is too strong, causing the white pixel unit to be saturated or close to saturation, the pixel value of the white pixel unit can be ignored, and the pixel value of the color pixel unit can be directly processed by the mosaic reconstruction algorithm to output RGB data. This method can ensure that the generated target image is not too bright in a strong light environment and is closer to the real effect.

[0107] When the light intensity is less than the first preset threshold and greater than or equal to the second preset threshold, the processor may use process 2 to generate a target image.

[0108] The processor can determine the texture information and guess the color according to the pixel value of the white pixel unit, and generate the target image according to the pixel value of the color pixel unit. In this process, the white pixel unit does not participate in the fusion process, but is only used to provide the texture information and guess the color guidance of the target image.

[0109] For example, the grayscale information of the target image may be generated according to the pixel value of the white pixel unit, and the processor may be used to determine the texture information of the target image according to the grayscale information.

[0110] In addition, when determining the color information at the position of the white pixel unit, color estimation can be performed based on the pixel value of the white pixel unit and the colors of the pixel units surrounding the white pixel unit to improve the accuracy of color restoration of the target image. For example, the color information at the position of the white pixel unit and the pixel value of the color can be determined based on the pixel value of the white pixel unit and the colors of the pixel units surrounding the white pixel unit.

[0111] When the light intensity is less than the second preset threshold and greater than or equal to the third preset threshold, the processor may use process 3 to generate a target image.

[0112] The processor can generate first image data according to the pixel value of the white pixel unit, generate second image data according to the pixel value of the color pixel unit, and use the correction coefficient α to correct the first image data to obtain the corrected first image data; and fuse the corrected first image data and the second image data to obtain the target image.

[0113] The correction coefficient α is determined according to the light intensity, 0<α<1. It can be understood that the greater the light intensity, the smaller α is, indicating that fewer W pixels need to be fused; the smaller the light intensity, the larger α is, indicating that more W pixels need to be fused.

[0114] When the light intensity is between the second preset threshold and the third preset threshold, the processor can weaken the problem of excessive brightness enhancement introduced when white pixels are fused, and only fuse part of the white pixel units, so that the brightness of the generated target image will not be too high or too low, and it will be closer to the real effect.

[0115] When the light intensity is less than the third preset threshold, the processor may use process 4 to generate a target image.

[0116] The processor may generate first image data according to the pixel value of the white pixel unit, generate second image data according to the pixel value of the color pixel unit, and fuse the first image data and the second image data to obtain a target image.

[0117] When the light intensity is relatively low, the pixel value of the white pixel unit can be completely integrated into the target image, thereby ensuring the brightness of the target image in the low-light environment, so that the image sensor can accurately restore the image in the low-light environment.

[0118] The image sensor in the embodiment of the present application may further include a light intensity determination module, which may be used to determine the intensity of the light signal sensed by the pixel unit array, so that the processor may determine a method for generating a target image.

[0119] Combine the following Figure 15-18 , the fusion process in the embodiment of the present application is described.

[0120] Fig.15 The light is saturated or close to saturation. Fig.15 The fusion process shown corresponds to process 1. In this case, the pixel values ​​of the color pixel units can be directly used, and the target image of the photographed object can be generated by reconstructing the mosaic algorithm.

[0121] Fig.16 The figure shows the situation with sufficient light. Fig.16 The fusion process shown corresponds to process 2. In this case, the W pixel unit can be separated from the pixel unit array, and a 4×4 full W pixel array can be obtained by interpolation. The full W pixel array can be used to provide texture information of the image and guess the color. In the process of the color pixel unit generating the target image, the texture information and guessed color information can be referred to. In this process, the color pixel unit can generate the target image by reconstructing the mosaic algorithm.

[0122] Fig.17 The figure shows the low light condition. Fig.17 The fusion process shown corresponds to process 3. In this case, some W pixel units can be fused. The specific process is as follows: the white pixel units in the pixel unit array can be separated to obtain discrete W pixel units, and then interpolation can be performed through an interpolation algorithm to obtain a 4×4 full W pixel array. The full W pixel array can provide texture information and guess the color in the process of generating the second image by the color pixel unit through the mosaic reconstruction algorithm. The light intensity judgment module can determine the light intensity information, and the light intensity information can be used to determine the correction coefficient α. The full W pixel array can be used to generate a first image. The pixel value of the full W pixel array can be multiplied by the correction coefficient α to obtain a corrected first image. The corrected first image is fused with the second image to obtain a target image of the photographed object.

[0123] Fig.18 The figure shows the situation of low light. Fig.18 The fusion process shown corresponds to process 4. In this case, all W pixel units can be fused. The specific process is as follows: the white pixel units in the pixel unit array can be separated to obtain discrete W pixel units, and then interpolation can be performed through an interpolation algorithm to obtain a 4×4 full W pixel array. The full W pixel array can provide texture information and guess the color in the process of generating the second image by the color pixel unit through the mosaic reconstruction algorithm. The light intensity judgment module can determine the correction coefficient α=1. The full W pixel array can be used to generate a first image, and the first image is fused with the second image to obtain a target image of the photographed object.

[0124] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0129] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image sensor, characterized in that: include: A filter unit array, comprising a plurality of filter unit groups, each of which comprises a white filter unit and a color filter unit; a pixel unit array, located below the filter unit array, wherein the pixel units in the pixel unit array correspond one-to-one to the filter units in the plurality of filter unit groups, the white pixel units in the pixel unit array are used to receive light signals passing through the white filter units corresponding thereto, and the color pixel units in the pixel unit array are used to receive light signals passing through the color filter units corresponding thereto, and the light intensity of the light signals sensed by the pixel unit array is used to determine the pixel values ​​of the color pixel units and the pixel values ​​of the white pixel units to generate a target image of the photographed object; A processor, the processor is used to: when the light intensity is less than a first preset threshold and greater than or equal to a second preset threshold, determine the texture information according to the pixel value of the white pixel unit, determine the color information and pixel value of the position of the white pixel unit according to the pixel value of the white pixel unit and the color of the pixel units around the white pixel unit, and generate the target image according to the color information and pixel value of the position of the white pixel unit and the pixel value of the color pixel unit.

2. The image sensor according to claim 1, characterized in that Each of the plurality of filter unit groups includes 4×4 filter units, the ratio of white filter units to color filter units in each filter unit group is 1:1, and the white filter units and the color filter units are arranged alternately.

3. The image sensor according to claim 2, characterized in that The color filter unit includes 2 red filter units, 2 blue filter units and 4 green filter units.

4. The image sensor according to claim 3, characterized in that: The filter units on one diagonal line in each filter unit group are white filter units, and the filter units on the other diagonal line are two red filter units and two blue filter units.

5. The image sensor according to claim 4, characterized in that: The two red filter units on the other diagonal line are arranged at a common vertex angle, and the two blue filter units are also arranged at a common vertex angle.

6. The image sensor according to claim 4, characterized in that: The white filter units are located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; The red filter unit is located in the third row and second column and the fourth row and first column of each filter unit group; The blue filter unit is located in the first row and fourth column and the second row and third column of each filter unit group; The green filter units are located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

7. The image sensor according to claim 4, characterized in that: The two red filter units on the other diagonal line are arranged at the same vertex angle, and the two blue filter units are arranged separately; or The two blue filter units on the other diagonal line are arranged at the same vertex angle, and the two red filter units are arranged separately.

8. The image sensor according to claim 4, characterized in that: The white filter units are located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; The red filter unit is located in the first row and fourth column and the fourth row and first column of each filter unit group; The blue filter unit is located in the second row and third column, and the third row and second column in each filter unit group; The green filter units are located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

9. The image sensor according to claim 4, characterized in that: The white filter units are located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; The red filter unit is located in the second row and third column, and the third row and second column in each filter unit group; The blue filter unit is located in the first row and fourth column and the fourth row and first column of each filter unit group; The green filter units are located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

10. The image sensor according to claim 4, characterized in that: The red filter units and the blue filter units on the other diagonal line are arranged alternately.

11. The image sensor according to claim 4, characterized in that: The white filter units are located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; The red filter unit is located in the third column of the second row and the first column of the fourth row in each filter unit group; The blue filter unit is located in the first row and fourth column and the third row and second column of each filter unit group; The green filter units are located in the first row and second column, the second row and first column, the third row and fourth column, and the fourth row and third column of each filter unit group.

12. The image sensor according to claim 3, characterized in that: The filter units on one diagonal line in each filter unit group are white filter units, and the filter units on the other diagonal line are green filter units.

13. The image sensor according to claim 12, characterized in that: The white filter units are located in the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of each filter unit group; The red filter unit is located in the second row and first column and the fourth row and third column of each filter unit group; The blue filter unit is located in the first row and second column and the third row and fourth column of each filter unit group; The green filter units are located in the first row and the fourth column, the second row and the third column, the third row and the second column, and the fourth row and the first column of each filter unit group.

14. The image sensor according to any one of claims 1 to 13, characterized in that: Also included is a processor configured to: determining light intensity according to the light signal sensed by the pixel unit array; When the light intensity is greater than or equal to the first preset threshold, the target image is generated using the pixel values ​​of the color pixel units.

15. The image sensor according to any one of claims 1 to 13, characterized in that: Also included is a processor configured to: When the light intensity is less than the second preset threshold and greater than or equal to the third preset threshold, first image data is generated according to the pixel value of the white pixel unit, second image data is generated according to the pixel value of the color pixel unit, and the first image data is corrected using a correction coefficient α to obtain corrected first image data; And the corrected first image data and the second image data are merged to obtain the target image, wherein the correction coefficient α is determined according to the magnitude of the light intensity, and 0<α<1.

16. The image sensor according to claim 15, characterized in that Also included is a processor configured to: When the light intensity is less than the third preset threshold, first image data is generated according to the pixel value of the white pixel unit, second image data is generated according to the pixel value of the color pixel unit, and the first image data and the second image data are fused to obtain the target image.

17. The image sensor according to any one of claims 1 to 13, characterized in that: Also includes: The microlens array includes a plurality of microlenses, which are located above the filter unit array and are used to converge the light signal returned by the photographed object to the filter unit array, wherein one microlens among the plurality of microlenses corresponds to at least one filter unit in the filter unit array.

18. The image sensor according to claim 17, characterized in that: The microlenses in the microlens array correspond one to one to the filter units in the filter unit array.

19. An electronic device, characterized in that: include: An image sensor as claimed in any one of claims 1 to 18.

Images