Image acquisition method, camera component and mobile terminal

Abstract

The present application discloses an image acquisition method, a camera component and a mobile terminal. A pixel array is exposed to obtain a first color raw image and a second color raw image. The data of the first color raw image is generated by at least one color photosensitive pixel in a subunit, and the data of the second color raw image is generated by at least one panchromatic photosensitive pixel and at least one color photosensitive pixel in the subunit. The image acquisition method includes: interpolating the first color raw image to obtain a first interpolated image for each color channel, and interpolating the second color raw image to obtain a second interpolated image for at least one color channel; fusing the second interpolated image with the first interpolated image for each color channel to obtain a fused image for each color channel; and obtaining a target image according to the fused image for each color channel. The implementation manner of the present application fuses the first interpolated image and the second interpolated image, improves the signal-to-noise ratio and clarity of the image, and thus improves the image quality captured in a dark environment.

Description

Technical Field

[0001] The present application relates to the field of imaging technologies, and particularly to an image acquisition method, a camera assembly, and a mobile terminal. Background Art

[0002] A camera can be installed in an electronic device such as a mobile phone to enable the electronic device to have an image acquisition function. A color filter array can be assembled in the camera to achieve the acquisition of a color image. Currently, the color filter array in a camera usually takes the form of a Bayer array. In a color filter array in the form of a Bayer array, each color filter can only pass light of a single color. Thus, most of the light will be filtered out, affecting the quality of the image acquired by the camera. Summary of the Invention

[0003] Embodiments of the present application provide an image acquisition method, a camera assembly, and a mobile terminal.

[0004] The image acquisition method of the embodiments of the present application is used for an image sensor. The image sensor includes a pixel array. The pixel array includes a plurality of subunits. Each subunit includes at least one panchromatic photosensitive pixel and at least one color photosensitive pixel. The color photosensitive pixel has a narrower spectral response than the panchromatic photosensitive pixel. The pixel array is exposed to obtain a first color raw image and a second color raw image. The first color raw image is composed of a plurality of first color raw image data. Each first color raw image data is generated by the at least one color photosensitive pixel in one subunit. The second color raw image is composed of a plurality of second color raw image data. Each second color raw image data is generated by the at least one panchromatic photosensitive pixel and the at least one color photosensitive pixel in one subunit. The image acquisition method includes: performing interpolation processing on the first color raw image to obtain a plurality of first interpolated images, each first interpolated image corresponding to a color channel respectively, and performing interpolation processing on the second color raw image to obtain a second interpolated image of at least one color channel; fusing the second interpolated image with the first interpolated images of each color channel respectively to obtain a fused image of each color channel; and obtaining a target image according to the fused images of each color channel.

[0005] The camera component of the embodiment of the present application includes an image sensor and a processor. The image sensor includes a pixel array, and the pixel array includes a plurality of subunits. Each subunit includes at least one panchromatic photosensitive pixel and at least one color photosensitive pixel. The color photosensitive pixel has a narrower spectral response than the panchromatic photosensitive pixel. The pixel array is exposed to obtain a first color raw image and a second color raw image. The first color raw image is composed of a plurality of first color raw image data, and each of the first color raw image data is generated by the at least one color photosensitive pixel in one subunit. The second color raw image is composed of a plurality of second color raw image data, and each of the second color raw image data is generated by the at least one panchromatic photosensitive pixel and the at least one color photosensitive pixel in one subunit. The processor is configured to perform interpolation processing on the first color raw image to respectively obtain first interpolation images of each color channel, perform interpolation processing on the second color raw image to obtain second interpolation images of at least one color channel; fuse the second interpolation images with the first interpolation images of each color channel respectively to obtain fused images of each color channel; and obtain a target image according to the fused images of each color channel.

[0006] The mobile terminal of the embodiment of the present application includes a housing and a camera component. The camera component is combined with the housing. The camera component includes an image sensor and a processor. The image sensor includes a pixel array, and the pixel array includes a plurality of subunits. Each subunit includes at least one panchromatic photosensitive pixel and at least one color photosensitive pixel. The color photosensitive pixel has a narrower spectral response than the panchromatic photosensitive pixel. The pixel array is exposed to obtain a first color raw image and a second color raw image. The first color raw image is composed of a plurality of first color raw image data, and each of the first color raw image data is generated by the at least one color photosensitive pixel in one subunit. The second color raw image is composed of a plurality of second color raw image data, and each of the second color raw image data is generated by the at least one panchromatic photosensitive pixel and the at least one color photosensitive pixel in one subunit. The processor is configured to perform interpolation processing on the first color raw image to obtain a plurality of first interpolation images, each of the first interpolation images corresponding to one color channel respectively, perform interpolation processing on the second color raw image to obtain second interpolation images of at least one color channel; fuse the second interpolation images with the first interpolation images of each color channel respectively to obtain fused images of each color channel; and obtain a target image according to the fused images of each color channel.

[0007] The image acquisition method, camera component, and mobile terminal according to the embodiments of the present application obtain a first color raw image including only color color channels and a second color raw image including both color color channels and panchromatic color channels by exposing a pixel array, and perform interpolation processing and fusion processing on the first color raw image and the second color raw image, so as to improve the signal-to-noise ratio and clarity of the image by means of the image data of the panchromatic color channels, thereby improving the quality of the image taken in a dark environment.

[0008] Additional aspects and advantages of the embodiments of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0010] Figure 1 is a schematic diagram of a camera component according to some embodiments of the present application;

[0011] Figure 2 is a schematic diagram of a pixel array according to some embodiments of the present application;

[0012] Figure 3 is a schematic cross-sectional view of a photosensitive pixel according to some embodiments of the present application;

[0013] Figure 4 is a pixel circuit diagram of a photosensitive pixel according to some embodiments of the present application;

[0014] Figures 5 to 15 is a schematic layout diagram of the smallest repeating unit in a pixel array according to some embodiments of the present application;

[0015] Figure 16 is a schematic diagram of the principle of a camera component in the present application for an image sensor to obtain a raw image;

[0016] Figure 17 and 18 is a schematic diagram of the principle of interpolation processing for a first color raw image according to some embodiments of the present application;

[0017] Figures 19 to 22 is a schematic diagram of the principle of filtering processing for a first interpolated image according to some embodiments of the present application;

[0018] Figure 23 is a schematic diagram of the principle of filtering processing for a second interpolated image according to some embodiments of the present application;

[0019] Figure 24It is a schematic structural diagram of a mobile terminal according to an embodiment of the present application;

[0020] Figure 25 It is a schematic flowchart of an image acquisition method according to some embodiments of the present application. Specific embodiments

[0021] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application and should not be construed as a limitation to the embodiments of the present application.

[0022] In the related art, the color filter array in a camera usually has a Bayer array form. However, in the color filter array arranged in the Bayer array form, each color filter can only pass light of a single color. Thus, most of the light will be filtered out, affecting the quality of the image obtained by the camera.

[0023] For the above reasons, please refer to Figure 1 、 Figure 2 and Figure 5 , the present application provides a camera assembly 100. The camera assembly 100 includes an image sensor 10 and a processor 20. The image sensor 10 includes a pixel array 11. The pixel array 11 includes a plurality of subunits. Each subunit includes at least one panchromatic photosensitive pixel W and at least one color photosensitive pixel. The color photosensitive pixel has a narrower spectral response than the panchromatic photosensitive pixel W. The pixel array 11 is exposed to obtain a first color raw image and a second color raw image. The first color raw image is composed of a plurality of first color raw image data. Each first color raw image data is generated by at least one color photosensitive pixel in a subunit. The second color raw image is composed of a plurality of second color raw image data. Each second color raw image data is generated by at least one panchromatic photosensitive pixel W and at least one color photosensitive pixel in a subunit. The processor 20 is electrically connected to the image sensor 10. The processor 20 is configured to perform interpolation processing on the first color raw image to obtain first interpolation images of respective color channels, perform interpolation processing on the second color raw image to obtain second interpolation images of at least one color channel, fuse the second interpolation images with the first interpolation images of respective color channels to obtain fused images of respective color channels, and fuse the fused images of respective color channels to obtain a target image.

[0024] The camera component 100 according to the embodiments of the present application exposes through the pixel array 11 to obtain a first color raw image including only color color channels and a second color raw image including both color color channels and panchromatic color channels of the image data, interpolates the first color raw image to obtain first interpolated images of each color channel, interpolates the second color raw image to obtain second interpolated images of at least one color channel, and fuses the first interpolated images and the second interpolated images to improve the signal-to-noise ratio and clarity of the image, thereby improving the quality of the images taken in a dark environment.

[0025] The camera component 100 according to the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0026] Please refer to Figure 2 , the image sensor 10 includes a pixel array 11, a vertical driving unit 12, a control unit 13, a column processing unit 14, and a horizontal driving unit 15.

[0027] For example, the image sensor 10 may employ a complementary metal oxide semiconductor (CMOS) photosensitive element or a charge-coupled device (CCD) photosensitive element.

[0028] For example, the pixel array 11 includes a plurality of photosensitive pixels 110 arranged in a two-dimensional array (i.e., arranged in a two-dimensional matrix form) ( Figure 3 as shown), and each photosensitive pixel 110 includes a photoelectric conversion element 1111 ( Figure 4 as shown). Each photosensitive pixel 110 converts light into charge according to the intensity of the light incident thereon.

[0029] For example, the vertical driving unit 12 includes a shift register and an address decoder. The vertical driving unit 12 includes a readout scan and a reset scan function. The readout scan refers to sequentially scanning each unit photosensitive pixel 110 row by row and reading signals from these unit photosensitive pixels 110 row by row. For example, the signals output by each photosensitive pixel 110 in the selected and scanned photosensitive pixel row are transmitted to the column processing unit 14. The reset scan is used to reset the charge, and the optical charge of the photoelectric conversion element 1111 is discarded, so that the accumulation of new optical charge can be started.

[0030] For example, the signal processing performed by the column processing unit 14 is correlated double sampling (CDS) processing. In the CDS processing, the reset level and the signal level output from each photosensitive pixel 110 in the selected photosensitive pixel row are taken out, and the level difference is calculated. Thus, the signals of the photosensitive pixels 110 in one row are obtained. The column processing unit 14 may have an analog-to-digital (A / D) conversion function for converting the analog pixel signal into a digital format.

[0031] For example, the horizontal driving unit 15 includes a shift register and an address decoder. The horizontal driving unit 15 sequentially scans the pixel array 11 column by column. Through the selection scanning operation performed by the horizontal driving unit 15, each photosensitive pixel column is sequentially processed by the column processing unit 14 and sequentially output.

[0032] For example, the control unit 13 configures timing signals according to the operation mode, and uses various timing signals to control the vertical driving unit 12, the column processing unit 14, and the horizontal driving unit 15 to work together.

[0033] Please refer to Figure 3 , the photosensitive pixel 110 includes a pixel circuit 111, a filter 112, and a microlens 113. Along the light-receiving direction of the photosensitive pixel 110, the microlens 113, the filter 112, and the pixel circuit 111 are sequentially arranged. The microlens 113 is used to converge light, and the filter 112 is used to allow light of a certain wavelength band to pass through and filter out light of the remaining wavelength bands. The pixel circuit 111 is used to convert the received light into an electrical signal and provide the generated electrical signal to Figure 2 the column processing unit 14 shown in

[0034] Please refer to Figure 4 , the pixel circuit 111 can be applied to each photosensitive pixel 110 ([[]] Figure 2 shown in Figure 3 ) in the pixel array 11 shown in Figures 2 to 4 . The working principle of the pixel circuit 111 will be described below in conjunction with

[0035] As Figure 4 shown, the pixel circuit 111 includes a photoelectric conversion element 1111 (for example, a photodiode), an exposure control circuit (for example, a transfer transistor 1112), a reset circuit (for example, a reset transistor 1113), an amplification circuit (for example, an amplification transistor 1114), and a selection circuit (for example, a selection transistor 1115). In the embodiments of the present application, the transfer transistor 1112, the reset transistor 1113, the amplification transistor 1114, and the selection transistor 1115 are, for example, MOS transistors, but are not limited thereto.

[0036] For example, the photoelectric conversion element 1111 includes a photodiode, and the anode of the photodiode is connected to ground, for example. The photodiode converts the received light into charges. The cathode of the photodiode is connected to the floating diffusion unit FD via an exposure control circuit (for example, transfer transistor 1112). The floating diffusion unit FD is connected to the gate of the amplification transistor 1114 and the source of the reset transistor 1113.

[0037] For example, the exposure control circuit is the transfer transistor 1112, and the control terminal TG of the exposure control circuit is the gate of the transfer transistor 1112. When a pulse of an effective level (for example, VPIX level) is transmitted to the gate of the transfer transistor 1112 through the exposure control line, the transfer transistor 1112 is turned on. The transfer transistor 1112 transfers the charges photoelectrically converted by the photodiode to the floating diffusion unit FD.

[0038] For example, the drain of the reset transistor 1113 is connected to the pixel power supply VPIX. The source of the reset transistor 113 is connected to the floating diffusion unit FD. Before the charges are transferred from the photodiode to the floating diffusion unit FD, a pulse of an effective reset level is transmitted to the gate of the reset transistor 113 via the reset line, and the reset transistor 113 is turned on. The reset transistor 113 resets the floating diffusion unit FD to the pixel power supply VPIX.

[0039] For example, the gate of the amplification transistor 1114 is connected to the floating diffusion unit FD. The drain of the amplification transistor 1114 is connected to the pixel power supply VPIX. After the floating diffusion unit FD is reset by the reset transistor 1113, the amplification transistor 1114 outputs a reset level through the output terminal OUT via the selection transistor 1115. After the charges of the photodiode are transferred by the transfer transistor 1112, the amplification transistor 1114 outputs a signal level through the output terminal OUT via the selection transistor 1115.

[0040] For example, the drain of the selection transistor 1115 is connected to the source of the amplification transistor 1114. The source of the selection transistor 1115 is connected to the Figure 2 column processing unit 14 in. When a pulse of an effective level is transmitted to the gate of the selection transistor 1115 through the selection line, the selection transistor 1115 is turned on. The signal output by the amplification transistor 1114 is transmitted to the column processing unit 14 through the selection transistor 1115.

[0041] It should be noted that the pixel structure of the pixel circuit 111 in the embodiments of the present application is not limited to Figure 4The structures shown. For example, the pixel circuit 111 may also have a three-transistor pixel structure, in which the functions of the amplification transistor 1114 and the selection transistor 1115 are completed by one transistor. For example, the exposure control circuit is not limited to the manner of a single transfer transistor 1112, and other electronic devices or structures having a control terminal control conduction function can be used as the exposure control circuit in the embodiments of the present application. The embodiment of the single transfer transistor 1112 in the embodiments of the present application is simple in implementation, low in cost, and easy to control.

[0042] Please refer to Figures 5 to 15 , the photosensitive pixels 110 ( Figure 2 shown) in the pixel array 11 ( Figure 3 shown) of some embodiments of the present application. The photosensitive pixels 110 include two types, one is a panchromatic photosensitive pixel W, and the other is a color photosensitive pixel. The pixel array 11 includes a plurality of minimum repeating units, and each minimum repeating unit includes a plurality of subunits. Figures 5 to 15 Only the arrangement of a plurality of photosensitive pixels 110 in one minimum repeating unit composed of four subunits is shown. In other examples, the number of subunits in each minimum repeating unit may also be two, three, five, ten, etc., which is not limited herein. For Figures 5 to 17 the minimum repeating unit composed of the four subunits shown, by replicating it multiple times in rows and columns, the pixel array 11 can be formed. Each subunit includes at least one panchromatic photosensitive pixel W and at least one color photosensitive pixel. Among them, in each subunit, the panchromatic photosensitive pixel W and the color photosensitive pixel can be alternately arranged; or, in each subunit, a plurality of photosensitive pixels 110 in the same row can have the same color channel; or, in each subunit, a plurality of photosensitive pixels 110 in the same column can have the same color channel; or, in each minimum repeating unit, a plurality of photosensitive pixels 110 in the same row and having the same color channel and a plurality of photosensitive pixels 110 in the same column and having the same color channel can be alternately arranged; or, in each subunit, when the number of panchromatic photosensitive pixels W is one and the number of color photosensitive pixels is multiple, the panchromatic photosensitive pixel W can be located at any position in the subunit; or, in each subunit, when the number of panchromatic photosensitive pixels W is multiple and the number of color photosensitive pixels is one, the color photosensitive pixel can be located at any position in the subunit.

[0043] Specifically, for example, please refer to Figure 5 , Figure 5 is the arrangement schematic diagram of the photosensitive pixels 110 ( Figure 3 shown) in a minimum repeating unit of an embodiment of the present application. Among them, the minimum repeating unit is 16 photosensitive pixels 110 in 4 rows and 4 columns, and each subunit is 4 photosensitive pixels 110 in 2 rows and 2 columns. The arrangement method is:

[0044] W A W B

[0045] A W B W

[0046] W B W C

[0047] B W C W

[0048] W represents a full-color photosensitive pixel W; A represents a first-color photosensitive pixel among a plurality of color photosensitive pixels; B represents a second-color photosensitive pixel among a plurality of color photosensitive pixels; C represents a third-color photosensitive pixel among a plurality of color photosensitive pixels.

[0049] As Figure 5 shown, for each subunit, the full-color photosensitive pixel W and the color photosensitive pixels are alternately arranged.

[0050] As Figure 5 shown, the categories of the subunits include three categories. Among them, the first type of subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first-color photosensitive pixels A; the second type of subunit UB includes a plurality of full-color photosensitive pixels W and a plurality of second-color photosensitive pixels B; the third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, namely one first-type subunit UA, two second-type subunits UB, and one third-type subunit UC. Among them, one first-type subunit UA and one third-type subunit UC are arranged in the first diagonal direction D1 (for example Figure 5 the direction connecting the upper left corner and the lower right corner), and two second-type subunits UB are arranged in the second diagonal direction D2 (for example Figure 5 the direction connecting the upper right corner and the lower left corner). The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0051] It should be noted that in other embodiments, the first diagonal direction D1 may also be the direction connecting the upper right corner and the lower left corner, and the second diagonal direction D2 may also be the direction connecting the upper left corner and the lower right corner. In addition, the "direction" here does not refer to a single direction, and can be understood as the concept of a "straight line" indicating the arrangement, and there can be a two-way pointing at both ends of the straight line. The following Figures 6 to 10 explanation of the first diagonal direction D1 and the second diagonal direction D2 is the same as that here.

[0052] For another example, please refer to Figure 6 , Figure 6 is the photosensitive pixel 110 in the minimum repeating unit of another embodiment of the present application ( Figure 3The layout schematic diagram as shown). Among them, the smallest repeating unit is 16 photosensitive pixels 110 arranged in 4 rows and 4 columns, and each sub-unit is 4 photosensitive pixels 110 arranged in 2 rows and 2 columns. The arrangement method is as follows:

[0053] W A W B

[0054] A W B W

[0055] B W W C

[0056] W B C W

[0057] W represents the panchromatic photosensitive pixel W; A represents the first-color photosensitive pixel among multiple color photosensitive pixels; B represents the second-color photosensitive pixel among multiple color photosensitive pixels; C represents the third-color photosensitive pixel among multiple color photosensitive pixels.

[0058] Figure 6 The arrangement of the photosensitive pixels 110 in the smallest repeating unit shown is Figure 5 substantially the same as the arrangement of the photosensitive pixels 110 in the smallest repeating unit shown, the difference being that Figure 6 in the second type of sub-unit UB located in the lower left corner in, the alternating order of the panchromatic photosensitive pixel W and the single-color photosensitive pixel is Figure 5 in the second type of sub-unit UB located in the lower left corner in inconsistent. Specifically, Figure 5 in the second type of sub-unit UB located in the lower left corner in, the alternating order of the photosensitive pixels 110 in the first row is the panchromatic photosensitive pixel W, the color photosensitive pixel (i.e., the second-color photosensitive pixel B), and the alternating order of the photosensitive pixels 110 in the second row is the single-color photosensitive pixel (i.e., the second-color photosensitive pixel B), the panchromatic photosensitive pixel W; while Figure 6 in the second type of sub-unit UB located in the lower left corner in, the alternating order of the photosensitive pixels 110 in the first row is the color photosensitive pixel (i.e., the second-color photosensitive pixel B), the panchromatic photosensitive pixel W, and the alternating order of the photosensitive pixels 110 in the second row is the panchromatic photosensitive pixel W, the color photosensitive pixel (i.e., the second-color photosensitive pixel B).

[0059] As Figure 6 shown, Figure 6 in the first type of sub-unit UA and the third sub-unit UC in, the alternating order of the panchromatic photosensitive pixel W and the single-color photosensitive pixel is inconsistent with the alternating order of the panchromatic photosensitive pixel W and the color photosensitive pixel in the second type of sub-unit UB located in the lower left corner. Specifically, Figure 6 in the first type of sub-unit UA and the third sub-unit UC shown in, the alternating order of the photosensitive pixels 110 in the first row is the panchromatic photosensitive pixel W, the color photosensitive pixel, and the alternating order of the photosensitive pixels 110 in the second row is the color photosensitive pixel, the panchromatic photosensitive pixel W; while Figure 6In the second type of sub-unit UB located at the lower left corner as shown, the alternating order of the photosensitive pixels 110 in the first row is a color photosensitive pixel (i.e., the third color photosensitive pixel B) and a panchromatic photosensitive pixel W, and the alternating order of the photosensitive pixels 110 in the second row is a panchromatic photosensitive pixel W and a color photosensitive pixel (i.e., the third color photosensitive pixel B).

[0060] Therefore, according to Figure 5 and Figure 6 shown, within the same minimum repeating unit, the alternating order of the panchromatic photosensitive pixels W and the color photosensitive pixels in different sub-units can be the same (as shown in Figure 5 ), or can be different (as shown in Figure 6 ).

[0061] For another example, please refer to Figure 7 , Figure 7 which is a schematic layout diagram of the photosensitive pixels 110 (shown in Figure 3 ) in the minimum repeating unit of another embodiment of the present application. Among them, the minimum repeating unit is 36 photosensitive pixels 110 in 6 rows and 6 columns, and the sub-unit is 9 photosensitive pixels 110 in 3 rows and 3 columns.

[0062] The layout method is as follows:

[0063] W A W B W B

[0064] A W A W B W

[0065] W A W B W B

[0066] B W B W C W

[0067] W B W C W C

[0068] B W B W C W

[0069] W represents a panchromatic photosensitive pixel; A represents the first color photosensitive pixel among multiple color photosensitive pixels; B represents the second color photosensitive pixel among multiple color photosensitive pixels; C represents the third color photosensitive pixel among multiple color photosensitive pixels.

[0070] As shown in Figure 7 , for each sub-unit, the panchromatic photosensitive pixel W and the color photosensitive pixels are alternately arranged.

[0071] As shown in Figure 7As shown, the categories of sub-units include three types. Among them, the first type of sub-unit UA includes a plurality of panchromatic photosensitive pixels W and a plurality of first-color photosensitive pixels A; the second type of sub-unit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; the third type of sub-unit UC includes a plurality of panchromatic photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four sub-units, namely one first-type sub-unit UA, two second-type sub-units UB, and one third-type sub-unit UC. Among them, one first-type sub-unit UA and one third-type sub-unit UC are arranged in the first diagonal direction D1, and two second-type sub-units UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0072] For another example, please refer to Figure 8 , Figure 8 which is the layout schematic diagram of the photosensitive pixels 110 ( Figure 3 as shown) in the minimum repeating unit of another embodiment of the present application. Among them, the minimum repeating unit is 64 photosensitive pixels 110 with 8 rows and 8 columns, and the sub-unit is 16 photosensitive pixels 110 with 4 rows and 4 columns. The layout method is:

[0073] W A W A W B W B

[0074] A W A W B W B W

[0075] W A W A W B W B

[0076] A W A W B W B W

[0077] W B W B W C W C

[0078] B W B W C W C W

[0079] W B W B W C W C

[0080] B W B W C W C W

[0081] W represents a panchromatic photosensitive pixel; A represents the first-color photosensitive pixel among a plurality of color photosensitive pixels; B represents the second-color photosensitive pixel among a plurality of color photosensitive pixels; C represents the third-color photosensitive pixel among a plurality of color photosensitive pixels.

[0082] As Figure 8 shown, for each sub-unit, the panchromatic photosensitive pixels W and the single-color photosensitive pixels are alternately arranged.

[0083] As Figure 8As shown, the sub-units are classified into three categories. Among them, the first type of sub-unit UA includes a plurality of panchromatic photosensitive pixels W and a plurality of first-color photosensitive pixels A; the second type of sub-unit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; the third type of sub-unit UC includes a plurality of panchromatic photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four sub-units, namely one first-type sub-unit UA, two second-type sub-units UB, and one third-type sub-unit UC. Among them, one first-type sub-unit UA and one third-type sub-unit UC are arranged in the first diagonal direction D1, and two second-type sub-units UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0084] For another example, please refer to Figure 9 , Figure 9 which is the layout schematic diagram of the photosensitive pixels 110 ( Figure 3 as shown) in the minimum repeating unit of another embodiment of the present application. Among them, the minimum repeating unit is 16 photosensitive pixels 110 arranged in 4 rows and 4 columns, and the sub-unit is 4 photosensitive pixels 110 arranged in 2 rows and 2 columns.

[0085] The arrangement method is:

[0086] W W W W

[0087] A A B B

[0088] W W W W

[0089] B B C C

[0090] W represents the panchromatic photosensitive pixel W; A represents the first-color photosensitive pixel among the plurality of color photosensitive pixels; B represents the second-color photosensitive pixel among the plurality of color photosensitive pixels; C represents the third-color photosensitive pixel among the plurality of color photosensitive pixels.

[0091] As Figure 9 shown, in each sub-unit, the plurality of photosensitive pixels 110 in the same row have the same color channel (that is, the plurality of photosensitive pixels 110 in the same row are photosensitive pixels 110 of the same category). Among them, the photosensitive pixels 110 of the same category include: (1) all are panchromatic photosensitive pixels W; (2) all are first-color photosensitive pixels A; (3) all are second-color photosensitive pixels B; (4) all are third-color photosensitive pixels C.

[0092] As Figure 9As shown, the categories of sub-units include three types. Among them, the first type of sub-unit UA includes a plurality of panchromatic photosensitive pixels W and a plurality of first-color photosensitive pixels A; the second type of sub-unit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; the third type of sub-unit UC includes a plurality of panchromatic photosensitive pixels W and a plurality of third-color photosensitive pixels C. Among them, a plurality of photosensitive pixels 110 having the same color channel can be located in the first row of the sub-unit or the second row of the sub-unit, and there is no limitation here. Each minimum repeating unit includes four sub-units, namely one first-type sub-unit UA, two second-type sub-units UB, and one third-type sub-unit UC. Among them, one first-type sub-unit UA and one third-type sub-unit UC are arranged in the first diagonal direction D1, and two second-type sub-units UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0093] For another example, please refer to Figure 10 , Figure 10 This is the layout schematic diagram of the photosensitive pixels 110 ( Figure 3 as shown) in the minimum repeating unit of another embodiment of the present application. Among them, the minimum repeating unit is 16 photosensitive pixels 110 with 4 rows and 4 columns, and the sub-unit is 4 photosensitive pixels 110 with 2 rows and 2 columns.

[0094] The layout method is:

[0095] A W B W

[0096] A W B W

[0097] B W C W

[0098] B W C W

[0099] W represents the panchromatic photosensitive pixel W; A represents the first-color photosensitive pixel among the plurality of color photosensitive pixels; B represents the second-color photosensitive pixel among the plurality of color photosensitive pixels; C represents the third-color photosensitive pixel among the plurality of color photosensitive pixels.

[0100] As Figure 10 shown, in each sub-unit, a plurality of photosensitive pixels 110 in the same column have the same color channel (that is, a plurality of photosensitive pixels 110 in the same column are photosensitive pixels 110 of the same category). Among them, the photosensitive pixels 110 of the same category include: (1) all are panchromatic photosensitive pixels W; (2) all are first-color photosensitive pixels A; (3) all are second-color photosensitive pixels B; (4) all are third-color photosensitive pixels C.

[0101] As Figure 10As shown, the sub-units are classified into three categories. Among them, the first type of sub-unit UA includes a plurality of panchromatic photosensitive pixels W and a plurality of first-color photosensitive pixels A; the second type of sub-unit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; the third type of sub-unit UC includes a plurality of panchromatic photosensitive pixels W and a plurality of third-color photosensitive pixels C. Among them, a plurality of photosensitive pixels 110 having the same color channel can be located in the first column of the sub-unit or the second column of the sub-unit, and there is no limitation here. Each minimum repeating unit includes four sub-units, namely one first-type sub-unit UA, two second-type sub-units UB, and one third-type sub-unit UC. Among them, one first-type sub-unit UA and one third-type sub-unit UC are arranged in the first diagonal direction D1, and two second-type sub-units UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0102] For another example, please refer to Figure 11 , Figure 11 which is a schematic layout diagram of the photosensitive pixels 110 in the minimum repeating unit of another embodiment of the present application ( Figure 3 as shown). Among them, the minimum repeating unit is 16 photosensitive pixels 110 in 4 rows and 4 columns, and the sub-unit is 4 photosensitive pixels 110 in 2 rows and 2 columns.

[0103] The layout is as follows:

[0104] A W W W

[0105] A W B B

[0106] W W C W

[0107] B B C W

[0108] W represents the panchromatic photosensitive pixel W; A represents the first-color photosensitive pixel among the plurality of color photosensitive pixels; B represents the second-color photosensitive pixel among the plurality of color photosensitive pixels; C represents the third-color photosensitive pixel among the plurality of color photosensitive pixels.

[0109] As Figure 11 shown, in each sub-unit, a plurality of photosensitive pixels 110 in the same column or the same row have the same color channel (that is, a plurality of photosensitive pixels 110 in the same column or the same row are photosensitive pixels 110 of the same category). Among them, the photosensitive pixels 110 of the same category include: (1) all are panchromatic photosensitive pixels W; (2) all are first-color photosensitive pixels A; (3) all are second-color photosensitive pixels B; (4) all are third-color photosensitive pixels C.

[0110] As Figure 11As shown, the categories of sub-units include three types. Among them, the first type of sub-unit UA includes a plurality of panchromatic photosensitive pixels W and a plurality of first-color photosensitive pixels A. The plurality of panchromatic photosensitive pixels W are in the same column, and the plurality of first-color photosensitive pixels A are in the same column. The second type of sub-unit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B. The plurality of panchromatic photosensitive pixels W are in the same row, and the plurality of first-color photosensitive pixels A are in the same row. The third type of sub-unit UC includes a plurality of panchromatic photosensitive pixels W and a plurality of third-color photosensitive pixels C. The plurality of panchromatic photosensitive pixels W are in the same column, and the plurality of first-color photosensitive pixels A are in the same column. Each minimum repeating unit includes four sub-units, namely one first-type sub-unit UA, two second-type sub-units UB, and one third-type sub-unit UC. Among them, one first-type sub-unit UA and one third-type sub-unit UC are arranged in the first diagonal direction D1, and two second-type sub-units UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0111] Therefore, as Figure 11 shown, in the same minimum repeating unit, it can be that multiple photosensitive pixels 110 in the same row within some sub-units are photosensitive pixels 110 of the same category, and multiple photosensitive pixels 110 in the same column within the remaining sub-units are photosensitive pixels 110 of the same category.

[0112] For another example, please refer to Figure 12 , Figure 12 which is a schematic layout diagram of the photosensitive pixels 110 ( Figure 3 shown) in the minimum repeating unit of another embodiment of the present application. Among them, the minimum repeating unit is 16 photosensitive pixels 110 with 4 rows and 4 columns, and each sub-unit is 4 photosensitive pixels 110 with 2 rows and 2 columns. The layout is as follows:

[0113] A W B W

[0114] W W W W

[0115] B W C W

[0116] W W W W

[0117] W represents the panchromatic photosensitive pixel W; A represents the first-color photosensitive pixel among the plurality of color photosensitive pixels; B represents the second-color photosensitive pixel among the plurality of color photosensitive pixels; C represents the third-color photosensitive pixel among the plurality of color photosensitive pixels.

[0118] As Figure 12 shown, only one color photosensitive pixel is included in the 4 photosensitive pixels 110 of each sub-unit. In each minimum repeating unit, the color photosensitive pixel can be located at any position within the sub-unit (for example,Figure 12 shown at the upper left corner of the subunit).

[0119] As Figure 12 shown, the categories of the subunits include three types. Among them, the first type of subunit UA includes a plurality of panchromatic photosensitive pixels W and a first color photosensitive pixel A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a second color photosensitive pixel B; the third type of subunit UC includes a plurality of panchromatic photosensitive pixels W and a third color photosensitive pixel C. Each minimum repeating unit includes four subunits, namely, one first type of subunit UA, two second type of subunits UB, and one third type of subunit UC. Among them, one first type of subunit UA and one third type of subunit UC are arranged in the first diagonal direction D1, and two second type of subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0120] For another example, please refer to Figure 13 , Figure 13 which is the layout schematic diagram of the photosensitive pixel 110 ( Figure 3 shown) in the minimum repeating unit of another embodiment of the present application. Among them, the minimum repeating unit is 16 photosensitive pixels 110 arranged in 4 rows and 4 columns, and each subunit is 4 photosensitive pixels 110 arranged in 2 rows and 2 columns. The layout is as follows:

[0121] A W W B

[0122] W W W W

[0123] W W W W

[0124] B W W C

[0125] W represents the panchromatic photosensitive pixel W; A represents the first color photosensitive pixel among the plurality of color photosensitive pixels; B represents the second color photosensitive pixel among the plurality of color photosensitive pixels; C represents the third color photosensitive pixel among the plurality of color photosensitive pixels.

[0126] As Figure 13 shown, only one color photosensitive pixel is included in the 4 photosensitive pixels 110 of each subunit. In each minimum repeating unit, the color photosensitive pixel can be located at any position of the subunit (for example, at the upper left corner position, lower left corner position, upper right corner position, or lower right corner position shown in Figure 12 ).

[0127] As Figure 13As shown, the categories of sub-units include three types. Among them, the first type of sub-unit UA includes a plurality of panchromatic photosensitive pixels W and one first-color photosensitive pixel A; the second type of sub-unit UB includes a plurality of panchromatic photosensitive pixels W and one second-color photosensitive pixel B; the third type of sub-unit UC includes a plurality of panchromatic photosensitive pixels W and one third-color photosensitive pixel C. Each minimum repeating unit includes four sub-units, namely one first-type sub-unit UA, two second-type sub-units UB, and one third-type sub-unit UC. Among them, one first-type sub-unit UA and one third-type sub-unit UC are arranged in the first diagonal direction D1, and two second-type sub-units UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0128] For another example, please refer to Figure 14 , Figure 14 which is the layout schematic diagram of the photosensitive pixels 110 ( Figure 3 as shown) in the minimum repeating unit of another embodiment of the present application. Among them, the minimum repeating unit is 16 photosensitive pixels 110 arranged in 4 rows and 4 columns, and each sub-unit is 4 photosensitive pixels 110 arranged in 2 rows and 2 columns. The arrangement method is:

[0129] W A W B

[0130] A A B B

[0131] W B W C

[0132] B B C C

[0133] W represents the panchromatic photosensitive pixel W; A represents the first-color photosensitive pixel among the plurality of color photosensitive pixels; B represents the second-color photosensitive pixel among the plurality of color photosensitive pixels; C represents the third-color photosensitive pixel among the plurality of color photosensitive pixels.

[0134] Such as Figure 14 as shown, only one panchromatic photosensitive pixel W is included in the 4 photosensitive pixels 110 of each sub-unit. In each minimum repeating unit, the panchromatic photosensitive pixel W can be located at any position of the sub-unit (for example, at the upper left corner position of the sub-unit as Figure 14 shown).

[0135] Such as Figure 14As shown, the sub-units are classified into three categories. Among them, the first type of sub-unit UA includes a panchromatic photosensitive pixel W and a plurality of first-color photosensitive pixels A; the second type of sub-unit UB includes a panchromatic photosensitive pixel W and a plurality of second-color photosensitive pixels B; the third type of sub-unit UC includes a panchromatic photosensitive pixel W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four sub-units, namely one first-type sub-unit UA, two second-type sub-units UB, and one third-type sub-unit UC. Among them, one first-type sub-unit UA and one third-type sub-unit UC are arranged in the first diagonal direction D1, and two second-type sub-units UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0136] For another example, please refer to Figure 15 , Figure 15 which is the layout schematic diagram of the photosensitive pixels 110 ( Figure 3 as shown) in the minimum repeating unit of another embodiment of the present application. Among them, the minimum repeating unit is 16 photosensitive pixels 110 arranged in 4 rows and 4 columns, and each sub-unit is 4 photosensitive pixels 110 arranged in 2 rows and 2 columns. The arrangement method is:

[0137] W A B W

[0138] A A B B

[0139] B B C C

[0140] W B C W

[0141] W represents the panchromatic photosensitive pixel W; A represents the first-color photosensitive pixel among the plurality of color photosensitive pixels; B represents the second-color photosensitive pixel among the plurality of color photosensitive pixels; C represents the third-color photosensitive pixel among the plurality of color photosensitive pixels.

[0142] As Figure 15 shown, only one panchromatic photosensitive pixel W is included in the 4 photosensitive pixels 110 of each sub-unit. In each minimum repeating unit, the panchromatic photosensitive pixel W can be located at any position of the sub-unit (for example, at the upper left corner position, lower left corner position, upper right corner position, or lower right corner position of the sub-unit as Figure 15 shown).

[0143] As Figure 15As shown, the categories of the sub-units include three types. Among them, the first type of sub-unit UA includes a panchromatic photosensitive pixel W and a plurality of first-color photosensitive pixels A; the second type of sub-unit UB includes a panchromatic photosensitive pixel W and a plurality of second-color photosensitive pixels B; the third type of sub-unit UC includes a panchromatic photosensitive pixel W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four sub-units, namely one first-type sub-unit UA, two second-type sub-units UB, and one third-type sub-unit UC. Among them, one first-type sub-unit UA and one third-type sub-unit UC are arranged in the first diagonal direction D1, and two second-type sub-units UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

[0144] In some embodiments, as Figures 5 to 15 shown in the minimum repeating unit, the first-color photosensitive pixel A can be a red photosensitive pixel R; the second-color photosensitive pixel B can be a green photosensitive pixel G; the third-color photosensitive pixel C can be a blue photosensitive pixel Bu.

[0145] In some embodiments, as Figures 5 to 15 shown in the minimum repeating unit, the first-color photosensitive pixel A can be a red photosensitive pixel R; the second-color photosensitive pixel B can be a yellow photosensitive pixel Y; the third-color photosensitive pixel C can be a blue photosensitive pixel Bu.

[0146] In some embodiments, as Figures 5 to 15 shown in the minimum repeating unit, the first-color photosensitive pixel A can be a magenta photosensitive pixel M; the second-color photosensitive pixel B can be a cyan photosensitive pixel Cy; the third-color photosensitive pixel C can be a yellow photosensitive pixel Y.

[0147] It should be noted that in some embodiments, the response band of the panchromatic photosensitive pixel W can be the visible light band (for example, 400nm - 760nm). For example, an infrared filter is provided on the panchromatic photosensitive pixel W to achieve the filtering of infrared light. In other embodiments, the response band of the panchromatic photosensitive pixel W is the visible light band and the near-infrared band (for example, 400nm - 1000nm), which matches the response band of the photoelectric conversion element 1111 ( Figure 1 shown) in the image sensor 10 ( Figure 4 shown). For example, the panchromatic photosensitive pixel W can be not provided with a filter or provided with a filter that allows light of all bands to pass through. The response band of the panchromatic photosensitive pixel W is determined by the response band of the photoelectric conversion element 1111, that is, the two match. The embodiments of the present application include but are not limited to the above band ranges.

[0148] Please refer to Figure 1 、 Figure 2and Figure 16 In some embodiments, the control unit 13 controls the pixel array 11 to be exposed to obtain a first color raw image and a second color raw image. The first color raw image is composed of a plurality of first color raw image data, and each first color raw image data is generated by at least one color photosensitive pixel in a sub-unit; the second color raw image is composed of a plurality of second color raw image data, and each second color raw image data is generated by at least one panchromatic photosensitive pixel W and at least one color photosensitive pixel in a sub-unit.

[0149] Please refer to Figure 2 , in an example, each sub-unit of the pixel array 11 includes a plurality of panchromatic photosensitive pixels W and a plurality of color photosensitive pixels (as Figures 5 to 11 shown). After the control unit 13 controls the pixel array 11 to be exposed, the sum value or average value of a plurality of electrical signals generated by the plurality of color photosensitive pixels in a sub-unit after receiving light is used as a first color raw image data, and the first color raw image data of all sub-units in the pixel array 11 constitutes the first color raw image; the sum value or average value of a plurality of electrical signals generated by the plurality of panchromatic photosensitive pixels W in a sub-unit after receiving light and all electrical signals generated by all color photosensitive pixels in the sub-unit after receiving light is used as a second color raw image data, and the second color raw image data of all sub-units in the pixel array 11 constitutes the second color raw image.

[0150] Please refer to Figure 16, for example, the sum or average of the two electrical signals generated after two first-color photosensitive pixels A in the first type of subunit UA receive light is obtained to get a first color raw image data; the sum or average of the two electrical signals generated after two second-color photosensitive pixels B in two second type of subunits UB receive light is obtained to get two first color raw image data; the sum or average of the two electrical signals generated after two third-color photosensitive pixels C in the third type of subunit UC receive light is obtained to get a first color raw image data. These four first color raw image data together form a first image unit in the first color raw image, and the multiple image pixels in this first image unit are arranged in the form of ABBC. The sum or average of the two electrical signals generated after two panchromatic photosensitive pixels W in the first type of subunit UA receive light and the two electrical signals generated after two first-color photosensitive pixels A in this subunit receive light is obtained to get a second color raw image data; the sum or average of the two electrical signals generated after two panchromatic photosensitive pixels W in two second type of subunits UB receive light and the two electrical signals generated after two second-color photosensitive pixels B in this subunit receive light is obtained to get two second color raw image data; the sum or average of the two electrical signals generated after two panchromatic photosensitive pixels W in the third type of subunit UC receive light and the two electrical signals generated after two third-color photosensitive pixels C in this subunit receive light is obtained to get a second color raw image data; these four second raw image data together form a second image unit in the second color raw image, and the multiple image pixels in this second image unit are arranged in the form of ABBC.

[0151] It should be noted that in another example, when each subunit includes one color photosensitive pixel, the one electrical signal generated after this one color photosensitive pixel receives light is used as a first color raw image data, and the sum or average of the one electrical signal generated after this one color photosensitive pixel receives light and the electrical signals generated after all panchromatic photosensitive pixels receive light is used as a second color raw image data. In yet another example, when each subunit includes one panchromatic photosensitive pixel W, the sum or average of the one electrical signal generated after this one panchromatic photosensitive pixel W receives light and all the electrical signals generated after all color photosensitive pixels in this subunit receive light is used as a second color raw image data.

[0152] Please refer to Figure 1 and Figure 17 , after the image sensor 10 obtains the first color raw image and the second color raw image, interpolation processing is performed on the first color raw image to respectively obtain first interpolation images of each color channel, and interpolation processing is performed on the second color raw image to obtain second interpolation images of at least one color channel.

[0153] In one example, the size of the reference region 112 for interpolating the first color raw image may be a 7x7 region ( Figure 17 as shown). A represents the first color photosensitive pixel in the first color raw image; B represents the second color photosensitive pixel in the first color raw image; C represents the third color photosensitive pixel in the first color raw image. Based on the principles of local color difference constancy and adjacent pixel interpolation constancy along the interpolation direction, a demosaicing algorithm can be used to interpolate the first color raw image to obtain the first interpolated image of the second color channel. If the pixel in the first color raw image has the image data of the second color channel, then the image data of the second color channel of the pixel in the first color raw image is the image data of the corresponding pixel in the first interpolated image of the second color channel. For example, B12' = B12. If the pixel in the first color raw image has the image data of the first color channel or the third color channel, then interpolation processing is required. For example, for the pixel C44 in the first color raw image that has the image data of the third color channel, interpolation processing is performed. The missing second color photosensitive pixel B44' at C44 can be obtained through the following two relational expressions, and the first interpolated image of the second color channel is obtained by filling it in through interpolation.

[0154]

[0155]

[0156]

[0157]

[0158] And:

[0159]

[0160]

[0161]

[0162]

[0163] The missing second color photosensitive pixel B44' at C44 can be obtained through two relational expressions:

[0164]

[0165] Interpolating the pixels of the image data with the first color channel in the first color original image to obtain the first interpolated image of the second color channel is similar to interpolating the pixels of the image data with the third color channel to obtain the first interpolated image of the second color channel, which will not be elaborated here.

[0166] In another example, the size of the reference region 113 for interpolating the first color original image is a 7x7 region ( Figure 18 as shown). A represents the first color photosensitive pixel in the first color original image; B represents the second color photosensitive pixel in the first color original image; C represents the third color photosensitive pixel in the first color original image. Based on the principle of local color difference constancy and adjacent pixel interpolation constancy along the interpolation direction, a demosaicing algorithm can be used to interpolate the first color original image to obtain the first interpolated image of the first color channel. If the pixel in the first color original image has the image data of the first color channel, then the image data of the first color channel of the pixel in the first color original image is the image data of the corresponding pixel in the first interpolated image of the first color channel. For example, A11' = A11. If the pixel in the first color original image has the image data of the third color channel, then interpolation processing is required. For example, for interpolating the pixel C44, the missing second color photosensitive pixel A44' at C44 can be obtained through the following two relational expressions, and the first interpolated image of the first color channel is completed by interpolation. Among them, the image data of the pixels B33', B35', B44', B53', B55' can be obtained through Figure 17 the calculation method shown.

[0167]

[0168]

[0169]

[0170]

[0171] And:

[0172]

[0173]

[0174]

[0175]

[0176] The missing second color photosensitive pixel A44' at C44 can be obtained through two relational expressions:

[0177]

[0178] Please refer to again Figure 18 , interpolate the first color original image to obtain the first interpolated image of the first color channel. If the pixel points in the first color original image have image data of the second color channel, interpolation processing is required. For example: interpolate the pixel B34. The missing first color photosensitive pixel A34' at B34 can be obtained through the following two relational expressions, and the first interpolated image of the first color channel is completed by interpolation.

[0179]

[0180]

[0181]

[0182]

[0183]

[0184] In the above example, the weighting directions for interpolating the pixels with image data of the second color channel in the first color original image to obtain the interpolated image of the first color channel are weighted in the left and right directions. For example: the weighting direction for calculating the missing first color photosensitive pixel A34' at B34 is weighted in the left and right directions; the weighting directions for interpolating the pixels with image data of the second color channel in the first color original image to obtain the interpolated image of the first color channel can also be weighted in the up and down directions. For example: the weighting direction for calculating the missing first color photosensitive pixel A45' at B45 is weighted in the up and down directions. The calculation methods for weighting in the left and right directions are similar to those for weighting in the up and down directions, and will not be elaborated here.

[0185] Based on the principle of local color difference constancy and the interpolation constancy of adjacent pixels along the interpolation direction, a demosaicing algorithm can also be used to interpolate the first color raw image to obtain the first interpolated image of the third color channel. If the pixel in the first color raw image has the image data of the third color channel, then the image data of the third color channel of the pixel in the first color raw image is the image data of the corresponding pixel in the first interpolated image of the third color channel. If the pixel in the first color raw image has the image data of the first color channel or the second color channel, interpolation processing is required. Interpolating the pixels in the first color raw image that have the image data of the first color channel to obtain the first interpolated image of the third color channel is similar to interpolating the pixels that have the image data of the third color channel to obtain the interpolated image of the first color channel, which will not be elaborated here. Interpolating the pixels in the first color raw image that have the image data of the second color channel to obtain the first interpolated image of the third color channel is similar to interpolating the pixels that have the image data of the second color channel to obtain the interpolated image of the first color channel, which will not be elaborated here.

[0186] In the embodiment of the present application, the processor 20 can also use a demosaicing algorithm to interpolate the second color raw image to obtain the second interpolated image of at least one color channel. The processor 20 can perform interpolation processing on the second color raw image in a manner similar to that of the Figure 17 and Figure 18 illustrated embodiment to obtain the second interpolated image of the first color channel, the second interpolated image of the second color channel, and the second interpolated image of the third color channel, which will not be described in detail here.

[0187] In some embodiments, the shape of the window formed by the reference regions for interpolating the first color raw image and the second color raw image can be square, rectangular, or other shapes, which is not limited here; the size of the reference regions for interpolating the first color raw image and the second color raw image can be 3x3, 4x4, 5x5, 3x5, 5x7, 7x7, 9x5, etc., which is also not limited here. In some embodiments, the interpolation method for the first color raw image and the second color raw image can also be other common demosaicing algorithms, such as: nearest neighbor interpolation, linear interpolation, cubic interpolation, high-quality linear interpolation, smooth tone transition interpolation, pattern recognition interpolation, adaptive color plane interpolation, interpolation algorithm based on direction-weighted gradient, etc.

[0188] In some embodiments, after the image sensor 10 obtains the first interpolated image and the second interpolated image, the second interpolated image is fused with the first interpolated image of each color channel to obtain the fused image of each color channel. In one example, the second interpolated image of the second color channel can be fused with the first interpolated image of each color channel to obtain the fused image of each color channel: the second interpolated image of the second color channel is fused with the first interpolated image of the first color channel to obtain the fused image of the first color channel; the second interpolated image of the second color channel is fused with the first interpolated image of the second color channel to obtain the fused image of the second color channel; the second interpolated image of the second color channel is fused with the first interpolated image of the third color channel to obtain the fused image of the third color channel, and then the target image is obtained according to the fused image of each color channel.

[0189] In some embodiments, after the first interpolated image of each color channel and the second interpolated image of at least one color channel are obtained, subsequent processing can be performed on the first interpolated image of each color channel and the second interpolated image of at least one color channel. Specifically, the processor 20 can perform filtering processing on the first interpolated image of each color channel to obtain the first filtered image of each color channel, and the first filtered image is composed of a plurality of first filtered image data. The processor 20 can also perform filtering processing on the second interpolated image of at least one color channel to obtain the second filtered image, and the second filtered image is composed of a plurality of second filtered image data.

[0190] In some embodiments, the processor 20 can perform filtering processing on the first interpolated image of each color channel in the following manner to obtain the first filtered image of each color channel. Specifically, the processor 20 can determine the first image pixel to be filtered and the first region to be filtered in the first interpolated image of the first color channel, and the first image pixel to be filtered is located in the first region to be filtered. The processor 20 can determine the first reference image pixel and the first reference region in the second interpolated image, the first reference image pixel corresponds to the first image pixel to be filtered, and the first reference region corresponds to the first region to be filtered. For example: the processor 20 performs filtering processing on the first interpolated image of the first color channel (such as Figure 19As shown, the processor 20 determines that the pixel A44' is the first image pixel to be filtered. Subsequently, the processor 20 can determine the first region to be filtered 116 based on the first image pixel A44' to be filtered. The processor 20 can determine the first reference image pixel B'44' and the first reference region 119 in the second interpolated image. The first reference image pixel B'44' corresponds to the first image pixel A44' to be filtered in the first color channel; the first reference region 119 corresponds to the first region to be filtered 116 in the first color channel. It should be noted that the first image pixel A44' to be filtered in the first color channel can be located at any position within the first region to be filtered 116 in the first color channel; the shape of the window formed by the first reference region 119 can be square, rectangular or other shapes, which are not limited here; the size of the first reference region 119 can be 3x3, 4x4, 5x5, 3x5, 5x7, 7x7, 9x5, etc., which are also not limited here. In this embodiment, the first reference region 119 is a 3x3 region. The multiple first image pixels in the first reference region 119 include: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55'. Subsequently, the processor 20 calculates the weights of the multiple first image pixels in the first reference region 119, namely B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' relative to the first reference image pixel B'44'. The weights include the weights in the spatial domain and the weights in the pixel range domain. Then, based on the weights of the multiple first image pixels, namely B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' and the pixel values of the pixel points corresponding to the first region to be filtered 116 in the first color channel, the pixel value of the first image pixel to be filtered in the first color channel is corrected to obtain the first filtered image data in the first color channel.

[0191] Please refer to again Figure 21, the processor 20 can calculate multiple first image pixels: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' according to the weight function f(||p - q||) for the spatial domain weight relative to the first reference image pixel B'44'. Here, p is the coordinate of the first reference image pixel B'44' in the first reference area 119, q is the coordinate of multiple first image pixels B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' within the first reference area 119, and f is the weight function in the spatial domain. Among them, the smaller the coordinate difference between the first reference image pixel B'44' and multiple first image pixels (i.e., the closer the first reference image pixel B'44' is to multiple first image pixels), the higher the spatial domain weight of the first reference image pixel B'44' relative to multiple first image pixels. The processor 20 can calculate according to the weight function calculate multiple first image pixels: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' for the range domain weight relative to the first reference image pixel B'44'. Here, is the first interpolated image data (which can also be understood as the pixel value) of the first reference image pixel B'44', is the first interpolated image data (which can also be understood as the pixel value) of multiple first image pixels: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55', and g is the weight function in the pixel range domain. Among them, the greater the difference between the first interpolated image data of the first reference image pixel B'44' and the first interpolated image data of multiple first image pixels, the smaller the range domain weight.

[0192] After obtaining the weights of multiple first image pixels, the processor 20 can correct the pixel value of the first filtered image pixel in the first color channel according to the weights of multiple first image pixels and the pixel value of the pixel point corresponding to the first area to be filtered in the first color channel to obtain the first filtered image data in the first color channel. Please refer to Figure 19 and Figure 20 , the processor 20 can calculate according to the formula where, J p is the first filtered image data (i.e., the output pixel value) in the first color channel, k p is the total weight of the first reference area 119, Ω is the filtering window, I qAre the pixel values of the pixel points corresponding to the first area to be filtered 116: A33', A34', A35', A43', A44', A45', A53', A54', A55'. In this way, the processor 20 can calculate the first filtered image data of the first image pixels A33', A34', A35', A43', A44', A45', A53', A54', A55' in the first area to be filtered 116: A33'', A34'', A35'', A43'', A44'', A45'', A53'', A54'', A55''. The processor 20 can traverse each image pixel in the first interpolated image of the first color channel to obtain a plurality of first filtered image data of the first color channel. That is to say, the processor 20 can determine each image pixel in the first interpolated image of the first color channel as the first image pixel to be filtered, and perform Figure 20 The filtering process of the shown embodiment, so as to obtain the first filtered image data of the first color channel corresponding to the image pixel. After obtaining a plurality of first filtered image data of the first color channel, the plurality of first filtered image data of the first color channel can form the first filtered image of the first color channel.

[0193] It can be understood that the first interpolated images of each color channel are filtered to obtain the first filtered images of each color channel. Among them, the first filtered image is composed of a plurality of first filtered image data, and can be corrected by the weight of the first image pixel in the second interpolated image and the pixel value of the pixel point corresponding to the first area to be filtered to obtain the first filtered image data. The second interpolated image is obtained by interpolating the panchromatic photosensitive pixel W and at least one color channel of the second color original image. Therefore, using the weight of the first image pixel in the second interpolated image to filter the first interpolated images of each color channel can make the first filtered images of each color channel have higher light input and clarity, and at the same time, the calculated first filtered image data is more accurate.

[0194] Similarly, the processor 20 can also filter the first interpolated image of the second color channel. For example: the processor 20 determines the pixel B44' as the first image pixel to be filtered, and the processor 20 can determine the first area to be filtered 117 of the second color channel according to the first image pixel to be filtered B44' (such as Figure 21as shown). The processor 20 may determine the first reference image pixel B'44' and the first reference region 119 in the second interpolated image. The first reference image pixel B'44' corresponds to the first image pixel B44' to be filtered in the second color channel; the first reference region 119 corresponds to the first region 117 to be filtered in the second color channel. The processor 20 calculates the weights of multiple first image pixels in the first reference region 119, namely B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' relative to the first reference image pixel B'44'. The weights include the weights in the spatial domain and the weights in the pixel range domain. Then, according to the weights of the multiple first image pixels and the pixel values of the pixel points corresponding to the first region 117 to be filtered in the second color channel, the pixel value of the first image pixel to be filtered in the second color channel is corrected to obtain the first filtered image data in the second color channel. The processor 20 may perform filtering processing on the first interpolated image in the second color channel in a manner similar to that of the Figure 20 embodiment shown to obtain the first filtered image in the second color channel, which will not be elaborated here.

[0195] Similarly, the processor 20 may also perform filtering processing on the first interpolated image in the third color channel. For example, the processor 20 determines that the pixel C44' is the first image pixel to be filtered. The processor 20 may determine the first region 118 to be filtered in the third color channel according to the first image pixel C44' to be filtered (as Figure 22 shown). The processor 20 may determine the first reference image pixel B'44' and the first reference region 119 in the second interpolated image. The first reference image pixel B'44' corresponds to the first image pixel C44' to be filtered in the third color channel; the first reference region 119 corresponds to the first region 118 to be filtered in the third color channel. The processor 20 calculates the weights of multiple first image pixels in the first reference region 119, namely B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' relative to the first reference image pixel B'44'. The weights include the weights in the spatial domain and the weights in the pixel range domain. Then, according to the weights of the multiple first image pixels and the pixel values of the pixel points corresponding to the first region 118 to be filtered in the third color channel, the pixel value of the first image pixel to be filtered in the third color channel is corrected to obtain the first filtered image data in the third color channel. The processor 20 may perform filtering processing on the first interpolated image in the third color channel in a manner similar to that of the Figure 20 embodiment shown to obtain the first filtered image in the third color channel, which will not be elaborated here.

[0196] The second filtered image is composed of a plurality of second filtered image data, and the second interpolated image is filtered to obtain the second filtered image. Specifically, the processor 20 may determine a second image pixel to be filtered and a second region to be filtered in the second interpolated image. The second image pixel to be filtered is located within the second region to be filtered. The processor 20 may calculate the weights of a plurality of second image pixels in the second region to be filtered relative to the second image pixel to be filtered. The weights include weights in the spatial domain and weights in the pixel range domain. Subsequently, the processor 20 may correct the pixel value of the second image pixel to be filtered according to the pixel values and weights of the plurality of second image pixels to obtain the second filtered image data. The processor 20 may traverse each image pixel in the second interpolated image to obtain a plurality of second filtered image data. For example, the processor 20 filters the second interpolated image of the second color channel (such as Figure 23As shown, the processor 20 can determine that the pixel B'44' in the second interpolated image of the second color channel is the second pixel to be filtered in the second color channel. Subsequently, the processor can determine the second region to be filtered 119 in the second color channel based on the second pixel to be filtered B'44' in the second color channel. It should be noted that the second pixel to be filtered B'44' in the second color channel can be located at any position within the second region to be filtered 119 in the second color channel; the shape of the window formed by the second region to be filtered 119 in the second color channel can be square, rectangular or other shapes, which are not limited here; the size of the second region to be filtered 119 in the second color channel can be 3x3, 4x4, 5x5, 3x5, 5x7, 7x7, 9x5, etc., which are also not limited here. In this embodiment, the second region to be filtered 119 in the second color channel is a 3x3 region. The multiple second reference image pixels in the second region to be filtered 119 in the second color channel further include: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55'. Subsequently, the processor 20 calculates the weights of the multiple second reference image pixels: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' in the second color channel relative to the second pixel to be filtered B'44' in the second color channel. The weights include the weights in the spatial domain and the weights in the pixel range domain. Specifically, the processor 20 can calculate the weights in the spatial domain of the multiple second reference image pixels: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' in the second color channel relative to the second pixel to be filtered B'44' in the second color channel according to the weight function f(||p-q||), where p is the coordinate of the second pixel to be filtered B'44' in the second color channel within the second region to be filtered 119 in the second color channel, q is the coordinate of the multiple second reference image pixels: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' within the second region to be filtered 119 in the second color channel, and f is the weight function in the spatial domain. Among them, the smaller the coordinate difference between the second pixel to be filtered B'44' in the second color channel and the multiple second reference image pixels (i.e., the closer the second pixel to be filtered B'44' in the second color channel is to the multiple second reference image pixels), the higher the weight in the spatial domain of the second pixel to be filtered B'44' in the second color channel relative to the multiple second reference image pixels. The processor 20 can calculate according to the weight function Calculate the second reference image pixels of multiple second color channels: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55' with respect to the range domain weights of the second image pixel to be filtered B'44' of the second color channel. Among them, is the second interpolation image data (which can also be understood as the pixel value) of the second image pixel to be filtered B'44' of the second color channel, is the second interpolation image data (which can also be understood as the pixel value) of multiple second reference image pixels of the second color channel: B'33', B'34', B'35', B'43', B'44', B'45', B'53', B'54', B'55'. g is the weight function of the pixel range domain. Among them, the greater the difference between the second interpolation image data of the second image pixel to be filtered B'44' of the second color channel and the second interpolation image data of multiple second reference image pixels of the second color channel, the smaller the range domain weight. Subsequently, the processor 20 can correct the pixel value of the second image pixel to be filtered of the second color channel according to the pixel values and weights of multiple second reference image pixels of the second color channel to obtain the second filtered image data of the second color channel. Specifically: The processor 20 can calculate according to the formula where, J p is the second filtered image data of the second color channel (i.e., the output pixel value), k p is the total weight of the second area to be filtered 119 of the second color channel, Ω is the filtering window, and I q is the pixel value of the second image pixel point to be filtered of the second color channel. The processor 20 can traverse each image pixel in the second interpolation image of the second color channel to obtain multiple second filtered image data of the second color channel. The processor 20 can form the second filtered image of the second color channel with multiple second filtered image data of the second color channel.

[0197] The processor 20 can also perform filtering processing on the second interpolation image of the first color channel and the second interpolation image of the third color channel in a manner similar to the embodiment shown in Figure 23 to obtain the second filtered image of the first color channel and the second filtered image of the third color channel, which will not be elaborated here.

[0198] In the embodiment of the present application, by performing filtering processing on the first interpolation image and the second interpolation image, the flat areas in the filtered image are smoother, and at the same time, the edge areas of the image are protected from being blurred and thus more prominent, which is more conducive to improving the imaging quality of the camera module 100 ( Figure 1 shown).

[0199] After the processor 20 filters the first interpolation images of the respective color channels to obtain the first filtered images of the respective color channels, and filters the second interpolation images of at least one color channel to obtain the second filtered images of at least one color channel, the processor 20 can further fuse the second filtered images with the first filtered images of the respective color channels to obtain the fused images of the respective color channels. The fused images are composed of a plurality of fused image data. For example:

[0200] The processor 20 can calculate the fused image data based on the first filtered image data, the second filtered image data, and the interpolation image data. Among them, the fused image data is positively correlated with the first filtered image data, negatively correlated with the second filtered image data, and positively correlated with the interpolation image data. Subsequently, the processor 20 can traverse each image pixel in the first filtered image to obtain a plurality of fused image data. For example: the image data of the preset pixel point in the second interpolation image of the second color channel is b 1 , the image data of the pixel point corresponding to the preset pixel point in the first filtered image of the first color channel is a, the image data of the pixel point corresponding to the preset pixel point in the first filtered image of the second color channel is b, the image data of the pixel point corresponding to the preset pixel point in the first filtered image of the third color channel is c, and the second filtered image data of the second color channel is b 2 , it can be obtained that: the fused image data of the first color channel is ab 1 / b 2 , the fused image data of the second color channel is bb 1 / b 2 , the fused image data of the third color channel is cb 1 / b 2 .

[0201] In another embodiment, the image data of the preset pixel point in the second interpolation image of the first color channel is a 1 , the image data of the preset pixel point in the second interpolation image of the second color channel is b 1 , the image data of the preset pixel point in the second interpolation image of the third color channel is c 1 , the image data of the pixel point corresponding to the preset pixel point in the first filtered image of the first color channel is a, the image data of the pixel point corresponding to the preset pixel point in the first filtered image of the second color channel is b, the image data of the pixel point corresponding to the preset pixel point in the first filtered image of the third color channel is c, the second filtered image data of the first color channel is a 2 , the second filtered image data of the second color channel is b 2 , the second filtered image data of the third color channel is c 2, it can be obtained that the fused image data of the first color channel is aa 1 / a 2 , the fused image data of the second color channel is bb 1 / b 2 , the fused image data of the third color channel is cc 1 / c 2 .

[0202] In some embodiments, when the first filtered image data is greater than a preset pixel value, the processor 20 may determine the first filtered image data as the fused image data. When the first filtered image data is less than the preset pixel value, the processor 20 may calculate the fused image data based on the first filtered image data, the second filtered image data, and the interpolated image data. Among them, the fused image data is positively correlated with the first filtered image data, negatively correlated with the second filtered image data, and positively correlated with the interpolated image data. For example: The image sensor 10 is a 10-bit image sensor, and the processor 20 may set the preset pixel value to 959. If the first filtered image data is greater than the preset pixel value 959, it is considered that the first filtered image data is in an overexposed state, and the processor 20 does not perform fusion processing on the first filtered image data, and the processor 20 directly determines the first filtered image data as the fused image data. If the first filtered image data is less than the preset pixel value 959, the processor 20 calculates the fused image data based on the first filtered image data, the second filtered image data, and the interpolated image data.

[0203] After obtaining the fused images of each color channel, the fused images of each color channel can be directly converted into a YUV image as the target image, or the pixel points in the fused images of each color channel can be taken to form the target image of the Bayer array, and then the target image is transmitted to the image signal processor (ISP) for processing. In one embodiment, the processor 20 may include a processing circuit and an ISP. The processing circuit is integrated in the image sensor 10 and is used to implement the image acquisition method of the embodiments of the present application. After obtaining the target image, the target image is transmitted to the ISP, and the ISP performs subsequent image processing.

[0204] It can be understood that the target image obtained from the fused images of each color channel can be obtained through interpolation processing, filtering processing, and fusion processing of the first color raw image and the second color raw image. The target image is fused with the help of the panchromatic photosensitive pixel W. The panchromatic photosensitive pixel W has a higher light input, so that the signal-to-noise ratio of the target image is higher and the clarity is better; in this embodiment, the first color raw image inherits the higher signal-to-noise ratio and clarity of the second color raw image through fusion, thereby improving the photographing effect and image quality at night.

[0205] In summary, the camera assembly 100 according to the embodiment of the present application exposes the pixel array 11 to obtain a first color raw image including only color color channels and a second color raw image including both color color channels and panchromatic color channels of image data, and performs interpolation processing, filtering processing, and fusion processing on the first color raw image and the second color raw image, so as to improve the signal-to-noise ratio and clarity of the image by means of the image data of the panchromatic color channels, thereby improving the quality of the image taken in a dark environment, and also making the flat area in the image smoother and the edge area more prominent, thus further improving the quality of the image.

[0206] Please refer to Figure 24 , the present application further provides a mobile terminal 300. The mobile terminal 300 includes the camera assembly 100 and the housing 200 described in any one of the above embodiments. The camera assembly 100 is combined with the housing 200.

[0207] The mobile terminal 300 may be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, smart glasses, a smart helmet), a drone, a head-mounted display device, etc., which is not limited herein.

[0208] The mobile terminal 300 according to the embodiment of the present application exposes the pixel array 11 to obtain a first color raw image including only color color channels and a second color raw image including both color color channels and panchromatic color channels of image data, and fuses the first color raw image and the second color raw image, so as to improve the signal-to-noise ratio and clarity of the image by means of the image data of the panchromatic color channels, thereby improving the quality of the image taken in a dark environment.

[0209] Please refer to Figure 1 , Figure 2 , Figure 5 and Figure 25 , the present application further provides an image acquisition method that can be used for the image sensor 10 described in any one of the above embodiments. The image acquisition method includes:

[0210] 01: The pixel array 11 is exposed to obtain a first color raw image and a second color raw image. The first color raw image is composed of a plurality of first color raw image data, and each first color raw image data is generated by at least one color photosensitive pixel in a sub-unit. The second color raw image is composed of a plurality of second color raw image data, and each second color raw image data is generated by at least one panchromatic photosensitive pixel W and at least one color photosensitive pixel in a sub-unit; and

[0211] 02: Interpolate the first color original image to obtain first interpolated images for respective color channels, and interpolate the second color original image to obtain second interpolated images for at least one color channel;

[0212] 03: Fuse the second interpolated images with the first interpolated images for respective color channels to obtain fused images for respective color channels; and

[0213] 04: Obtain a target image based on the fused images for respective color channels.

[0214] In some embodiments, the image acquisition method further includes:

[0215] Filter the first interpolated images for respective color channels to obtain first filtered images for respective color channels; and

[0216] Filter the second interpolated images to obtain second filtered images;

[0217] Fuse the second interpolated images with the first interpolated images for respective color channels to obtain fused images for respective color channels.

[0218] Step 04 of obtaining a target image based on the fused images for respective color channels includes:

[0219] Fuse the second filtered images with the first filtered images for respective color channels to obtain fused images for respective color channels.

[0220] Please refer to Figure 18 , in some embodiments, the step of filtering the first interpolated images for respective color channels to obtain first filtered images for respective color channels includes:

[0221] Determine a first image pixel to be filtered and a first region to be filtered in the first interpolated image, where the first image pixel to be filtered is located within the first region to be filtered;

[0222] Determine a first reference image pixel and a first reference region in the second interpolated image, where the first reference image pixel corresponds to the first image pixel to be filtered, and the first reference region corresponds to the first region to be filtered;

[0223] Calculate weights of multiple first image pixels in the first reference region relative to the first reference image pixel, where the weights include weights in the spatial domain and weights in the pixel range domain;

[0224] Correct the pixel value of the first image pixel to be filtered based on the weights of the multiple first image pixels and the pixel value of the pixel point corresponding to the first region to be filtered to obtain first filtered image data; and

[0225] Traverse each image pixel in the first interpolated image to obtain a plurality of first filtered image data.

[0226] In some embodiments, the step of filtering the second interpolated image to obtain a second filtered image includes:

[0227] Determine the second image pixel to be filtered in the second interpolated image;

[0228] Determine the second area to be filtered in the second interpolated image, and the second image pixel to be filtered is located within the second area to be filtered;

[0229] Calculate the weights of a plurality of second image pixels in the second area to be filtered relative to the second image pixel to be filtered, where the weights include weights in the spatial domain and weights in the pixel range domain;

[0230] Correct the pixel value of the second image pixel to be filtered according to the pixel values and weights of the plurality of second image pixels to obtain second filtered image data; and

[0231] Traverse each image pixel in the second interpolated image to obtain a plurality of second filtered image data.

[0232] In some embodiments, the step of fusing the second filtered image with the first filtered image of each color channel respectively to obtain a fused image of each color channel includes:

[0233] Calculate and obtain fused image data according to the first filtered image data, the second filtered image data, and the interpolated image data. The fused image data is positively correlated with the first filtered image data, negatively correlated with the second filtered image data, and positively correlated with the interpolated image data;

[0234] Traverse each image pixel in the first filtered image to obtain a plurality of fused image data.

[0235] In some embodiments, the step of fusing the second filtered image with the first filtered image of each color channel respectively to obtain a fused image of each color channel further includes:

[0236] When the first filtered image data is greater than a preset pixel value, determine the first filtered image data as the fused image data;

[0237] When the first filtered image data is greater than a preset pixel value, calculate and obtain the fused image data according to the first filtered image data, the second filtered image data, and the interpolated image data. The fused image data is positively correlated with the first filtered image data, negatively correlated with the second filtered image data, and positively correlated with the interpolated image data.

[0238] Please refer to Figure 13, in some embodiments, when each subunit includes a plurality of color photosensitive pixels, the sum or average value of the plurality of electrical signals generated after the plurality of color photosensitive pixels receive light is used as a first color raw image data.

[0239] Please refer to Figure 14 , in some embodiments, when each subunit includes a panchromatic photosensitive pixel W, the sum or average value of an electrical signal generated after the panchromatic photosensitive pixel W receives light and all the electrical signals generated after all the color photosensitive pixels in the subunit receive light is used as a second color raw image data.

[0240] Please refer to Figure 5 , when each subunit includes a plurality of panchromatic photosensitive pixels, the sum or average value of the plurality of electrical signals generated after the plurality of panchromatic photosensitive pixels receive light and all the electrical signals generated after all the color photosensitive pixels in the subunit receive light is used as a second color raw image data.

[0241] The specific implementation process of the image acquisition method described in any of the above embodiments is the same as the specific implementation process of the foregoing description of the camera assembly 100 ( Figure 1 shown) for acquiring the first color raw image and the second color raw image and performing interpolation processing to respectively obtain the first interpolation images of each color channel and the second interpolation images of at least one color channel, fusing the second interpolation images with the first interpolation images of each color channel to obtain the fused images of each color channel, and obtaining the target image according to the fused images of each color channel, which will not be elaborated here.

[0242] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0243] Any process or method description set forth in the flowchart or otherwise described herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations in which functions may be performed in a substantially simultaneous manner or in an order opposite to that shown or discussed, according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0244] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. An image acquisition method for an image sensor, characterized in that, the image sensor includes a pixel array, the pixel array includes a plurality of subunits, each subunit includes at least one panchromatic photosensitive pixel and at least one color photosensitive pixel, and the color photosensitive pixel has a narrower spectral response than the panchromatic photosensitive pixel; the pixel array is exposed to obtain a first color raw image and a second color raw image, the first color raw image is composed of a plurality of first color raw image data, and each first color raw image data is generated by the at least one color photosensitive pixel in one subunit, and the second color raw image is composed of a plurality of second color raw image data, and each second color raw image data is generated by the at least one panchromatic photosensitive pixel and the at least one color photosensitive pixel in one subunit; the image acquisition method includes: performing interpolation processing on the first color raw image to obtain a plurality of first interpolated images, each first interpolated image corresponding to a color channel respectively, and performing interpolation processing on the second color raw image to obtain a second interpolated image of at least one color channel; fusing the second interpolated image with the first interpolated images of each color channel respectively to obtain a fused image of each color channel; and obtaining a target image according to the fused images of each color channel; wherein, when each subunit includes one color photosensitive pixel, an electrical signal generated by the one color photosensitive pixel after receiving light is used as one first color raw image data; when each subunit includes a plurality of color photosensitive pixels, a sum value or an average value of a plurality of electrical signals generated by the plurality of color photosensitive pixels after receiving light is used as one first color raw image data; when each subunit includes one panchromatic photosensitive pixel, a sum value or an average value of an electrical signal generated by the one panchromatic photosensitive pixel after receiving light and all electrical signals generated by all color photosensitive pixels in the subunit is used as one second color raw image data; when each subunit includes a plurality of panchromatic photosensitive pixels, a sum value or an average value of a plurality of electrical signals generated by the plurality of panchromatic photosensitive pixels after receiving light and all electrical signals generated by all color photosensitive pixels in the subunit is used as one second color raw image data.

2. The image acquisition method according to claim 1, characterized in that, the image acquisition method further includes: performing filtering processing on the first interpolated images of each color channel to obtain first filtered images of each color channel; and performing filtering processing on the second interpolated image to obtain a second filtered image; the step of fusing the second interpolated image with the first interpolated images of each color channel respectively to obtain a fused image of each color channel includes: fusing the second filtered image with the first filtered images of each color channel respectively to obtain a fused image of each color channel.

3. The image acquisition method according to claim 2, characterized in that, The first filtered image is composed of a plurality of first filtered image data. Filtering the first interpolated image of each color channel to obtain the first filtered image of each color channel includes: Determining a first image pixel to be filtered and a first region to be filtered in the first interpolated image, where the first image pixel to be filtered is located within the first region to be filtered; Determining a first reference image pixel and a first reference region in the second interpolated image, where the first reference image pixel corresponds to the first image pixel to be filtered, and the first reference region corresponds to the first region to be filtered; Calculate the weights of a plurality of first image pixels in the first reference region relative to the first reference image pixel, where the weights include weights in the spatial domain and weights in the pixel range domain, and the plurality of first image pixels are all image pixels in the first reference region; the weights in the spatial domain are calculated according to the weight function f(||p - q||), where p is the coordinate of the first reference image pixel in the first reference region, q is the coordinate of the plurality of first image pixels in the first reference region, f is the weight function in the spatial domain, and the smaller the coordinate difference between the first reference image pixel and the plurality of first image pixels, the higher the weight in the spatial domain; the weights in the pixel range domain are calculated according to the weight function obtained, where is the first interpolated image data of the first reference image pixel, is the first interpolated image data of the plurality of first image pixels, g is the weight function in the pixel range domain, and the greater the difference between the first interpolated image data of the first reference image pixel and the first interpolated image data of the plurality of first image pixels, the smaller the weight in the pixel range domain; Modify the pixel value of the first image pixel to be filtered according to the weight of the multiple first image pixels and the pixel value of the pixel corresponding to the first area to be filtered, so as to obtain the first filtered image data. The first filtered image data is calculated according to the formula which is calculated, where J p is the first filtered image data, k p is the sum of the weights of the first reference area, Ω is the filtering window, and I q is the pixel value of the pixel corresponding to the first area to be filtered; and Determining each image pixel in the first interpolated image of each color channel as the first filtered image data, and performing the above filtering process to obtain a plurality of the first filtered image data of each color channel.

4. The image acquisition method according to claim 2, wherein, The second filtered image is composed of a plurality of second filtered image data. Filtering the second interpolated image to obtain the second filtered image includes: Determining a second image pixel to be filtered in the second interpolated image; Determining a second region to be filtered in the second interpolated image, where the second image pixel to be filtered is located within the second region to be filtered; Calculate the weights of multiple second image pixels in the second area to be filtered relative to the second image pixel to be filtered. The weights include the weights in the spatial domain and the weights in the pixel range domain. The multiple second image pixels are all the image pixels in the second area to be filtered. The weights in the spatial domain are calculated according to the weight function f(||p - q||), where p is the coordinate of the second image pixel to be filtered in the second area to be filtered, q is the coordinate of the multiple second image pixels in the second area to be filtered, f is the weight function in the spatial domain, and the smaller the coordinate difference between the second image pixel to be filtered and the multiple second image pixels, the higher the weight in the spatial domain. The weights in the pixel range domain are calculated according to the weight function obtained, where is the second interpolated image data of the second image pixel to be filtered, is the second interpolated image data of the multiple second image pixels, g is the weight function in the pixel range domain, and the greater the difference between the second interpolated image data of the second image pixel to be filtered and the second interpolated image data of the multiple second image pixels, the smaller the weight in the pixel range domain; The pixel value of the second image pixel to be filtered is corrected according to the pixel values of the multiple second image pixels and the weights to obtain the second filtered image data. The second filtered image data is calculated according to the formula where J p is the second filtered image data, k p is the total weight of the second area to be filtered, Ω is the filtering window, and I q is the pixel value of the pixel point corresponding to the second area to be filtered; and Determining each image pixel in the second interpolated image of each color channel as the second filtered image data, and performing the above filtering process to obtain a plurality of the second filtered image data of each color channel.

5. The image acquisition method according to claim 2, wherein, The first filtered image is composed of a plurality of first filtered image data, the second filtered image is composed of a plurality of second filtered image data, the second interpolated image is composed of a plurality of interpolated image data, and the fused image is composed of a plurality of fused image data. Fusing the second filtered image with the first filtered image of each color channel to obtain the fused image of each color channel includes: When the first filtered image data is greater than a preset pixel value, determining the first filtered image data as the fused image data; when the first filtered image data is less than the preset pixel value, calculating the fused image data based on the first filtered image data, the second filtered image data, and the interpolated image data. The fused image data is positively correlated with the first filtered image data, negatively correlated with the second filtered image data, and positively correlated with the interpolated image data; Processing each image pixel in the first filtered image to obtain a plurality of the fused image data.

6. A camera assembly, wherein, comprising: An image sensor, the image sensor comprising a pixel array, the pixel array comprising a plurality of subunits, each subunit comprising at least one panchromatic photosensitive pixel and at least one color photosensitive pixel, the color photosensitive pixel having a narrower spectral response than the panchromatic photosensitive pixel; the pixel array is exposed to obtain a first color raw image and a second color raw image, the first color raw image consisting of a plurality of first color raw image data, each of the first color raw image data being generated by the at least one color photosensitive pixel in one of the subunits, the second color raw image consisting of a plurality of second color raw image data, each of the second color raw image data being generated by the at least one panchromatic photosensitive pixel and the at least one color photosensitive pixel in one of the subunits; and a processor, the processor being configured to perform interpolation processing on the first color raw image to obtain a plurality of first interpolated images, each of the first interpolated images corresponding to one color channel respectively, perform interpolation processing on the second color raw image to obtain a second interpolated image of at least one color channel; fuse the second interpolated image with the first interpolated images of each color channel respectively to obtain a fused image of each color channel; obtain a target image according to the fused images of each color channel; wherein, when each subunit comprises one color photosensitive pixel, an electrical signal generated by the one color photosensitive pixel after receiving light is used as one of the first color raw image data; when each subunit comprises a plurality of color photosensitive pixels, a sum value or an average value of a plurality of electrical signals generated by the plurality of color photosensitive pixels after receiving light is used as one of the first color raw image data; when each subunit comprises one panchromatic photosensitive pixel, a sum value or an average value of an electrical signal generated by the one panchromatic photosensitive pixel after receiving light and all electrical signals generated by all the color photosensitive pixels in the subunit after receiving light is used as one of the second color raw image data; when each subunit comprises a plurality of panchromatic photosensitive pixels, a sum value or an average value of a plurality of electrical signals generated by the plurality of panchromatic photosensitive pixels after receiving light and all electrical signals generated by all the color photosensitive pixels in the subunit after receiving light is used as one of the second color raw image data.

7. The camera assembly according to claim 6, wherein, the processor is further configured to: perform filtering processing on the first interpolated images of each color channel to obtain first filtered images of each color channel; perform filtering processing on the second interpolated image to obtain a second filtered image; and fuse the second filtered image with the first filtered images of each color channel respectively to obtain a fused image of each color channel.

8. The camera assembly according to claim 7, wherein, the first filtered image consists of a plurality of first filtered image data, and the processor is further configured to: Determine a first image pixel to be filtered and a first region to be filtered in the first interpolated image, where the first image pixel to be filtered is located within the first region to be filtered; Determine a first reference image pixel and a first reference region in the second interpolated image, where the first reference image pixel corresponds to the first image pixel to be filtered, and the first reference region corresponds to the first region to be filtered; Calculate the weights of multiple first image pixels in the first reference region relative to the first reference image pixel. The weights include the weights in the spatial domain and the weights in the pixel range domain. The multiple first image pixels are all the image pixels in the first reference region. The weights in the spatial domain are calculated according to the weight function f(||p - q||), where p is the coordinate of the first reference image pixel in the first reference region, q is the coordinate of the multiple first image pixels in the first reference region, f is the weight function in the spatial domain, and the smaller the coordinate difference between the first reference image pixel and the multiple first image pixels, the higher the weight in the spatial domain. The weights in the pixel range domain are calculated according to the weight function obtained, where is the first interpolated image data of the first reference image pixel, is the first interpolated image data of the multiple first image pixels, g is the weight function in the pixel range domain, and the greater the difference between the first interpolated image data of the first reference image pixel and the first interpolated image data of the multiple first image pixels, the smaller the weight in the pixel range domain; Modify the pixel value of the first pixel to be filtered according to the weights of the multiple first image pixels and the pixel value of the pixel corresponding to the first region to be filtered to obtain the first filtered image data. The first filtered image data is calculated according to the formula is calculated, where J p is the first filtered image data, k p is the total weight of the first reference region, Ω is the filtering window, and I q is the pixel value of the pixel corresponding to the first region to be filtered; and Determine each image pixel in the first interpolated image of each color channel as the first filtered image data, and perform the above filtering process to obtain a plurality of the first filtered image data of each color channel.

9. The camera assembly according to claim 7, wherein, The second filtered image is composed of a plurality of second filtered image data, and the processor is further configured to: Determine a second image pixel to be filtered in the second interpolated image; Determine a second region to be filtered in the second interpolated image, where the second image pixel to be filtered is located within the second region to be filtered; Calculate the weights of multiple second image pixels in the second area to be filtered relative to the second image pixel to be filtered. The weights include the weights in the spatial domain and the weights in the pixel range domain. The multiple second image pixels are all the image pixels in the second area to be filtered. The weights in the spatial domain are calculated according to the weight function f(||p - q||), where p is the coordinate of the second image pixel to be filtered in the second area to be filtered, q is the coordinate of the multiple second image pixels in the second area to be filtered, f is the weight function in the spatial domain, and the smaller the coordinate difference between the second image pixel to be filtered and the multiple second image pixels, the higher the weight in the spatial domain. The weights in the pixel range domain are calculated according to the weight function obtained, where is the second interpolated image data of the second image pixel to be filtered, is the second interpolated image data of the multiple second image pixels, g is the weight function in the pixel range domain, and the greater the difference between the second interpolated image data of the second image pixel to be filtered and the second interpolated image data of the multiple second image pixels, the smaller the weight in the pixel range domain; The pixel value of the second image pixel to be filtered is corrected according to the pixel values of the multiple second image pixels and the weight to obtain the second filtered image data. The second filtered image data is calculated according to the formula where J p is the second filtered image data, k p is the total weight of the second area to be filtered, Ω is the filtering window, and I q is the pixel value of the pixel point corresponding to the second area to be filtered; and Determine each image pixel in the second interpolated image of each color channel as the second filtered image data, and perform the above filtering process to obtain a plurality of the second filtered image data of each color channel.

10. The camera assembly according to claim 7, wherein, The first filtered image is composed of a plurality of first filtered image data, the second filtered image is composed of a plurality of second filtered image data, the second interpolated image is composed of a plurality of interpolated image data, and the fused image is composed of a plurality of fused image data. The processor is further configured to: When the first filtered image data is greater than a preset pixel value, determine the first filtered image data as the fused image data; when the first filtered image data is less than the preset pixel value, calculate the fused image data based on the first filtered image data, the second filtered image data, and the interpolated image data. The fused image data is positively correlated with the first filtered image data, negatively correlated with the second filtered image data, and positively correlated with the interpolated image data; Process each image pixel in the first filtered image to obtain a plurality of the fused image data.

11. The camera assembly according to claim 6, wherein, In each of the sub-units, the panchromatic photosensitive pixels and the color photosensitive pixels are alternately arranged; or In each of the sub-units, a plurality of photosensitive pixels in the same row have the same color channel; or In each of the sub-units, a plurality of photosensitive pixels in the same column have the same color channel.

12. A mobile terminal, wherein, comprising: a housing; and the camera assembly according to any one of claims 6 to 11, the camera assembly being combined with the housing.

Images