Image sensors, image processing methods, camera modules and electronic devices

Abstract

This disclosure relates to an image sensor, an image processing method, a camera module, and an electronic device. The image sensor includes a light-transmitting layer, a narrow-band gating filter, and a photosensitive substrate. The narrow-band gating filter is disposed between the light-transmitting layer and the photosensitive substrate. The narrow-band gating filter includes multiple spectral units, each spectral unit including multiple first pixels. The wavelength range of the gating light from each first pixel within the same spectral unit differs from the wavelength range of the gating light from other first pixels. The wavelength ranges of the multiple gating light rays corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range. The photosensitive substrate is used to sense the total amount of light energy transmitted through each first pixel. The multiple total light energy amounts corresponding to the same spectral unit are used to process and obtain the relative light energy of the gating light rays from each first pixel within the spectral unit, and to obtain the spectral information of each spectral unit. The multiple spectral information are used to process and obtain an image.

Description

Technical Field

[0001] This disclosure relates to the field of terminal technology, and in particular to an image sensor, an image processing method, a camera module, and an electronic device. Background Technology

[0002] Currently, electronic devices are typically equipped with cameras to enable their shooting functions. In related technologies, such cameras usually include a CCD sensor and a lens. The CCD sensor can typically be configured with pixels that transmit red, green, and blue light. However, red, green, and blue light are three disconnected bands within the visible wavelength range, making it impossible to obtain complete spectral information within the visible light wavelength range. Summary of the Invention

[0003] This disclosure provides an image sensor, an image processing method, a camera module, and an electronic device to address the shortcomings of related technologies.

[0004] According to a first aspect of the present disclosure, an image sensor is provided, including a light-transmitting layer, a narrow-band gating filter, and a photosensitive substrate, wherein the narrow-band gating filter is disposed between the light-transmitting layer and the photosensitive substrate.

[0005] The narrowband gating filter includes multiple spectral units, each of which includes multiple first pixels. The wavelength range of the gating light of each first pixel in the same spectral unit is different from the wavelength range of the gating light of other first pixels. The wavelength ranges of the multiple gating lights corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range.

[0006] The photosensitive substrate is used to sense the total amount of light energy transmitted through each first pixel. Multiple total amounts of light energy corresponding to the same spectral unit are used to process and obtain the relative light energy of the gated light rays of each first pixel within the spectral unit, and to obtain the spectral information of each spectral unit. Multiple spectral information are used to process and obtain an image.

[0007] Optionally, the first transmittance of the selected ray corresponding to each first pixel is greater than zero, and the second transmittance of the non-selected ray with a wavelength range within the preset wavelength range are all equal.

[0008] Optionally, each of the spectral units includes a second pixel, the second pixel having equal third transmittance for light with wavelengths within the preset wavelength range, and the total light energy of each of the first pixels and the total light energy of the second pixels within the same spectral unit are used to determine the relative light energy of the gated light of the first pixel.

[0009] Optionally, the second transmittance of the unselected light rays of each first pixel whose wavelength is within the preset wavelength range is equal to the third transmittance of the second pixel within the same spectral unit.

[0010] Optionally, the first transmittance of the first pixel in the same spectral unit is greater than the third transmittance of the second pixel.

[0011] Optionally, the preset wavelength range is 400nm-720nm.

[0012] Optionally, the bandwidth of the wavelength range of the gating light corresponding to each of the first pixels within the same spectral unit is equal.

[0013] Optionally, the bandwidth of the wavelength range of the gating ray corresponding to each first pixel within the same spectral unit is 30 nm or 40 nm.

[0014] Optionally, the wavelength range distribution of the multiple gated rays of the multiple first pixels in each of the spectral units is the same as the wavelength range distribution of the multiple gated rays of the multiple first pixels in any other spectral unit.

[0015] Optionally, each spectral unit includes at least four of the first pixels.

[0016] According to a second aspect of the present disclosure, a camera module is provided, comprising:

[0017] Lens;

[0018] The image sensor described in any of the above embodiments has its light-transmitting layer facing the exit end of the lens.

[0019] Optional, also includes:

[0020] A cutoff filter is disposed between the exit end of the lens and the light-transmitting layer of the image sensor. The cutoff filter is used to cut off light with wavelengths less than the minimum value of the preset wavelength range and wavelengths greater than the maximum value of the preset wavelength range.

[0021] Optionally, the cut-off filter is used to block infrared and ultraviolet light.

[0022] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0023] case;

[0024] A processor, wherein the processor is disposed within the housing;

[0025] As described in any of the above embodiments, the camera module is disposed within the housing, and the lens of the camera module faces the outside of the electronic device. The processor is connected to the image sensor. The processor is used to process and obtain the relative light energy of the gated light of each first pixel in the spectral unit according to the total light energy of multiple light units corresponding to the same spectral unit, and to obtain the spectral information of each spectral unit. The multiple spectral information is used to process and obtain an image.

[0026] According to a fourth aspect of the present disclosure, an image processing method is provided, applied to an electronic device, the electronic device including an image sensor, the image sensor including a light-transmitting layer, a narrow-band gating filter and a photosensitive substrate, the narrow-band gating filter being disposed between the light-transmitting layer and the photosensitive substrate;

[0027] The narrowband gating filter includes multiple spectral units, each of which includes multiple first pixels. The wavelength range of the gating light of each first pixel in the same spectral unit is different from the wavelength range of the gating light of other first pixels. The wavelength ranges of the multiple gating lights corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range.

[0028] The image processing method includes:

[0029] Obtain the total amount of light energy transmitted through each first pixel within each spectral unit;

[0030] Based on the total light energy of multiple first pixels, the relative light energy of the gating ray corresponding to each first pixel is obtained;

[0031] The spectral information of each spectral unit is obtained based on the relative light energy of the gating rays corresponding to multiple first pixels within the same spectral unit.

[0032] An image is obtained by processing the spectral information of multiple spectral units.

[0033] Optionally, the transmittance of the selected ray corresponding to each first pixel is a first transmittance, and the transmittance of the non-selected ray with a wavelength range within the preset wavelength range is a second transmittance;

[0034] The step of obtaining the relative light energy of the gating ray corresponding to each first pixel based on the total light energy of the plurality of first pixels includes:

[0035] Based on the first transmittance and total light energy of each first pixel, the total light energy of the second pixel when the first transmittance is equal to the standard first transmittance is obtained.

[0036] Based on the second transmittance of each first pixel and the sum of light energy of the unselected rays, obtain the sum of secondary light energy of the unselected rays when the second transmittance of the second pixel is equal to the standard second transmittance;

[0037] The relative light energy of the gated ray of the corresponding first pixel is obtained based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-gated rays.

[0038] Optionally, each of the spectral units includes a second pixel, the second pixel having equal third transmittance for light with wavelengths within the preset wavelength range;

[0039] The step of obtaining the relative light energy of the gating ray corresponding to each first pixel based on the total light energy of the plurality of first pixels includes:

[0040] Based on the first transmittance and total light energy of the gating ray of each first pixel, the total light energy of the second pixel when the first transmittance is equal to the standard first transmittance is obtained.

[0041] Based on the third transmittance of the second pixel and the wavelength of the transmitted light, the secondary light energy of the non-gated light ray of each first pixel is obtained;

[0042] The relative light energy of the gated ray of the corresponding first pixel is obtained based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-gated rays.

[0043] Optionally, processing the spectral information of the plurality of spectral units to obtain an image includes:

[0044] Based on the spectral information, the brightness of each first pixel is obtained;

[0045] A black and white image is obtained based on the brightness level.

[0046] Optionally, within the same spectral information, each first pixel transmits multiple non-gated rays, the wavelength range of the multiple non-gated rays being within a preset wavelength range, and the bandwidth being equal;

[0047] The process of processing the spectral information of multiple spectral units to obtain an image includes:

[0048] Obtain the relative light energy of each non-gated ray corresponding to the first pixel;

[0049] Based on the relative light energy of the gated ray and the relative light energy of the non-gated ray of each first pixel, the color displayed by each first pixel is determined to obtain a color image.

[0050] Optionally, the number of non-gated rays corresponding to each first pixel is equal to the number of first pixels in the same spectral unit minus 1, and the wavelength range of each non-gated ray is equal to the wavelength range of the gated rays of any other first pixel.

[0051] The step of obtaining the relative light energy of each non-gated ray corresponding to the first pixel includes:

[0052] Based on the relative light energy of the gated rays of adjacent first pixels, the relative light energy of each non-gated ray of the first pixel is obtained, and the wavelength range of the gated rays of adjacent first pixels is the same as the wavelength range of the non-gated rays obtained by the target.

[0053] Optionally, obtaining the relative light energy of each non-gated ray corresponding to the first pixel includes:

[0054] The relative light energy of each non-gated ray corresponding to the first pixel is obtained based on the relative light energy and weighting coefficient of the gated ray of each first pixel.

[0055] According to a fifth aspect of the present disclosure, an image processing apparatus is provided for use in an electronic device, the electronic device including an image sensor, the image sensor including a light-transmitting layer, a narrow-band gating filter and a photosensitive substrate, the narrow-band gating filter being disposed between the light-transmitting layer and the photosensitive substrate;

[0056] The narrowband gating filter includes multiple spectral units, each of which includes multiple first pixels. The wavelength range of the gating light of each first pixel in the same spectral unit is different from the wavelength range of the gating light of other first pixels. The wavelength ranges of the multiple gating lights corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range.

[0057] The image processing device includes:

[0058] The first acquisition module acquires the total amount of light energy transmitted through each first pixel within each spectral unit.

[0059] The second acquisition module acquires the relative light energy of the gating ray corresponding to each first pixel based on the total light energy of the multiple first pixels.

[0060] The third acquisition module obtains the spectral information of each spectral unit based on the relative light energy of the gating rays corresponding to multiple first pixels within the same spectral unit.

[0061] The processing module processes the spectral information of multiple spectral units to obtain an image.

[0062] Optionally, the transmittance of the selected ray corresponding to each first pixel is a first transmittance, and the transmittance of the non-selected ray with a wavelength range within the preset wavelength range is a second transmittance;

[0063] The second acquisition module includes:

[0064] The first acquisition unit acquires the total light energy when the first light transmittance of the first pixel is equal to the standard first transmittance, based on the first light transmittance and total light energy of each first pixel.

[0065] The second acquisition unit acquires, based on the second transmittance of each first pixel and the sum of light energy of the unselected light rays, the sum of the secondary light energy of the unselected light rays when the second transmittance of the second pixel is equal to the standard second transmittance;

[0066] The third acquisition unit acquires the relative light energy of the gated ray of the corresponding first pixel based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-gated rays.

[0067] Optionally, each of the spectral units includes a second pixel, the second pixel having equal third transmittance for light with wavelengths within the preset wavelength range;

[0068] The second acquisition module includes:

[0069] The fourth acquisition unit acquires the total light energy of the first pixel when the first transmittance is equal to the standard first transmittance, based on the first transmittance and total light energy of the selected light rays of each first pixel.

[0070] The fifth acquisition unit acquires the secondary light energy of the non-gated light rays of each first pixel based on the third transmittance of the second pixel and the wavelength of the transmitted light rays;

[0071] The sixth acquisition unit acquires the relative light energy of the gated ray of the corresponding first pixel based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-gated rays.

[0072] Optionally, the processing module includes:

[0073] The seventh acquisition unit acquires the brightness of each first pixel based on the spectral information;

[0074] The eighth acquisition unit acquires a black and white image based on the brightness level.

[0075] Optionally, within the same spectral information, each first pixel transmits multiple non-gated rays, the wavelength range of the multiple non-gated rays being within a preset wavelength range, and the bandwidth being equal;

[0076] The processing module includes:

[0077] The ninth acquisition unit acquires the relative light energy of each non-gated ray corresponding to the first pixel;

[0078] The tenth acquisition unit determines the color displayed by each first pixel based on the relative light energy of the gated ray and the relative light energy of the non-gated ray of each first pixel, thereby obtaining a color image.

[0079] Optionally, the number of non-gated rays corresponding to each first pixel is equal to the number of first pixels in the same spectral unit minus 1, and the wavelength range of each non-gated ray is equal to the wavelength range of the gated rays of any other first pixel.

[0080] The ninth acquisition unit includes:

[0081] The first acquisition subunit acquires the relative light energy of each non-gated ray of the first pixel based on the relative light energy of the gated rays of the adjacent first pixels, wherein the wavelength range of the gated rays of the adjacent first pixels is the same as the wavelength range of the non-gated rays acquired by the target.

[0082] Optionally, the ninth acquisition unit includes:

[0083] The second acquisition subunit acquires the relative light energy of each non-gated ray corresponding to the first pixel based on the relative light energy and weighting coefficient of the gated ray of each first pixel.

[0084] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in any of the foregoing embodiments.

[0085] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0086] As can be seen from the above embodiments, this disclosure, through the combined action of the photosensitive substrate and the narrowband gate filter, can obtain the relationship between wavelength and light energy of each spectral unit within a preset wavelength range, and can obtain complete spectral information of each spectral unit within the preset wavelength range, which is beneficial to improving the color accuracy of the image obtained subsequently based on the spectral information.

[0087] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0088] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0089] Figure 1 This is a schematic cross-sectional view of an image sensor according to an exemplary embodiment.

[0090] Figure 2 This is a schematic diagram of the pixel distribution of a narrowband gating filter according to an exemplary embodiment.

[0091] Figure 3 This is a schematic diagram of a partial structure of an image sensor according to an exemplary embodiment.

[0092] Figure 4 This is a schematic diagram of the pixel distribution of another narrowband gating filter according to an exemplary embodiment.

[0093] Figure 5 yes Figure 4 A schematic diagram of the light transmission in the first local area.

[0094] Figure 6 yes Figure 4 A schematic diagram of the light transmission in the second local area.

[0095] Figure 7 yes Figure 4 A schematic diagram of the light transmission in the third local area.

[0096] Figure 8 This is a cross-sectional schematic diagram of a camera module according to an exemplary embodiment.

[0097] Figure 9 yes Figure 8 A schematic diagram of the AA cross-section of the camera module in the diagram.

[0098] Figure 10 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0099] Figure 11 This is a flowchart illustrating another image processing method according to an exemplary embodiment.

[0100] Figure 12 This is a schematic diagram illustrating the relative light energy of the gated and ungated rays of each first pixel in a fifth spectral unit according to an exemplary embodiment.

[0101] Figure 13 This is one of the block diagrams of an image processing apparatus according to an exemplary embodiment.

[0102] Figure 14This is a second block diagram of an image processing apparatus according to an exemplary embodiment.

[0103] Figure 15 This is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0104] Figure 16 This is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0105] Figure 17 This is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0106] Figure 18 This is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0107] Figure 19 This is block diagram seven of an image processing apparatus according to an exemplary embodiment.

[0108] Figure 20 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. Detailed Implementation

[0109] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0110] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0111] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0112] Figure 1 This is a cross-sectional schematic diagram of an image sensor 100 according to an exemplary embodiment. Figure 2 This is a schematic diagram of the pixel distribution of a narrowband gating filter 2 according to an exemplary embodiment. Figure 3 This is a schematic diagram of a partial structure of an image sensor 100 according to an exemplary embodiment. For example... Figures 1-3 As shown, the image sensor 100 may include a light-transmitting layer 1, a narrow-band gating filter 2, and a photosensitive substrate 3. The light-transmitting layer 1 may include a microlens to focus light incident toward the light-transmitting layer 1. The narrow-band gating filter 2 may be disposed between the light-transmitting layer 1 and the photosensitive substrate 3. The light emitted from the light-transmitting layer 1 can be sensed by the photosensitive substrate 3 after passing through the narrow-band gating filter 2. The photosensitive substrate 3 can convert the optical signal into an electrical signal for output, so that the back-end can perform corresponding processing based on the received electrical signal.

[0113] like Figure 2 As shown, the narrowband gated filter 2 may include multiple spectral units 21, each of which includes multiple first pixels. For example... Figure 2 As shown, the plurality of spectral units 21 may include a first spectral unit 211, a second spectral unit 212, a third spectral unit 213, a fourth spectral unit 214, a fifth spectral unit 215, a sixth spectral unit 216, a seventh spectral unit 217, an eighth spectral unit 218, and a ninth spectral unit 219. The nine spectral units 21 can be arranged in a 3×3 array. Each of the nine spectral units 21 may include a plurality of first pixels. For example, taking the first spectral unit 211 as an example, the first spectral unit 211 may include nine first pixels, namely first pixel A, first pixel B, first pixel C, first pixel D, first pixel E, first pixel F, first pixel G, first pixel H, and first pixel J. The nine first pixels can be arranged in a 3×3 array. Similarly, the other spectral units of the narrowband gating filter 2 may also adopt the same structure as the first spectral unit 211, which will not be described in detail here.

[0114] Within the same spectral unit, the wavelength range of the selected ray of each first pixel differs from the wavelength range of the selected ray of other first pixels. The wavelength ranges of multiple selected rays corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range. The selected ray of a first pixel refers to light with relatively high transmittance, while the non-selected ray of a first pixel refers to light with relatively low transmittance. (Continuing with...) Figure 2For example, assume that the gate ray of the first pixel A is gate ray A, the gate ray of the first pixel B is gate ray B, the gate ray of the first pixel C is gate ray C, the gate ray of the first pixel D is gate ray D, the gate ray of the first pixel E is gate ray E, the gate ray of the first pixel F is gate ray F, the gate ray of the first pixel G is gate ray G, the gate ray of the first pixel H is gate ray H, and the gate ray of the first pixel J is gate ray J; among gate rays A, B, C, D, E, F, G, H, and J, the wave of gate ray A is... The wavelength range of a gate ray A differs from that of other gate rays B, C, D, E, F, G, H, and J. Similarly, the wavelength range of any one of the gate rays B, C, D, E, F, G, H, and J differs from that of the other gate rays. Furthermore, the multiple wavelength ranges corresponding to the gate ray A, B, C, D, E, F, G, H, and J can be continuously and uniformly distributed within a preset wavelength range. For example, assuming the preset wavelength range is 440nm-710nm, the bandwidth of each gated ray can be designed to be 30nm. For instance, the wavelength range of gated ray A is 440nm-470nm; the wavelength range of gated ray B is 471nm-500nm, and so on. Multiple first pixels within the first spectral unit 211 can divide the light within the preset wavelength range into multiple segments for gated selection. Subsequently, the total light energy of each transmitted pixel can be sensed by the photosensitive substrate 3. The multiple total light energy corresponding to the same spectral unit can be used to process and obtain the relative light energy of the gated ray of each first pixel within the spectral unit, and obtain the spectral information of each spectral unit. Multiple spectral information are used to process and obtain an image.

[0115] Based on this, through the combined action of the photosensitive substrate 3 and the narrowband gate filter 2, the relationship between wavelength and light energy of each spectral unit within a preset wavelength range can be obtained. This allows for the acquisition of complete spectral information for each spectral unit within the preset wavelength range, which is beneficial for improving the color accuracy of the image subsequently obtained based on this spectral information. In the embodiments provided in this disclosure, each spectral unit includes nine first line pixels as an example. In other embodiments, each spectral unit may also include other numbers of first pixels, such as at least four first pixels. Therefore, when the image sensor 100 acquires visible light, since the wavelength range of visible light is fixed, using four or more first pixels allows for a narrower design of the wavelength range of the gated light for each first pixel, which is beneficial for the color accuracy of subsequent imaging. The above description uses the narrowband gate filter 2 including nine spectral units as an example. In other embodiments, the narrowband gate filter 2 may also include other numbers of spectral units, and this disclosure does not impose any limitations on this. The above description uses a 3×3 array distribution of both the first pixel and the spectral unit as an example. In other embodiments, the array distribution can also be 4×5 or 6×7, depending on the number of units. This disclosure does not limit this distribution.

[0116] In the above embodiments, to facilitate subsequent image processing, the wavelength range distribution of the multiple gated rays of multiple first pixels within each spectral unit is the same as the wavelength range distribution of the multiple gated rays of multiple first pixels within any other spectral unit. For example, the wavelength range of the nine gated rays corresponding to the nine first pixels within the first spectral unit 211 can be the same as the wavelength range of the nine gated rays corresponding to the nine first pixels in any of the other eight spectral units. For instance, the wavelength range of the gated ray of the first pixel A in the first spectral unit 211 can be the same as the wavelength range of the gated ray of the first pixel A in the second spectral unit 212, and the wavelength range of the gated ray of the first pixel B in the first spectral unit 211 can be the same as the wavelength range of the gated ray of the first pixel B in the second spectral unit 212, and so on for other first pixels. In this way, the subsequent preset wavelength range can be divided into multiple interval ranges. Each interval range can be selected by any first pixel to obtain the corresponding spectrum within that interval range. This makes it convenient to obtain the light conditions of other wavelength ranges transmitted by the first pixel when using algorithms for color processing (this effect will be clearly understood in the subsequent explanation of image processing).

[0117] In the following embodiments, the example is taken with a preset wavelength range of 400nm-720nm and a bandwidth of 40nm for the wavelength range of the selected light corresponding to each first pixel in the same spectral unit.

[0118] In one embodiment, such as Figure 4 As shown, it is assumed that the multiple spectral units 21 of the narrowband gated filter 2 may include a first spectral unit 211, a second spectral unit 212, a third spectral unit 213, a fourth spectral unit 214, a fifth spectral unit 215, a sixth spectral unit 216, a seventh spectral unit 217, an eighth spectral unit 218, and a ninth spectral unit 219; each spectral unit includes eight first pixels and one second pixel. Taking the first spectral unit 211 as an example, the first spectral unit 211 may include first pixel A, first pixel B, first pixel C, first pixel D, first pixel F, first pixel G, first pixel H, first pixel J, and second pixel K. The gate ray of the first pixel A is gate ray A, and the wavelength range of gate ray A is 400nm-440nm; the gate ray of the first pixel B is gate ray B, and the wavelength range of gate ray B is 441nm-480nm; the gate ray of the first pixel C is gate ray C, and the wavelength range of gate ray B is... The wavelength range of the first pixel D is 481nm-520nm; the selected ray of the first pixel D is selected ray D, and the wavelength range of the selected ray B is 521nm-560nm; the selected ray of the first pixel F is selected ray F, and the wavelength range of the selected ray B is 561nm-600nm; the selected ray of the first pixel G is selected ray G, and the wavelength range of the selected ray B is 601nm-640nm; the selected ray of the first pixel H is selected ray H, and the wavelength range of the selected ray B is 641nm-680nm; the selected ray of the first pixel J is selected ray J, and the wavelength range of the selected ray B is 681nm-720nm; the second pixel K can transmit light in the range of 400nm-720nm, and the third transmittance of the second pixel K for light in the wavelength range of 400nm-720nm is equal. The total light energy of each first pixel and the total light energy of the second pixel within the same spectral unit are used to determine the relative light energy of the selected ray of the first pixel.

[0119] For example, such as Figures 5-7 As shown, assuming that the first transmittance of each first pixel for the selected ray is T1, and the second transmittance for the non-selected ray (i.e., ray whose wavelength range is within the preset wavelength range but different from the wavelength range of the selected ray) is equal to the third transmittance of the second pixel K, and both are represented as second transmittance = third transmittance = T3, then we can obtain as shown in 5- Figure 7The curve relationship between wavelength and transmittance is shown in the graph. In this graph, the relationship between the first pixel A and the second pixel K is used. In the curve relationship between wavelength and transmittance corresponding to the first pixel A, the area S1 enclosed by the curve and the coordinate system can be used to characterize the total light energy of the first pixel A sensed by the photosensitive substrate 3. This total light energy includes the light energy of the selected ray A and the light energy of the non-selected ray of the first pixel A. In the curve relationship between wavelength and transmittance corresponding to the second pixel K, the area S2 enclosed by the curve and the coordinate system can be used to characterize the total light energy of the second pixel K sensed by the photosensitive substrate 3. Therefore, based on the area S2 corresponding to the second pixel K, the area P1 corresponding to the gate ray A in the first pixel A can be obtained as P1 = S1 - 7 / 8S2. This area P1 can be used to characterize the relative light energy of the gate ray A of the first pixel A. Similarly, the relative light energy of the gate ray B of the first pixel B, the relative light energy of the gate ray C of the first pixel C, the relative light energy of the gate ray D of the first pixel D, the relative light energy of the gate ray F of the first pixel F, the relative light energy of the gate ray G of the first pixel G, the relative light energy of the gate ray H of the first pixel H, and the relative light energy of the gate ray A of the first pixel J can be obtained. Based on this, the spectral information of the first spectral unit 211 can be obtained. Similarly, the spectral information of other units of the narrowband gate filter 2 can be obtained.

[0120] It should be noted that in the above embodiments, the second transmittance of the non-gated ray of the first pixel A is equal to the third transmittance of the second pixel K as an example. In other embodiments, the second transmittance of the non-gated ray of the first pixel A may not be equal to the third transmittance of the second pixel K. In this case, the relative light energy of the gated ray of each first pixel can still be calculated by using the method P1 = S1 - 7 / 8S2, so that the relative light energy of the gated ray of multiple first pixels can be obtained based on the second transmittance being equal to the third transmittance. Similarly, when the second transmittance and the third transmittance of other first pixels are not equal, the embodiment for the first pixel A can be referred to to obtain the relative light energy of the gated ray of each first pixel. This will not be elaborated here.

[0121] In this embodiment, the first transmittance of the first pixel within the same spectral unit is greater than the third transmittance of the second pixel. In the above embodiments, the example given is that the first transmittance of multiple first pixels for their corresponding gated rays is equal. In other embodiments, the first transmittance of multiple first pixels may not be equal. Subsequently, the first transmittance of multiple first pixels can be converted to a standard first transmittance, and then the relative light energy of each first pixel's gated ray when it has the standard first transmittance can be calculated.

[0122] In another embodiment, the first transmittance of the selected ray of each first pixel is greater than zero, and the second transmittance of the non-selected rays whose wavelength range is within a preset wavelength range is equal. For example, the second transmittance can be equal to 0, in other words, only the selected ray passes through the same first pixel, and other non-selected rays cannot pass through. In this case, the area enclosed by the curve and the coordinate system in the curve relationship between the transmittance and wavelength of each first pixel can be used to characterize the relative light energy of the selected ray of the first pixel. Alternatively, the second transmittance can be greater than zero. In this case, the difference between the area enclosed by the curve and the coordinate system and the area corresponding to the non-selected ray in the curve relationship between the transmittance and wavelength of each first pixel can be used to characterize the relative light energy of the selected ray of the first pixel.

[0123] In some cases, when the first transmittance of multiple first pixels is equal, the spectral information of the same spectral unit can be obtained. In other cases, when the first transmittance of multiple first pixels is not equal, it can be converted to the case where the first transmittance of each first pixel equals the standard first transmittance, and the area used to characterize the relative light energy of the gated ray of that first pixel can be obtained. This allows the spectral information of each spectral unit to be obtained. (Relative to...) Figure 4 The embodiment uses a second pixel to define the light energy of non-gated rays. This embodiment's calculation method is simple and fast. Figure 4 The scheme that uses the second pixel to calibrate the light energy of ungated rays is more accurate than the previous embodiment, and the color can be more precise in the later stages.

[0124] In the above embodiments, the example given is that the preset wavelength range is 400nm-720nm, and the bandwidth of the wavelength range of the gating light corresponding to each first pixel in the same spectral unit is equal and equal to 40nm. In other embodiments, the bandwidth of the wavelength range of the gating light corresponding to each first pixel in the same spectral unit may also be equal to 30nm or 35nm, etc., and this disclosure does not impose any limitation on this. When the bandwidth of the wavelength range of the gating light of each first pixel is equal, the number of first pixels can be adjusted according to the size of the preset wavelength range, and this disclosure does not impose any limitation on this.

[0125] Based on the image sensor 100 provided in the above embodiments, such as Figure 8 and Figure 9As shown, this disclosure also provides a camera module 200, which may include a lens 201 and an image sensor 100. The light-transmitting layer 1 of the image sensor 100 faces the emitting end of the lens 201, so that light transmitted from the lens 201 can enter the image sensor 100 and be sensed by the photosensitive substrate 3 after passing through the narrowband gate filter 2. Further, the camera module 200 may also include a cutoff filter 202, which is disposed between the emitting end of the lens 201 and the light-transmitting layer 1 of the image sensor 100. The cutoff filter 202 can be used to cut off light with wavelengths less than the minimum value of a preset wavelength range of the image sensor 100 and wavelengths greater than the maximum value of the preset wavelength range. Based on this, after external light passes through the cutoff filter 202, the wavelength of the light entering the image sensor 100 will be within the preset wavelength range. Subsequently, the narrowband gate filter 2 can be used to further refine the division of each wavelength band to obtain accurate spectral information. For example, the cutoff filter can be used to cut off infrared and ultraviolet light, that is, the cutoff filter 202 can cut off light with wavelengths less than 400nm and wavelengths greater than 760nm.

[0126] This disclosure also provides an electronic device (not shown) that may include a housing, a processor, and a camera module 200 as described in any of the above embodiments. The processor may be disposed within the housing, and the lens 201 of the camera module 200 may be positioned facing outwards from the electronic device, for example, through an opening in the housing or through an opening in the display panel of the electronic device. This disclosure does not impose any limitations on this. The processor may also be electrically connected to the image sensor 100 of the camera module 200, for example, by one or more of a printed circuit board, a flexible circuit board, copper wires, and metal springs to connect the processor and the image sensor 100. The processor may be used to process and obtain the relative light energy of the gated rays of each first pixel within the same spectral unit based on the total light energy corresponding to the same spectral unit, and to obtain the spectral information of each spectral unit. The multiple spectral information are used to process and obtain an image.

[0127] like Figure 10 As shown, this disclosure also provides an image processing method that can be applied to an electronic device. The electronic device may include an image sensor, which includes a light-transmitting layer, a narrow-band gating filter, and a photosensitive substrate. The narrow-band gating filter is disposed between the light-transmitting layer and the photosensitive substrate.

[0128] The narrowband gating filter includes multiple spectral units, each spectral unit including multiple first pixels. The wavelength range of the gating light from each first pixel within the same spectral unit differs from the wavelength range of the gating light from other first pixels. The wavelength ranges of the multiple gating light rays corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range. The photosensitive substrate is used to sense the total amount of light energy transmitted through each first pixel. The image processing method may include the following steps:

[0129] In step 101, the total amount of light energy transmitted through each first pixel within each spectral unit is obtained.

[0130] In this embodiment, the total light energy of each pixel output by the image sensor 100 can be obtained, which can be output through an interface or through a single interface, and this disclosure does not limit this.

[0131] In step 102, the relative light energy of the gating ray corresponding to each first pixel is obtained based on the total light energy of the multiple first pixels.

[0132] In this embodiment, in some cases, the transmittance of the selected ray corresponding to each first pixel is a first transmittance, and the transmittance of the non-selected ray with a wavelength range within the preset wavelength range is a second transmittance. The total secondary light energy when the first transmittance of the first pixel equals the standard first transmittance can be obtained based on the first transmittance and the total light energy of each first pixel. Specifically, this can be achieved according to the method described in the previous embodiment, based on the relationship curve between transmittance and wavelength, and the principle that the ratio of the area enclosed by the wavelength relationship curve and the coordinate system to the transmittance is equal. Similarly, based on the second transmittance of each first pixel and the sum of the secondary light energy of the non-selected ray, the sum of the secondary light energy of the non-selected ray when the second transmittance of the second pixel equals the standard second transmittance can be obtained. This can also be achieved based on the relationship curve between transmittance and wavelength, and the principle that the ratio of the area enclosed by the wavelength relationship curve and the coordinate system to the transmittance is equal. Finally, the relative light energy of the selected ray corresponding to the first pixel is obtained based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-selected ray. The first transmittance and the second transmittance of the standard are both greater than zero and less than 1, and the first transmittance of the standard is greater than the second transmittance of the standard.

[0133] It should be noted that when the first transmittance of multiple first pixels within the same spectral unit is equal, this first transmittance is the standard first transmittance. When the first transmittance of multiple first pixels is unequal, one of the multiple unequal first transmittances can be selected as the standard first transmittance, or a value different from the first transmittance of the multiple first pixels can be taken as the standard first transmittance. Similarly, when the second transmittance of multiple first pixels within the same spectral unit is equal, this second transmittance is the standard second transmittance. When the second transmittance of multiple first pixels is unequal, one of the multiple unequal second transmittances can be selected as the standard second transmittance, or a value different from the second transmittance of the multiple first pixels can be taken as the standard second transmittance. The main purpose is to calculate the relative light energy based on equal first and second transmittances.

[0134] In other cases, each spectral unit also includes a second pixel, the second pixel having the same third transmittance for light within a preset wavelength range; thus, based on the first transmittance and total light energy of each first pixel, the total secondary light energy when the first transmittance of the first pixel equals the standard first transmittance can be obtained. Specifically, this can be done according to the aforementioned embodiments, based on the relationship curve between transmittance and wavelength, and the principle that the ratio of the area enclosed by the wavelength relationship curve and the coordinate system to the transmittance is equal, to obtain the total secondary light energy; based on the third transmittance of the second pixel and the wavelength of the transmitted light, the sum of secondary light energy of the non-gated light rays can be obtained. In fact, the sum of secondary light energy can be represented by seven-eighths of the area in the relationship curve between the transmittance and wavelength of the second pixel; based on the difference between the total secondary light energy and the sum of secondary light energy of the non-gated light rays, the relative light energy of the gated light rays corresponding to the first pixel can be obtained.

[0135] It should be noted that when the first transmittance of multiple first pixels within the same spectral unit is equal, this first transmittance is the standard first transmittance. When the first transmittance of multiple first pixels is not equal, one of the multiple unequal first transmittances can be selected as the standard first transmittance, or a value different from the first transmittance of the multiple first pixels can be taken as the standard first transmittance. Similarly, when the second transmittance of a first pixel is equal to or not equal to the third transmittance of the second pixel, the light energy can be represented by seven-eighths of the area in the curve of the relationship between the transmittance and wavelength of the second pixel, thus obtaining the relative light energy of the selected light rays when the second transmittance of multiple first pixels is equal to the third transmittance.

[0136] In step 103, the spectral information of each spectral unit is obtained based on the relative light energy of the gating rays corresponding to multiple first pixels within the same spectral unit.

[0137] In this embodiment, after obtaining the relative light energy of each gated ray in step 102, the spectral information within the preset wavelength range corresponding to each spectral unit can be obtained. The narrowband gated filter includes multiple spectral units, so the spectral information of multiple spectral units can be obtained according to the above implementation method.

[0138] In step 104, the spectral information of multiple spectral units is processed to obtain an image.

[0139] In this embodiment, the image can be a black and white image or a color image. Specifically, in some embodiments, the brightness of each first pixel can be obtained based on the spectral information, and a black and white image can be obtained based on the brightness. In other embodiments, each first pixel within the same spectral information is transmitted through multiple non-gated rays, the wavelength range of the multiple non-gated rays being within a preset wavelength range and having equal bandwidth; thereby, the relative light energy of each non-gated ray corresponding to the first pixel can be obtained, and the color displayed by each first pixel can be determined based on the relative light energy of the gated rays and the relative light energy of each non-gated ray, thus obtaining a color image. That is, based on the relative light energy of the gated rays and the relative light energy of each non-gated ray, the mixing ratio of the gated rays and non-gated rays can be obtained, thereby obtaining the color to be displayed by the first pixel.

[0140] In one embodiment, the relative light energy of each non-gated ray corresponding to a first pixel can be obtained based on the relative light energy and weighting coefficient of the gated ray of each first pixel. In another embodiment, the number of non-gated rays corresponding to each first pixel is equal to the number of first pixels in the same spectral unit minus 1, and the wavelength range of each non-gated ray is the same as the wavelength range of the gated ray of any other first pixel. For example, the wavelength range of a non-gated ray of first pixel A is the same as the wavelength range of the gated ray of first pixel B, the wavelength range of a non-gated ray of first pixel A is the same as the wavelength range of the gated ray of first pixel C, and so on. The wavelength range of each non-gated ray can be the same as the range of the gated rays of other first pixels located in the same spectral unit besides first pixel A. For the setting of the wavelength range of the non-gated rays of first pixels other than first pixel A in the same spectral unit, the aforementioned first pixel A can be referred to, and will not be repeated here. Based on this, the relative light energy of each non-gated ray of a first pixel can be obtained from the relative light energy of the gated rays within adjacent first pixels. The wavelength range of the gated rays of adjacent first pixels is the same as the wavelength range of the non-gated rays obtained by the target. That is, taking the first pixel A as an example, the relative light energy of a non-gated ray of the first pixel A can be equal to the relative light energy of the gated ray of another first pixel adjacent to the first pixel A, and the wavelength range of the gated ray of the other adjacent first pixel is the same as the wavelength range of a non-gated ray of the first pixel A.

[0141] To provide a detailed description of the embodiments of this disclosure, such as Figure 11 As shown, taking each spectral unit as an example including a first pixel and a second pixel, the image processing method may include the following steps:

[0142] In step 111, the total light energy of each first pixel and the total light energy of the second pixel are obtained.

[0143] In step 112, the secondary light energy of the non-gated rays in the first pixel is obtained based on the total light energy of the second pixel.

[0144] In step 113, the relative light energy of the gated ray in the first pixel is determined based on the total light energy and the sum of the secondary light energy.

[0145] In this embodiment, the relative light energy of the gated ray in the first pixel can be determined based on the difference between the total light energy and the sum of secondary light energy. It should be noted that in this disclosure, light energy can be characterized by the area enclosed by the relationship between the transmittance and wavelength of each first pixel or each second pixel and the coordinate system; therefore, the total light energy and the sum of secondary light energy mentioned above are relative concepts.

[0146] In this embodiment, the same spectral unit includes a first pixel and a second pixel as an example. In other embodiments, the same spectral unit may also include only the first pixel. In this case, the relative light energy of the gated ray of each first pixel may be equal to the difference between the total light energy of the first pixel and the sum of the secondary light energy of the non-gated rays.

[0147] In step 114, spectral information of multiple spectral units is obtained.

[0148] In step 115, the relative light energy of the ungated ray of each first pixel is obtained.

[0149] In this embodiment, in some cases, the relative light energy of the non-gated ray of each first pixel can be equal to the relative light energy of the gated ray multiplied by a weighting coefficient. In other cases, ... Figure 4 Taking the illustrated embodiment as an example, consider the fifth spectral unit from left to right and top to bottom. This fifth spectral unit may include first pixel A, first pixel B, first pixel C, first pixel D, first pixel F, first pixel G, first pixel H, first pixel J, and second pixel K. Assume the relative light energy of the gate ray A of first pixel A is A5, the relative light energy of the gate ray B of first pixel B is B5, the relative light energy of the gate ray C of first pixel C is C5, the relative light energy of the gate ray D of first pixel D is D5, the energy of the light passing through second pixel K is K5, the relative light energy of the gate ray F of first pixel F is F5, the relative light energy of the gate ray G of first pixel G is G5, and the relative light energy of the gate ray H of first pixel H is H5. In this configuration, the first pixel A can transmit not only the gated ray A, but also other non-gated rays with wavelengths within a preset wavelength range. Assuming the preset wavelength range, excluding the wavelength of gated ray A, is divided into 7 segments, each segment corresponding to a non-gated ray, i.e., the first pixel A can transmit non-gated rays B', C', D', F', G', H', and J'. Furthermore, the wavelength range of non-gated ray B' corresponds to that of the gated ray of the first pixel B. The wavelength range of non-passed ray C' is the same as that of the passed ray C of the first pixel C; the wavelength range of non-passed ray D' is the same as that of the passed ray D of the first pixel D; the wavelength range of non-passed ray F' is the same as that of the passed ray F of the first pixel F; the wavelength range of non-passed ray G' is the same as that of the passed ray G of the first pixel G; the wavelength range of non-passed ray H' is the same as that of the passed ray H of the first pixel H; and the wavelength range of non-passed ray J' is the same as that of the passed ray J of the first pixel J.

[0150] Therefore, for the first pixel A in the fifth spectral unit, the relative light energy of the non-gated ray B' can be equal to the relative light energy of the gated ray B of the first pixel B adjacent to the first pixel A, according to... Figure 4 It can be seen that the relative light energy of the non-gated ray B' is equal to B5, where B5 refers to the relative light energy of the gated ray B of the first pixel B, which is the fifth spectral unit from left to right and from top to bottom. The relative light energy of the non-gated ray C' can be equal to the relative light energy of the gated ray C of the first pixel C adjacent to the first pixel A. Based on the same principle... Figure 4 It can be deduced that the relative light energy of the non-gated ray C' is C5; and so on, we can obtain... Figure 12 The table shown provides the relative light energy of the gated rays and the relative light energy of the non-gated rays for each first pixel within the fifth spectral unit. This embodiment uses the fifth spectral unit as an example; other spectral units can be referred to in the aforementioned embodiments, and will not be described in detail here.

[0151] In step 116, the display color of the first pixel is determined based on the relative light energy of the non-gated ray and the relative light energy of the gated ray of each first pixel to obtain an image.

[0152] Corresponding to the aforementioned embodiments of the image processing method, this disclosure also provides embodiments of the image processing apparatus.

[0153] Figure 13 This is one of the block diagrams illustrating an image processing apparatus according to an exemplary embodiment. (Refer to...) Figure 13 This device is applied to an electronic device, which includes an image sensor. The image sensor includes a light-transmitting layer, a narrow-band gating filter, and a photosensitive substrate. The narrow-band gating filter is disposed between the light-transmitting layer and the photosensitive substrate. The narrow-band gating filter includes multiple spectral units, each spectral unit including multiple first pixels. The wavelength range of the gating light from each first pixel within the same spectral unit differs from the wavelength range of the gating light from other first pixels. The wavelength ranges of the multiple gating light rays corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range. The photosensitive substrate is used to sense the total amount of light energy transmitted through each first pixel. The device includes a first acquisition module 121, a second acquisition module 122, a third acquisition module 123, and a processing module 124, wherein:

[0154] The first acquisition module 121 acquires the total amount of light energy transmitted through each first pixel within each spectral unit;

[0155] The second acquisition module 122 acquires the relative light energy of the gating ray corresponding to each first pixel based on the total light energy of the multiple first pixels.

[0156] The third acquisition module 123 obtains the spectral information of each spectral unit based on the relative light energy of the gating rays corresponding to multiple first pixels within the same spectral unit.

[0157] The processing module 124 processes the spectral information of multiple spectral units to obtain an image.

[0158] like Figure 14 As shown, Figure 14 This is a second block diagram of an image processing apparatus according to an exemplary embodiment, which is described above. Figure 13 Based on the illustrated embodiment, the transmittance of the selected ray corresponding to each first pixel is a first transmittance, and the transmittance of the non-selected ray with a wavelength range within the preset wavelength range is a second transmittance; the second acquisition module 122 includes a first acquisition unit 1221, a second acquisition unit 1222, and a third acquisition unit 1223, wherein:

[0159] The first acquisition unit 1221 acquires the total light energy when the first light transmittance of the first pixel is equal to the standard first transmittance, based on the first light transmittance and total light energy of each first pixel.

[0160] The second acquisition unit 1222 acquires, based on the second transmittance of each first pixel and the sum of light energy of the non-gated rays, the sum of the secondary light energy of the non-gated rays when the second transmittance of the second pixel is equal to the standard second transmittance;

[0161] The third acquisition unit 1223 acquires the relative light energy of the gated ray corresponding to the first pixel based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-gated rays.

[0162] like Figure 15 As shown, Figure 15 This is a block diagram of an image processing apparatus according to an exemplary embodiment, which is described above. Figure 13 Based on the illustrated embodiment, each spectral unit further includes a second pixel, the second pixel having equal third transmittance for light with wavelengths within the preset wavelength range; the second acquisition module 122 includes a fourth acquisition unit 1224, a fifth acquisition unit 1225, and a sixth acquisition unit 1226, wherein:

[0163] The fourth acquisition unit 1224 acquires the total light energy of the first pixel when the first transmittance is equal to the standard first transmittance, based on the first transmittance and total light energy of the gating light of each first pixel.

[0164] The fifth acquisition unit 1225 acquires the secondary light energy of the non-gated light ray based on the third transmittance of the second pixel and the wavelength of the transmitted light ray.

[0165] The sixth acquisition unit 1226 acquires the relative light energy of the gated ray corresponding to the first pixel based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-gated rays.

[0166] like Figure 16 As shown, Figure 16 This is a block diagram four of an image processing apparatus according to an exemplary embodiment, which is described above. Figure 13 Based on the illustrated embodiment, the processing module 124 includes a seventh acquisition unit 1241 and an eighth acquisition unit 1242, wherein:

[0167] The seventh acquisition unit 1241 acquires the brightness of each first pixel based on the spectral information;

[0168] The eighth acquisition unit 1242 acquires a black and white image based on the brightness level.

[0169] It should be noted that the above Figure 15 The structures of the seventh acquisition unit 1241 and the eighth acquisition unit 1242 in the illustrated device embodiment can also be included in the aforementioned... Figure 13 or Figure 14 In the device embodiments, this disclosure is not limited thereto.

[0170] like Figure 17 As shown, Figure 17 This is a block diagram five illustrating an image processing apparatus according to an exemplary embodiment, which is described above. Figure 13 Based on the illustrated embodiment, each first pixel within the same spectral information transmits multiple non-gated rays, the wavelength range of which is within a preset wavelength range and the bandwidth is equal; the processing module 124 includes a ninth acquisition unit 1243 and a tenth acquisition unit 1244, wherein:

[0171] The ninth acquisition unit 1243 acquires the relative light energy of each non-gated ray corresponding to the first pixel;

[0172] The tenth acquisition unit 1244 determines the color displayed by each first pixel based on the relative light energy of the gated ray and the relative light energy of each non-gated ray, thereby obtaining a color image.

[0173] It should be noted that the above Figure 17 The structures of the ninth acquisition unit 1243 and the tenth acquisition unit 1244 in the illustrated device embodiment can also be included in the aforementioned... Figures 14-16 This disclosure is not limited in any of the device embodiments.

[0174] like Figure 18 As shown, Figure 18This is a block diagram of an image processing apparatus according to an exemplary embodiment, which is described above. Figure 17 Based on the illustrated embodiment, the number of non-gated rays corresponding to each first pixel is equal to the number of first pixels in the same spectral unit minus 1, and the wavelength range of each non-gated ray is equal to the wavelength range of the gated rays of any other first pixel; the ninth acquisition unit 1243 includes:

[0175] The first acquisition subunit 12431 acquires the relative light energy of each non-gated ray of the first pixel based on the relative light energy of the gated rays within adjacent first pixels. The wavelength range of the gated rays of the adjacent first pixels is the same as the wavelength range of the non-gated rays acquired by the target.

[0176] like Figure 19 As shown, Figure 19 This is block diagram seven of an image processing apparatus according to an exemplary embodiment, which is described above. Figure 17 Based on the illustrated embodiment, the ninth acquisition unit 1243 includes:

[0177] The second acquisition subunit 12432 acquires the relative light energy of each non-gated ray corresponding to the first pixel based on the relative light energy and weighting coefficient of the gated ray of each first pixel.

[0178] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0179] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0180] Accordingly, this disclosure also provides an image processing apparatus applied to an electronic device. The electronic device includes an image sensor, which includes a light-transmitting layer, a narrow-band gating filter, and a photosensitive substrate. The narrow-band gating filter is disposed between the light-transmitting layer and the photosensitive substrate. The narrow-band gating filter includes multiple spectral units, each spectral unit including multiple first pixels. The wavelength range of the gating light of each first pixel within the same spectral unit differs from the wavelength range of the gating light of other first pixels. The wavelength ranges of the multiple gating light rays corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range. The photosensitive substrate is used to sense the total amount of light energy transmitted through each first pixel. The apparatus includes: a processor; and a memory for storing processor-executable instructions. The processor is configured to: acquire the total amount of light energy transmitted through each first pixel within each spectral unit; acquire the relative light energy of the gating light rays corresponding to each first pixel based on the multiple total light energy of the multiple first pixels; obtain the spectral information of each spectral unit based on the relative light energy of the gating light rays corresponding to the multiple first pixels within the same spectral unit; and process the spectral information of the multiple spectral units to obtain an image.

[0181] Accordingly, this disclosure also provides a terminal, which is applied to an electronic device. The electronic device includes an image sensor, which includes a light-transmitting layer, a narrow-band gating filter, and a photosensitive substrate. The narrow-band gating filter is disposed between the light-transmitting layer and the photosensitive substrate. The narrow-band gating filter includes multiple spectral units, each spectral unit including multiple first pixels. The wavelength range of the gating light of each first pixel within the same spectral unit is different from the wavelength range of the gating light of other first pixels. The wavelength ranges of the multiple gating light rays corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range. The photosensitive substrate is used to sense the total amount of light energy transmitted through each first pixel. The terminal includes a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: obtaining the total light energy transmitted through each first pixel within each spectral unit; obtaining the relative light energy of the gated ray corresponding to each first pixel based on the multiple total light energy values ​​of the multiple first pixels; obtaining the spectral information of each spectral unit based on the relative light energy of the gated ray corresponding to the multiple first pixels within the same spectral unit; and processing the spectral information of the multiple spectral units to obtain an image.

[0182] Figure 20 This is a block diagram illustrating an image processing apparatus 2000 according to an exemplary embodiment. For example, apparatus 2000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0183] Reference Figure 20 The device 2000 may include one or more of the following components: a processing component 2002, a memory 2004, a power supply component 2006, a multimedia component 2008, an audio component 2010, an input / output (I / O) interface 2012, a sensor component 2014, and a communication component 2016.

[0184] Processing component 2002 typically controls the overall operation of device 2000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 2002 may include one or more processors 2020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 2002 may include one or more modules to facilitate interaction between processing component 2002 and other components. For example, processing component 2002 may include a multimedia module to facilitate interaction between multimedia component 2008 and processing component 2002.

[0185] Memory 2004 is configured to store various types of data to support the operation of device 2000. Examples of this data include instructions for any application or method operating on device 2000, contact data, phonebook data, messages, pictures, videos, etc. Memory 2004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0186] Power supply component 2006 provides power to various components of device 2000. Power supply component 2006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 2000.

[0187] The multimedia component 2008 includes a screen that provides an output interface between the device 2000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 2008 includes a front-facing camera and / or a rear-facing camera. When the device 2000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0188] Audio component 2010 is configured to output and / or input audio signals. For example, audio component 2010 includes a microphone (MIC) configured to receive external audio signals when device 2000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 2004 or transmitted via communication component 2016. In some embodiments, audio component 2010 also includes a speaker for outputting audio signals.

[0189] I / O interface 2012 provides an interface between processing component 2002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0190] Sensor assembly 2014 includes one or more sensors for providing status assessments of various aspects of device 2000. For example, sensor assembly 2014 can detect the on / off state of device 2000, the relative positioning of components such as the display and keypad of device 2000, changes in position of device 2000 or a component of device 2000, the presence or absence of user contact with device 2000, orientation or acceleration / deceleration of device 2000, and temperature changes of device 2000. Sensor assembly 2014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 2014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 2014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0191] The communication component 2016 is configured to facilitate wired or wireless communication between the device 2000 and other devices. The device 2000 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR, or combinations thereof. In one exemplary embodiment, the communication component 2016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 2016 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0192] In an exemplary embodiment, the apparatus 2000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0193] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 2004 including instructions, which can be executed by a processor 2020 of the device 2000 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0194] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0195] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image sensor, characterized in that, It includes a light-transmitting layer, a narrow-band gating filter, and a photosensitive substrate, wherein the narrow-band gating filter is disposed between the light-transmitting layer and the photosensitive substrate; The narrowband gating filter includes multiple spectral units, each of which includes a second pixel and multiple first pixels. The wavelength range of the gating light of each first pixel within the same spectral unit is different from the wavelength range of the gating light of other first pixels. The wavelength ranges of the multiple gating lights corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range. The third transmittance of the second pixel for light with wavelengths within the preset wavelength range is equal. The photosensitive substrate is used to sense the total amount of light energy transmitted through each first pixel. The first transmittance and the total amount of light energy of the selected light rays of the first pixel are used to obtain the total amount of secondary light energy when the first transmittance of the first pixel is equal to the standard first transmittance. The third transmittance of the second pixel and the wavelength of the transmitted light rays are used to obtain the sum of the secondary light energy of the non-selected light rays of each first pixel. The difference between the total amount of secondary light energy and the sum of the secondary light energy of the non-selected light rays is used to obtain the relative light energy of the selected light rays of the corresponding first pixel, and to obtain the spectral information of each spectral unit. The multiple spectral information are used to process and obtain an image.

2. The image sensor according to claim 1, characterized in that, The first transmittance of the selected ray corresponding to each first pixel is greater than zero, and the second transmittance of the non-selected ray whose wavelength range is within the preset wavelength range are all equal.

3. The image sensor according to claim 1, characterized in that, Each of the spectral units includes a second pixel, the second pixel having equal third transmittance for light with wavelengths within the preset wavelength range, and the total light energy of each first pixel and the total light energy of the second pixel within the same spectral unit are used to determine the relative light energy of the gated ray of the first pixel.

4. The image sensor according to claim 3, characterized in that, The second transmittance of the unselected light rays of each first pixel whose wavelength is within the preset wavelength range is equal to the third transmittance of the second pixel within the same spectral unit.

5. The image sensor according to claim 3, characterized in that, The first transmittance of the first pixel in the same spectral unit is greater than the third transmittance of the second pixel.

6. The image sensor according to claim 1, characterized in that, The preset wavelength range is 400nm-720nm.

7. The image sensor according to claim 1, characterized in that, Within the same spectral unit, the bandwidth of the wavelength range of the gating ray corresponding to each first pixel is equal.

8. The image sensor according to claim 1, characterized in that, The bandwidth of the wavelength range of the selected light corresponding to each first pixel within the same spectral unit is 30 nm or 40 nm.

9. The image sensor according to claim 1, characterized in that, The wavelength range distribution of the multiple gated rays of the multiple first pixels in each of the spectral units is the same as the wavelength range distribution of the multiple gated rays of the multiple first pixels in any other spectral unit.

10. The image sensor according to claim 1, characterized in that, Each spectral unit includes at least four of the first pixels.

11. A camera module, characterized in that, include: Lens; The image sensor according to any one of claims 1-10, wherein the light-transmitting layer of the image sensor faces the exit end of the lens.

12. The camera module according to claim 11, characterized in that, include: A cutoff filter is disposed between the exit end of the lens and the light-transmitting layer of the image sensor. The cutoff filter is used to cut off light with wavelengths less than the minimum value of the preset wavelength range and wavelengths greater than the maximum value of the preset wavelength range.

13. The camera module according to claim 12, characterized in that, The cut-off filter is used to block infrared and ultraviolet light.

14. An electronic device, characterized in that, include: case; A processor, wherein the processor is disposed within the housing; The camera module as described in any one of claims 11-13 is disposed within the housing, and the lens of the camera module faces the outside of the electronic device. The processor is connected to the image sensor, and the processor is configured to obtain the total light energy when the first transmittance of the first pixel is equal to the standard first transmittance, based on the first transmittance and total light energy of the gated light of each first pixel. Based on the third transmittance of the second pixel and the wavelength of the transmitted light, the sum of the secondary light energy of the non-gated light rays of each first pixel is obtained; based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-gated light rays, the relative light energy of the gated light rays of the corresponding first pixel is obtained, and the spectral information of each spectral unit is obtained. The multiple spectral information are used to process and obtain an image.

15. An image processing method, characterized in that, The invention is applied to an electronic device, the electronic device including an image sensor, the image sensor including a light-transmitting layer, a narrow-band gating filter and a photosensitive substrate, the narrow-band gating filter being disposed between the light-transmitting layer and the photosensitive substrate; The narrowband gating filter includes multiple spectral units, each of which includes a second pixel and multiple first pixels. The wavelength range of the gating light of each first pixel within the same spectral unit is different from the wavelength range of the gating light of other first pixels. The wavelength ranges of the multiple gating lights corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range. The third transmittance of the second pixel for light with wavelengths within the preset wavelength range is equal. The image processing method includes: Obtain the total amount of light energy transmitted through each first pixel within each spectral unit; Based on the total light energy of multiple first pixels, the relative light energy of the gating ray corresponding to each first pixel is obtained; The spectral information of each spectral unit is obtained based on the relative light energy of the gating rays corresponding to multiple first pixels within the same spectral unit. An image is obtained by processing the spectral information of multiple spectral units; The step of obtaining the relative light energy of the gating ray corresponding to each first pixel based on the total light energy of the plurality of first pixels includes: Based on the first transmittance and total light energy of the gating ray of each first pixel, the total light energy of the second pixel when the first transmittance is equal to the standard first transmittance is obtained. Based on the third transmittance of the second pixel and the wavelength of the transmitted light, the secondary light energy of the non-gated light ray of each first pixel is obtained; The relative light energy of the gated ray of the corresponding first pixel is obtained based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-gated rays.

16. The image processing method according to claim 15, characterized in that, The process of processing the spectral information of multiple spectral units to obtain an image includes: Based on the spectral information, the brightness of each first pixel is obtained; A black and white image is obtained based on the brightness level.

17. The image processing method according to claim 15, characterized in that, Within the same spectral information, each first pixel transmits multiple non-gated rays, the wavelength range of the multiple non-gated rays is within a preset wavelength range, and the bandwidth is equal; The process of processing the spectral information of multiple spectral units to obtain an image includes: Obtain the relative light energy of each non-gated ray corresponding to the first pixel; Based on the relative light energy of the gated ray and the relative light energy of the non-gated ray of each first pixel, the color displayed by each first pixel is determined to obtain a color image.

18. The image processing method according to claim 17, characterized in that, The number of non-gated rays corresponding to each first pixel is equal to the number of first pixels in the same spectral unit minus 1, and the wavelength range of each non-gated ray is equal to the wavelength range of the gated rays of any other first pixel; The step of obtaining the relative light energy of each non-gated ray corresponding to the first pixel includes: Based on the relative light energy of the gated rays of adjacent first pixels, the relative light energy of each non-gated ray of the first pixel is obtained, and the wavelength range of the gated rays of adjacent first pixels is the same as the wavelength range of the non-gated rays obtained by the target.

19. The image processing method according to claim 17, characterized in that, The step of obtaining the relative light energy of each non-gated ray corresponding to the first pixel includes: The relative light energy of each non-gated ray corresponding to the first pixel is obtained based on the relative light energy and weighting coefficient of the gated ray of each first pixel.

20. An image processing apparatus, characterized in that, The invention is applied to an electronic device, the electronic device including an image sensor, the image sensor including a light-transmitting layer, a narrow-band gating filter and a photosensitive substrate, the narrow-band gating filter being disposed between the light-transmitting layer and the photosensitive substrate; The narrowband gating filter includes multiple spectral units, each of which includes a second pixel and multiple first pixels. The wavelength range of the gating light of each first pixel within the same spectral unit is different from the wavelength range of the gating light of other first pixels. The wavelength ranges of the multiple gating lights corresponding to the same spectral unit are uniformly and continuously distributed within a preset wavelength range. The third transmittance of the second pixel for light with wavelengths within the preset wavelength range is equal. The image processing device includes: The first acquisition module acquires the total amount of light energy transmitted through each first pixel within each spectral unit. The second acquisition module acquires the relative light energy of the gating ray corresponding to each first pixel based on the total light energy of the multiple first pixels. The third acquisition module obtains the spectral information of each spectral unit based on the relative light energy of the gating rays corresponding to multiple first pixels within the same spectral unit. The processing module processes the spectral information of multiple spectral units to obtain an image; The second acquisition module includes: The fourth acquisition unit acquires the total light energy of the first pixel when the first transmittance is equal to the standard first transmittance, based on the first transmittance and total light energy of the selected light rays of each first pixel. The fifth acquisition unit acquires the secondary light energy of the non-gated light rays of each first pixel based on the third transmittance of the second pixel and the wavelength of the transmitted light rays; The sixth acquisition unit acquires the relative light energy of the gated ray of the corresponding first pixel based on the difference between the total secondary light energy and the sum of the secondary light energy of the non-gated rays.

21. The image processing apparatus according to claim 20, characterized in that, The processing module includes: The seventh acquisition unit acquires the brightness of each first pixel based on the spectral information; The eighth acquisition unit acquires a black and white image based on the brightness level.

22. The image processing apparatus according to claim 20, characterized in that, Within the same spectral information, each first pixel transmits multiple non-gated rays, the wavelength range of the multiple non-gated rays is within a preset wavelength range, and the bandwidth is equal; The processing module includes: The ninth acquisition unit acquires the relative light energy of each non-gated ray corresponding to the first pixel; The tenth acquisition unit determines the color displayed by each first pixel based on the relative light energy of the gated ray and the relative light energy of the non-gated ray of each first pixel, thereby obtaining a color image.

23. The image processing apparatus according to claim 22, characterized in that, The number of non-gated rays corresponding to each first pixel is equal to the number of first pixels in the same spectral unit minus 1, and the wavelength range of each non-gated ray is equal to the wavelength range of the gated rays of any other first pixel; The ninth acquisition unit includes: The first acquisition subunit acquires the relative light energy of each non-gated ray of the first pixel based on the relative light energy of the gated rays of the adjacent first pixels, wherein the wavelength range of the gated rays of the adjacent first pixels is the same as the wavelength range of the non-gated rays acquired by the target.

24. The image processing apparatus according to claim 22, characterized in that, The ninth acquisition unit includes: The second acquisition subunit acquires the relative light energy of each non-gated ray corresponding to the first pixel based on the relative light energy and weighting coefficient of the gated ray of each first pixel.

25. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 15-19.

Images