Image sensor, image data acquisition method, imaging device

Abstract

An image sensor comprises: a pixel array and a color filter array (420) covering the pixel array; wherein: the color filter array (420) comprises a plurality of color filter units, the plurality of color filter units comprising at least one first color filter unit (501) and at least one second color filter unit (502), each first color filter unit (501) comprising a basic color filter and an extended color filter, and each second color filter unit (502) comprising a plurality of basic color filters, wherein the color of the extended color filter is different from the color of the basic color filter, and a light signal passing through the basic color filter is used at least for imaging, and a light signal passing through the extended color filter is used for spectral measurement. The image sensor increases the sampling points, improves the accuracy of color restoration, reduces the occurrence of metamerism, and reduces the calculation difficulty of white balance coefficient and color conversion matrix.

Description

Technical Field

[0001] This application relates to the field of image processing technology. In particular, it relates to an image sensor, an image data acquisition method, and an imaging device. Background Technology

[0002] With the widespread use of digital cameras and mobile phones, image sensors (CCD / CMOS) have received extensive attention and application in recent years. Commonly used image sensors include color filter arrays, which consist of multiple identical color filter units, each of which includes multiple color filters.

[0003] Because commonly used color filter arrays consist of identical color filter units, and each unit contains three or four color filters, the array effectively has only three or four sampling points. However, in real-world scenarios, even a tiny change in light intensity across any spectral band can alter the color perception of the human eye. Therefore, commonly used image sensors can only perform coarse color restoration. Furthermore, restoring the 400–700 nm spectrum using only three or four sampling points leads to numerous metamerism issues, posing challenges to the calculation of white balance coefficients and color restoration matrices. Summary of the Invention

[0004] This application provides an image sensor, an image data acquisition method, and an imaging device to address the problems that commonly used image sensors can only perform rough color reproduction, have many metamerisms, and face increasing challenges in calculating white balance coefficients and color conversion matrices.

[0005] In a first aspect, this application provides an image sensor, comprising: a pixel array and a color filter array covering the pixel array; wherein: the color filter array includes a plurality of color filter units, the plurality of color filter units including at least one first color filter unit and at least one second color filter unit, each first color filter unit including a basic color filter and an extended color filter, each second color filter unit including a plurality of basic color filters; wherein the color of the extended color filter is different from the color of the basic color filter, the light signal passing through the basic color filter is used for at least imaging, and the light signal passing through the extended color filter is used for spectral measurement.

[0006] Since the first color filter unit includes a basic color filter and an extended color filter, and the second color filter unit includes multiple basic color filters, and the colors of the extended color filters are different from those of the basic color filters, the color combinations of the color filters in the first color filter unit are different from those in the second color filter unit. In other words, the color filter array includes two types of color filter units. Compared to commonly used color filter arrays composed of multiple identical color filter units, this increases the number of sampling points, improves the accuracy of color reproduction, reduces the occurrence of metamerism, and simplifies the calculation of white balance coefficients and color conversion matrices. Furthermore, compared to related technologies, this improvement in the color filter array, without adding any additional components, facilitates the miniaturization and low-cost widespread adoption of image sensors, making them more suitable for use in mobile terminal devices such as smartphones. Moreover, since the light signal passing through the basic color filter is used for imaging, and the light signal passing through the extended color filter is used for spectral measurement, imaging and spectral measurement can be performed simultaneously in a single exposure, shortening the spectral measurement time and thus improving the color reproduction and imaging efficiency of the image sensor.

[0007] In one possible implementation, each of the first color filter units contains 3 basic color filters, each of the first color filter units contains 1 extended color filter, and each of the second color filter units contains 4 basic color filters.

[0008] In one possible implementation, the colors of the basic color filters in the second color filter unit include red, green, and blue, and the colors of the basic color filters in the first color filter unit include red, green, and blue.

[0009] In one possible implementation, the different extended color filters have different colors.

[0010] Since different extended color filters have different colors, the number of sampling points is further increased, the accuracy of color reproduction is improved, the occurrence of metamerism is reduced, and the calculation difficulty of white balance coefficient and color reproduction matrix is ​​reduced.

[0011] In one possible implementation, the extended color filter includes a first extended color filter and a second extended color filter, wherein: at least one of the thickness of the first extended color filter, the effective area of ​​the pixel covered by the first extended color filter, and the exposure time of the pixel covered by the first extended color filter is determined according to the upper limit of the dynamic range of the image sensor; at least one of the thickness of the second extended color filter, the effective area of ​​the pixel covered by the second extended color filter, and the exposure time of the pixel covered by the second extended color filter is determined according to the lower limit of the dynamic range of the image sensor.

[0012] As can be seen from the above, the relevant parameters of the first extended color filter, the pixels covered by the first extended color filter, the second extended color filter, and the pixels covered by the second extended color filter can be determined according to the upper and lower limits of the dynamic range of the image sensor. This makes the dynamic range of the image sensor unrestricted, thereby taking into account the dynamic range of both bright and dark objects and improving the imaging effect.

[0013] Secondly, this application provides an image data acquisition method applied to an image sensor, the image sensor including a pixel array and a color filter array covering the pixel array; wherein: the color filter array includes a plurality of color filter units, the plurality of color filter units including at least one first color filter unit and at least one second color filter unit, each first color filter unit including a basic color filter and an extended color filter, each second color filter unit including a plurality of basic color filters, and the color of the extended color filter is different from the color of the basic color filter;

[0014] The method includes: obtaining a first electrical signal and a second electrical signal; determining first spectral measurement data of the current frame based on the first electrical signal, and determining imaging data of the current frame based on the second electrical signal; wherein, the first electrical signal is an electrical signal obtained by photoelectric conversion of a first light signal by a first pixel, the first light signal is a light signal passing through the extended color filter, the first pixel is a pixel covered by the extended color filter, the second electrical signal is an electrical signal obtained by photoelectric conversion of a second light signal by a second pixel, the second light signal is a light signal passing through the basic color filter, and the second pixel is a pixel covered by the basic color filter.

[0015] As shown above, a single exposure can simultaneously acquire the imaging data of the current frame and the first spectral measurement data, shortening the data acquisition time. Furthermore, because this image data acquisition method utilizes an image sensor with increased sampling points, it improves color reproduction accuracy and reduces the computational complexity of white balance coefficient and color conversion accuracy.

[0016] In one possible implementation, determining the first spectral measurement data of the current frame based on the first electrical signal includes: determining motion information of the current frame relative to the historical frames based on the imaging data of the current frame and the imaging data of historical frames; determining second spectral measurement data of the current frame based on the first electrical signal; and correcting the second spectral measurement data based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

[0017] In one possible implementation, determining the first spectral measurement data of the current frame based on the first electrical signal includes: generating the first spectral measurement data based on at least a portion of the second electrical signal and the first electrical signal.

[0018] In one possible implementation, determining the first spectral measurement data of the current frame based on the first electrical signal includes: determining motion information of the current frame relative to the historical frames based on the imaging data of the current frame and the imaging data of historical frames; generating second spectral measurement data of the current frame based on at least a portion of the second electrical signal and the first electrical signal; and correcting the second spectral measurement data based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

[0019] In one possible implementation, obtaining the first electrical signal and the second electrical signal includes: obtaining the first electrical signal based on the position of the extended color filter in the color filter array; and obtaining the second electrical signal based on the position of the basic color filter in the color filter array.

[0020] In one possible implementation, the method further includes: segmenting the current frame based on imaging data of the current frame to obtain at least one object in the current frame; determining the original light source spectrum of the current frame based on the color distribution of the at least one object; determining spectral measurement data of each object in the first spectral measurement data of the current frame, and correcting the original light source spectrum according to the spectral measurement data of each object to obtain a target light source spectrum; determining the white balance coefficient and / or color conversion matrix of the current frame according to the target light source spectrum; and processing the imaging data of the current frame according to the white balance coefficient and / or color conversion matrix of the current frame.

[0021] In one possible implementation, the method further includes: segmenting the current frame based on imaging data of the current frame to obtain at least one object in the current frame; determining spectral measurement data of each object in first spectral measurement data of the current frame, and determining the spectrum of each object based on the spectral measurement data of each object; and diagnosing and / or classifying and / or identifying the corresponding object based on the spectrum of each object.

[0022] In one possible implementation, the extended color filter includes a first extended color filter and a second extended color filter, wherein: at least one of the thickness of the first extended color filter, the effective area of ​​the pixel covered by the first extended color filter, and the exposure time of the pixel covered by the first extended color filter is determined according to the upper limit of the dynamic range of the image sensor; at least one of the thickness of the second extended color filter, the effective area of ​​the pixel covered by the second extended color filter, and the exposure time of the pixel covered by the second extended color filter is determined according to the lower limit of the dynamic range of the image sensor.

[0023] Thirdly, this application provides an imaging device, comprising: an image sensor and an image processor, wherein: the image sensor includes a pixel array and a color filter array covering the pixel array; the color filter array includes a plurality of color filter units, the plurality of color filter units including at least one first color filter unit and at least one second color filter unit, each first color filter unit including a basic color filter and an extended color filter, each second color filter unit including a plurality of basic color filters, the color of the extended color filter being different from the color of the basic color filter; the extended color filter is used to filter incident light signals. The system performs filtering to obtain a first light signal; the basic color filter is used to filter the incident light signal to obtain a second light signal; a first pixel is used to perform photoelectric conversion on the first light signal to obtain a first electrical signal, and the first pixel is a pixel covered by the extended color filter; a second pixel is used to perform photoelectric conversion on the second light signal to obtain a second electrical signal, and the second pixel is a pixel covered by the basic color filter; the image processor is used to obtain the first electrical signal and the second electrical signal, determine the first spectral measurement data of the current frame based on the first electrical signal, and determine the imaging data of the current frame based on the second electrical signal.

[0024] In one possible implementation, the image processor determines the first spectral measurement data by: determining motion information of the current frame relative to the historical frames based on the imaging data of the current frame and the imaging data of the historical frames; determining second spectral measurement data of the current frame based on the first electrical signal; and correcting the second spectral measurement data based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

[0025] In one possible implementation, the image processor specifically determines the first spectral measurement data by generating the first spectral measurement data based on at least a portion of the second electrical signal and the first electrical signal.

[0026] In one possible implementation, the image processor determines the first spectral measurement data by: determining motion information of the current frame relative to the historical frames based on imaging data of the current frame and imaging data of historical frames; generating second spectral measurement data of the current frame based on at least a portion of the second electrical signal and the first electrical signal; and correcting the second spectral measurement data based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

[0027] In one possible implementation, the image processor obtains the first electrical signal and the second electrical signal specifically by: obtaining the first electrical signal based on the position of the extended color filter in the color filter array; and obtaining the second electrical signal based on the position of the basic color filter in the color filter array.

[0028] In one possible implementation, the image processor is further configured to: segment the current frame based on the imaging data of the current frame to obtain at least one object in the current frame; determine the original light source spectrum of the current frame based on the color distribution of the at least one object; determine the spectral measurement data of each object in the first spectral measurement data of the current frame, and correct the original light source spectrum according to the spectral measurement data of each object to obtain the target light source spectrum; determine the white balance coefficient and / or color conversion matrix of the current frame according to the target light source spectrum; and process the imaging data of the current frame according to the white balance coefficient and / or color conversion matrix of the current frame.

[0029] In one possible implementation, the image processor is further configured to: segment the current frame based on imaging data of the current frame to obtain at least one object in the current frame; determine spectral measurement data of each object in first spectral measurement data of the current frame, and determine the spectrum of each object based on the spectral measurement data of each object; and diagnose and / or classify and / or identify the corresponding object based on the spectrum of each object.

[0030] In one possible implementation, the extended color filter includes a first extended color filter and a second extended color filter, wherein: at least one of the thickness of the first extended color filter, the effective area of ​​the pixel covered by the first extended color filter, and the exposure time of the pixel covered by the first extended color filter is determined according to the upper limit of the dynamic range of the image sensor; at least one of the thickness of the second extended color filter, the effective area of ​​the pixel covered by the second extended color filter, and the exposure time of the pixel covered by the second extended color filter is determined according to the lower limit of the dynamic range of the image sensor.

[0031] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the method as described in any one of the second aspects.

[0032] Fifthly, this application provides a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to perform the method as described in any one of the second aspects. Attached Figure Description

[0033] Figure 1 A schematic diagram of a commonly used color filter array. Figure 1 ;

[0034] Figure 2 A schematic diagram of a commonly used color filter array. Figure 2 ;

[0035] Figure 3 This is a schematic diagram of the imaging device provided in the embodiments of this application;

[0036] Figure 4 This is a schematic diagram of the structure of the image sensor provided in the embodiments of this application;

[0037] Figure 5 A schematic diagram of the structure of the color filter array in the image sensor provided in this application embodiment. Figure 1 ;

[0038] Figure 6A schematic diagram of the structure of the color filter array in the image sensor provided in this application embodiment. Figure 2 ;

[0039] Figure 7 A schematic diagram of the structure of the color filter array in the image sensor provided in this application embodiment. Figure 3 ;

[0040] Figure 8 This is a schematic diagram illustrating the workflow of the image processor provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0044] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0045] Commonly used image sensors include pixel arrays and color filter arrays overlaid on the pixel arrays.

[0046] Figure 1 A schematic diagram of a commonly used color filter array. Figure 1 .like Figure 1 As shown, the color filter array includes 16 identical color filter units 101, arranged in a four-row, four-column configuration. Each color filter unit 101 includes four color filters, arranged in a two-row, two-column configuration. Each color filter unit 101 is a 3-channel RGB color filter, meaning it consists of three color filters: red (R), green (G), blue (B), and green (G).

[0047] Figure 2 A schematic diagram of a commonly used color filter array. Figure 2 .like Figure 2 As shown, the color filter array includes one color filter unit. The color filter unit contains 64 color filters. The color filter unit is a 4-channel RGYB array, meaning it consists of color filters of four colors. The 64 color filters in the color filter unit are divided into four filter groups, with each group containing 16 color filters. The four filter groups are arranged in two rows and two columns, with the 16 color filters in each group arranged in four rows and four columns. The color filter in the first filter group is red (R), the color filter in the second filter group is green (G), the color filter in the third filter group is yellow (Y), and the color filter in the fourth filter group is blue (B).

[0048] In summary, because commonly used color filter arrays consist of multiple identical color filter units, and each color filter unit typically has 3 or 4 channels, this means that the color filter array only has three or four sampling points. This limited number of sampling points means that commonly used image sensors can only perform coarse color reproduction. Furthermore, the limited number of sampling points exacerbates metamerism, posing challenges to the calculation of white balance coefficients and color conversion matrices.

[0049] To address the aforementioned technical problems, a related technology provides an image sensor. This image sensor adds a grating device to a commonly used image sensor. Thus, while the image sensor is imaging, the grating device can also disperse the incident light, enabling spectral measurement. The spectral measurement results can then be used to assist in the color reproduction process of the image sensor, as well as the calculation of the white balance coefficient and color conversion matrix.

[0050] Clearly, by using grating devices, the number of sampling points is increased, the color reproduction accuracy of the image sensor is improved, the occurrence of metamerism is reduced, and the calculation difficulty of white balance coefficient and color conversion matrix is ​​reduced.

[0051] However, while grating devices can increase the number of sampling points, they also add an extra layer of complexity to a commonly used structure. Because grating devices are relatively large and expensive, they hinder the miniaturization and cost-effective widespread adoption of image sensors. Furthermore, the reliability of grating devices is affected by vibration, making them unsuitable for use in mobile devices such as smartphones. In addition, the longer time required for a single spectral measurement with a grating device results in lower color reproduction and imaging efficiency for the image sensor.

[0052] To address the aforementioned technical problems, this application provides an imaging device. Figure 3 This is a schematic diagram of the imaging device provided in an embodiment of this application. Figure 3 As shown, the imaging device includes: an image sensor 310, a memory 320, an image processor 330, an I / O interface 340, and a display 350. Wherein:

[0053] Image sensor 310 can also be called an imaging sensor, which is a sensor that senses and transmits image information. Image sensor 310 can be used in electronic imaging devices, including digital cameras, camera modules, medical imaging devices such as thermal imaging devices, radar, sonar, and other night vision devices. For example, image sensor 310 can be an active pixel sensor in complementary metal-oxide-semiconductor (CMOS) or N-type metal-oxide-semiconductor (NMOS) technology, and the embodiments of the present invention are not limited thereto. For example, image sensor 310 can also be a charge-coupled device (CCD), etc.

[0054] Figure 4 This is a schematic diagram of the structure of the image sensor provided in the embodiments of this application, such as... Figure 4 As shown, the image sensor 310 includes a pixel array 410, a color filter array 420 covering the pixel array 410, and a readout circuit 430. Wherein:

[0055] The color filter array 420 includes multiple color filter units, each including at least one first color filter unit and at least one second color filter unit. Each first color filter unit includes a basic color filter and an extended color filter. Each second color filter unit includes multiple basic color filters. The color of the extended color filter is different from the color of the basic color filter.

[0056] An extended color filter is used to filter incident light (allowing only light rays within the passband of the extended color filter to pass through), resulting in a first optical signal. This first optical signal (i.e., the optical signal after passing through the extended color filter) is used for spectral measurements.

[0057] A primary color filter is used to filter the incident light signal (allowing only light rays within the passband of the primary color filter to pass through), resulting in a second light signal. This second light signal (i.e., the light signal after passing through the primary color filter) is used for imaging or for both imaging and spectral measurements.

[0058] The pixel array 410 includes multiple pixels, wherein the pixel covered by the extended color filter is called the first pixel, and the pixel covered by the basic color filter is called the second pixel. The first pixel is used to perform photoelectric conversion on the first optical signal to obtain the first electrical signal, and the second pixel is used to perform photoelectric conversion on the second optical signal to obtain the second electrical signal.

[0059] The readout circuit 430 is used to read out a first electrical signal from a first pixel and a second electrical signal from a second pixel.

[0060] The image processor 330 is used to acquire a first electrical signal and a second electrical signal, determine first spectral measurement data of the current frame based on the first electrical signal, and determine imaging data of the current frame based on the second electrical signal. The image processor 330 can also determine the white balance coefficient and / or color and transformation matrix of the current frame and / or diagnose and / or classify and / or identify objects in the current frame based on the first spectral data and imaging data of the current frame.

[0061] The memory 320 is used to store data corresponding to the signals output by the image sensor 310. The display 350 is used to display images and / or spectra based on the signals output from the image sensor 310 or stored in the memory 320. The I / O interface 340 is used to communicate with other electronic devices, such as mobile phones, smartphones, tablet phones, tablet computers, or personal computers.

[0062] In addition to the imaging devices mentioned above, those skilled in the art should understand that the techniques disclosed herein are also applicable to other electronic devices with imaging capabilities, such as mobile phones, smartphones, phablets, tablet computers, personal assistants, etc.

[0063] As can be seen from the above, since the first color filter unit includes a basic color filter and an extended color filter, and the second color filter unit includes multiple basic color filters, and the colors of the extended color filters are different from those of the basic color filters, the color combinations of the color filters in the first color filter unit are different from those in the second color filter unit. That is, the color filter array includes two types of color filter units. Compared to commonly used color filter arrays composed of multiple identical color filter units, this increases the number of sampling points, improves the accuracy of color reproduction, reduces the occurrence of metamerism, and simplifies the calculation of white balance coefficients and color conversion matrices. Furthermore, compared to related technologies, this improvement in the color filter array, without adding any additional components, facilitates the miniaturization and low-cost widespread adoption of image sensors, and also makes image sensors more suitable for use in mobile terminal devices such as smartphones. In addition, since the light signal passing through the basic color filter is used for imaging, and the light signal passing through the extended color filter is used for spectral measurement, imaging and spectral measurement can be performed simultaneously with a single exposure, shortening the spectral measurement time and thus improving the color reproduction and imaging efficiency of the image sensor.

[0064] Figure 5 A schematic diagram of the structure of the color filter array in the image sensor provided in this application embodiment. Figure 1 .like Figure 5 As shown, the color filter array 420 includes multiple color filter units, each including at least one first color filter unit 501 and at least one second color filter unit 502. Each first color filter unit 501 includes a basic color filter and an extended color filter, and each second color filter unit 502 includes multiple basic color filters.

[0065] The more color filter units in a color filter array, the higher the resolution, and vice versa. The more color types a color filter unit can hold, the higher its channel count, and vice versa. A larger number of color filters allows for a wider variety of colors to be included within the unit. Since, with a fixed number of color filters in the array, more color filter units result in fewer individual color filters within each unit, and vice versa, the number of color filter units and the number of individual color filters can be adjusted based on the required resolution and channel count.

[0066] It should be noted that if the color filter unit is the first color filter unit 501, then the number of color filters in the color filter unit refers to the sum of the number of basic color filters and extended color filters. If the color filter unit is the second color filter unit 502, then the number of color filters in the color filter unit refers to the number of basic color filters.

[0067] The number of color filters in the first color filter unit 501 can be the same as or different from the number of color filters in the second color filter unit 502.

[0068] Since the light signal passing through the extended color filter is used for spectral measurement, and the light signal passing through the basic color filter is used for imaging, and all color filters in the second color filter unit 502 are basic color filters, while the color filters in the first color filter unit 501 include both basic and extended color filters, the number of the first color filter unit 501, the number of the second color filter unit 502, and the number of extended color filters in the first color filter unit 501 can be determined according to the imaging effect and spectral measurement requirements. The first color filter unit 501 contains at least one extended color filter.

[0069] For example, each first color filter unit 501 has 3 basic color filters, each first color filter unit 501 has 1 extended color filter, and each second color filter unit 502 has 4 basic color filters.

[0070] The colors and number of basic color filters in the second color filter unit 502 are determined according to imaging requirements. The number of basic color filters in the second color filter unit 502 can be, for example, 3, 4, or more, and this application does not impose any special limitations on this. The colors of the basic color filters in the second color filter unit 502 can include, for example, three, four, or more of the following: red, green, blue, yellow, magenta, and near-infrared, and this application does not impose any special limitations on this.

[0071] The number of color types of the basic color filters in the second color filter unit 502 is equal to or less than the total number of basic color filters in the second color filter unit 502. Specifically, if the number of color types of the basic color filters in the second color filter unit 502 is equal to the total number of basic color filters in the second color filter unit 502, then the colors of the different basic color filters in the second color filter unit 502 are different; if the number of color types of the basic color filters in the second color filter unit 502 is less than the total number of basic color filters in the second color filter unit 502, then some of the basic color filters in the second color filter unit 502 have the same color.

[0072] The number of channels in the second color filter unit 502 is equal to the number of color types of the basic color filters therein. For example, if the basic color filters in the second color filter unit 502 include red, green, and blue, then the number of channels in the second color filter unit 502 is 3; if the basic color filters in the second color filter unit 502 include red, yellow, green, and blue, then the number of channels in the second color filter unit 502 is 4.

[0073] The basic color filters in the first color filter unit 501 are a subset of the multiple basic color filters in the second color filter unit 502. The colors of the basic color filters in different first color filter units 501 can be completely different, not completely different, or completely the same. For example, the second color filter unit 501 may have four basic color filters, and the colors of these four basic color filters are red, green, blue, and yellow, respectively. If the first color filter unit 501 has three basic color filters, then the colors of the three basic color filters in each first color filter unit 501 can be red, green, and blue, respectively. Obviously, the basic color filters in different first color filter units 501 have the same color. If the number of basic color filters in the first color filter unit 501 is 2, then the colors of the two basic color filters in some of the first color filter units 501 can be red and green respectively, the colors of the two basic color filters in some of the first color filter units 501 can be blue and yellow respectively, and the colors of the two basic color filters in some of the first color filter units 501 can be green and yellow respectively. Obviously, the colors of the basic color filters in different first color filter units 501 can be exactly the same, completely different, or partially the same.

[0074] Since the color of the base color filter is different from that of the extended color filter, the color of the extended color filter can be any color other than that of the base color filter. Specifically, it can be determined according to the spectral measurement requirements.

[0075] The colors of different extended color filters in the same first color filter unit 501 can be exactly the same, completely different, or not exactly the same.

[0076] For example, the first color filter unit 501 includes an extended color filter, and the extended color filters in different first color filter units 501 have different colors. As another example, the first color filter unit 501 includes multiple extended color filters, and the colors of different extended color filters within the same first color filter unit 501 are completely different, and the colors of extended color filters in different first color filter units 501 are also completely different. Clearly, in both of the above examples, for the color filter array, different extended color filters have different colors.

[0077] In a color filter array, the more colors the extended color filters can represent, the more sampling points are required. Therefore, in the two examples above, because different extended color filters have different colors, the number of sampling points is further increased, improving the accuracy of color reproduction, reducing the occurrence of metamerism, and simplifying the calculation of the white balance coefficient and color reproduction matrix.

[0078] The number of channels in the first color filter unit 501 is equal to the number of color types in the basic color filter and extended color filter within the first color filter unit 501. The number of channels in the first color filter unit 501 can be the same as or different from the number of channels in the second color filter unit 502.

[0079] At least one first color filter unit 501 and at least one second color filter unit 502 are arranged in N rows and M columns. N and M can be the same or different, and both N and M are integers greater than or equal to 1. The first color filter units 501 can be regularly distributed in N rows and M columns, or they can be randomly distributed in N rows and M columns; this application does not impose any special limitations on this.

[0080] The extended color filter can be located at any position in its respective first color filter unit 501, or it can be located at a fixed position in its respective first color filter unit 501.

[0081] It should be noted that the position information of the extended color filters also needs to be recorded for subsequent data processing. Specifically, if the extended color filters are regularly distributed within the color filter array, the position of the first extended color filter and its distribution pattern should be recorded. This allows the position of each subsequent extended color filter to be determined based on the position and distribution pattern of the first extended color filter. If the extended color filters are randomly distributed within the color filter array, the position of each individual extended color filter needs to be recorded.

[0082] Figure 6 A schematic diagram of the structure of the color filter array in the image sensor provided in this application embodiment. Figure 2 .like Figure 6As shown, the color filter array includes 16 color filter units. These 16 color filter units include 4 first color filter units 601 and 12 second color filter units 602. Each first color filter unit 601 includes 3 basic color filters and 1 extended color filter. Each second color filter unit 602 includes 4 basic color filters.

[0083] The four primary color filters in each second color filter unit 602 have colors of RGGB (red, green, green, blue). The three primary color filters in each first color filter unit 601 have colors of RGB (red, green, blue). The extended color filters in different first color filter units 601 have different colors, and the extended color filters in the four first color filter units 601 have the following colors: cyan, orange, purple, and gray.

[0084] As can be seen from the above, since the second color filter unit 602 includes three basic color filters, the number of channels in the second color filter unit 602 is 3. The first color filter unit 601 includes three basic color filters and one extended color filter, therefore, the number of channels in the first color filter unit 601 is 4.

[0085] Figure 7 A schematic diagram of the structure of the color filter array in the image sensor provided in this application embodiment. Figure 3 . Figure 7 The color filter array in Figure 6 The difference in the color filter arrays is that the four basic color filters in each second color filter unit 702 have the colors RGYB (red, green, yellow, blue). The three basic color filters in each first color filter unit 701 have the colors RGB (red, green, blue).

[0086] As can be seen from the above, since the second color filter unit 702 includes four basic color filters, the number of channels in the second color filter unit 702 is four. The first color filter unit 701 includes three basic color filters and one extended color filter, therefore, the number of channels in the first color filter unit 701 is four.

[0087] It should be noted that the above Figure 6 and Figure 7 The description of the color filter array is merely exemplary and is not intended to limit this application.

[0088] Because commonly used image sensors have limited dynamic range, they cannot simultaneously capture the dynamic range of both bright and dark objects, resulting in poor image quality. A typical scenario where limited dynamic range leads to poor image quality is as follows:

[0089] In a shooting scene where both a light source and an object are present, the light source is brighter than the object. Therefore, if you focus on the light source, the details of the light source can be clearly displayed in the image, but the details of the object will be lost. If you focus on the object, the details of the object can be clearly displayed in the image, but the light source will be overexposed.

[0090] To address this technical problem, this application improves the extended color filter in the color filter array. The principle of the improvement is as follows:

[0091] Since overexposure can be avoided by reducing pixel photon accumulation, and signal-to-noise ratio can be improved by increasing pixel photon accumulation, object detail loss can be avoided. Furthermore, changes in at least one of the effective area of ​​a pixel, exposure time, and the thickness of the color filter covering the pixel will alter the pixel's photon accumulation. Therefore, this application can use one portion of the extended color filters in the color filter array as the first extended color filter and another portion as the second extended color filter. Then, the dynamic range of the image sensor is determined based on the dynamic range of the brightest object and the dynamic range of the darkest object in the scene where the image sensor is applied. For example, the union of the dynamic ranges of the brightest and darkest objects is calculated, the maximum value of the union is determined as the upper limit of the dynamic range of the image sensor, and the minimum value of the union is determined as the lower limit of the dynamic range of the image sensor. Finally, at least one of the following is determined based on the upper limit of the dynamic range of the image sensor: the thickness of the first extended color filter, the effective area of ​​the pixel covered by the first extended color filter, and the exposure time of the pixel covered by the first extended color filter. At least one of the following is determined based on the lower limit of the dynamic range of the image sensor: the thickness of the second extended color filter, the exposure time of the pixel covered by the second extended color filter, and the effective area of ​​the pixel covered by the second extended color filter.

[0092] The thickness of the extended color filter is negatively correlated with the photon accumulation effect of the pixel. The exposure time of the pixel is positively correlated with the photon accumulation effect of the pixel, and the effective area of ​​the pixel is positively correlated with the photon accumulation effect of the pixel.

[0093] Methods to reduce the effective area of ​​a pixel include reducing the fill factor and partial occlusion.

[0094] As can be seen from the above, the dynamic range of the image sensor can be determined according to the application scenario, and the relevant parameters of the first extended color filter, the pixels covered by the first extended color filter, the second extended color filter, and the pixels covered by the second extended color filter can be determined according to the upper and lower limits of the dynamic range of the image sensor. This makes the dynamic range of the image sensor unrestricted, so as to simultaneously take into account the dynamic range of objects with high brightness and objects with low brightness, thereby improving the imaging effect.

[0095] Next, combined Figure 8 The workflow of the image processor is described. For example... Figure 8 As shown, the workflow of an image processor includes the following steps:

[0096] 801. Obtain the first electrical signal and the second electrical signal.

[0097] For example, the image sensor obtains a first telecommunication signal and a second electrical signal from the signal read out by the readout circuit.

[0098] The process of obtaining the first and second electrical signals can be, for example, as follows: The first electrical signal is obtained based on the position of the extended color filter in the color filter array, and the second electrical signal is obtained based on the position of the basic color filter in the color filter array. Specifically, the position of the first pixel in the pixel array is determined based on the position of the extended color filter in the color filter array, and the first electrical signal is obtained from the electrical signals read from the pixel array by the readout circuit based on the position of the first pixel in the pixel array. Similarly, the position of the second pixel in the pixel array is determined based on the position of the basic color filter in the color filter array, and the second electrical signal is obtained from the electrical signals read from the pixel array by the readout circuit based on the position of the second pixel in the pixel array.

[0099] 802. Determine the first spectral measurement data of the current frame based on the first electrical signal, and determine the imaging data of the current frame based on the second electrical signal.

[0100] For example, the process of determining the imaging data of the current frame based on the second electrical signal can be as follows:

[0101] The second electrical signal read from the second pixel is converted into a pixel value to obtain the pixel value of the second pixel. The pixel value of the second pixel and the position of the second pixel in the pixel array are determined as the imaging data of the current frame.

[0102] In another possible implementation of this application, after obtaining the imaging data, the imaging data can be preprocessed. This preprocessing includes, but is not limited to, one or more of black level correction, lens correction, and dead pixel compensation. The preprocessed imaging data is then determined as the imaging data obtained in step 802.

[0103] As can be seen from the above, the imaging data of the current frame can be either the imaging data before preprocessing or the imaging data after preprocessing.

[0104] For example, the methods for determining the first spectral measurement data of the current frame based on the first electrical signal may include the following four:

[0105] The first method involves converting the first electrical signal read from the first pixel into a pixel value. The pixel value of the first pixel and its position in the pixel array are then used as the first spectral measurement data for the current frame.

[0106] The second method involves first determining the motion information of the current frame relative to the historical frames based on the imaging data of the current frame and the imaging data of the historical frames.

[0107] A historical frame is any frame preceding the current frame. For example, a historical frame is a video frame that is preceding and adjacent to the current frame. The imaging data of a historical frame can be raw imaging data, which includes the pixel value of the second pixel and the position of the second pixel in the pixel array obtained when the historical frame was captured. Alternatively, the imaging data of a historical frame can be data obtained by performing the aforementioned preprocessing and / or white balance coefficient processing and / or color conversion matrix processing on the raw imaging data of the historical frame.

[0108] The process of determining the motion information of the current frame relative to historical frames may include: calculating the position difference between the second pixels indicating the same feature in the current frame and historical frames based on the pixel value of the second pixel in the imaging data of the current frame and the position of the second pixel in the pixel array, as well as the position and pixel value of the second pixel in the imaging data of historical frames; averaging the position differences of all second pixels indicating the same feature in the current frame and historical frames; and determining this average value as the motion information of the current frame relative to historical frames. Features include, but are not limited to, semantics and characteristics. It should be noted that the above method for determining the motion information of the current frame relative to historical frames is merely exemplary and is not intended to limit this application.

[0109] Then, the second spectral measurement data for the current frame is determined based on the first electrical signal. Here, the second spectral measurement data can be understood as the first spectral measurement data in the first method.

[0110] Finally, the second spectral measurement data of the current frame is corrected based on the motion information of the current frame relative to the historical frames and the spectral measurement data of the historical frames to obtain the first spectral measurement data of the current frame.

[0111] The spectral measurement data of historical frames can be the raw spectral measurement data of the historical frames, which includes the pixel value of the first pixel and the position of the first pixel in the pixel array obtained when the historical frame was captured. The spectral measurement data of historical frames can also be data obtained after processing the raw spectral measurement data. This processing includes, but is not limited to, correcting the raw spectral measurement data of the historical frames using spectral measurement data from video frames preceding the historical frames, performing black level correction, lens correction, and dead pixel compensation on the raw spectral measurement data of the historical frames.

[0112] The process of correcting the second spectral measurement data of the current frame can be, for example, as follows: Based on motion information, align the first pixels indicating the same feature in the historical frame and the current frame; calculate the average pixel value of the aligned first pixel in the historical frame and the current frame; determine this average value as the corrected pixel value of the first pixel in the current frame; and replace the aligned first pixel value in the second spectral measurement data of the current frame with the corrected pixel value to obtain the first spectral measurement data of the current frame. It should be noted that the above-described process for correcting the second spectral measurement data of the current frame is merely exemplary and is not intended to limit this application.

[0113] The third method involves generating first spectral measurement data based on at least a portion of the second electrical signal and the first electrical signal.

[0114] At least a portion of the second electrical signal refers to the second electrical signal read from at least a portion of the second pixels. At least a portion of the second pixels refers to all or a portion of the second pixels. For example, a portion of the second pixels can be second pixels located around the first pixel.

[0115] The first electrical signal read from the first pixel is converted from an electrical signal to a pixel value to obtain the pixel value of the first pixel. The second electrical signal read from at least a portion of the second pixels is converted from an electrical signal to a pixel value to obtain the pixel values ​​of at least a portion of the second pixels. The pixel values ​​of the first pixel and the position of the first pixel in the pixel array, as well as the pixel values ​​of at least a portion of the second pixels and the positions of at least a portion of the second pixels in the pixel array, are determined as first spectral measurement data.

[0116] The fourth method involves first determining the motion information of the current frame relative to historical frames based on the imaging data of the current frame and the imaging data of historical frames. Since this process has already been explained above, it will not be repeated here.

[0117] Then, second spectral measurement data for the current frame is generated based on at least a portion of the second electrical signal and the first electrical signal. This second spectral data can be understood as the first spectral measurement data in the third approach.

[0118] Finally, the second spectral measurement data of the current frame is corrected based on motion information and spectral measurement data from historical frames to obtain the first spectral measurement data of the current frame. The spectral measurement data of historical frames can be the original spectral measurement data of the historical frames, which includes the pixel value of the first pixel and its position in the pixel array, as well as the pixel values ​​and positions of at least a portion of the second pixels, obtained when the historical frame was captured. The spectral measurement data of historical frames can also be data obtained after processing the original spectral measurement data. This processing includes, but is not limited to, correcting the original spectral data of the historical frames using spectral measurement data from video frames preceding the historical frames, performing black level correction, lens correction, and dead pixel compensation on the original spectral measurement data of the historical frames.

[0119] Since the principle of correcting the second spectral data of the current frame based on motion information and spectral measurement data of historical frames has been explained above, it will not be repeated here.

[0120] In another possible implementation of this application, after obtaining the first spectral measurement data, the first spectral measurement data can be preprocessed. This preprocessing includes, but is not limited to, one or more of black level correction, lens correction, and dead pixel compensation. The preprocessed first spectral measurement data is then determined as the first spectral measurement data obtained in step 802.

[0121] As can be seen from the above, the first spectral measurement data of the current frame can be either the first spectral measurement data before preprocessing or the first spectral measurement data after preprocessing.

[0122] It should be noted that, in the case where the first spectral measurement data of the current frame is the preprocessed first spectral measurement data, for methods two and four, the preprocessing process can be advanced to be performed after the second spectral measurement data is obtained. That is, after the second spectral measurement data is obtained, the above-mentioned preprocessing is performed on the second spectral measurement data, and subsequent processing is performed based on the preprocessed second spectral measurement data.

[0123] In methods two and four above, the second spectral measurement data of the current frame is corrected by using the spectral measurement data of historical frames to obtain the first spectral measurement data of the current frame. That is, the second spectral measurement data of the current frame is corrected by multi-frame noise reduction, which improves the accuracy of the first spectral measurement data, improves the accuracy of color reproduction, and improves the calculation accuracy of white balance coefficient and color conversion accuracy.

[0124] In summary, this method allows for the simultaneous acquisition of imaging data and first spectral measurement data for the current frame in a single exposure, thus reducing data acquisition time. Furthermore, because this image data acquisition method utilizes an image sensor with increased sampling points, it improves color reproduction accuracy and reduces the computational complexity of white balance coefficient and color conversion accuracy.

[0125] After obtaining the first spectral measurement data of the current frame, the application scenarios of the first spectral measurement data include the following two:

[0126] The first method involves calculating a white balance coefficient and / or a color conversion matrix based on the first spectral measurement data and imaging data of the current frame. This white balance coefficient and / or color conversion matrix are then used to process the imaging data of the current frame, improving color reproduction accuracy and thus enhancing the display effect. The specific execution steps are as follows:

[0127] 803. Based on the imaging data of the current frame, perform feature segmentation on the current frame to obtain at least one object in the current frame.

[0128] Elements include, but are not limited to, semantics and features. Element segmentation can be achieved, for example, through a model formed by a neural network. Objects include, but are not limited to, people, animals, flowers, grass, clouds, blue sky, tables, houses, faces, white blocks, and gray blocks.

[0129] 804. Determine the original light source spectrum of the current frame based on the color distribution of at least one object. Specifically, based on the imaging data of the current frame and the position of each object, analyze the color distribution of at least one object, evaluate the original light source color temperature of the current frame based on the color distribution, and determine the original light source spectrum based on the original light source color temperature.

[0130] 805, determine the spectral measurement data of each object in the first spectral measurement data of the current frame, and correct the original light source spectrum based on the spectral measurement data of each object to obtain the target light source spectrum.

[0131] The process of determining the spectral measurement data of an object can be as follows: determine the position of the boundary pixels of the object, and based on the position of the pixels in the first spectral measurement data of the current frame, determine the pixel value and position of the pixels located within the boundary pixels of the object in the first spectral measurement data as the spectral measurement data of the object.

[0132] The process of obtaining the target light source spectrum can be as follows: Based on the spectral measurement data of each object, analyze the color distribution of at least one object, determine the corrected color temperature based on the color distribution, and determine the corrected spectrum based on the corrected color temperature. The original light source spectrum is then corrected based on the corrected spectrum to obtain the target light source spectrum.

[0133] A light source spectrum is used to indicate the distribution of photometric or radiometric measurements at various wavelengths within a light source.

[0134] The above-described process for determining the target light source spectrum is merely exemplary and is not intended to limit this application. For example, after determining the original light source color temperature, a corrected color temperature can be determined. Then, the original light source color temperature can be corrected based on the corrected color temperature to obtain the target light source color temperature. Finally, the target light source spectrum can be determined based on the target light source color temperature. As another example, after obtaining at least one object in the current frame, imaging data and spectral measurement data for each object can be determined. Then, the imaging data for the corresponding object can be corrected based on the spectral measurement data for each object to obtain target imaging data for each object. Finally, based on the target imaging data for each object, color analysis of at least one object can be performed, the target light source color temperature can be evaluated based on the color distribution, and the target light source spectrum can be determined based on the target light source color temperature.

[0135] 806, Determine the white balance coefficient and / or color conversion matrix of the current frame based on the spectrum of the target light source.

[0136] 807. Process the imaging data of the current frame according to the color conversion matrix of the current frame and / or the color conversion matrix of the current frame.

[0137] For example, if the current frame is an RGB image, that is, the second color filter unit in the color filter array of the image sensor is 3-channel RGB, then the process of processing the imaging data by the white balance coefficient is as follows:

[0138]

[0139] Where R is the pixel value of the pixel covered by the red basic color filter, G is the pixel value of the pixel covered by the green basic color filter, B is the pixel value of the pixel covered by the blue basic color filter, Rg is the pixel value of the pixel covered by the red basic color filter, Gg is the pixel value of the pixel covered by the green basic color filter, Bg is the pixel value of the pixel covered by the blue basic color filter, and r, g, and b are white balance coefficients.

[0140] The principle of processing imaging data using a color conversion matrix is ​​as follows:

[0141]

[0142] Wherein, sR is the pixel value of the pixel covered by the red basic color filter after processing, sG is the pixel value of the pixel covered by the green basic color filter after processing, sB is the pixel value of the pixel covered by the blue basic color filter after processing, Rg is the pixel value of the pixel covered by the red basic color filter, Gg is the pixel value of the pixel covered by the green basic color filter, Bg is the pixel value of the pixel covered by the blue basic color filter, and a, b, c, d, e, f, g, h, i are the parameters in the color transformation matrix.

[0143] It should be noted that in other embodiments of this application, the local light source spectrum and local color conversion matrix in the current frame can also be obtained through the above principle, so as to process the local area in the current frame through the local light source spectrum and local color conversion matrix.

[0144] As can be seen from the above, the increase in sampling points allows the imaging data and the first spectral measurement data to reflect more information from the sampling points. In this way, the original light source spectrum of the current frame is determined by the imaging data, and the target light source spectrum obtained after correcting the original light source spectrum based on the spectral measurement data of each object reflects more information from the sampling points. This improves the accuracy of the target light source spectrum, reduces the occurrence of metamerism, and thus reduces the calculation difficulty of the white balance coefficient and color conversion matrix, thereby improving the accuracy of color reproduction.

[0145] The second method involves diagnosing and / or classifying and / or identifying objects in the current frame based on the first spectral test data of the current frame. The specific execution steps are as follows:

[0146] 808. Based on the imaging data of the current frame, perform feature segmentation on the current frame to obtain at least one object in the current frame. Since this step has already been explained above, it will not be repeated here.

[0147] 809. Determine the spectral measurement data of each object in the first spectral measurement data of the current frame, and determine the spectrum of each object based on the spectral measurement data of each object.

[0148] The process of determining the spectral measurement data for each object has already been explained above and will not be repeated here. The process of determining the object's spectrum can be, for example, as follows: analyze the object's color distribution based on the spectral measurement data, evaluate the object's color temperature based on the color distribution, and determine the object's spectrum based on the object's color temperature.

[0149] The spectrum of an object refers to the distribution of photometric or radiometric measurements of various wavelengths reflected by the object.

[0150] 810. Diagnose and / or classify and / or identify the corresponding objects based on the spectrum of each object.

[0151] For the diagnosis of an object, the spectrum of the object for different diagnostic results can be obtained in advance, the object's spectrum can be matched with the spectrum of the object for different diagnostic results, and the diagnostic result of the object can be determined based on the matching result.

[0152] For object classification, the spectra of the object for different classification results can be obtained in advance. The object's spectrum is then matched with the spectrum of the object for different classifications, and the classification result of the object is determined based on the matching result.

[0153] For object recognition, the spectra of each object can be obtained in advance, the spectra of the object can be matched with the spectra of each object, and the object recognition result can be determined based on the matching result.

[0154] It should be noted that the above-mentioned spectral matching can refer to the matching of spectra on a certain mathematical characteristic (such as mean, variance, statistical distribution, etc.).

[0155] As shown above, by determining the spectrum of each object in the current frame, objects can be diagnosed and / or classified and / or identified based on their spectra. This method is simple and easy to implement. Furthermore, it allows users to quickly and accurately diagnose and / or classify and / or identify objects using this method, improving the user experience.

[0156] This application also provides an image data acquisition method, applied to the aforementioned image sensor. The image data acquisition method includes the following steps:

[0157] First, a first electrical signal and a second electrical signal are obtained. Then, first spectral measurement data of the current frame are determined based on the first electrical signal, and imaging data of the current frame are determined based on the second electrical signal.

[0158] Wherein, the first electrical signal is the electrical signal obtained by the first pixel performing photoelectric conversion on the first light signal, the first light signal is the light signal passing through the extended color filter, the first pixel is the pixel covered by the extended color filter, the second electrical signal is the electrical signal obtained by the second pixel performing photoelectric conversion on the second light signal, the second light signal is the light signal passing through the basic color filter, and the second pixel is the pixel covered by the basic color filter.

[0159] In one possible implementation, determining the first spectral measurement data of the current frame based on the first electrical signal includes: determining motion information of the current frame relative to the historical frames based on the imaging data of the current frame and the imaging data of historical frames; determining second spectral measurement data of the current frame based on the first electrical signal; and correcting the second spectral measurement data based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

[0160] In one possible implementation, determining the first spectral measurement data of the current frame based on the first electrical signal includes: generating the first spectral measurement data based on at least a portion of the second electrical signal and the first electrical signal.

[0161] In one possible implementation, determining the first spectral measurement data of the current frame based on the first electrical signal includes: determining motion information of the current frame relative to the historical frames based on the imaging data of the current frame and the imaging data of historical frames; generating second spectral measurement data of the current frame based on at least a portion of the second electrical signal and the first electrical signal; and correcting the second spectral measurement data based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

[0162] In one possible implementation, obtaining the first electrical signal and the second electrical signal includes: obtaining the first electrical signal based on the position of the extended color filter in the color filter array; and obtaining the second electrical signal based on the position of the basic color filter in the color filter array.

[0163] In one possible implementation, the method further includes: segmenting the current frame based on imaging data of the current frame to obtain at least one object in the current frame; determining the original light source spectrum of the current frame based on the color distribution of the at least one object; determining spectral measurement data of each object in the first spectral measurement data of the current frame, and correcting the original light source spectrum according to the spectral measurement data of each object to obtain a target light source spectrum; determining the white balance coefficient and / or color conversion matrix of the current frame according to the target light source spectrum; and processing the imaging data of the current frame according to the white balance coefficient and / or color conversion matrix of the current frame.

[0164] In one possible implementation, the method further includes: segmenting the current frame based on imaging data of the current frame to obtain at least one object in the current frame; determining spectral measurement data of each object in first spectral measurement data of the current frame, and determining the spectrum of each object based on the spectral measurement data of each object; and diagnosing and / or classifying and / or identifying the corresponding object based on the spectrum of each object.

[0165] The implementation principle and technical effects of the above-mentioned image data acquisition method of this application have been explained above, and will not be repeated here.

[0166] This application also provides a computer-readable storage medium including a computer program, which, when executed on a computer, causes the computer to perform the technical solutions of any of the above-described method embodiments.

[0167] This application also provides a computer program, which, when executed by a computer or processor, is used to perform the technical solutions of any of the above-described method embodiments.

[0168] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some ports, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0173] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image sensor, characterized in that, include: A pixel array and a color filter array covering the pixel array; wherein: The color filter array includes multiple color filter units, each of which includes at least one first color filter unit and at least one second color filter unit. Each first color filter unit includes a basic color filter and an extended color filter, and each second color filter unit includes multiple basic color filters. The color of the extended color filter is different from that of the basic color filter. The light signal passing through the basic color filter is used for imaging at least, and the light signal passing through the extended color filter is used for spectral measurement. The extended color filter includes a first extended color filter and a second extended color filter, wherein: At least one of the thickness of the first extended color filter, the effective area of ​​the pixel covered by the first extended color filter, and the exposure time of the pixel covered by the first extended color filter is determined according to the upper limit of the dynamic range of the image sensor. The thickness of the second extended color filter, the effective area of ​​the pixel covered by the second extended color filter, and the exposure time of the pixel covered by the second extended color filter are determined according to the lower limit of the dynamic range of the image sensor.

2. The image sensor according to claim 1, characterized in that, Each of the first color filter units contains 3 basic color filters, each of the first color filter units contains 1 extended color filter, and each of the second color filter units contains 4 basic color filters.

3. The image sensor according to claim 1 or 2, characterized in that, The colors of the basic color filters in the second color filter unit include red, green, and blue, and the colors of the basic color filters in the first color filter unit include red, green, and blue.

4. The image sensor according to claim 1 or 2, characterized in that, Different of the extended color filters have different colors.

5. A method for acquiring image data, characterized in that, An image sensor is used, the image sensor including a pixel array and a color filter array covering the pixel array; wherein: the color filter array includes a plurality of color filter units, the plurality of color filter units including at least one first color filter unit and at least one second color filter unit, each first color filter unit including a basic color filter and an extended color filter, each second color filter unit including a plurality of basic color filters, and the color of the extended color filter is different from the color of the basic color filter; The extended color filter includes a first extended color filter and a second extended color filter, wherein: At least one of the thickness of the first extended color filter, the effective area of ​​the pixel covered by the first extended color filter, and the exposure time of the pixel covered by the first extended color filter is determined according to the upper limit of the dynamic range of the image sensor. At least one of the thickness of the second extended color filter, the effective area of ​​the pixel covered by the second extended color filter, and the exposure time of the pixel covered by the second extended color filter is determined according to the lower limit of the dynamic range of the image sensor. The method includes: Obtain the first electrical signal and the second electrical signal; The first spectral measurement data of the current frame is determined based on the first electrical signal, and the imaging data of the current frame is determined based on the second electrical signal; Wherein, the first electrical signal is the electrical signal obtained by the first pixel performing photoelectric conversion on the first light signal, the first light signal is the light signal passing through the extended color filter, the first pixel is the pixel covered by the extended color filter, the second electrical signal is the electrical signal obtained by the second pixel performing photoelectric conversion on the second light signal, the second light signal is the light signal passing through the basic color filter, and the second pixel is the pixel covered by the basic color filter.

6. The method according to claim 5, characterized in that, The step of determining the first spectral measurement data of the current frame based on the first electrical signal includes: The motion information of the current frame relative to the historical frames is determined based on the imaging data of the current frame and the imaging data of the historical frames. The second spectral measurement data of the current frame is determined based on the first electrical signal; The second spectral measurement data is corrected based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

7. The method according to claim 5, characterized in that, The step of determining the first spectral measurement data of the current frame based on the first electrical signal includes: The first spectral measurement data is generated based on at least a portion of the second electrical signal and the first electrical signal.

8. The method according to claim 5, characterized in that, The step of determining the first spectral measurement data of the current frame based on the first electrical signal includes: The motion information of the current frame relative to the historical frames is determined based on the imaging data of the current frame and the imaging data of the historical frames. The second spectral measurement data of the current frame is generated based on at least a portion of the second electrical signal and the first electrical signal; The second spectral measurement data is corrected based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

9. The method according to any one of claims 5 to 8, characterized in that, The acquisition of the first electrical signal and the second electrical signal includes: The first electrical signal is obtained based on the position of the extended color filter in the color filter array; The second electrical signal is obtained based on the position of the basic color filter in the color filter array.

10. The method according to any one of claims 5 to 8, characterized in that, The method further includes: Based on the imaging data of the current frame, feature segmentation is performed on the current frame to obtain at least one object in the current frame; Based on the color distribution of the at least one object, determine the original light source spectrum of the current frame; In the first spectral measurement data of the current frame, spectral measurement data of each object is determined, and the original light source spectrum is corrected according to the spectral measurement data of each object to obtain the target light source spectrum; The white balance coefficient and / or color conversion matrix of the current frame are determined based on the spectrum of the target light source; The imaging data of the current frame is processed based on the white balance coefficient and / or color conversion matrix of the current frame.

11. The method according to any one of claims 5 to 8, characterized in that, The method further includes: Based on the imaging data of the current frame, feature segmentation is performed on the current frame to obtain at least one object in the current frame; Spectral measurement data for each object is determined from the first spectral measurement data of the current frame, and the spectrum of each object is determined based on the spectral measurement data of each object; Diagnose and / or classify and / or identify the corresponding objects based on the spectrum of each object.

12. An imaging device, characterized in that, include: Image sensors and image processors, wherein: The image sensor includes a pixel array and a color filter array covering the pixel array; the color filter array includes a plurality of color filter units, the plurality of color filter units including at least one first color filter unit and at least one second color filter unit, each first color filter unit including a basic color filter and an extended color filter, each second color filter unit including a plurality of basic color filters, and the color of the extended color filter is different from the color of the basic color filter; The extended color filter includes a first extended color filter and a second extended color filter, wherein: At least one of the thickness of the first extended color filter, the effective area of ​​the pixel covered by the first extended color filter, and the exposure time of the pixel covered by the first extended color filter is determined according to the upper limit of the dynamic range of the image sensor. At least one of the thickness of the second extended color filter, the effective area of ​​the pixel covered by the second extended color filter, and the exposure time of the pixel covered by the second extended color filter is determined according to the lower limit of the dynamic range of the image sensor. The extended color filter is used to filter the incident light signal to obtain a first light signal; The basic color filter is used to filter the incident light signal to obtain the second light signal; The pixel array comprises multiple pixels, wherein: The first pixel is used to perform photoelectric conversion on the first optical signal to obtain a first electrical signal, and the first pixel is a pixel covered by the extended color filter; The second pixel is used to perform photoelectric conversion on the second optical signal to obtain a second electrical signal. The second pixel is a pixel covered by the basic color filter. The image processor is configured to obtain the first electrical signal and the second electrical signal, determine the first spectral measurement data of the current frame based on the first electrical signal, and determine the imaging data of the current frame based on the second electrical signal.

13. The device according to claim 12, characterized in that, The image processor determines the first spectral measurement data in the following manner: The motion information of the current frame relative to the historical frames is determined based on the imaging data of the current frame and the imaging data of the historical frames. The second spectral measurement data of the current frame is determined based on the first electrical signal; The second spectral measurement data is corrected based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

14. The device according to claim 12, characterized in that, The image processor determines the first spectral measurement data in the following manner: The first spectral measurement data is generated based on at least a portion of the second electrical signal and the first electrical signal.

15. The device according to claim 12, characterized in that, The image processor determines the first spectral measurement data in the following manner: The motion information of the current frame relative to the historical frames is determined based on the imaging data of the current frame and the imaging data of the historical frames. The second spectral measurement data of the current frame is generated based on at least a portion of the second electrical signal and the first electrical signal; The second spectral measurement data is corrected based on the motion information and the spectral measurement data of the historical frames to obtain the first spectral measurement data.

16. The device according to any one of claims 12 to 15, characterized in that, The image processor obtains the first and second electrical signals in the following manner: The first electrical signal is obtained based on the position of the extended color filter in the color filter array; The second electrical signal is obtained based on the position of the basic color filter in the color filter array.

17. The device according to any one of claims 12 to 15, characterized in that, The image processor is also used for: Based on the imaging data of the current frame, feature segmentation is performed on the current frame to obtain at least one object in the current frame; Based on the color distribution of the at least one object, determine the original light source spectrum of the current frame; In the first spectral measurement data of the current frame, spectral measurement data of each object is determined, and the original light source spectrum is corrected according to the spectral measurement data of each object to obtain the target light source spectrum; The white balance coefficient and / or color conversion matrix of the current frame are determined based on the spectrum of the target light source; The imaging data of the current frame is processed based on the white balance coefficient and / or color conversion matrix of the current frame.

18. The device according to any one of claims 12 to 15, characterized in that, The image processor is also used for: Based on the imaging data of the current frame, feature segmentation is performed on the current frame to obtain at least one object in the current frame; Spectral measurement data for each object is determined from the first spectral measurement data of the current frame, and the spectrum of each object is determined based on the spectral measurement data of each object; Diagnose and / or classify and / or identify the corresponding objects based on the spectrum of each object.

19. A computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the method as described in any one of claims 5 to 11.

20. A computer program product comprising instructions that, when run on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 5 to 11.

Images