An image sensor, a signal processing method, and an apparatus

Abstract

The embodiments of the present application disclose an image sensor, a signal processing method and a device, which can be used in the field of image processing. The image sensor includes at least one hyperspectral or multi-spectral filter array, which is used to obtain optical signals of a target object in at least two different spectral bands through m filters; the optical signals obtained by the m filters are converted into electrical signals; the electrical signals corresponding to the m filters are merged to obtain a merged electrical signal; the merged electrical signal is used to generate an image of the target object. By merging the electrical signals of at least two different spectral bands obtained by the hyperspectral or multi-spectral filter array, the N spectral bands of the multi-spectral or hyperspectral can be mapped to H spectral channels, where the number H is less than N, so that the compatibility of the hyperspectral or multi-spectral filter array with the low-spectral imaging technology can be achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image sensor, a signal processing method and a device. Background Art

[0002] Red green blue (RGB) imaging technology refers to collecting the red spectrum information, green spectrum information and blue spectrum information of the target object in the visible light spectrum band, and then generating an image of the target object.

[0003] Hyperspectral imaging technology or multispectral imaging technology has more spectral bands and can collect light signals of target objects in more spectral bands. It has the characteristics of high spectral resolution and wide spectral range, and the collected images have richer information.

[0004] However, the three primary colors of the human eye are red, green, and blue. In order to present an image that is convenient for the human eye to see, after collecting the spectral information of the target object through a hyperspectral or multispectral filter array, it is necessary to finally output an RGB image. Therefore, a solution that uses a hyperspectral or multispectral filter array to collect the light signal of the target object and is compatible with RGB imaging technology is urgently needed. Summary of the invention

[0005] The embodiments of the present application provide an image sensor, a signal processing method and a device. By merging the electrical signals of at least two different spectral bands acquired by a hyperspectral or multi-spectral filter array, N spectral bands of the multi-spectral or hyperspectral can be mapped to H spectral channels, where the number H is less than N, thereby achieving compatibility between the hyperspectral or multi-spectral filter array and the low-spectral imaging technology.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides an image sensor that can be used in the field of image processing. The image sensor includes at least one filter array, which is a high-spectrum or multi-spectrum filter array. The filter array includes m filters and a photoelectric conversion module, and the photoelectric conversion module includes a photoelectric conversion unit and a merging unit. Among them, the photoelectric conversion unit can be expressed as a photoelectric conversion circuit, and the merging unit can be expressed as a merging circuit. The m filters are used to obtain optical signals of the target object in at least two different spectral bands, and the at least two different spectral bands can be spectral bands within the same spectral channel range. For example, the at least two different spectral bands include two different spectral bands R1 and R2, and the two spectral bands R1 and R2 are both within the spectral band range of the spectral channel R in the RGB channel. The photoelectric conversion unit is used to convert the optical signals obtained by the m filters into electrical signals, and the merging unit is used to merge the electrical signals corresponding to the m filters to obtain a merged electrical signal, and the merging method can be to take the maximum value, take the average value or superposition. The photoelectric conversion module is used to output the merged electrical signal, and the merged electrical signal is used to generate an image of the target object.

[0008] In this implementation, the filter array included in the hyperspectral imaging technology or the multispectral imaging technology can obtain the optical signals of the target object in multiple spectral bands. When it is necessary to output a non-hyperspectral image or a multispectral image, the electrical signals corresponding to at least two different spectral bands can be merged to obtain a merged electrical signal, that is, the N spectral bands of the multispectral or hyperspectral can be mapped to H spectral channels, and the number of H is less than N, so that the compatibility of the hyperspectral or multispectral filter array with the low-spectral imaging technology can be achieved; when the output is an RGB image, the value of H is 3, and the 3 spectral channels correspond to the three types of electrical signals required to generate the RGB image. Through the above-mentioned method, it is possible to achieve compatibility with the RGB imaging technology when the spectral information of the target object is collected using a hyperspectral or multispectral filter array.

[0009] In a possible implementation of the first aspect, the merging unit is specifically used to superimpose the electrical signals corresponding to the m filters to obtain a merged electrical signal. In this implementation, the operation of merging the electrical signals corresponding to the m filters is achieved by superimposing the electrical signals corresponding to each filter, and the solution is simple and easy to implement.

[0010] In a possible implementation of the first aspect, the filter array includes M filters, the M filters are divided into K groups, and any group of filters in the K groups is used to obtain the optical signal of the target object in at least one spectral band. It should be noted that the situation that each group in the K groups is only used to obtain the optical signal of the target object in one spectral band is not achievable, that is, at least one group in the K groups is used to obtain the optical signal of the target object in at least two spectral bands. One group in the K groups includes m filters, and K is an integer greater than 1. The K groups of filters have a mapping relationship with H spectral channels, and the H spectral channels are divided according to any one of RGB channels, RGGB channels, RYB channels, RYYB channels, RWWB channels, or RBGIR channels. As an example, for example, the optical signals of N spectral bands are divided according to RGB channels, and the N spectral bands may include 6 different spectral bands of R1, R2, G1, G2, G3, G4, B1 and B2. Among them, the light signals of the target object in the two spectral bands R1 and R2 are mapped to the R channel in the RGB channel, the light signals of the target object in the four spectral bands G1, G2, G3 and G4 are mapped to the G channel in the RGB channel, and the light signals of the target object in the two spectral bands B1 and B2 are mapped to the B channel in the RGB channel.

[0011] In this implementation, multiple division methods of H spectral channels corresponding to K groups of filters are listed, which improves the implementation flexibility of this solution and expands the application scenarios of this solution.

[0012] In a possible implementation of the first aspect, the m filters have a mapping relationship with the first spectral channel among the H spectral channels, the first spectral channel is one of the H spectral channels, and the first spectral channel corresponds to the first spectral band. At least two different spectral bands corresponding to the m filters are located within the first spectral band range. As an example, for example, the at least two different spectral bands corresponding to the m filters are two different spectral bands G1 and G2, respectively, the first spectral channel is the G channel in the RGGB channel, and the two different spectral bands G1 and G2 are both located within the spectral range of the G channel. Alternatively, the spectral bands greater than or equal to a preset proportion of the at least two different spectral bands corresponding to the m filters are located within the first spectral band range. The value of the preset proportion may be ninety-eight percent, ninety-five percent, or ninety percent. As an example, at least two different spectral bands corresponding to m filters are B1 and B2, the first spectral channel is the B channel in the RBBB channel, and more than 90% of the spectral bands B1 and B2 are within the spectral range of the B channel.

[0013] In this implementation, at least two different spectral bands corresponding to the m filters are located within the first spectral band range, or, among the at least two different spectral bands corresponding to the m filters, spectral bands greater than or equal to a preset proportion are located within the first spectral band range, that is, the spectral bands of the hyperspectral or multi-spectral are highly similar to the H spectral bands after mapping, which is beneficial to improving the quality of the finally acquired image.

[0014] In a possible implementation of the first aspect, a group of filters consisting of m filters has a rectangular shape; further, any group of filters in the K groups has a rectangular shape. In this implementation, since the shape of each group of filters in the RGB channel, RGGB channel, RYB channel, RYYB channel, RWWB channel or RBGIR channel technology is rectangular, the shape of any group of filters in the K groups is also set to be rectangular, which helps to reduce the difficulty of the mapping process.

[0015] In a possible implementation of the first aspect, the m filters are adjacent filters; further, any group of filters in the K groups includes at least one filter, and the at least one filter includes adjacent filters. In this implementation, the optical signal acquired by a group is mapped to an optical signal of a channel. Since in imaging technologies such as RGB, RYB, RWWB or RBGIR, at least one filter corresponding to the same channel includes adjacent filters, that is, the dimension of filter position allocation increases the similarity with imaging technologies such as RGB, RYB, RWWB or RBGIR, which is conducive to reducing the difficulty in subsequent signal processing.

[0016] In a possible implementation of the first aspect, the filters included in different groups in the K groups of filters are completely different, that is, there are no overlapping filters included in the different groups in the K groups of filters. In this implementation, since there are no overlapping filters included in different groups in the K groups of filters, the information obtained by each filter will not be reused, there is no problem of information redundancy, and it is conducive to saving computer resources.

[0017] In a possible implementation of the first aspect, the same filters exist between different groups in the K groups of filters and are not completely the same, that is, there is at least one overlapping filter between different groups. In this implementation, since the same filters exist between different groups in the K groups of filters, the spatial structure is more compact, which is conducive to reducing the difficulty of subsequent signal processing.

[0018] In a possible implementation of the first aspect, any group of filters in the K groups is used to obtain a light signal of a target object in at least one spectral band, and the number of spectral bands corresponding to different groups is the same, or the number of spectral bands corresponding to different groups is different. In this implementation, the flexibility of the division process of the M filters is enhanced.

[0019] In a possible implementation of the first aspect, M filters are used to obtain optical signals of the target object in 12 different sub-spectral bands, K groups of filters are used to map to three spectral channels, and each group of filters in the K groups is used to obtain spectral information of the target object in four different sub-spectral bands, that is, each of the three spectral channels corresponds to four different sub-spectral bands, and the four different sub-spectral bands are all within the spectral band range of the corresponding spectral channel. As an example, for example, the 12 different sub-spectral bands are R1, R2, R3, R4, G1, G2, G3, G4, B1, B2, B3 and B4, and the three spectral channels are R channel, G channel and B channel. The four different spectral bands of R1, R2, R3 and R4 are all located in the spectral band of R channel, the four different spectral bands of G1, G2, G3 and G4 are all located in the spectral band of G channel, and the four different spectral bands of B1, B2, B3 and B4 are all located in the spectral band of B channel. Alternatively, M filters are used to obtain the optical signals of the target object in 6 different sub-spectral bands, K groups of filters are used to map to 3 spectral channels, and each group of filters in the K groups is used to obtain the spectral information of the target object in 2 different sub-spectral bands, that is, each spectral channel in the 3 spectral channels corresponds to 2 different sub-spectral bands, and the 2 different sub-spectral bands are both located in the spectral band range of the corresponding spectral channel. Alternatively, M filters are used to obtain optical signals of the target object in 6 different sub-spectral bands, K groups of filters are used to map to 3 spectral channels, the K group includes a second group and a third group, the filters in the second group are used to obtain spectral information of the target object in 4 spectral bands, the 4 different sub-spectral bands are all located within the spectral band range of the corresponding spectral channels, the filters in the third group are used to obtain spectral information of the target object in 2 spectral bands, the 2 different sub-spectral bands are all located within the spectral band range of the corresponding spectral channels, and the second group and the third group are two different groups in the K group.

[0020] In this implementation, the allocation of spectral bands corresponding to different groups is provided, which improves the implementation flexibility of this solution. In addition, since the number of spectral bands corresponding to the same group is greater, the spectral information obtained in the group is richer, but more computer resources are required in the subsequent processing process. Among the H channels corresponding to the K groups, the importance of different channels may be different. In this way, a larger number of sub-spectral bands are allocated to the groups corresponding to the channels with high importance, and a smaller number of sub-spectral bands are allocated to the groups corresponding to the channels with low importance, which is conducive to increasing the richness of the spectral information obtained while saving computer resources.

[0021] In a possible implementation of the first aspect, the filters included in different groups of K groups of filters are completely different, the value of M is 16, the value of K is 4, and each group of filters in the K group includes 4 filters. The value of H is 3 or 4, that is, the K group of filters is mapped to 3 spectral channels or 4 spectral channels. In this implementation, a variety of specific implementations of the filter array are provided, which improves the implementation flexibility of the scheme and expands the application scenarios of the scheme. In addition, the filters included in different groups in the above-mentioned several implementations are completely different, which avoids the processing of easy information and avoids the waste of computer resources.

[0022] In a possible implementation of the first aspect, the filters included in different groups of K groups of filters have the same filters and are not completely the same, the value of M is 9, the value of N is 6, the value of K is 4, each group of filters in the K group is used to obtain the spectral information of the target object in 4 spectral bands, each group of filters in the K group includes 4 filters, and one group of filters in the K group is in a matrix type. The value of H is 3 or 4, that is, the K groups of filters are mapped to 3 spectral channels or 4 spectral channels. In this implementation, a specific implementation method of the filter array is provided, which further improves the implementation flexibility of the present solution; in addition, the filters included in different groups have the same filters and are not completely the same, which reduces the difficulty of the subsequent signal processing process.

[0023] In a possible implementation of the first aspect, M filters are used to obtain optical signals of the target object in N sub-spectral bands. The filters included in different groups in the K groups of filters have the same filters and are not completely the same, the value of M is 9, the value of N is 9, the value of K is 4, each group of filters in the K group includes 4 filters, each group of filters in the K group is used to obtain the spectral information of the target object in 3 spectral bands, and one group of filters in the K group is L-shaped, inverted L-shaped or reverse L-shaped. Alternatively, the filters included in different groups in the K groups of filters are completely different, the value of M is 8, the value of N is 6, the value of K is 4, each group of filters in the K group includes 2 filters, and each group of filters in the K group is used to obtain the spectral information of the target object in 2 spectral bands. The value of H is 3 or 4, that is, the K groups of filters are mapped to 3 spectral channels or 4 spectral channels. In this implementation, two specific implementations of the filter array are provided, which further improves the implementation flexibility of this solution.

[0024] In a possible implementation of the first aspect, when the W channel (i.e., the white channel) is not included in the H channels, the filter array may further include a fully transparent light-transmitting sheet, which is located in the central area of ​​the filter array. The value of M is 8, the value of K is 4, each filter group in the K group includes 2 filters, and the fully transparent light-transmitting sheet is located in the center of the 8 filters. Furthermore, the value of N is 6.

[0025] In this implementation, the contour information of the target object can be obtained through the fully transparent light-transmitting film, providing assistance for mosaic processing in the subsequent signal processing process; and the light signal collected by the fully transparent light-transmitting film can be used as an auxiliary sensor of the light source spectrum, providing assistance for white balance processing in the subsequent signal processing process.

[0026] In a second aspect, an embodiment of the present application provides an electronic device that can be used in the field of image processing. The electronic device includes at least one filter array and a processor. The filter array is a hyperspectral or multi-spectral filter array. The filter array includes m filters and a photoelectric conversion unit. The m filters are used to obtain optical signals of the target object in at least two different spectral bands. The photoelectric conversion unit is used to convert the optical signals obtained by the m filters into electrical signals. The processor is used to merge the electrical signals corresponding to the m filters to obtain a merged electrical signal, and generate an image of the target object based on the merged electrical signal. The filter array of the second aspect is different from the filter array of the first aspect only in that there is no merging unit in the filter array of the second aspect.

[0027] In this implementation, no merging unit is added to the filter array. Instead, the electrical signals corresponding to all filters in a group are merged during data processing by the processor, providing another implementation scheme compatible with RGB imaging technology when a hyperspectral or multi-spectral filter array is used to collect spectral information of a target object, thereby improving the implementation flexibility of this scheme. In addition, in this implementation scheme, no hardware changes are required, which is conducive to reducing costs.

[0028] In a possible implementation of the second aspect, the electronic device further includes an analog-to-digital conversion unit, which is used to convert the analog electrical signal generated by the photoelectric conversion unit into a digital electrical signal; the processor is specifically used to superimpose the digital electrical signals corresponding to all the filters in the first group to obtain a merged digital electrical signal, that is, to add the digital electrical signals corresponding to all the filters in the first group to obtain a merged digital electrical signal. The processor performs the above operation on each group of filters in the K groups to obtain K merged digital electrical signals. Among them, a digital electrical signal can be specifically expressed as a numerical value, and the value range of the digital electrical signal is related to the bit width of the analog-to-digital conversion.

[0029] In a possible implementation manner of the second aspect, the electronic device is any one of the following devices: an image sensor, a camera, a mobile terminal, a smart car, and a monitoring device.

[0030] In a possible implementation of the second aspect, K groups of filters have a mapping relationship with H spectral bands, H is a positive integer less than or equal to K, and the H spectral channels are any one of RGB channels, RGGB channels, RYB channels, RYYB channels, RWWB channels or RBGIR channels; wherein R is a red channel, G is a green channel, B is a blue channel, Y is a yellow channel, W is a white channel, and IR is a near-infrared channel.

[0031] In a possible implementation of the second aspect, K groups of filters have a mapping relationship with H spectral channels, H spectral bands corresponding to the H spectral channels constitute a first spectral band, and the N spectral bands are located within the first spectral band range, or, spectral bands greater than or equal to a preset proportion of the N spectral bands are located within the first spectral band range.

[0032] In a possible implementation manner of the second aspect, the shape of any group of filters in the K groups is rectangular.

[0033] In a possible implementation manner of the second aspect, any group of filters in the K groups includes adjacent filters.

[0034] In a possible implementation manner of the second aspect, filters included in different groups in the K groups of filters are completely different.

[0035] For the concepts of the nouns in the second aspect of the embodiment of the present application and some possible implementation methods of the second aspect, specific implementation steps and beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the first aspect, and will not be repeated here one by one.

[0036] In a third aspect, an embodiment of the present application provides an image sensor that can be used in the field of image processing. The image sensor includes at least one filter array, and the filter array is the filter array of the first aspect. The filter array is a hyperspectral or multi-spectral filter array, and the filter array includes m filters and a photoelectric conversion module; the m filters are used to obtain optical signals of the target object in at least two different spectral bands; the photoelectric conversion module is used to convert the optical signals obtained by the m filters into electrical signals when the image sensor is in an image capture mode, merge the electrical signals corresponding to the m filters to obtain the merged electrical signals, and output the merged electrical signals, and the merged electrical signals are used to generate an image of the target object; the photoelectric conversion module is used to convert the optical signals obtained by the m filters into electrical signals when the image sensor is in a non-image capture mode, and output the electrical signals corresponding to the m filters to generate a hyperspectral image of the target object or a multi-spectral image of the target object. Further, if the image sensor is not configured with a processor, the electronic device where the image sensor is located can determine whether the electronic device starts the image sensor based on the image capture mode. When the electronic device is determined to be in the image shooting mode, that is, the image sensor works in the image shooting mode, the merging unit is controlled to be turned on, that is, the merging unit is controlled to work, so that the merging unit can merge and process the electrical signals corresponding to the m filters; when the electronic device is determined to be in the non-image shooting mode, that is, the image sensor works in the non-image shooting mode, the merging unit is controlled to be turned off, that is, the merging unit is controlled not to work, so that the photoelectric conversion module in the image sensor directly outputs the electrical signals corresponding to the m filters. If a processor is configured in the image sensor, when the electronic device is determined to be in the image shooting mode, the electronic device sends a first instruction to the image sensor, and the image sensor responds to the first instruction and controls the merging unit to be turned on, that is, the merging unit is controlled to work, so that the merging unit can merge and process the electrical signals corresponding to the m filters. When the electronic device is determined to be in the non-image shooting mode, the electronic device sends a second instruction to the image sensor, and the image sensor responds to the second instruction and controls the merging unit to be turned off, that is, the merging unit is controlled not to work, so that the photoelectric conversion module in the image sensor directly outputs the electrical signals corresponding to the m filters.

[0037] In this implementation, not only can it be compatible with RGB imaging technology when using a hyperspectral or multi-spectral filter array to collect spectral information of the target object, but it can also directly output the electrical signal generated by the photoelectric conversion unit after the photoelectric conversion unit converts the optical signals of N spectral bands into electrical signals. That is, the output electrical signal can carry hyperspectral or multi-spectral information, which expands the application scenarios of this solution.

[0038] In a possible implementation manner of the third aspect, the non-image shooting mode is any one of the following modes: an object recognition mode, a light source estimation mode, and a depth detection mode.

[0039] The image sensor of the third aspect can also execute the steps executed by the image sensor in the first aspect. For the concepts of the nouns in the third aspect of the embodiment of the present application and some possible implementation methods of the third aspect, specific implementation steps and beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the first aspect, and will not be repeated here one by one.

[0040] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device includes an image sensor and a processor, the image sensor includes at least one filter array, and the filter array is the filter array of the second aspect. The filter array is a hyperspectral or multi-spectral filter array, and the filter array includes m filters and a photoelectric conversion unit; the m filters are used to obtain optical signals of the target object in at least two different spectral bands; the photoelectric conversion unit is used to convert the optical signals obtained by the m filters into electrical signals; the processor is used to merge the electrical signals corresponding to the m filters when the electronic device is in an image capture mode to obtain a merged electrical signal, and generate an image of the target object according to the merged electrical signal; the processor is used to no longer perform a merging operation on the electrical signals corresponding to the m filters when the electronic device is in a non-image capture mode, and generate a hyperspectral image or a multi-spectral image of the target object according to the electrical signals corresponding to the m filters. Further, the electronic device where the image sensor is located can determine whether the electronic device starts the image sensor based on the image capture mode. When the electronic device is determined to be in an image capturing mode, that is, the image sensor operates in the image capturing mode, the merging unit is controlled to be turned on, that is, the merging unit is controlled to operate, so that the merging unit can merge the electrical signals corresponding to the m filters; when the electronic device is determined to be in a non-image capturing mode, that is, the image sensor operates in the non-image capturing mode, the merging unit is controlled to be turned off, that is, the merging unit is controlled not to operate, so that the photoelectric conversion module in the image sensor directly outputs the electrical signals corresponding to the m filters.

[0041] In a possible implementation manner of the fourth aspect, the non-image shooting mode is any one of the following modes: an object recognition mode, a light source estimation mode, and a depth detection mode.

[0042] The image sensor of the fourth aspect can also execute the steps executed by the image sensor in the second aspect. For the concepts of the nouns in the fourth aspect of the embodiment of the present application and some possible implementation methods of the fourth aspect, specific implementation steps and beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the second aspect, and will not be repeated here one by one.

[0043] In a fifth aspect, an embodiment of the present application provides an electronic device that can be used in the field of image processing. The electronic device includes an image sensor and a processor, the image sensor includes at least one filter array, the image sensor includes at least one filter array, the image sensor is the image sensor of the first aspect above, and the processor is used to generate an image of the target object according to the electrical signal output by each filter array.

[0044] In a sixth aspect, an embodiment of the present application provides a signal processing method, which is applied to an electronic device, wherein the electronic device includes at least one filter array, the filter array is a hyperspectral or multi-spectral filter array, the filter array includes m filters, and the method includes: acquiring optical signals of a target object in at least two different spectral bands through m filters; converting the optical signals acquired by the m filters into electrical signals; merging the electrical signals corresponding to the m filters to obtain a merged electrical signal, and generating an image of the target object based on the merged electrical signal.

[0045] In a possible implementation of the sixth aspect, the electrical signals corresponding to the m filters are combined by a processor, including: superimposing the electrical signals corresponding to the m filters to obtain a combined electrical signal.

[0046] For the concepts of nouns in the possible implementation methods of the sixth aspect of the embodiment of the present application, the specific implementation steps and the beneficial effects brought about by each possible implementation method, you can refer to the descriptions of the various possible implementation methods in the first and second aspects, and will not be repeated here one by one.

[0047] In a seventh aspect, an embodiment of the present application provides a signal processing method, which is applied to an electronic device, wherein the electronic device includes at least one filter array, which is a hyperspectral or multi-spectral filter array, and the filter array includes m filters. The method includes: acquiring optical signals of a target object in at least two different spectral bands through m filters; converting the optical signals acquired by the m filters into electrical signals; when the electronic device is in an image capture mode, merging the electrical signals corresponding to the m filters to obtain a merged electrical signal, and generating an image of the target object based on the merged electrical signal; when the electronic device is in a non-image capture mode, generating a hyperspectral image or a multi-spectral image of the target object based on the electrical signals corresponding to the m filters.

[0048] In a possible implementation manner of the seventh aspect, the non-image shooting mode is any one of the following modes: an object recognition mode, a light source estimation mode, and a depth detection mode.

[0049] For the concepts of nouns, specific implementation steps and beneficial effects brought about by each possible implementation method in the possible implementation methods of the seventh aspect of the embodiment of the present application, please refer to the descriptions of the various possible implementation methods in the first aspect and the second aspect, and will not be repeated here one by one.

[0050] In an eighth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the signal processing method described in the sixth or seventh aspect above.

[0051] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the signal processing method described in the sixth or seventh aspect above.

[0052] In a tenth aspect, an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the signal processing method described in the sixth aspect or the seventh aspect above.

[0053] In the eleventh aspect, an embodiment of the present application provides a chip system, which includes a processor for supporting the functions involved in the above aspects, for example, sending or processing the data and / or information involved in the above methods. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for a server or communication device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1A schematic diagram of the structure of a filter array provided in an embodiment of the present application;

[0055] Figure 2 Three structural schematic diagrams of filters in the filter array provided in the embodiments of the present application;

[0056] Figure 3 Two schematic diagrams of the corresponding relationship between the filter and the photoelectric conversion unit in the filter array provided in the embodiment of the present application;

[0057] Figure 4 A schematic diagram of a correspondence between N spectral bands and H spectral bands of spectral channels in a filter array provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of another correspondence between N spectral bands and spectral bands of H spectral channels in a filter array provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of another correspondence between N spectral bands and spectral bands of H spectral channels in the filter array provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of another correspondence between N spectral bands and spectral bands of H spectral channels in the filter array provided in an embodiment of the present application;

[0061] Figure 8 Three schematic diagrams of the positional relationship between multiple filters in a group of filters in a filter array provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of the positional relationship between filters included in different groups in the filter array provided in an embodiment of the present application;

[0063] Fig.10 Another schematic diagram of the positional relationship between filters included in different groups in the filter array provided in an embodiment of the present application;

[0064] Fig.11 A schematic diagram of the positional relationship between filters included in different groups in the filter array provided in an embodiment of the present application;

[0065] Fig.12 Another schematic diagram of the positional relationship between filters included in different groups in the filter array provided in an embodiment of the present application;

[0066] Fig.13 A schematic diagram of a grouping of a filter array provided in an embodiment of the present application;

[0067] Fig.14 Another schematic diagram of grouping of the filter array provided in an embodiment of the present application;

[0068] Fig.15 A schematic diagram of another grouping of the filter array provided in the embodiment of the present application;

[0069] Fig.16 A schematic diagram of another grouping of the filter array provided in the embodiment of the present application;

[0070] Fig.17 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0071] Fig.18 Another structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0072] Fig.19 A schematic diagram of the structure of an image sensor provided in an embodiment of the present application;

[0073] Fig. 20 A schematic diagram of another structure of an electronic device provided in an embodiment of the present application;

[0074] Fig.21 A flowchart of a signal processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The embodiments of the present application provide an image sensor, a signal processing method and a device. By merging the electrical signals of at least two different spectral bands acquired by a hyperspectral or multi-spectral filter array, N spectral bands of the multi-spectral or hyperspectral can be mapped to H spectral channels, where the number H is less than N, thereby achieving compatibility between the hyperspectral or multi-spectral filter array and the low-spectral imaging technology.

[0076] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0077] In the description of the present application, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0078] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "set" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0079] In the description of the present application, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the application.

[0080] Certain words are used in the specification and claims to refer to specific components. Those skilled in the art will understand that hardware manufacturers may use different terms to refer to the same component. This specification and the following claims do not use differences in names as a way to distinguish components, but rather use differences in the functions of the components as the criteria for distinction. The terms "include" or "comprises" mentioned in the specification and claims are open-ended terms and should be interpreted as "includes but not limited to" or "includes but not limited to".

[0081] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0082] The embodiments of the present application can be applied to various application scenarios that require image acquisition. As an example, in a smart phone, the mobile phone can be used to capture images of a target object, or it can be used to identify the material of the target object. When the mobile phone is used to identify the material, it is necessary to collect more information about the target object. In this case, the image sensing module of the mobile phone will be configured with a high-spectral or multi-spectral filter array. When the mobile phone is used to capture images of the target object, the mobile phone does not need to obtain the spectral information of the target object in too many spectral bands, that is, the mobile phone only needs to generate an RGB image of the target object. As another example, in a smart car, the smart car can be used to capture images of surrounding objects, or it can be used to extract the intrinsic reflectivity of surrounding objects through multi-spectral information for accurate object recognition. It is necessary to collect more spectral information about the surrounding objects. In this case, the image sensing module of the smart car will be configured with a high-spectral or multi-spectral filter array. When the smart car is used to capture images of surrounding objects, the smart car does not need to obtain the spectral information of the target object in too many spectral bands, that is, the smart car only needs to generate an RGB image of the surrounding objects. In both of the above application scenarios, it is necessary to use a hyperspectral or multispectral filter array to obtain the light signal of the target object and be compatible with RGB imaging technology. It should be understood that the application scenarios of the embodiments of the present application are not exhaustive here, and the RGB image shown above is only an example.

[0083] In order to solve the above problems, an embodiment of the present application provides an image sensor, which includes at least one filter array, which is a hyperspectral or multi-spectral filter array, that is, the filter array can obtain the optical signal of the target object in N first spectral bands, where N is an integer greater than 3. The filter array includes m filters and at least one photoelectric conversion unit, and the m filters can obtain the optical signal of the target object in at least two different spectral bands, and the photoelectric conversion unit can convert the optical signal obtained by the filter into an electrical signal. When it is not necessary to obtain the hyperspectral or multi-spectral information of the target object, the electrical signals corresponding to the m filters are merged to obtain the merged electrical signal, and the electrical signals of at least two different spectral bands in the hyperspectral or multi-spectral filter array are merged and processed, so that the mapping between the optical signals of the N first spectral bands and the H spectral channels can be realized, and the value of H is less than N. Among them, the value of H can be set according to the type of image to be output. If the output is an RGB image, H can be 3. The three spectral channels are used to obtain the light signal of the red channel, the light signal of the green channel, and the light signal of the blue channel in the RGB channel, respectively. It should be understood that the RGB image is used as an example here only to facilitate the understanding of this scheme and is not used to limit this scheme.

[0084] After obtaining the combined electrical signal, the electronic device equipped with the above image sensor needs to perform illumination estimation, de-mosaic processing and noise reduction processing on the combined signal to obtain a raw format image, and then perform white balance processing, etc., to obtain the image of the target object. The following first introduces the structure of the hardware such as the filter array, image sensor module, and electronic equipment, and then introduces the process of generating the image of the target object after obtaining the electrical signal.

[0085] Furthermore, an image sensor may include one or more filter arrays, and a merging unit may be provided in the filter array to merge the electrical signals corresponding to a group of filters; or the merging operation may be performed after the processor obtains the electrical signal corresponding to each filter. The above two situations are different, and the above two situations are introduced respectively below.

[0086] 1. Setting a merging unit in the filter array to merge the electrical signals corresponding to a group of filters

[0087] In the present application, please refer to Figure 1 , Figure 1 A schematic diagram of the structure of a filter array provided in an embodiment of the present application. The filter array 100 is a high-spectrum filter array, or the filter array 100 is a multi-spectrum filter array. A filter array 100 includes M filters and a photoelectric conversion module, and the photoelectric conversion module includes at least one photoelectric conversion unit and at least one merging unit. The M filters are used to obtain optical signals of a target object in N different spectral bands, where N is an integer greater than 3. The photoelectric conversion unit is used to convert the optical signals obtained by the M filters into electrical signals.

[0088] In the first case, M filters are divided into K groups, and any group of filters in the K groups is used to obtain the optical signal of the target object in at least one spectral band. It should be noted that it is not possible to realize that each group in the K groups is only used to obtain the optical signal of the target object in one spectral band, that is, at least one group in the K groups is used to obtain the optical signal of the target object in at least two spectral bands. One group in the K groups includes m filters, and the m filters are used to obtain the optical signal of the target object in at least two spectral bands. The merging unit is used to merge the electrical signals corresponding to the m filters to obtain a merged electrical signal; it can also be called pixel merging of the pixel units corresponding to the m filters to obtain a merged electrical signal.

[0089] The first case refers to a case where a hyperspectral or multi-spectral light signal is not required. In the aforementioned case, the merging unit is in an on state. As an example, when a mobile phone calls the "camera" function to take a photo, that is, when the image sensor is in an image capture mode, a hyperspectral or multi-spectral light signal is not required. It should be understood that the examples here are only for the convenience of understanding this solution.

[0090] In an embodiment of the present application, the filter array 100 included in the hyperspectral imaging technology or the multispectral imaging technology can obtain optical signals of the target object in multiple spectral bands. When a non-hyperspectral image or a multispectral image needs to be output, the electrical signals corresponding to at least two different spectral bands can be merged to obtain a merged electrical signal, that is, the N spectral bands of the multispectral or hyperspectral image can be mapped to H spectral channels, and the number H is less than N, so that the compatibility of the hyperspectral or multispectral filter array with the low-spectral imaging technology can be achieved; when an RGB image needs to be output, the value of H is 3, and the 3 spectral channels correspond to the three types of electrical signals required to generate an RGB image. In the above manner, it is possible to achieve compatibility with the RGB imaging technology when a hyperspectral or multispectral filter array is used to collect the spectral information of the target object.

[0091] Optionally, in the second case, the merging unit is in a closed state, and the photoelectric conversion module is used to output the electrical signal generated by the photoelectric conversion unit and corresponding to the M filters. The second case refers to a case where a high-spectrum or multi-spectrum light signal is required. As an example, for example, when the electronic device where the filter array 100 is located performs object recognition, depth detection, light source estimation or other tasks that require rich spectral information, it is considered to be in the second case, that is, the second case refers to the case of non-image shooting mode, and the non-image shooting mode can be any of the following modes: object recognition mode, light source estimation mode, depth detection mode or other modes, which are not exhaustive here. Object recognition refers to the identification of the type of the target object. Depth detection refers to the detection of the depth value of the target object.

[0092] When the filter array 100 is configured in an electronic device, if the photoelectric conversion unit is specifically manifested as a photoelectric conversion circuit, the electronic device is also configured with an analog-to-digital conversion unit, which is used to convert the analog electrical signal generated by the photoelectric conversion unit and corresponding to the M filters into a digital electrical signal to perform subsequent operations such as illumination estimation, de-mosaicing, noise reduction, and white balance processing.

[0093] Furthermore, if the image sensor is not configured with a processor, the electronic device in which the image sensor is located can determine whether the electronic device starts the image sensor based on the image capture mode. When the electronic device is determined to be in the image capture mode, that is, the image sensor works in the image capture mode, the merging unit is controlled to be turned on, that is, the merging unit is controlled to work, so that the merging unit can merge and process the electrical signals corresponding to the m filters; when the electronic device is determined to be in the non-image capture mode, that is, the image sensor works in the non-image capture mode, the merging unit is controlled to be turned off, that is, the merging unit is controlled not to work, so that the photoelectric conversion module in the image sensor will directly output the electrical signals corresponding to the m filters.

[0094] If a processor is configured in the image sensor, when the electronic device determines that it is in an image capture mode, the electronic device sends a first instruction to the image sensor, and the image sensor responds to the first instruction to control the merging unit to turn on, that is, to control the merging unit to work, so that the merging unit can merge and process the electrical signals corresponding to the m filters. When the electronic device determines that it is in a non-image capture mode, the electronic device sends a second instruction to the image sensor, and the image sensor responds to the second instruction to control the merging unit to turn off, that is, to control the merging unit not to work, so that the photoelectric conversion module in the image sensor will directly output the electrical signals corresponding to the m filters.

[0095] In the embodiments of the present application, not only can the RGB imaging technology be compatible when a hyperspectral or multispectral filter array is used to collect the spectral information of the target object, but also the electrical signal generated by the photoelectric conversion unit can be directly output after the photoelectric conversion unit converts the optical signals of N spectral bands into electrical signals. That is, the output electrical signal can carry hyperspectral or multispectral information, thereby expanding the application scenarios of the present solution.

[0096] Among them, for the structure of the filter. The filter is sometimes also called a color filter, which is an optical device used to select the required spectral band. The light within the spectral band that the filter can pass through is considered to be the light signal of the target object within the spectral band that the filter can obtain.

[0097] In one implementation, the filter is made of structural optics. For example, the filter may be a nanopore structure, a waveguide grating structure, a multi-layer interference film structure, etc., which is not limited here.

[0098] For a more intuitive understanding of this solution, please refer to Figure 2 , Figure 2 Three schematic diagrams of structures of filters in the filter array 100 provided in the embodiments of the present application. Figure 2 It includes three sub-schematic diagrams (a), (b) and (c). Figure 2 In the schematic diagram (a), the filter is a nanopore structure, and the interference of surface plasmon polaritons of adjacent holes is used to trigger the selective transmission of light. The nanopore type is designed by adjusting the aperture size of the nanopores processed by the metal material and the spacing between the nanopores to design different filters. Figure 2 In the (b) schematic diagram, the filter is a waveguide grating structure. The micro-nano grating structure introduces surface plasmon metal, so that the micro-nano grating also superimposes the absorption and scattering of surface plasmons on the basis of optical diffraction. The micro-nano periodic grating structure can be used as a dispersion medium to select the wavelength band. By changing the parameters such as the period and duty cycle of the grating structure, the absorption, scattering, diffraction and polarization characteristics of light can be controlled, thereby achieving filtering characteristics.

[0099] Figure 2 In the sub-schematic diagram (c), the filter is a multilayer interference thin film structure. According to the main physical mechanism causing optical filtering, this structure can be divided into metal / dielectric / metal (MIM) type based on localized surface plasmon resonance, and Fabry-Perot (FP) type based on multi-beam interference. The top metal of the MIM filter structure generally has a micro-nano structure. The localized surface plasmon resonance of the top layer and the antiparallel surface currents of the upper and lower metal layers work together to make the MIM structure have absorption characteristics for light in a specific band, thereby realizing the filtering function. The FP filter structure uses two layers of semi-transparent and semi-reflective films to form a reflective layer, and a cutoff layer in the middle to form a resonator. Multi-beam interference is used to allow constructive interference light to be transmitted or reflected. This mode does not require a nanostructure on the top layer, and generally only considers the optical properties and manufacturing process of each film layer. It should be understood that Figure 2 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0100] In another implementation, the filter is made of chemical dye materials. Specifically, the filter can be made of plastic or glass with special dyes added. Furthermore, the transmittance of the glass is originally similar to that of air, and all colored light can pass through, so it is transparent. However, after being dyed, the molecular structure changes, the refractive index also changes, and the passage of certain colored light changes. As an example, a red plastic filter can only allow red light to pass; as another example, a beam of white light passes through a blue glass filter, and a beam of blue light is emitted.

[0101] M is an integer greater than or equal to N, and N is an integer greater than 3. As an example, the value of N can be 4, 6, 8, 9, 12, 16, 27, 36 or other values. The value of M can be 4, 8, 9, 12, 16, 20, 36 or other values. Optionally, N is an integer greater than 3 and less than or equal to 25, and correspondingly, the value of M can be an integer greater than 3 and less than or equal to 25. Among them, the larger the value of N, the richer the spectral information obtained, but correspondingly, the higher the complexity of the image generation process, then the values ​​of M and N can be determined in combination with the requirements for the richness of spectral information in actual application scenarios, the processing power of the processor, etc., and are not limited here.

[0102] A filter array 100 may be rectangular, “L”-shaped, “concave”-shaped, “E”-shaped or other shapes, which are not limited here.

[0103] Each of the M filters corresponds to at least one photoelectric conversion unit. The photoelectric conversion unit can be specifically expressed as a photoelectric conversion circuit or a computer code. The specific form of the photoelectric conversion unit is not limited here. If the photoelectric conversion unit is specifically expressed as a photoelectric conversion circuit, the photoelectric conversion unit outputs an analog electrical signal. In one case, each filter corresponds to a photoelectric conversion unit, that is, each photoelectric conversion unit is used to convert an optical signal obtained by a filter into an electrical signal. Alternatively, each of the M filters corresponds to multiple photoelectric conversion units, that is, multiple photoelectric conversion units jointly convert an optical signal obtained by a filter into an electrical signal.

[0104] For a more intuitive understanding of this solution, please refer to Figure 3 , Figure 3 Two schematic diagrams of the corresponding relationship between filters and photoelectric conversion units in the filter array provided in the embodiments of the present application. Figure 3 It includes two sub-schematic diagrams (a) and (b). Figure 3 In the sub-schematic diagram (a), a filter is arranged above the photoelectric conversion unit, and the filter and the photoelectric conversion unit are in a one-to-one correspondence, that is, one filter is arranged above one photoelectric conversion unit. Figure 3 In the sub-schematic diagram (b), a filter is arranged above the plurality of photoelectric conversion units. Figure 3 It is shown that a filter is arranged above every two photoelectric conversion units. It should be understood that Figure 3 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0105] A filter array 100 may include K merging units corresponding to K groups of filters, and each merging unit may be specifically expressed as a merging circuit or as a computer code. The specific form of the merging unit is not limited here. The merging method may be taking the maximum value, taking the average value, and superposition. Optionally, the merging unit is specifically used to superimpose the electrical signals corresponding to all filters in the first group to obtain a merged analog electrical signal; each of the K merging units performs the aforementioned operation, thereby obtaining K merged analog electrical signals. Any group in the K groups includes at least two filters, and the number of filters included in different groups in the K groups may be the same or different. In the embodiment of the present application, the merging operation of the electrical signals corresponding to all filters in the first group is achieved by superimposing the electrical signals corresponding to each filter. The scheme is simple and easy to implement.

[0106] When the filter array 100 is configured in an electronic device, the electronic device may also be provided with an analog-to-digital conversion unit, which is used to convert the K merged analog electrical signals output by the merging unit into K merged digital electrical signals to perform subsequent signal processing operations such as illumination estimation, de-mosaicing, noise reduction, and white balance processing.

[0107] The light signals of the target object in N spectral bands obtained by M filters need to be mapped to H spectral channels, and the H spectral channels are divided according to any one of RGB channels, RGGB channels, RYB channels, RYYB channels, RWWB channels or RBGIR channels. As an example, for example, the light signals of N spectral bands are divided according to RGB channels, and the N spectral bands may include 6 different spectral bands of R1, R2, G1, G2, G3, G4, B1 and B2. Among them, the light signals of the target object in the two spectral bands R1 and R2 are mapped to the R channel in the RGB channel, the light signals of the target object in the four spectral bands G1, G2, G3 and G4 are mapped to the G channel in the RGB channel, and the light signals of the target object in the two spectral bands B1 and B2 are mapped to the B channel in the RGB channel. It should be understood that the examples here are only for the convenience of understanding this solution and are not used to limit this solution. In the embodiment of the present application, a variety of division methods of H spectral channels corresponding to K groups of filters are listed, which improves the implementation flexibility of the present solution and expands the application scenarios of the present solution.

[0108] Among them, the M filters are divided into K groups, so that the optical signals of the target object in N spectral bands acquired by the M filters are also divided into K groups. The concept of spectral channel is used to reflect the ability of the filter array 100 to acquire spectral bands. Each working band of the filter array 100 is called a channel. If the filter array 100 can generate information of the target object in several spectral bands, it is called a filter array 100 with several spectral channels. Although the filter array 100 can acquire the optical signals of the target object in N spectral bands, it is necessary to map the optical signals of the N spectral bands into H spectral bands, that is, the optical signals of the N spectral bands need to be mapped to H spectral channels.

[0109] K is an integer greater than or equal to 1 and less than N. For example, the value of K can be 3, 4 or 5. The specific value of K needs to be determined in combination with the type of image that needs to be output in the actual application scenario, and is not limited here. R is the red channel, G is the green channel, B is the blue channel, Y is the yellow channel, W is the white channel, and IR is the near-infrared channel. The filter corresponding to the white channel can be a fully transparent light-transmitting sheet, through which the contour information of the target object can be collected.

[0110] Furthermore, the above-mentioned K groups of filters correspond to H spectral bands, and filters of different groups can correspond to the same spectral band, that is, filters of different groups in the K group can correspond to different types of spectral channels, or they can correspond to the same type of spectral channels. As an example, when the optical signals of the target object in N spectral bands acquired through M filters are mapped to the RGGB channel, that is, the value of K is 4, the value of H is 3, and the four groups are mapped to the R channel, the G channel, the G channel and the B channel respectively. The spectral band corresponding to the R channel is the red visible light band with a wavelength of 0.630-0.680, the spectral band corresponding to the G channel is the green visible light band with a wavelength of 0.525-0.600, and the spectral band corresponding to the B channel is the blue visible light band with a wavelength of 0.450-0.515; then the spectral bands corresponding to the two groups of filters mapped to the G channel are the same. It should be understood that the examples given here are only for the convenience of understanding the corresponding relationship between K groups of filters and H spectral bands, and are not used to limit this solution.

[0111] Next, the correspondence between the N spectral bands and the H spectral channels is introduced. The H spectral bands corresponding to the H spectral channels can constitute a first spectral band. In one case, the N spectral bands are within the first spectral band range, and the N spectral bands are subdivisions of the H spectral bands, or the N spectral bands are sub-spectral bands of the H spectral bands. Then the m filters have a mapping relationship with the first spectral channel of the H spectral channels, the first spectral channel is one of the H spectral channels, the first spectral channel corresponds to the first spectral band, and the at least two different spectral bands corresponding to the m filters are within the first spectral band range.

[0112] For a more intuitive understanding of this solution, please refer to Figures 4 to 7 , Figures 4 to 7 Schematic diagram of four correspondences between N spectral bands and H spectral bands of spectral channels in the filter array provided in an embodiment of the present application, Figures 4 to 7 In the examples, H spectral channels correspond to 3 spectral bands. Figure 4 , Figure 4 Taking the value of H as 3 and the value of N as 12 as an example, the three spectral channels are used to collect the optical signals of the target object in the spectral bands of blue visible light, green visible light and red visible light respectively. The optical signals of the target object in 12 spectral bands can be obtained through M filters. The 12 spectral bands are all located in the three spectral bands to be mapped, that is, the N spectral bands are all located in the first spectral band. Figure 4 There are four sub-spectral bands corresponding to the spectral band of green visible light (i.e. Figure 4 For example, G1, G2, G4 and G4 in Figure 1. Figure 4 Not shown in the figure, the correspondence between the spectral band of blue visible light and the spectral band of red visible light and the sub-spectral bands is similar to the correspondence between the spectral band of green visible light and the sub-spectral bands.

[0113] Continue reading Figure 5 , Figure 5 Taking the value of H as 3 and the value of N as 6 as an example, the three spectral channels are used to collect the optical signals of the target object in the spectral bands of blue visible light, green visible light and red visible light respectively. The optical signals of the target object in the six spectral bands can be obtained through M filters. The six spectral bands are all located in the three spectral bands to be mapped. The spectral bands of blue visible light, green visible light and red visible light each correspond to two sub-spectral bands.

[0114] Continue reading Figure 6 , Figure 6Taking the value of H as 3 and the value of N as 8 as an example, the three spectral channels are used to collect the optical signals of the target object in the spectral bands of blue visible light, green visible light and red visible light respectively. The optical signals of the target object in 8 spectral bands can be obtained through M filters, and the 8 spectral bands are all located in the 3 spectral bands to be mapped. The spectral band of green visible light corresponds to 4 sub-spectral bands, and the spectral bands of blue visible light and red visible light each correspond to 2 sub-spectral bands.

[0115] Continue reading Figure 7 , Figure 7 Taking the value of H as 3 and the value of N as 8 as an example, the three spectral channels are used to collect the light signals of the target object in the blue visible light spectral band, the green visible light spectral band and the red visible light spectral band respectively. The light signals of the target object in the eight spectral bands can be obtained through M filters. The eight spectral bands are all located in the three spectral bands to be mapped. The blue visible light spectral band, the green visible light spectral band and the red visible light spectral band each correspond to three sub-spectral bands. It should be understood that Figures 4 to 7 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0116] In summary, the filters of the third group in the K group correspond to the second spectral channel among the H spectral channels, the third group is any group in the K group, the second spectral channel is the spectral channel corresponding to the third group among the H spectral channels, and the filters of the third group are used to obtain the optical signal of the target object in at least one spectral band, and at least one spectral band corresponding to the third group is located within the spectral band range of the second spectral channel.

[0117] In another case, the spectral bands greater than or equal to the preset ratio among the N spectral bands are located within the first spectral band range, and the spectral bands greater than or equal to the preset ratio among the at least two different spectral bands corresponding to the m filters are located within the first spectral band range. The value of the preset ratio can be 98%, 95%, 90% or other values, that is, most of the spectral bands among the N spectral bands are located within the first spectral band. In the embodiment of the present application, at least two different spectral bands corresponding to the m filters are located within the first spectral band range, or, the spectral bands greater than or equal to the preset ratio among the at least two different spectral bands corresponding to the m filters are located within the first spectral band range, that is, the similarity between the N spectral bands and the mapped H spectral bands is very high, which is conducive to improving the quality of the image finally acquired.

[0118] In another case, the relationship between the N spectral bands and the first spectral band is random, or the spectral bands less than a preset proportion of the N spectral bands are located within the range of the first spectral band.

[0119] Furthermore, K groups of filters are used to obtain light signals of the target image in H spectral bands, and any group of filters in the K groups is used to obtain light signals of the target object in at least one spectral band, and the number of spectral bands corresponding to different groups is the same, or the number of spectral bands corresponding to different groups is different.

[0120] Furthermore, in one implementation, the value of N is 12, and each filter group in the K group is used to obtain the spectral information of the target object in four spectral bands. Since the spectral bands corresponding to different groups can be the same, the value of K in this implementation can be 3, 4, 5 or other values. As an example, see Figure 4 , specifically Figure 4 For understanding, please refer to the above description, which will not be repeated here.

[0121] In another implementation, the value of N is 6, and each filter group in the K group is used to obtain the spectral information of the target object in two spectral bands. Similarly, the value of K in this implementation can be 3, 4, 5 or other values. As an example, refer to Figure 5 , specifically Figure 5 For understanding, please refer to the above description, which will not be repeated here.

[0122] In another implementation, the value of N is 8, and the K group includes at least one first group and at least one second group. The filters in the first group are used to obtain the spectral information of the target object in 4 spectral bands, and the filters in the second group are used to obtain the spectral information of the target object in 2 spectral bands. The first group and the second group are two different groups in the K group. As an example, refer to Figure 6 , specifically Figure 6 For understanding, please refer to the above description, which will not be repeated here.

[0123] In the embodiment of the present application, the allocation of spectral bands corresponding to different groups under different values ​​of N is provided, which improves the implementation flexibility of the present solution. In addition, since the number of spectral bands corresponding to the same group is greater, the richer the spectral information obtained in the group is, but the more computer resources are required in the subsequent processing process. Among the H channels corresponding to the K group, the importance of different channels may be different. In this case, a larger number of sub-spectral bands are allocated to the groups corresponding to the channels with high importance, and a smaller number of sub-spectral bands are allocated to the groups corresponding to the channels with low importance, which is beneficial to increase the richness of the obtained spectral information while saving computer resources.

[0124] As an example, when an RGB image needs to be generated, the three channels corresponding to the K group are the R channel, the G channel, and the B channel. In actual imaging, the G channel is the main brightness channel and an important information channel that determines the color restoration effect of the image. That is, the importance of the G channel is higher than that of the R channel and the B channel. Then, a larger number of sub-spectral bands can be allocated to the group that has a mapping relationship with the G channel. It should be understood that the examples here are only for the convenience of understanding this solution. When the three channels corresponding to the K group are the RYB channel, the RWWB channel, or the RBGIR channel, the mapping relationship between the N sub-spectral bands and the K spectral channels can be adjusted according to the actual situation.

[0125] Any group of filters in the K group includes at least one filter, and the aforementioned at least one filter includes adjacent filters. In the embodiment of the present application, the optical signal acquired by a group is mapped to the optical signal of a channel. Since in imaging technologies such as RGB, RYB, RWWB or RBGIR, at least one filter corresponding to the same channel includes adjacent filters, that is, the dimension of the filter position allocation increases the similarity with imaging technologies such as RGB, RYB, RWWB or RBGIR, which is conducive to reducing the difficulty in the subsequent signal processing process.

[0126] For a more intuitive understanding of this solution, please refer to Figure 8 , Figure 8 Three schematic diagrams of the positional relationship between multiple filters in a group of filters in the filter array provided in the embodiments of the present application, the filters included in the two groups A1 and A2 are both adjacent filters, and the group of filters shown in A3 contains non-adjacent filters. It should be understood that Figure 8 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0127] Regarding the positional relationship between the filters included in different groups in the K group of filters. In one implementation, the filters included in different groups in the K group of filters are completely different, that is, there are no overlapping filters included in the filters included in different groups in the K group of filters. In the embodiment of the present application, since there are no overlapping filters included in different groups in the K group of filters, the information obtained by each filter will not be reused, there is no problem of information redundancy, and it is conducive to saving computer resources.

[0128] For a more intuitive understanding of this solution, please refer to Fig. 9 and Fig.10 , Fig. 9 and Fig.10 Two schematic diagrams of the positional relationship between filters included in different groups in the filter array provided in the embodiment of the present application, first refer to Fig. 9 , Fig. 9 and Fig.10 In the example above, the K value is 4. The four filters in the upper left corner belong to the first group, the four filters in the upper right corner belong to the second group, the four filters in the lower left corner belong to the third group, and the four filters in the lower right corner belong to the fourth group. There are no overlapping filters in the four groups. Fig.10 , C1, C2, C3, and C4 represent four groups respectively, and there are no overlapping filters in the filters included in the four groups. It should be understood that Fig. 9 and Fig.10 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0129] In another implementation, the same filters exist between different groups in the K groups of filters, but they are not completely the same, that is, there is at least one overlapping filter between different groups. In the embodiment of the present application, since the same filters exist between different groups in the K groups of filters, the spatial structure is more compact, which is conducive to reducing the difficulty of subsequent signal processing.

[0130] For a more intuitive understanding of this solution, please refer to Fig.11 and Fig.12 , Fig.11 and Fig.12 Two schematic diagrams of the positional relationship between filters included in different groups in the filter array provided in the embodiments of the present application. Fig.11 , Fig.11 Taking the value of M as 9, the value of N as 8, and the value of K as 4 as an example, D1, D2, D3, and D4 represent four groups respectively, but the group represented by D1 and the group represented by D3 both include a filter labeled 2, and the group represented by D2 and the group represented by D4 both include a filter labeled 6, that is, the group represented by D1 and the group represented by D3 have the same filters but are not completely the same, and the group represented by D2 and the group represented by D4 have the same filters but are not completely the same.

[0131] Continue reading Fig.12 , Fig.12Taking the value of M as 9, the value of N as 6, and the value of K as 4 as an example, E1, E2, E3, and E4 represent four groups respectively. The group represented by E1 and the group represented by E2 both include two filters in E5, the group represented by E1 and the group represented by E3 both include two filters in E6, the group represented by E1 and the group represented by E4 both include a filter labeled 4, the group represented by E2 and the group represented by E3 both include a filter labeled 4, and the group represented by E2 and the group represented by E4 both include two filters in E7, that is, there are the same filters among the groups represented by E1, E2, E3, and E4, but they are not completely the same. It should be understood that Fig.10 and Fig.11 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0132] The shape of any group of filters in the K group can be a rectangle, a vertical "I" shape, a horizontal "I" shape, a positive "L" shape, a reverse "L" shape, an inverted "L" shape, a "concave" shape or other shapes, etc. The specific shape of each group of filters is not limited here. In the embodiment of the present application, since the shape of each group of filters in the RGB channel, RGGB channel, RYB channel, RYYB channel, RWWB channel or RBGIR channel technology is a rectangle, the shape of any group of filters in the K group is also set to a rectangle, which is conducive to reducing the difficulty of the mapping process.

[0133] For a more intuitive understanding of this solution, please refer to the above Figure 10 to Figure 11 , Fig.10 and Fig.11 Each filter group is rectangular in shape. Fig.10 The shape of each filter group is vertical "I" or horizontal "I". Fig.11 In the group represented by D1, a group of filters has an inverted "L" shape, in the group represented by D2, a group of filters has a reverse "L" shape, and in the groups represented by D3 and D4, a group of filters has an "L" shape. It should be understood that the examples given here are only for a more convenient understanding of the present scheme and are not used to limit the present scheme.

[0134] Furthermore, several different implementations of the filter array 100 are disclosed below.

[0135] In one implementation, the filters included in different groups of K groups of filters are completely different, the value of M is 16, the value of N is 12, the value of K is 4, each group of filters in K group includes 4 filters, and each group of filters in K group is used to obtain the spectral information of the target object in 4 spectral bands. The value of H is 3 or 4, that is, the K group of filters is mapped to 3 or 4 spectral channels. When H is 3, the 3 spectral channels are any one of RGB channels, RGGB channels, RYB channels, RYYB channels or RWWB channels. When H is 4, the 4 spectral channels are RBGIR channels or other types of spectral channels.

[0136] For a more intuitive understanding of this solution, please refer to Figure 4 and Fig. 9 In this implementation, the correspondence between the 12 sub-spectral bands and the 3 spectral channels is as follows: Figure 4 As shown, the grouping of the 4 groups of filters is as follows Fig. 9 As shown, Fig. 9 Taking H as 3 as an example, 4 groups of filters are mapped to the three channels of R, G and B. Fig. 9 G1, G2, G3 and G4 in the figure can respectively collect the optical signals of the target object in four spectral bands, and the four spectral bands corresponding to G1, G2, G3 and G4 are all within the spectral band of green visible light. Fig. 9 R1, R2, R3 and R4 in the figure can respectively collect light signals of the target object in four spectral bands, and the four spectral bands corresponding to R1, R2, R3 and R4 are all within the spectral band of red visible light. Fig. 9 B1, B2, B3 and B4 can respectively collect optical signals of the target object in four spectral bands, and the four spectral bands corresponding to B1, B2, B3 and B4 are all within the spectral band of blue visible light. It should be understood that the RGGB channel can also be replaced by the RYYB channel or the RWWB channel, or, in other embodiments, the value of H can also be 4. The above examples are only for the convenience of understanding the present scheme and are not used to limit the present scheme.

[0137] In another implementation, the filters included in different groups in the K groups of filters are completely different, the value of M is 16, the value of N is 6, the value of K is 4, and each group of filters in the K groups is used to obtain the spectral information of the target object in two spectral bands. The value of H is 3 or 4, that is, the K groups of filters are mapped to three spectral channels or four spectral channels.

[0138] For a more intuitive understanding of this solution, please refer to Fig.13 , Fig.13 A schematic diagram of grouping of a filter array provided in an embodiment of the present application. Fig.13 It includes a left sub-diagram and a right sub-diagram. The left sub-diagram is the same as the above Figure 5 Similarly, the correspondence between the six sub-spectral bands and the three spectral channels is shown, which will not be repeated here. Fig.13 The right sub-schematic diagram shows the grouping of 4 groups of filters. The four filters in the upper left corner belong to the first group, the four filters in the upper right corner belong to the second group, the four filters in the lower left corner belong to the third group, and the four filters in the lower right corner belong to the fourth group. There are no overlapping filters in the four groups. Fig.13 Taking H as 3 as an example, 4 groups of filters are mapped to the three channels of R, G and B. Fig.13 G1 and G2 in the figure can respectively collect the light signals of the target object in two spectral bands, and the two spectral bands corresponding to G1 and G2 are both within the spectral band of green visible light. Fig.13 R1 and R2 in the figure can respectively collect light signals of the target object in two spectral bands, and the two spectral bands corresponding to R1 and R2 are both within the spectral band of red visible light. Fig.13 B1 and B2 in the figure can respectively collect the optical signals of the target object in two spectral bands, and the two spectral bands corresponding to B1 and B2 are both located in the spectral band of blue visible light. It should be understood that the RGGB channel can also be replaced by the RYYB channel or the RWWB channel, or, in other embodiments, the value of H can also be 4. The above examples are only for the convenience of understanding this solution and are not used to limit this solution.

[0139] In another implementation, the filters included in different groups in the K group of filters are completely different, the value of M is 16, the value of N is 8, the value of K is 4, the K group includes a first group and a second group, the filters in the first group are used to obtain the spectral information of the target object in 4 spectral bands, the filters in the second group are used to obtain the spectral information of the target object in 2 spectral bands, and the first group and the second group are two different groups in the K group. The value of H is 3 or 4, that is, the K group of filters is mapped to 3 spectral channels or 4 spectral channels.

[0140] For a more intuitive understanding of this solution, please refer to Fig.14 , Fig.14 A schematic diagram of grouping of the filter array provided in an embodiment of the present application. Fig.14 It includes a left sub-diagram and a right sub-diagram. The left sub-diagram is the same as the above Figure 6 Similarly, the corresponding relationship between the 8 sub-spectral bands and the 3 spectral channels is shown, which will not be repeated here. Fig.14The right sub-schematic diagram shows the grouping of 4 groups of filters. The four filters in the upper left corner belong to the first group, the four filters in the upper right corner belong to the second group, the four filters in the lower left corner belong to the third group, and the four filters in the lower right corner belong to the fourth group. There are no overlapping filters in the four groups. Fig.14 Taking H as 3 as an example, 4 groups of filters are mapped to the three channels of R, G and B. Fig.14 G1, G2, G3 and G4 in the figure can respectively collect the optical signals of the target object in four spectral bands, and the four spectral bands corresponding to G1, G2, G3 and G4 are all within the spectral band of green visible light. Fig.14 R1 and R2 in the figure can respectively collect light signals of the target object in two spectral bands, and the two spectral bands corresponding to R1 and R2 are both within the spectral band of red visible light. Fig.14 B1 and B2 in the figure can respectively collect optical signals of the target object in two spectral bands. The two spectral bands corresponding to B1 and B2 are both located in the spectral band of blue visible light. It should be understood that the RGGB channel can also be replaced by the RYYB channel or the RWWB channel, or, in other embodiments, the value of H can also be 4. The above examples are only for the convenience of understanding the present scheme and are not used to limit the present scheme.

[0141] In the embodiments of the present application, a plurality of specific implementations of the filter array 100 are provided, which improves the implementation flexibility of the present solution and expands the application scenarios of the present solution. In addition, the filters included in different groups in the above-mentioned implementations are completely different, which avoids the processing of simple information and avoids the waste of computer resources.

[0142] In another implementation, the filters included in different groups in the K groups of filters have the same filters and are not completely the same, the value of M is 9, the value of N is 6, the value of K is 4, each group of filters in the K group is used to obtain the spectral information of the target object in 4 spectral bands, each group of filters in the K group includes 4 filters, and a group of filters in the K group is in a matrix type. The value of H is 3 or 4, that is, the K group of filters is mapped to 3 spectral channels or 4 spectral channels.

[0143] For a more intuitive understanding of this solution, please refer to Fig.12 , Fig.12 E1, E2, E3, and E4 represent four groups respectively. Fig.12 The numbers 1-6 in the figure represent six sub-spectral bands respectively. The filters included in the group represented by E1 are used to obtain the optical signals of the target object in the four spectral bands 3, 2, 5 and 4. For the understanding of E2, E3 and E4, please refer to the above description of E1, which will not be repeated here. Fig.12In the example, the value of H is 3. E1 and E4 are mapped to the same spectral channel. It should be understood that Fig.12 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0144] In the embodiment of the present application, a specific implementation method of the filter array 100 is provided, which further improves the implementation flexibility of the present solution; in addition, the filters included in different groups contain the same filters but are not completely the same, which reduces the difficulty of the subsequent signal processing process.

[0145] In another implementation, the filters included in different groups in the K group of filters have the same filters and are not completely the same, the value of M is 9, the value of N is 9, the value of K is 4, each group of filters in the K group includes 4 filters, each group of filters in the K group is used to obtain the spectral information of the target object in 3 spectral bands, and a group of filters in the K group is L-shaped, inverted L-shaped or reversed L-shaped. The value of H is 3 or 4, that is, the K group of filters is mapped to 3 spectral channels or 4 spectral channels.

[0146] For a more intuitive understanding of this solution, please refer to Fig.11 , Fig.11 D1, D2, D3, and D4 represent four groups respectively. Fig.11 The numbers 1-8 in the figure represent eight sub-spectral bands respectively. The filters included in the group represented by D1 are used to obtain the optical signals of the target object in the three spectral bands 4, 5 and 2. For the understanding of D2, D3 and D4, please refer to the above description of D1, which will not be repeated here. Fig.11 In the example, the value of H is still 3. D1 and D4 map to the same spectral channel, but the same spectral channel corresponds to two different spectral bands. It should be understood that Fig.11 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0147] In another implementation, the filters included in different groups in the K groups of filters are completely different, the value of M is 8, the value of N is 6, the value of K is 4, each group of filters in the K groups includes 2 filters, and each group of filters in the K groups is used to obtain spectral information of the target object in 2 spectral bands.

[0148] For a more intuitive understanding of this solution, please refer to Fig.10 and Figure 5 , Fig.10 C1, C2, C3, and C4 represent four groups respectively. Fig.10 The numbers 1-6 in the figure represent six sub-spectral bands respectively. The filters included in the group represented by C1 are used to obtain the optical signals of the target object in the two spectral bands 3 and 4. For the understanding of C2, C3 and C4, please refer to the above description of C1, which will not be repeated here. Fig.10 and Figure 5 In the example, the value of H is still 3. C1 and C4 are mapped to the same spectral channel. It should be understood that Fig.10 and Figure 5 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0149] In the embodiment of the present application, two specific implementations of the filter array 100 are provided, which further improves the implementation flexibility of the present solution.

[0150] Optionally, when the W channel (i.e., the white channel) is not included in the H channels, the filter array 100 may further include a fully transparent light-transmitting sheet, which is located in the center area of ​​the filter array 100. Specifically, in one implementation, the value of M is 8, the value of K is 4, each filter group in the K groups includes 2 filters, and the fully transparent light-transmitting sheet is located in the center of the 8 filters. For a more intuitive understanding of this solution, please refer to Fig.15 , Fig.15 A schematic diagram of grouping of a filter array provided in an embodiment of the present application. Fig.15 The grouping of the four groups can be found in the above Fig.10 The introduction of will not be elaborated here. Fig.15 and Fig.10 The difference is that a fully transparent light-transmitting sheet is set at the center of the 8 filters. It should be understood that Fig.15 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0151] In one implementation, the value of M is 16, the value of K is 4, each filter group in the K group includes 4 filters, and the fully transparent light-transmitting filter is located at the center of the 8 filters. For a more intuitive understanding of this solution, please refer to Fig.16 , Fig.16 A schematic diagram of a grouping of the filter array provided in the embodiment of the present application. F1, F2, F3, and F4 represent four groups respectively. A fully transparent light-transmitting sheet is arranged at the center of the 16 filters. It should be understood that Fig.16 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0152] Furthermore, another implementation of the filter array 100 is disclosed below. The value of N is 6, the value of K is 4, and the value of M is 8. The filter array 100 also includes a fully transparent light-transmitting sheet, which is located in the central area of ​​the 8 filters. In the embodiment of the present application, the contour information of the target object can be obtained through the fully transparent light-transmitting sheet, which provides assistance for mosaic processing in the subsequent signal processing process; and the light signal collected by the fully transparent light-transmitting sheet can be used as an auxiliary sensor of the light source spectrum, which provides assistance for white balance processing in the subsequent signal processing process.

[0153] After obtaining the digital electrical signal or the combined digital electrical signal, the electronic device where the filter array 100 is located needs to perform signal processing to generate an image of the target object. The aforementioned signal processing process includes but is not limited to the following steps: illumination estimation, demosaicing, noise reduction, and white balance processing. It should be noted that in actual situations, the signal processing process may include more or fewer steps, which are not limited here.

[0154] The illumination estimation is to make the signal meet a preset standard (International Standardization Organization, ISO). For example, if the electronic device is a camera, the standard is the standard of the camera's sensitivity. In order to meet the preset standard, the camera's processor will gain the combined digital signal.

[0155] The process of approximating the original value of each pixel unit based on the colors around it and finally obtaining a color image is called demosaicing. There are different ways to implement demosaicing. For example, a simple interpolation method, for example, a pixel unit including a green filter accurately measures the green component, and the red value of the pixel unit including the green filter can be calculated by interpolating the two adjacent red pixel units.

[0156] There are many ways to reduce noise. For example, noise is often more abrupt, so blurring can be used to reduce noise. However, blurring will also affect details, so consider filling in the area with stronger signals in the removed part.

[0157] Color deviation can be corrected by white balance processing. For example, taking H spectral channels as RGB channels, white balance is to correct the point (white point) where RGB is equal, which can be corrected by scaling the three channels separately. White balance processing is to adjust the color circuit inside the camera under different color temperature conditions to make the captured image offset the color cast and be closer to the visual habits of the human eye. White balance can also be simply understood as the standard white captured by the camera under any color temperature condition, which is adjusted by the circuit so that it remains white after imaging. It should be noted that the above process is only an exemplary description, and more steps may be included in the actual application scenario. By processing the data obtained by the first type of filter array, processing the data obtained by the second type of filter array, and fusing the processed data through the above method, a high-spectral and high-resolution image can be obtained. Exemplarily, a method for fusing the processed data to obtain a high-spectral and high-resolution image is given below.

[0158] 2. Merging the electrical signals corresponding to a group of filters in a filter array

[0159] In the present application, please refer to Fig.17 , Fig.17 The electronic device 2 includes at least one optical filter array 20 and a processor 21. The optical filter array 20 and the processor 21 are connected. Figures 1 to 17 The difference between the filter array 100 described is that the filter array 20 in this embodiment does not include the merging unit in the filter array 100. The rest of the description can be referred to above and will not be repeated here. The electronic device 2 can be specifically any of the following devices: an image sensor, a camera, a mobile terminal, a smart car, and a monitoring device.

[0160] The processor 21 is used to merge the electrical signals corresponding to the m filters in the first case to obtain the merged electrical signal, and generate an image of the target object according to the merged electrical signal; the processor is used to generate a hyperspectral image or a multispectral image of the target object according to the electrical signals corresponding to the m filters in the second case. In the embodiment of the present application, a merging unit is not added to the filter array, but the electrical signals corresponding to all filters in a group are merged during the data processing by the processor, which provides another implementation scheme compatible with RGB imaging technology when a hyperspectral or multispectral filter array is used to collect the spectral information of the target object, thereby improving the implementation flexibility of the present scheme; and in the present implementation scheme, complex signal processing is not required, which is conducive to reducing costs.

[0161] For details, please refer to Fig.18 , Fig.18 A structural schematic diagram of an electronic device provided in an embodiment of the present application. The electronic device 2 also includes an analog-to-digital conversion unit 22, which is used to convert the analog electrical signal generated by the photoelectric conversion unit in the filter array 20 into a digital electrical signal. The processor 21 is specifically used to superimpose the digital electrical signals corresponding to the m filters in the first case, that is, to add the digital electrical signals corresponding to the m filters to obtain a merged digital electrical signal. The processor 21 performs the above-mentioned operation on each group of filters in the K groups to obtain K merged digital electrical signals, and then can perform subsequent signal processing according to the K merged numerical electrical signals to generate an image of the target object. Among them, a digital electrical signal can be specifically expressed as a numerical value, and the value range of the digital electrical signal is related to the bit width of the analog-to-digital conversion. If the bit width is 8, the value range of the digital electrical signal is 0-255. If the bit width is 12, the value range of the digital electrical signal is 0-4096.

[0162] In the second case, after the analog-to-digital conversion unit 22 converts the analog electrical signal generated by the photoelectric conversion unit in the filter array 20 into a digital electrical signal, the processor 21 does not need to merge the data electrical signal, but directly generates a hyperspectral image or multispectral image of the target object based on the acquired digital electrical signal carrying information of N spectral bands.

[0163] For further information on the first and second situations, please refer to Figures 1 to 16 The description in is not repeated here.

[0164] exist Figures 1 to 18 Based on the corresponding embodiment, the present application embodiment also provides an image sensor, see Fig.19 , Fig.19 The image sensor 10 includes at least one filter array 100. Different filter arrays 100 are fixedly connected. The filter arrays 100 can refer to the above-mentioned Figures 1 to 16 The description in the corresponding embodiment is not repeated here. Optionally, the image sensor 10 may further include a low-spectrum filter array, and the filter array 100 is fixedly connected to the low-spectrum filter array.

[0165] The image sensor 10 can be configured in various types of electronic devices, and the electronic device is any of the following devices: a camera, a mobile terminal, a smart car, and a monitoring device.

[0166] Further optionally, the image sensor may also include a processor, the processor being used to generate an image of the target object based on the combined electrical signal generated by the filter array 100 in the first case. Optionally, the processor 11 is used to generate an image of the target object based on the combined electrical signal generated by each filter array 100 and the electrical signal generated by the low-spectrum filter array. The processor is also used to generate a hyperspectral image or a multispectral image of the target object based on the electrical signal corresponding to the M filters generated by the photoelectric conversion unit in the filter array 100 in the second case. The specific implementation process can be found in the above Figures 1 to 16 The description in the corresponding embodiments will not be repeated here.

[0167] The present application also provides an electronic device. Fig. 20 , Fig. 20 The electronic device 1 includes an image sensor 10 and a processor 11. The image sensor 10 includes at least one filter array 100. The filter array 100 can refer to the above Figures 1 to 16The description in the corresponding embodiment is not repeated here. Optionally, the image sensor 10 may further include a low-spectrum filter array, and the filter array 100 is fixedly connected to the low-spectrum filter array.

[0168] The processor 11 is used to generate an image of the target object based on the combined electrical signal generated by the filter array 100 in the first case. Optionally, the processor 11 is used to generate an image of the target object based on the combined electrical signal generated by each filter array 100 and the electrical signal generated by the low-spectrum filter array. The processor 11 is also used to generate a hyperspectral image or a multispectral image of the target object based on the electrical signal corresponding to the M filters generated by the photoelectric conversion unit in the filter array 100 in the second case. The specific implementation process can be referred to above Figures 1 to 16 The description in the corresponding embodiments will not be repeated here.

[0169] The processor 21 and the processor 11 include but are not limited to one or more of a central processing unit (CPU), a network processor (NP), an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0170] The present application also provides a signal processing method, which is applied to an electronic device 2, wherein the electronic device 2 includes at least one filter array, and the filter array can be Figures 1 to 16 The filter array 100 in Fig.17 and Fig.18 The specific structures of the filter array 20, the filter array 100 and the filter array 20 can be referred to above. Figures 1 to 16 The description in the corresponding embodiment. Fig.21 , Fig.21 A flow chart of a signal processing method provided in an embodiment of the present application. The signal processing method provided in an embodiment of the present application may include:

[0171] 2101. An electronic device obtains optical signals of a target object in N spectral bands through M filters.

[0172] 2102. The electronic device converts the optical signals acquired by the M filters into electrical signals.

[0173] In the embodiment of the present application, the specific implementation of step 2102 to step 2102 can refer to the above Figures 1 to 18 Corresponding to the description in the embodiment. If the filter array in this embodiment is the filter array 100, the electrical signal in step 2102 refers to the electrical signal generated by the photoelectric conversion unit. If the filter array in this embodiment is the filter array 20, the electrical signal in step 2102 refers to the electrical signal generated by the analog-to-digital conversion unit.

[0174] 2103. In the first case, the electronic device combines the electrical signals corresponding to the m filters to obtain a combined electrical signal.

[0175] In the embodiment of the present application, the specific implementation of steps 2102 to 2103 can be referred to above. Figures 1 to 18 Corresponding to the description in the embodiment.

[0176] Among them, M filters are divided into K groups, any group in the K groups includes at least two filters, and any group of filters in the K groups is used to obtain the optical signal of the target object in one spectral band. It should be noted that it is not possible to realize that each group in the K groups is only used to obtain the optical signal of the target object in one spectral band, that is, at least one group in the K groups is used to obtain the optical signal of the target object in at least two spectral bands. One group in the K groups includes m filters, and the m filters are used to obtain the optical signal of the target object in at least two spectral bands.

[0177] Specifically, if the filter array in this embodiment is Figures 1 to 16 The filter array 100 in the optical filter array, step 2103 may include: in the first case, the electronic device superimposes the electrical signals corresponding to the m filters through the merging unit in the optical filter array to obtain a merged electrical signal, and the electronic device performs the above operation on each group in the K groups through the merging unit in the optical filter array to obtain K merged electrical signals.

[0178] If the filter array in this embodiment is Fig.17 and Fig.18 In the filter array 20, step 2103 may include: in the first case, the electronic device superimposes the electrical signals corresponding to the m filters through the processor to obtain a combined electrical signal, and the electronic device performs the above operation on each group in the K groups through the processor to obtain K combined electrical signals.

[0179] 2104. The electronic device generates an image of the target object based on the combined electrical signal.

[0180] In the embodiment of the present application, the specific implementation of step 2104 can be referred to above. Figures 1 to 18 The description in the corresponding embodiment is not repeated here.

[0181] 2105. In a second case, the electronic device generates a hyperspectral image or a multispectral image of the target object according to the electrical signals corresponding to the m filters.

[0182] In the embodiment of the present application, in the second case, the electronic device no longer performs a merging process on the electrical signals corresponding to the m filters. Figures 1 to 16 The corresponding embodiments have been described. The interaction process between the processor and the image sensor in the electronic device, as well as the concepts of various terms in steps 2101 to 2105 can all be found in the description of the above embodiments, and will not be repeated here.

[0183] The present application also provides a computer-readable storage medium in which a program is stored. When the program is run on a computer, the computer executes the above-mentioned Fig.21 The illustrated embodiment describes the steps performed by the processor 21 in the method.

[0184] The present application also provides a computer program which, when executed on a computer, enables the computer to execute the above-mentioned Fig.21 The illustrated embodiment describes the steps performed by the processor 21 in the method.

[0185] The present application also provides a circuit system, which includes a processing circuit, wherein the processing circuit is configured to perform the above Fig.21 The illustrated embodiment describes the steps performed by the processor 21 in the method.

[0186] The processor 21 and the processor 11 provided in the embodiment of the present application can be specifically chips, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip to execute the above-mentioned Fig.21 The steps performed by the electronic device in the method described in the embodiment shown, or performing the above Fig.21The steps performed by the input device in the method described in the illustrated embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0187] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

[0188] Through the description of the above implementation mode, the technicians in the relevant field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CLUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0189] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program.

[0190] The computer program includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. An image sensor, characterized in that: The image sensor includes at least one filter array, which is a hyperspectral or multi-spectral filter array, and includes M filters, a photoelectric conversion module, and a fully transparent light-transmitting sheet. The M filters are divided into K groups, and one group in the K groups includes m filters, where K is an integer greater than 1. The photoelectric conversion module includes a photoelectric conversion unit and a merging unit, and the fully transparent light-transmitting sheet is located in the central area of ​​the filter array; The m filters are used to obtain optical signals of the target object in at least two different spectral bands; The photoelectric conversion unit is used to convert the optical signals acquired by the m optical filters into electrical signals; The merging unit is used to merge the electrical signals corresponding to the m filters to obtain a merged electrical signal; The photoelectric conversion module is further used to output the combined electrical signal, and the combined electrical signal is used to generate an image of the target object; The fully transparent light-transmitting sheet is used to obtain the contour information of the target object and / or the light signal of the target object in the spectral band; The K groups of filters have a mapping relationship with the H spectral channels; The M filters are used to obtain the optical signals of the target object in N spectral bands; The H is 3, the N is 12, and each filter group in the K groups is used to obtain the optical signal of the target object in 4 spectral bands; or, The H is 3, the N is 6, and each filter group in the K groups is used to obtain the optical signal of the target object in two spectral bands; or, The H is 3, the N is 8, the K group includes at least one first group and at least one second group, the filters in the first group are used to obtain the optical signals of the target object in 4 spectral bands, and the filters in the second group are used to obtain the optical signals of the target object in 2 spectral bands; or, The H is 3, the N is 8, and each filter group in the K groups is used to obtain the optical signal of the target object in three spectral bands.

2. The image sensor according to claim 1, characterized in that The merging unit is specifically used to superimpose the electrical signals corresponding to the m filters to obtain the merged electrical signal.

3. The image sensor according to claim 1, characterized in that The m filters have a mapping relationship with a first spectral channel among the H spectral channels, the first spectral channel is a spectral channel among the H spectral channels, and the first spectral channel corresponds to a first spectral band; The at least two different spectral bands corresponding to the m filters are located within the first spectral band range, or, among the at least two different spectral bands corresponding to the m filters, spectral bands greater than or equal to a preset proportion are located within the first spectral band range.

4. The image sensor according to claim 1 or 2, characterized in that: The shape of a group of filters consisting of the m filters is rectangular.

5. The image sensor according to claim 1 or 2, characterized in that: The m filters are adjacent filters.

6. The image sensor according to claim 1, characterized in that: The filters included in different groups in the K group of filters are completely different.

7. An electronic device, characterized in that: The electronic device comprises a processor and at least one filter array, wherein the filter array is a hyperspectral or multi-spectral filter array, and comprises M filters, a photoelectric conversion unit, and a fully transparent light-transmitting sheet, wherein the M filters are divided into K groups, and one group in the K groups comprises m filters, wherein K is an integer greater than 1, and the fully transparent light-transmitting sheet is located in the central area of ​​the filter array; The m filters are used to obtain optical signals of the target object in at least two different spectral bands; The photoelectric conversion unit is used to convert the optical signals acquired by the m optical filters into electrical signals; The processor is used to combine the electrical signals corresponding to the m filters to obtain a combined electrical signal, and generate an image of the target object according to the combined electrical signal; The fully transparent light-transmitting sheet is used to obtain the contour information of the target object and / or the light signal of the target object in the spectral band; The K groups of filters have a mapping relationship with the H spectral channels; The M filters are used to obtain the optical signals of the target object in N spectral bands; The H is 3, the N is 12, and each filter group in the K groups is used to obtain the optical signal of the target object in 4 spectral bands; or, The H is 3, the N is 6, and each filter group in the K groups is used to obtain the optical signal of the target object in two spectral bands; or, The H is 3, the N is 8, the K group includes at least one first group and at least one second group, the filters in the first group are used to obtain the optical signals of the target object in 4 spectral bands, and the filters in the second group are used to obtain the optical signals of the target object in 2 spectral bands; or, The H is 3, the N is 8, and each filter group in the K groups is used to obtain the optical signal of the target object in three spectral bands.

8. The electronic device according to claim 7, characterized in that: The processor is specifically used to superimpose the electrical signals corresponding to the m filters to obtain the combined electrical signal.

9. The electronic device according to claim 7, characterized in that: The m filters have a mapping relationship with a first spectral channel among the H spectral channels, the first spectral channel is a spectral channel among the H spectral channels, and the first spectral channel corresponds to a first spectral band; The at least two different spectral bands corresponding to the m filters are located within the first spectral band range, or, among the at least two different spectral bands corresponding to the m filters, spectral bands greater than or equal to a preset proportion are located within the first spectral band range.

10. The electronic device according to claim 7 or 8, characterized in that: The electronic device is any one of the following devices: a mobile terminal, a smart car, or a monitoring device.

11. An image sensor, characterized in that: The image sensor includes at least one filter array, which is a hyperspectral or multi-spectral filter array, and includes M filters, a photoelectric conversion module, and a fully transparent light-transmitting sheet. The M filters are divided into K groups, and one group in the K groups includes m filters, where K is an integer greater than 1, and the fully transparent light-transmitting sheet is located in the central area of ​​the filter array; The m filters are used to obtain optical signals of the target object in at least two different spectral bands; The photoelectric conversion module is used to convert the optical signals acquired by the m filters into electrical signals when the image sensor is in an image capture mode, merge the electrical signals corresponding to the m filters to obtain a merged electrical signal, and output the merged electrical signal, wherein the merged electrical signal is used to generate an image of the target object; The photoelectric conversion module is also used to convert the optical signals acquired by the m filters into electrical signals when the image sensor is in a non-image shooting mode, and output electrical signals corresponding to the m filters to generate a hyperspectral image of the target object or a multispectral image of the target object; The fully transparent light-transmitting sheet is used to obtain the contour information of the target object and / or the light signal of the target object in the spectral band; The K groups of filters have a mapping relationship with the H spectral channels; The M filters are used to obtain the optical signals of the target object in N spectral bands; The H is 3, the N is 12, and each filter group in the K groups is used to obtain the optical signal of the target object in 4 spectral bands; or, The H is 3, the N is 6, and each filter group in the K groups is used to obtain the optical signal of the target object in two spectral bands; or, The H is 3, the N is 8, the K group includes at least one first group and at least one second group, the filters in the first group are used to obtain the optical signals of the target object in 4 spectral bands, and the filters in the second group are used to obtain the optical signals of the target object in 2 spectral bands; or, The H is 3, the N is 8, and each filter group in the K groups is used to obtain the optical signal of the target object in three spectral bands.

12. The image sensor according to claim 11, characterized in that The non-image capturing mode is any one of the following modes: an object recognition mode, a light source estimation mode, and a depth detection mode.

13. An electronic device, characterized in that: The electronic device comprises a processor and at least one filter array, wherein the filter array is a hyperspectral or multi-spectral filter array, and comprises M filters, a photoelectric conversion unit, and a fully transparent light-transmitting sheet, wherein the M filters are divided into K groups, and one group in the K groups comprises m filters, wherein K is an integer greater than 1, and the fully transparent light-transmitting sheet is located in the central area of ​​the filter array; The m filters are used to obtain optical signals of the target object in at least two different spectral bands; The photoelectric conversion unit is used to convert the optical signals acquired by the m optical filters into electrical signals; The processor is used for merging the electrical signals corresponding to the m filters to obtain a merged electrical signal when the electronic device is in an image capture mode, and generating an image of the target object according to the merged electrical signal; The processor is further configured to generate a hyperspectral image or a multispectral image of the target object according to the electrical signals corresponding to the m filters when the electronic device is in a non-image capturing mode; The fully transparent light-transmitting sheet is used to obtain the contour information of the target object and / or the light signal of the target object in the spectral band; The K groups of filters have a mapping relationship with the H spectral channels; The M filters are used to obtain the optical signals of the target object in N spectral bands; The H is 3, the N is 12, and each filter group in the K groups is used to obtain the optical signal of the target object in 4 spectral bands; or, The H is 3, the N is 6, and each filter group in the K groups is used to obtain the optical signal of the target object in two spectral bands; or, The H is 3, the N is 8, the K group includes at least one first group and at least one second group, the filters in the first group are used to obtain the optical signals of the target object in 4 spectral bands, and the filters in the second group are used to obtain the optical signals of the target object in 2 spectral bands; or, The H is 3, the N is 8, and each filter group in the K groups is used to obtain the optical signal of the target object in three spectral bands.

14. The electronic device according to claim 13, characterized in that: The non-image capturing mode is any one of the following modes: an object recognition mode, a light source estimation mode, and a depth detection mode.

15. An electronic device, characterized in that: The electronic device comprises an image sensor and a processor, wherein the image sensor comprises at least one filter array. The image sensor is the image sensor according to any one of claims 1 to 6; The processor is used to generate an image of the target object according to the electrical signal output by each filter array.

16. The electronic device according to claim 15, characterized in that: The electronic device is any one of the following devices: a mobile terminal, a smart car, or a monitoring device.

17. A signal processing method, characterized in that: The method is applied to an electronic device, wherein the electronic device comprises at least one filter array, wherein the filter array is a hyperspectral or multi-spectral filter array, wherein the filter array comprises M filters and a fully transparent light-transmitting plate, wherein the fully transparent light-transmitting plate is located in the central area of ​​the filter array, wherein the M filters are divided into K groups, wherein one group in the K groups comprises m filters, wherein K is an integer greater than 1, and the K groups of filters have a mapping relationship with H spectral channels, and the method comprises: Acquire the optical signal of the target object in at least two different spectral bands through the m optical filters; Converting the optical signals acquired by the m optical filters into electrical signals; Combining the electrical signals corresponding to the m filters to obtain a combined electrical signal, and generating an image of the target object according to the combined electrical signal; Acquiring contour information of the target object and / or a light signal of the target object in a spectral band through the fully transparent light-transmitting sheet; Acquire the optical signal of the target object in N spectral bands through the M optical filters; The H is 3, the N is 12, and the optical signal of the target object in 4 spectral bands is obtained through each filter group in the K groups; or, The H is 3, the N is 6, and the optical signal of the target object in two spectral bands is obtained through each filter group in the K groups; or, The H is 3, the N is 8, the K group includes at least one first group and at least one second group, the optical signal of the target object in 4 spectral bands is obtained through the filters in the first group, and the optical signal of the target object in 2 spectral bands is obtained through the filters in the second group; or, The H is 3, the N is 8, and the optical signal of the target object in three spectral bands is obtained through each filter group in the K groups.

18. The method according to claim 17, characterized in that The combining and processing of the electrical signals corresponding to the m filters comprises: The electrical signals corresponding to the m filters are superimposed to obtain the combined electrical signal.

19. A signal processing method, characterized in that: The method is applied to an electronic device, wherein the electronic device comprises at least one filter array, wherein the filter array is a hyperspectral or multi-spectral filter array, wherein the filter array comprises M filters and a fully transparent light-transmitting plate, wherein the fully transparent light-transmitting plate is located in the central area of ​​the filter array, wherein the M filters are divided into K groups, wherein one group in the K groups comprises m filters, wherein K is an integer greater than 1, and the K groups of filters have a mapping relationship with H spectral channels, and the method comprises: Acquire the optical signal of the target object in at least two different spectral bands through the m optical filters; Converting the optical signals acquired by the m optical filters into electrical signals; When the electronic device is in an image capture mode, the electrical signals corresponding to the m filters are combined to obtain a combined electrical signal, and an image of the target object is generated according to the combined electrical signal; When the electronic device is in a non-image capturing mode, generating a hyperspectral image or a multispectral image of the target object according to the electrical signals corresponding to the m filters; Acquiring contour information of the target object and / or a light signal of the target object in a spectral band through the fully transparent light-transmitting sheet; Acquire the optical signal of the target object in N spectral bands through the M optical filters; The H is 3, the N is 12, and the optical signal of the target object in 4 spectral bands is obtained through each filter group in the K groups; or, The H is 3, the N is 6, and the optical signal of the target object in two spectral bands is obtained through each filter group in the K groups; or, The H is 3, the N is 8, the K group includes at least one first group and at least one second group, the optical signal of the target object in 4 spectral bands is obtained through the filters in the first group, and the optical signal of the target object in 2 spectral bands is obtained through the filters in the second group; or, The H is 3, the N is 8, and the optical signal of the target object in three spectral bands is obtained through each filter group in the K groups.

20. The method according to claim 19, characterized in that The non-image capturing mode is any one of the following modes: an object recognition mode, a light source estimation mode, and a depth detection mode.

21. A computer program product, characterized in that When the method is executed on a computer, the method enables the computer to execute the method according to claim 17 or 18, or enables the computer to execute the method according to claim 19 or 20.

Images