Filter Structure, Shooting Method, Device, Terminal and Storage Medium

Abstract

The present disclosure relates to a filter structure, a shooting method, a device, a terminal, and a storage medium. The filter structure includes a plurality of filter unit groups, and each filter unit group includes a plurality of filters; in each filter unit group, the plurality of filters are used to transmit light in a plurality of set wavelength bands, and the plurality of filters correspond to the plurality of set wavelength bands one by one; among the plurality of set wavelength bands, at least two set wavelength bands correspond to the same set channel. In this filter structure, a plurality of set wavelength bands with a narrower wavelength range are set, which can meet different requirements and improve the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to a filter structure, a shooting method, a device, a terminal, and a storage medium. Background Art

[0002] In an image sensor in a current terminal such as a mobile phone, within the wavelength band of the same color channel, only one image can be captured, which has great limitations, restricts the diversification of shooting effects and shooting results, generally can only meet the daily shooting needs, and it is very difficult to improve the imaging effect. Summary of the Invention

[0003] To overcome the problems in the related art, the present disclosure provides a filter structure, a shooting method, a device, a terminal, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a filter structure for an image sensor, the filter structure including a plurality of filter unit groups, and each of the filter unit groups including a plurality of filters;

[0005] In each of the filter unit groups, the plurality of filters are configured to transmit light of a plurality of set wavelength bands, and the plurality of filters correspond one-to-one to the plurality of set wavelength bands;

[0006] Among the plurality of set wavelength bands, at least two of the set wavelength bands correspond to the same set channel.

[0007] Optionally, each of the filter unit groups includes a plurality of filter units, the plurality of filter units are configured to transmit light of the plurality of set channels, and each of the filter units includes a plurality of the filters;

[0008] In each of the filter unit groups, at least one of the set channels corresponds to a plurality of the filters.

[0009] Optionally, in each of the filter unit groups, all the filters in at least one of the filter units are configured to transmit light of the same set channel; and / or,

[0010] In each of the filter units, the plurality of filters correspond to at least two of the set channels.

[0011] Optionally, in each of the filter unit groups, the plurality of set wavelength bands corresponding to the set channel corresponding to the plurality of filters are linearly distributed.

[0012] Optionally, in each of the filter unit groups, the plurality of set wavelength bands are linearly distributed.

[0013] Optionally, the plurality of set channels includes at least one of the following: a near-infrared light channel, a red light channel, a green light channel, and a blue light channel.

[0014] According to a second aspect of embodiments of the present disclosure, there is provided an image sensor, which includes a filter structure of the image sensor as described in the first aspect.

[0015] According to a third aspect of embodiments of the present disclosure, there is provided a terminal, which includes the image sensor as described in the second aspect.

[0016] According to a fourth aspect of embodiments of the present disclosure, there is provided a shooting method, which is applied to a terminal, and the method includes:

[0017] Obtain a plurality of sub-images; wherein, each of the sub-images corresponds to a set wavelength band, and among the plurality of set wavelength bands, at least two of the set wavelength bands correspond to the same set channel;

[0018] Determine a shooting result according to the plurality of sub-images.

[0019] Optionally, the determining a shooting result according to the plurality of sub-images includes:

[0020] Based on a received first control instruction, determine at least one target sub-image corresponding to a target object; wherein, the target sub-image refers to a sub-image used to determine the contour information of the target object;

[0021] Determine the contour information of the target object according to the at least one target sub-image; wherein, the contour information refers to information used to determine the contour of the target object;

[0022] Determine a highlighted image of the target object according to the contour information and the plurality of sub-images; wherein, the highlighted image is used as the shooting result.

[0023] Optionally, the determining at least one target sub-image corresponding to a target object based on a received first control instruction includes:

[0024] Based on a received first control instruction, determine a target pixel region corresponding to the target object; wherein, the target pixel region refers to a region in the sub-image where the pixels for displaying the target object are located;

[0025] Determine at least one target sub-image according to the plurality of target pixel regions and the plurality of sub-images; wherein, the target sub-image refers to a sub-image in which the brightness information of more than a set proportion of the pixels in the target pixel region is less than or equal to a brightness threshold.

[0026] Optionally, determining the contour information of the object according to the at least one target sub-image includes:

[0027] Determining the contour information of the target object according to the luminance information of the at least one target sub-image.

[0028] Optionally, determining the highlighted image of the target object according to the contour information and the multiple sub-images includes:

[0029] Performing a fusion process on the multiple sub-images to determine a base image;

[0030] Determining the highlighted image of the target object according to the contour information and the base image.

[0031] Optionally, determining the shooting result according to the multiple sub-images includes:

[0032] Determining an enhancement band;

[0033] Obtaining at least one enhancement sub-image corresponding to the enhancement band;

[0034] Performing a fusion process on the multiple sub-images to determine a base image;

[0035] Performing a fusion process on the at least one enhancement sub-image and the base image to determine an enhanced image; wherein, the enhanced image is used as the shooting result, and the enhanced image refers to an image in which the image information of the enhancement band is improved for the base image.

[0036] Optionally, determining the shooting result according to the multiple sub-images includes:

[0037] Determining the spectral characteristic curve of the object to be recognized according to the multiple sub-images;

[0038] Determining the type of the object to be recognized according to the spectral characteristic curve and a first pre-stored mapping table; wherein, the type of the object to be recognized is used as the shooting result, and the first pre-stored mapping table includes the corresponding relationship between the spectral characteristic curve and the type of the object.

[0039] Optionally, determining the shooting result according to the multiple sub-images includes:

[0040] Determining the spectral characteristic curve of the object to be determined according to the multiple sub-images;

[0041] Determining the composition of the object to be determined according to the spectral characteristic curve and a second pre-stored mapping table; wherein, the composition of the object to be determined is used as the shooting result, and the second pre-stored mapping table includes the corresponding relationship between the spectral characteristic curve and the composition of the object.

[0042] According to a fifth aspect of the embodiments of the present disclosure, a photographing device is provided, which is applied to a terminal. The device includes:

[0043] An acquisition module, configured to acquire a plurality of sub-images; wherein, each of the sub-images corresponds to a set band, and among the plurality of set bands, at least two of the set bands correspond to the same set channel;

[0044] A determination module, configured to determine a photographing result according to the plurality of sub-images.

[0045] Optionally, the determination module is further configured to:

[0046] Based on a received first control instruction, determine at least one target sub-image corresponding to a target object; wherein, the target sub-image refers to a sub-image used to determine the contour information of the target object;

[0047] Determine the contour information of the target object according to the at least one target sub-image; wherein, the contour information refers to information used to determine the contour of the target object;

[0048] Determine a highlighted image of the target object according to the contour information and the plurality of sub-images; wherein, the highlighted image serves as the photographing result.

[0049] Optionally, the determination module is further configured to:

[0050] Based on a received first control instruction, determine a target pixel region corresponding to a target object; wherein, the target pixel region refers to a region in the sub-image where the pixels for displaying the target object are located;

[0051] Determine at least one target sub-image according to the plurality of target pixel regions and the plurality of sub-images; wherein, the target sub-image refers to a sub-image in which the brightness information of more than a set proportion of the pixels in the target pixel region is less than or equal to a brightness threshold.

[0052] Optionally, the determination module is further configured to:

[0053] Determine the contour information of the target object according to the brightness information of the at least one target sub-image.

[0054] Optionally, the determination module is further configured to: perform a fusion process on the plurality of sub-images to determine a base image;

[0055] Determine a highlighted image of the target object according to the contour information and the base image.

[0056] Optionally, the determination module is further configured to:

[0057] Determine the enhancement band;

[0058] Obtain at least one enhanced sub-image corresponding to the enhancement band;

[0059] Fuse the multiple sub-images to determine a base image;

[0060] Fuse the at least one enhanced sub-image with the base image to determine an enhanced image; wherein, the enhanced image is used as the shooting result, and the enhanced image refers to an image in which the image information of the enhancement band is improved for the base image.

[0061] Optionally, the determining module is further configured to:

[0062] Determine the spectral characteristic curve of the object to be recognized according to the multiple sub-images;

[0063] Determine the type of the object to be recognized according to the spectral characteristic curve and a first pre-stored mapping table; wherein, the type of the object to be recognized is used as the shooting result, and the first pre-stored mapping table includes the corresponding relationship between the spectral characteristic curve and the type of the object.

[0064] Optionally, the determining module is further configured to:

[0065] Determine the spectral characteristic curve of the object to be determined according to the multiple sub-images;

[0066] Determine the composition of the object to be determined according to the spectral characteristic curve and a second pre-stored mapping table; wherein, the composition of the object to be determined is used as the shooting result, and the second pre-stored mapping table includes the corresponding relationship between the spectral characteristic curve and the composition of the object.

[0067] According to the sixth aspect of the embodiments of the present disclosure, there is provided a terminal, characterized in that the terminal includes:

[0068] A processor;

[0069] A memory for storing instructions executable by the processor;

[0070] Wherein, the processor is configured to execute the shooting method as described in the fourth aspect.

[0071] According to the seventh aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the terminal, enabling the terminal to execute the shooting method as described in the fourth aspect.

[0072] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: In this filter structure, a plurality of more refined set wavelength bands are provided, and the wavelength band of the light passing through each filter is narrower. Therefore, more subdivided sub-images can be obtained, and then the obtained multiple sub-images are processed according to requirements, so as to meet different requirements and improve the user experience.

[0073] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present invention and used together with the specification to explain the principles of the present invention.

[0075] Figure 1 is a schematic diagram of the principle of a filter structure shown according to the related art.

[0076] Figure 2 is a schematic diagram of the principle of a filter structure shown according to an exemplary embodiment.

[0077] Figure 3 is a schematic diagram of the transmittance of a filter structure shown according to an exemplary embodiment.

[0078] Figure 4 is a flowchart of a shooting method shown according to an exemplary embodiment.

[0079] Figure 5 is a flowchart of a shooting method shown according to an exemplary embodiment.

[0080] Figure 6 is a flowchart of a shooting method shown according to an exemplary embodiment.

[0081] Figure 7 is a flowchart of a shooting method shown according to an exemplary embodiment.

[0082] Figure 8 is a flowchart of a shooting method shown according to an exemplary embodiment.

[0083] Figure 9 is a flowchart of a shooting method shown according to an exemplary embodiment.

[0084] Figure 10 is a flowchart of a shooting method shown according to an exemplary embodiment.

[0085] Figure 11 is a flowchart of a shooting method shown according to an exemplary embodiment.

[0086] Figure 12It is a block diagram of a photographing device shown according to an exemplary embodiment.

[0087] Figure 13 It is a block diagram of a terminal shown according to an exemplary embodiment. Detailed implementation manners

[0088] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0089] In the related art, as shown in Figure 1 , in the Bayer array pixel structure of an image sensor, relying on the arrangement of color filters, the light in nature is disassembled into three channels of red, green, and blue to sense the light in corresponding bands. In this filter structure, each filter unit group includes four filters, namely a, b, c, and d in sequence, where a, b, and d are color filters, c is a transparent filter, a corresponds to the red light channel, b corresponds to the green light channel, d corresponds to the blue light channel, and c is used to transmit all visible light. Then, a complete color image is obtained through algorithms such as demosaicing and white balance.

[0090] Since in this image sensor, the filters in each filter unit group are only roughly divided into filters corresponding to the light in the three channels of red, green, and blue and filters corresponding to visible light, without further finer division, images corresponding to more finely divided bands cannot be obtained, and more detailed image information cannot be obtained, resulting in a relatively single function, being only applicable to conventional photographing, and also having a poor imaging effect.

[0091] The present disclosure provides a filter structure for an image sensor. In this filter structure, a plurality of more refined set bands are provided, and the band of light passing through each filter is narrower. Therefore, more finely divided sub-images can be obtained. Then, according to requirements, the obtained multiple sub-images are processed, which can meet different requirements and improve the user experience. For example, a spectral characteristic curve of a photographed object can be generated through multiple sub-images for object recognition; for example, according to the different brightnesses of sub-images formed by different objects in different bands, direct local highlighting processing can be performed on the photographed object during the photographing process, etc.

[0092] In one exemplary embodiment, a filter structure is provided. Refer to Figure 2 and 3As shown, the filter structure includes a plurality of filter unit groups, and each filter unit group includes a plurality of filters. In each filter unit group, the plurality of filters are used to transmit light of a plurality of set wavelength bands, and the plurality of filters correspond to the plurality of set wavelength bands one by one. That is to say, each filter corresponds to a set wavelength band, and the set wavelength bands corresponding to each filter are all different, and the number of filters is the same as the number of types of set wavelength bands. Among them, among the plurality of set wavelength bands, at least two set wavelength bands correspond to the same set channel. That is to say, among the plurality of filters, at least two filters correspond to the same set channel.

[0093] It can be understood that in this exemplary embodiment, the same set channel can be a near-infrared light channel. At this time, the set wavelength bands can include the wavelength band where the near-infrared light is located, and at least two set wavelength bands are set within the wavelength band where the near-infrared light is located; or, in this exemplary embodiment, the same set channel can also be a color channel within the visible light range. At this time, the set wavelength bands can include the wavelength band where a certain color light in the visible light is located, and at least two set wavelength bands are set within the wavelength band where the certain color light is located. Thus, richer image information can be obtained, more shooting functions can be realized, more diverse shooting results can be obtained according to requirements, so as to better meet the needs of users and improve the user experience.

[0094] In this filter structure, two sub-images can be obtained within the same set channel. For example, in the traditional solution, only one sub-image within the blue light channel can be obtained. At this time, only the general information of the blue light reflected by the shooting object can be obtained. If at least two set wavelength bands correspond to the same set channel, which is the blue light channel, two sub-images can be obtained within the wavelength band corresponding to the blue light channel, that is, the image information of two sub-bands within the wavelength range corresponding to the blue light can be obtained, and then the sub-image within the blue light channel can be subdivided to better determine richer image information, which is convenient for image analysis and realizes more functions. If the shooting result is an image type result, then more types of images and higher-quality images can be obtained.

[0095] It can be understood that the more set wavelength bands a certain set channel corresponds to, the more sub-images can be obtained, so more detailed image information can also be obtained, and then more image processing functions can be realized and higher-quality shooting results can be obtained. For example, when the shooting result is an image, an image with a better image effect can be obtained. Correspondingly, the more set channels corresponding to a plurality of set wavelength bands, the more detailed image information can be obtained within more set channels, and then more image processing functions can be realized and higher-quality shooting results can be obtained. That is to say, if the wavelength range corresponding to a single set wavelength band is small, more detailed image information can be obtained, and then more image processing functions can be realized and higher-quality shooting results can be obtained.

[0096] In an exemplary embodiment, a filter structure is provided. Refer to Figure 2 and 3 As shown, in this filter structure, each filter unit group includes a plurality of filter units, and the plurality of filter units are used to transmit light of a plurality of set channels. Each filter unit includes a plurality of filters. In each filter unit group, at least one set channel corresponds to a plurality of filters.

[0097] The plurality of set channels here include at least one of the following: a near-infrared light channel, a red light channel, a green light channel, and a blue light channel. That is, the plurality of set channels can be respectively a near-infrared light channel, a red light channel, a green light channel, and a blue light channel, or there can be only one of the above four channels, and the remaining set channels can be set as color channels for transmitting other visible lights, such as a purple light channel for transmitting purple light, to meet different requirements. Of course, if necessary, the set channel can also be set as a far-infrared light channel for transmitting far-infrared light for more complex spectral analysis.

[0098] In this filter structure, a plurality of set channels are provided, and at least one set channel corresponds to a plurality of filters, where "a plurality" means more than one. When an image sensor provided with this filter structure works, sub-images of a plurality of set channels can be obtained, and moreover, a plurality of sub-images can be obtained within at least one set channel. Compared with the prior art, richer image information can be obtained, and more shooting functions can be realized.

[0099] Among them, the structure of the filter unit group includes the following two ways.

[0100] Way 1: In each filter unit group, all the filters in at least one filter unit are used to transmit light of the same said set channel.

[0101] That is to say, in the same filter unit group, it can be that all the filters in one filter unit are used to transmit light of the same set channel; it can also be that all the filters in a plurality of filter units are used to transmit light of the same set channel; even more, all the filters in each filter unit are used to transmit light of the same set channel, that is, a plurality of filter units correspond one-to-one to a plurality of set channels.

[0102] In one embodiment, refer to Figure 2 As shown, all the filters in each filter unit are used to transmit light of the same set channel.

[0103] In the filter unit group, the set channels corresponding to each filter unit group include a near-infrared light channel for transmitting near-infrared light, a blue light channel (B channel) for transmitting blue light, a red light channel (R channel) for transmitting red light, and a green light channel (G channel) for transmitting green light. Each filter unit group includes four filter element units, which are sequentially used to transmit the light of the above-mentioned near-infrared light channel, blue light channel, red light channel, and green light channel, that is, sequentially used to transmit near-infrared light, blue light, red light, and green light. Each filter element unit includes four filters. In the filter element unit for transmitting near-infrared light, all four filters are used to transmit near-infrared light. In the filter element unit for transmitting blue light, all four filters are used to transmit blue light, and so on.

[0104] In this embodiment, it is equivalent that each set channel corresponds to four set wavelength bands, that is, four sub-images can be obtained within each set channel. Compared with the prior art, only one sub-image can be obtained in each set channel. Obviously, this embodiment can obtain richer and more accurate image information, which is very helpful for improving the shooting effect. In addition, due to obtaining richer image information, more functions can be realized and more shooting results can be obtained. For example, in this embodiment, since 16 set wavelength bands are set, 16 sub-images corresponding to the 16 set wavelength bands can be obtained. According to the 16 sub-images, a spectral characteristic curve is determined, and then according to the spectral characteristic curve, the type or composition of the corresponding object can be determined, that is, the shooting result of the type or composition of the object can be obtained by using this filter structure.

[0105] Embodiment 2: In each filter element unit, multiple filters correspond to at least two set channels.

[0106] That is to say, in each filter element unit, it can be that some filters correspond to the same set channel, and some other filters correspond to another set channel; or it can be that each filter corresponds to a different set channel. When each filter corresponds to a different set channel, the number of set channels corresponding to each filter unit group and the number of set channels corresponding to multiple filters in each filter element unit can be the same or different.

[0107] In one embodiment (this embodiment is not shown in the figure), in each filter element unit, each filter corresponds to a different set channel, and the number of set channels corresponding to each filter unit group is the same as the number of set channels corresponding to multiple filters in each filter element unit.

[0108] Each filter unit group corresponds to four set channels, namely the near-infrared light channel, the blue light channel, the red light channel, and the green light channel in sequence. Each filter unit group includes four filter element units, and the four filter element units also correspond to the above four set channels. Each filter element unit includes four filters, and in each filter element unit, each filter also corresponds to the above four set channels.

[0109] Specifically, each filter unit group includes 16 filters, which are respectively denoted as a1, a2, a3, a4, b1, b2, b3, b4, c1, c2, c3, c4, d1, d2, d3, and d4. Among them, a1, a2, a3, and a4 correspond to the red light channel, b1, b2, b3, and b4 correspond to the green light channel, c1, c2, c3, and c4 correspond to the near-infrared light channel, and d1, d2, d3, and d4 correspond to the blue light channel. Among the four filter element units, the first filter element unit includes filters a1, b1, c1, and d1 respectively, the second filter element unit includes filters a2, b2, c2, and d2 respectively, the third filter element unit includes filters a3, b3, c3, and d3 respectively, and the fourth filter element unit includes filters a4, b4, c4, and d4 respectively.

[0110] In an exemplary embodiment, a filter structure is provided. The filter structure includes a plurality of filter unit groups, and each filter unit group includes one of the following two structures:

[0111] Structure 1 (this structure is not shown in the figure): Each filter unit group includes a plurality of filters. In each filter unit group, the plurality of filters are used to transmit light of a plurality of set bands distributed linearly, and the filters correspond to the set bands.

[0112] In this structure, in each filter unit group, each filter is only used to transmit light of one set band, and moreover, the plurality of set bands corresponding to the plurality of filters are distributed linearly, so that sub-images corresponding to the plurality of set bands distributed linearly can be generated, and then the above-mentioned plurality of sub-images are processed to meet different user requirements.

[0113] Structure 2: Each filter unit group includes a plurality of filter element units. The plurality of filter element units are used to transmit light of a plurality of set channels. Each filter element unit includes a plurality of filters. The plurality of filters are used to transmit light of a plurality of set bands, and the filters correspond to the set bands. Moreover, in each filter unit group, the plurality of set bands corresponding to the same set channel are distributed linearly.

[0114] In this structure, there are at least two ways.

[0115] Way 1: Refer to Figure 2 and 3As shown, in each filter unit group, each filter unit is used to transmit light of the same set channel, and the filter unit corresponds to the set channel. That is to say, the number of filter units included in each filter unit group is the same as the number of set channels, and each filter unit corresponds to one set channel.

[0116] For example, referring to Figure 2 As shown, the set channels corresponding to each filter unit group include a near-infrared light channel, a red light channel, a green light channel, and a blue light channel. Then, each filter unit group includes four filter units respectively corresponding to the four set channels. In each filter unit, the set bands corresponding to the multiple filters all belong to the band where the corresponding channel is located. And, in each filter unit, the multiple set bands corresponding to the multiple filters are linearly distributed to ensure that more refined multiple sub-images can be determined through this filter structure.

[0117] Method 2 (not shown in the figure): Each filter unit group includes multiple filter units. The multiple filter units are used to transmit light of multiple set channels. Each filter unit includes multiple filters. The multiple filters are used to transmit light of multiple set bands. The filter corresponds to the set band. And, in each filter unit group, the multiple set bands corresponding to the same set channel are linearly distributed. That is to say, the types of set channels corresponding to each filter unit group are the same as the types of set channels corresponding to each filter unit. The number of filters included in each filter unit is the same as the number of types of set channels corresponding to each filter unit group.

[0118] For example, the set channels corresponding to each filter unit group include a near-infrared light channel, a red light channel, a green light channel, and a blue light channel. Then, each filter unit also corresponds to a near-infrared light channel, a red light channel, a green light channel, and a blue light channel. Among them, each filter unit includes four filters respectively corresponding to the above four set channels. And, in each filter unit group, the multiple set bands corresponding to the multiple filters used to transmit light of the same set channel are linearly distributed to ensure that more refined multiple sub-images can be determined through this filter structure.

[0119] It should be noted that the above-mentioned multiple set bands being linearly distributed means being linearly distributed within a certain band. Among them, linear distribution means that the wavelength range of each set band is the same. For example, a set band is set every 10nm / 20nm / 30nm.

[0120] In one embodiment, within the wavelength range of 350 nm - 510 nm, one set band is set every 10 nm, and a total of 16 bands are set, namely 350 nm - 360 nm, 360 nm - 370 nm, 370 nm - 380 nm,..., 480 nm - 490 nm, 490 nm - 500 nm, 500 nm - 510 nm. Among them, each set band may include the end values of the wavelength range within the set band, or may only include the smaller end value, or only include the larger end value, or does not include the end values.

[0121] In this filter structure, more refined set bands are set, and the band of light passing through each filter is narrower. Therefore, more subdivided sub-images can be obtained. Then, according to requirements, the multiple obtained sub-images are processed to meet different requirements and improve the user experience. For example, the spectral characteristic curve of the photographed object can be generated from multiple sub-images for object recognition; for example, the object in the photograph can be directly cropped according to the different brightness of the sub-images formed by different objects in different bands, and so on.

[0122] In an exemplary embodiment, a filter structure is provided. Referring to Figure 2 and 3 as shown, in this filter structure,

[0123] the multiple filters in each filter unit are distributed in an N*N array, where N is an integer greater than or equal to 2; and / or,

[0124] the multiple filter units in each filter unit group are distributed in an M*M array, where M is an integer greater than or equal to 2.

[0125] In this filter structure, each filter unit includes at least 4 filters, and each filter unit group includes at least 16 filters. In the image sensor manufactured using this filter structure, when taking an image, each pixel corresponds to at least 16 sub-images of set bands. Compared with the three sub-images of the traditional R channel, G channel, and B channel, the solution of the present application can obviously obtain more sub-images, perform imaging of the photographed object in more set bands, so as to better analyze the photographed object. At the same time, more accurate and more diverse image processing can be performed according to the 16 sub-images of the set bands to meet different user requirements.

[0126] It should be noted that within the same filter unit group, the wavelength range of the light covered by multiple set bands is generally 300 nm - 1000 nm, or it can also be designed in the range of 8000 - 14000 nm. The material of the corresponding image sensor is made of a material matching the corresponding wavelength, from silicon germanium to vanadium oxide, gallium arsenide, etc.

[0127] In one embodiment, referring to Figure 2 As shown, the filter unit group corresponds to four set channels, namely the near-infrared light channel, the red light channel, the green light channel, and the blue light channel. Each filter unit group includes four filter unit arrays arranged in a 2×2 array, and each filter unit array is respectively used to transmit light of one set channel among the above four set channels. Each filter unit array includes four filters arranged in a 2×2 array, and all four filters are used to transmit light of the same set channel.

[0128] That is to say, each filter unit group includes 16 filters, and the size of each filter is the same to ensure that the light transmission area of each set wavelength band is the same. As shown in the reference figure, they are respectively denoted as a1, a2, a3, a4, b1, b2, b3, b4, c1, c2, c3, c4, d1, d2, d3, and d4. Among them, a1, a2, a3, and a4 correspond to the red light channel, b1, b2, b3, and b4 correspond to the green light channel, c1, c2, c3, and c4 correspond to the near-infrared light channel, and d1, d2, d3, and d4 correspond to the blue light channel. The set wavelength band corresponding to a1 is 600nm±10nm, the set wavelength band corresponding to a2 is 630nm±10nm, the set wavelength band corresponding to a3 is 660nm±10nm, the set wavelength band corresponding to a4 is 690nm±10nm, the set wavelength band corresponding to c1 is 800nm±10nm, the set wavelength band corresponding to c2 is 850nm±10nm, the set wavelength band corresponding to c3 is 900nm±10nm, the set wavelength band corresponding to c4 is 950nm±10nm, the set wavelength band corresponding to b1 is 500nm±10nm, the set wavelength band corresponding to b2 is 530nm±10nm, the set wavelength band corresponding to b3 is 560nm±10nm, the set wavelength band corresponding to b4 is 590nm±10nm, the set wavelength band corresponding to d1 is 400nm±10nm, the set wavelength band corresponding to d2 is 430nm±10nm, the set wavelength band corresponding to d3 is 460nm±10nm, and the set wavelength band corresponding to d4 is 490nm±10nm. The light transmittance of this filter structure refers to Figure 4 As shown.

[0129] In this embodiment, in each filter unit group, multiple filters of the same set channel are arranged together to form a 2×2 array. During operation, sub-images of the same set channel of multiple pixels can be synthesized into one image for output, improving the light input amount of the light corresponding to the set channel. Alternatively, according to the mosaic algorithm, sub-images of the same set channel of multiple pixels can be interpolated to obtain images of other set channels, and then fusion processing can be performed to achieve high-resolution output and improve the detail performance.

[0130] In this embodiment, the wavelength range of the light transmitted through the conventional filter is narrowed, that is, in this embodiment, the wavelength range of the band corresponding to the filter in the set band ratio technology is narrower. Specifically, the bands corresponding to the four channels of the original near-infrared light channel, red light channel, green light channel, and blue light channel are linearly distributed and subdivided to facilitate more refined and diversified shooting processing.

[0131] It should be noted that the filter structure provided in this application can be used not only in the field of cameras, but also in other technical fields, such as petrochemical, medical and health, environmental protection, metallurgy, geological exploration and other fields. As long as image information acquisition is required, the filter structure proposed in this application can be applied.

[0132] In addition, the filter structure mainly subdivides the band into a narrower range, so that the wavelength range of the set band corresponding to each filter is narrower. By setting multiple set bands, more detailed, rich and accurate image information can be obtained, so as to achieve more functions and obtain more shooting results. That is to say, the arrangement form of the multiple filters in the filter structure can also be other forms than the forms introduced above, as long as it can be ensured that each filter corresponds to a set band, and among the multiple set bands, at least two set bands correspond to the same set channel.

[0133] The present disclosure also proposes an image sensor, including the above filter structure, so that the image sensor has an effect corresponding to the above filter structure.

[0134] The present disclosure also proposes a terminal, which is, for example, a device with a shooting function such as a mobile phone, a camera, a camera, a notebook computer, a tablet computer, etc. The terminal includes the above image sensor, so that the terminal has an effect corresponding to the above image sensor, that is to say, the terminal has an effect corresponding to the above filter structure.

[0135] In one embodiment, the terminal is a mobile phone. In the image sensor included in the mobile phone, refer to Figure 2 and 3 As shown, the filter unit group corresponds to four set channels, namely the near-infrared light channel, the red light channel, the green light channel, and the blue light channel. Each filter unit group includes four filter unit arrays arranged in a 2*2 array, and each filter unit is respectively used to transmit the light of one set channel among the above four set channels. Each filter unit includes four filters arranged in a 2-* array, and the four filters are all used to transmit the light of the same set channel.

[0136] That is to say, each filter unit group includes 16 filters, and the size of each filter is the same to ensure that the light transmission area of each set wavelength band is the same. As shown in the reference figure, they are respectively denoted as a1, a2, a3, a4, b1, b2, b3, b4, c1, c2, c3, c4, d1, d2, d3 and d4. Among them, a1, a2, a3 and a4 correspond to the red light channels, b1, b2, b3 and b4 correspond to the green light channels, c1, c2, c3, c4 correspond to the near-infrared light channels, and d1, d2, d3 and d4 correspond to the blue light channels. The set wavelength band corresponding to a1 is 600nm ± 10nm, the set wavelength band corresponding to a2 is 630nm ± 10nm, the set wavelength band corresponding to a3 is 660nm ± 10nm, the set wavelength band corresponding to a4 is 690nm ± 10nm, the set wavelength band corresponding to c1 is 800nm ± 10nm, the set wavelength band corresponding to c2 is 850nm ± 10nm, the set wavelength band corresponding to c3 is 900nm ± 10nm, the set wavelength band corresponding to c4 is 950nm ± 10nm, the set wavelength band corresponding to b1 is 500nm ± 10nm, the set wavelength band corresponding to b2 is 530nm ± 10nm, the set wavelength band corresponding to b3 is 560nm ± 10nm, the set wavelength band corresponding to b4 is 590nm ± 10nm, the set wavelength band corresponding to d1 is 400nm ± 10nm, the set wavelength band corresponding to d2 is 430nm ± 10nm, the set wavelength band corresponding to d3 is 460nm ± 10nm, and the set wavelength band corresponding to d4 is 490nm ± 10nm.

[0137] When using this mobile phone for shooting, each pixel receives the reflected light of the object in the corresponding wavelength band and converts it into an electrical signal to determine the image information of each set wavelength band corresponding to each pixel. According to the image information of all pixels corresponding to the same set wavelength band, the sub-image corresponding to this set wavelength band is determined. By flexibly using the sub-images of each set wavelength band, the image or other shooting results required by the user are determined.

[0138] Specifically, this mobile phone can be used in the following scenarios.

[0139] Scenario 1: Complete the daily photo-taking function.

[0140] In this scenario, directly fuse the obtained multiple sub-images to obtain a normal image. For example, it can be used to shoot people, landscapes, buildings, artworks, etc. The specific shooting method in this scenario is the same as the principle of the traditional shooting method, so it will not be elaborated here.

[0141] Scenario 2: Implement the multi-spectral image acquisition function. In this scenario, it can be used for object recognition and component determination.

[0142] Determine the sub-images of an object under different preset wavelength bands. For example, when photographing an apple, determine the sub-images of different preset wavelength bands reflected from the apple's surface, and then perform spectral analysis on the specified positions of the sub-images of each preset wavelength band to determine the spectral characteristic curves at these positions. By comparing with a pre-stored mapping table, determine the object type corresponding to this position, or determine the composition corresponding to this position. Among them, the pre-stored mapping table includes the mapping relationship between the spectral characteristic curve and the object type, or includes the mapping relationship between the spectral characteristic curve and the object composition.

[0143] Specifically, object recognition refers to determining the object type based on the different intensities of light reflected from the object surface detected by the image sensor at different wavelength bands, and the intensity matches the characteristics of the object's inherent color. For example, distinguish apples, pears, oranges, etc.

[0144] Composition determination refers to determining the object composition based on the intensity of light reflected by the object at different wavelength bands, such as the sugar content and acidity of an apple, or determining internal defects of the object, such as whether an apple has deteriorated.

[0145] Scenario 3: Implement the image enhancement function.

[0146] Since more refined preset wavelength bands are set in the filter unit group, different image enhancement effects can be achieved through the sub-images of different wavelength bands.

[0147] For example, at night, there is little visible light. At this time, use the sub-images of wavelength bands such as 850nm or 940nm and nearby wavelength bands (wavelength bands corresponding to the near-infrared light channel) as the enhancement sub-images, and perform fusion processing with the sub-images of the wavelength band where visible light is located to achieve the night scene enhancement effect.

[0148] For another example, during the day, use the sub-images of the wavelength band where red light is located (wavelength band corresponding to the red light channel) as the enhancement sub-images, and perform fusion processing with the sub-images of the wavelength band where visible light is located to achieve the enhancement effect of a single channel to achieve a specific filter effect.

[0149] Scenario 4: Achieve the highlighting effect of a specific target.

[0150] After determining the sub-images of each preset wavelength band, by comparing the sub-images, the contour image of the target object to be highlighted can be determined, and then the contour image is fused with multiple sub-images to highlight the target object, cut out the target object, and blur the background.

[0151] Due to the different absorption and reflection of light of different wavelengths by the material of the object itself, there will be obvious differences (such as brightness differences) in the multiple sub-images obtained at the junction of objects with different materials. Based on the above differences, the contour of the target object can be determined.

[0152] For example, photograph a green plant in a flowerpot. The green plant absorbs green light and reflects light of other colors. Since the material of the flowerpot is different from that of the green plant, in the multiple sub-images obtained by the mobile phone, in the wavelength band where green light is located, the image information of the green plant is weak, while the image information of the flowerpot is strong. Thus, based on the fact that the brightness value of the pixel points corresponding to the green plant in the green light channel is low, the contour image of the green plant can be determined to achieve prominent display of the green plant, or the green plant can be directly cut out to achieve an accurate image cut-out effect.

[0153] Specifically, for example, if the obtained sub-images are three, namely the first sub-image, the second sub-image, and the third sub-image, by comparing the first sub-image with the second sub-image to determine the contour information of the target object, and then matching the result with the fused image of the first sub-image, the second sub-image, and the third sub-image, the main body of the target object can be cut out to achieve a more accurate and faster image cut-out effect.

[0154] Of course, in actual use, more than three sub-images will be obtained. More sub-images can determine more accurate contour information, and thus achieve a more precise image cut-out.

[0155] Scenario Five: Flexibly extract an image in a certain wavelength band.

[0156] According to the user's needs, in the multiple obtained sub-images, the sub-images in the corresponding set wavelength band can be directly extracted.

[0157] Scenario Six: Extract images with different resolutions.

[0158] According to the user's resolution requirements, the multiple obtained sub-images are appropriately combined to determine the image with the required resolution. For example, in each of the multiple set wavelength bands corresponding to a set channel, some of the sub-images corresponding to the set wavelength bands can be extracted and then fused to obtain an image with a lower resolution. Another example is to directly fuse all the sub-images to obtain an image with a higher resolution.

[0159] In this terminal, while ensuring the ordinary shooting function, it can bring more abundant functions, with low cost and high feasibility, enhancing the user experience.

[0160] The present disclosure also proposes a shooting method applied to the above terminal. Specifically, as shown in Figure 4 the method includes:

[0161] S11. Obtain multiple sub-images; where each sub-image corresponds to a set wavelength band, and among the multiple set wavelength bands, at least two set wavelength bands correspond to the same set channel;

[0162] S12. Determine the shooting result according to the multiple sub-images.

[0163] In step S11, first, the image information of each pixel in each set band is determined, and then, according to the image information of all pixels in the same set band, a sub-image corresponding to the set band is determined to obtain sub-images of each set band. Multiple sub-images of set bands that are more accurately linearly distributed can be obtained, and then corresponding features of the photographed object are determined through different processing methods to better obtain the corresponding photographing result.

[0164] In one embodiment, multiple set bands corresponding to each pixel are linearly distributed to obtain multiple sub-images of set bands that are more accurately linearly distributed, and then corresponding features of the photographed object are determined through different processing methods to better obtain the corresponding photographing result.

[0165] In one embodiment, each pixel corresponds to multiple set channels, and among the multiple set bands of each pixel, the multiple set bands belonging to the same set channel are linearly distributed. Multiple sub-images of a certain set channel can be obtained, more detailed and accurate image information of the set channel can be determined, and then corresponding features of the photographed object are determined through different processing methods to better obtain the corresponding photographing result.

[0166] In step S12, corresponding setting methods can be determined by setting different control instructions, that is, different control instructions correspond to different setting methods, and then different photographing results can be achieved. The corresponding relationship between the control instructions and the setting methods is preset in the terminal. For example, the setting method corresponding to the highlighting sub-control instruction is used to highlight the image of the target object corresponding to the highlighting sub-control instruction. For another example, the setting method corresponding to the enhanced waveband sub-control instruction is used to perform image enhancement processing on the sub-image of the waveband to be enhanced by using the sub-image of the enhanced waveband, and so on.

[0167] Among them, the photographing result is not limited to the result of the image type in the prior art. In this method, the photographing result includes at least one of the following: the basic image obtained by regular photographing, the highlighting image highlighting a certain object, the enhanced image with improved image information of a certain waveband, the type of the photographed object, the composition of the photographed object, etc.

[0168] In this photographing method, more refined set bands are set, more subdivided sub-images can be obtained, and then the multiple obtained sub-images are processed according to requirements, so as to meet different requirements and improve the user experience.

[0169] In an exemplary embodiment, a photographing method is provided. This method is an improvement on step S12 in the above method. Specifically, referring to Figure 5 as shown, according to multiple sub-images, the photographing result is determined, including:

[0170] S21. Determine the enhancement band;

[0171] S22. Obtain at least one enhanced sub-image corresponding to the enhancement band;

[0172] S23. Perform fusion processing on multiple sub-images to determine the base image;

[0173] S24. Perform fusion processing on at least one enhanced sub-image and the base image to determine the enhanced image; wherein, the enhanced image is used as the shooting result, and the enhanced image refers to an image that has enhanced the image information of the base image in the enhancement band.

[0174] In step S21, the enhancement band can be determined according to the user's selection. For example, according to the second control instruction input by the user for determining the enhancement band, the corresponding enhancement band is determined. The enhancement band can also be determined according to the current environmental information. For example, when the current environment is at night, in order to achieve the shooting effect of night scene enhancement, the enhancement band is automatically determined to be the band where visible light is located, and night scene enhancement is performed through sub-images of multiple set bands in the band where visible light is located.

[0175] In one embodiment, when shooting a night image and the user wants to perform night scene enhancement, the user can select the function corresponding to "night scene enhancement" on the shooting interface. Based on the user's selection, the terminal determines to execute the night scene enhancement function, and then determines the enhancement band (for example, the band corresponding to near-infrared light).

[0176] In one embodiment, when shooting an image at night, the terminal automatically identifies the current shooting scene as a night shooting according to the current time, and can automatically start the night scene enhancement function, and then determines the enhancement band (for example, the band corresponding to near-infrared light).

[0177] In step S22, after determining the enhancement band, search for the set band corresponding to the enhancement band from multiple set bands. Since the wavelength range set by the set band is small, there may be a situation where the enhancement band includes multiple set bands. If the enhancement band corresponds to only one set band, only one enhanced sub-image is obtained; if the enhancement band corresponds to multiple set bands, multiple enhanced sub-images are obtained. It can be understood that the narrower the wavelength range of the set band, the more set bands the enhancement band corresponds to, and the more enhanced sub-images for enhancement are obtained. That is to say, the more abundant the image information for enhancement is obtained, and more accurate and better image enhancement can be achieved.

[0178] In this method, step S23 can also be performed simultaneously with step S22, or prior to step S22. Of course, step S23 can also be performed simultaneously with step S21, or prior to step S21. In step S23, the method of fusing multiple sub-images to determine the base image can be directly implemented by existing technologies and will not be elaborated here. It should be noted that when obtaining the base image, all sub-images can be fused to obtain it, or only some sub-images can be fused to obtain it. The specific sub-images used for fusion processing can be determined according to the user's selection.

[0179] In step S24, the enhancement sub-images can be fused with the multiple sub-images to determine the enhanced image. According to the enhancement requirements, determine the number of images in the enhancement sub-images participating in the fusion processing. It can be understood that the higher the enhancement requirements, the more enhancement sub-images are used for the fusion processing.

[0180] Of course, if the terminal has a normal shooting function and a base image obtained by normal shooting can be determined each time shooting is performed, in this embodiment, the enhanced image can also be determined by fusing the enhancement sub-images with the base image obtained by normal shooting. It can be understood that the base image is also determined according to some or all of the multiple sub-images.

[0181] In one embodiment, during night shooting, since there is little visible light, sub-images in a set band near a wavelength of 850 nm or a set band near a wavelength of 940 nm (the band corresponding to near-infrared light) are determined as the enhancement sub-images to achieve the night scene enhancement effect.

[0182] In one embodiment, sub-images in the band where red light is located are used as the enhancement sub-images to achieve the enhancement effect of a single channel to achieve a specific filter effect.

[0183] In this method, different image enhancement effects can be achieved through sub-images of different set bands to meet different needs of users.

[0184] In an exemplary embodiment, a shooting method is provided. This method is an improvement of step S12 in the above method. Specifically, as shown in Figure 6 According to multiple sub-images, determine the shooting result, including:

[0185] S31. Determine the spectral characteristic curve of the object to be recognized according to multiple sub-images;

[0186] S32. Determine the type of the object to be recognized according to the spectral characteristic curve and the first pre-stored mapping table; wherein, the first pre-stored mapping table includes the correspondence between the spectral characteristic curve and the type of the object.

[0187] The intensities of light of different wavelength bands reflected by different objects are different. Based on this characteristic of the object, the type of the object can be determined, such as differentiating apples, pears, oranges, etc.

[0188] In this method, a first pre-stored mapping table is pre-stored in the terminal in advance. The first pre-stored mapping table includes the correspondence between the spectral characteristic curve and the type of the object. The above-mentioned first pre-stored mapping table can determine the correspondence between the spectral characteristic curve and the type of the object based on multiple tests. Then, the method is used to determine multiple sub-images of the object to be recognized in multiple set wavelength bands, and then the spectral characteristic curve of the object to be recognized is determined based on the above-mentioned multiple sub-images. The spectral characteristic curve corresponding to the spectral characteristic curve of the object to be recognized is searched from the first pre-stored mapping table, and then the type of the object to be recognized is determined.

[0189] In an exemplary embodiment, a shooting method is provided. This method is an improvement on step S12 in the above method. Specifically, refer to Figure 7 As shown, based on multiple sub-images, the shooting result is determined, including:

[0190] S41. Determine the spectral characteristic curve of the object to be determined based on multiple sub-images;

[0191] S41. Determine the composition of the object to be determined based on the spectral characteristic curve and the second pre-stored mapping table; wherein, the second pre-stored mapping table includes the correspondence between the spectral characteristic curve and the composition of the object.

[0192] The intensities of light of different wavelength bands reflected by different compositions are different. Based on this characteristic, the composition of the object can be determined, such as determining the sugar content and acidity of an apple, or determining whether the apple has deteriorated, etc.

[0193] In this method, a second pre-stored mapping table is pre-stored in the terminal in advance. The second pre-stored mapping table includes the correspondence between the spectral characteristic curve and the composition of the object. The above-mentioned second pre-stored mapping table can determine the correspondence between the spectral characteristic curve and the composition of the object based on multiple tests. Then, the method is used to determine multiple sub-images of the object to be determined in multiple set wavelength bands, and then the spectral characteristic curve of the object to be determined is determined based on the above-mentioned multiple sub-images. The spectral characteristic curve corresponding to the spectral characteristic curve of the determined object is searched from the second pre-stored mapping table, and then the composition of the object to be determined is determined.

[0194] In an exemplary embodiment, a shooting method is provided. This method is an improvement on step S12 in the above method. Specifically, refer to Figure 8 As shown, based on multiple sub-images, the shooting result is determined, including:

[0195] S51. Determine at least one target sub-image corresponding to the target object based on the received first control instruction; wherein, the target sub-image refers to the sub-image used to determine the contour information of the target object.

[0196] S52. Determine the contour information of the target object according to at least one target sub-image; wherein, the contour information refers to the information used to determine the contour of the target object.

[0197] S53. Determine the highlighted image of the target object according to the contour information and the multiple sub-images; wherein, the highlighted image is used as the shooting result.

[0198] In step S51, the user can directly select at least one corresponding sub-image from the multiple sub-images and determine it as at least one target sub-image corresponding to the target object.

[0199] For example, the user can directly input a certain set channel and use the sub-image corresponding to the set channel as the target sub-image.

[0200] For example, the user can directly input a certain set wavelength band, or some set wavelength bands, and determine the sub-image corresponding to the certain set wavelength band or the some set wavelength bands as the target sub-image.

[0201] For example, the user can directly click on a certain sub-image or some sub-images and determine the clicked sub-images as the target sub-images.

[0202] Of course, the user can also input the first control instruction for determining the target object, first determine the target object, and then automatically determine at least one target sub-image according to the algorithm.

[0203] When shooting, a preview image is displayed on the terminal. According to the position in the preview image clicked by the user, determine the target object corresponding to the above position; or according to the touch trajectory input by the user, determine the object corresponding to the image within the touch trajectory as the target object. It can be understood that the target object can also be determined in other ways in the existing technologies, which will not be elaborated here.

[0204] Specifically, as shown in Figure 9 Based on the received first control instruction, determining at least one target sub-image corresponding to the target object includes:

[0205] S511. Determine the target pixel region corresponding to the target object based on the received first control instruction.

[0206] S512. Determine at least one target sub-image according to the multiple target pixel regions and the multiple sub-images.

[0207] In step S511, each image includes a plurality of pixels. The target pixel region refers to the region where the pixels for displaying the target object are located in the sub-image. In each sub-image, the image information within the target pixel region is the image information of the target object in that sub-image, that is, the image information of the target object in the set wavelength band corresponding to the sub-image, and also the brightness information of the light within the set wavelength band reflected by the target object.

[0208] In one embodiment, when shooting, a preview image is displayed on the terminal. According to the touch trajectory input by the user, the object corresponding to the image within the touch trajectory is determined as the target object. Among all the sub-images, the pixel region corresponding to the region enclosed by the touch trajectory is the target pixel region.

[0209] It should be noted that determining the target pixel region of the target object from an image can also be achieved by other existing technologies, which are not limited herein.

[0210] In step S512, the target sub-image refers to the sub-image in which the brightness information of the pixels in the set proportion of the target pixel region is less than or equal to the brightness threshold. After determining the sub-images corresponding to each set wavelength band and the target pixel region, by comparing the brightness information of the target pixel region of each sub-image with the brightness threshold, at least one target sub-image that can reflect the contour information of the target object can be determined. In this step, the set proportion and the brightness threshold can be determined according to the user's selection, can also be automatically generated according to past records, or can be preset.

[0211] In one embodiment, the determined target pixel region includes ten thousand pixel points, the set proportion is 80%, and the brightness threshold is 100 lux. If in a certain sub-image, among the pixel points within the target pixel region, the brightness values of more than eight thousand pixel points are less than or equal to 100 lux, then this sub-image is determined as the target sub-image. If in a certain sub-image, among the pixel points within the target pixel region, only seven thousand pixel points have brightness values less than or equal to 100 lux, then this sub-image is determined not to be the target sub-image.

[0212] In step S52, after determining at least one target sub-image, the contour information of the target object can be determined according to the brightness information of the at least one target sub-image. Specifically, in the target sub-image, the image of the target object has a large difference in brightness information from the images corresponding to other images. According to the difference in their brightness values, the contour information of the target object in the target sub-image can be determined. The contour information is, for example, the coordinates of the contour trajectory. Then, based on the contour information, the effects of image matting or highlighting the target object can be achieved.

[0213] In step S53, according to the contour information and multiple sub-images, the highlighted display image of the target object is determined, including the following two methods.

[0214] Reference Figure 10 As shown, Method 1:

[0215] S531. Perform fusion processing on multiple sub-images to determine the base image of the multiple sub-images;

[0216] S532. Determine the highlighted image of the target object based on the contour information and the base image.

[0217] In this method, first perform fusion processing on the obtained multiple sub-images to determine the base image, and then determine the highlighted image of the target object based on the contour information and the fused image. For example, perform matte processing on the fused image according to the contour information to determine the boundary between the image of the target object and other images in the base image, and then reduce the brightness value of the area where other images are located, so as to determine a more prominently highlighted image of the target object; when the brightness value of the area where other images are located becomes zero, then determine the image with only the target object. In this method, based on the contour information and the base image information, the operation is simple. When using this method to determine the highlighted image with only the target object, only one matte operation is required to achieve it, the operation is simple, and the matte is more accurate.

[0218] It should be noted that the above base image is the image determined when taking a normal photo. Therefore, in this method, the base image can be directly saved as a normal image for the user to use elsewhere.

[0219] Reference Figure 11 As shown, Method 2:

[0220] S53-1. Determine multiple highlighted sub-images corresponding to the target object according to the contour information and multiple sub-images;

[0221] S53-2. Perform fusion processing on the multiple highlighted sub-images to determine the highlighted image of the target object.

[0222] In this method, the multiple highlighted sub-images correspond one-to-one with the multiple sub-images, that is, one sub-image corresponds to one generated highlighted sub-image. This method determines multiple highlighted sub-images, and all the multiple highlighted sub-images are images with the target object prominently highlighted. Then perform fusion processing on the multiple highlighted sub-images to determine the highlighted image of the target object.

[0223] It should be noted that the number of multiple highlighted sub-images determined in this method is the same as the number of multiple sub-images, and the only difference between them is that the highlighted sub-images prominently highlight the target object. Therefore, these multiple highlighted sub-images can be used to replace the original multiple sub-images for processing to obtain the corresponding shooting results.

[0224] For example, another target object can be highlighted for the multiple highlighted sub-images, and a highlighted image with both target objects highlighted can be obtained.

[0225] For another example, by performing low-resolution processing on the multiple highlighted sub-images, a highlighted image of the target object with low resolution can be obtained.

[0226] It can be understood that the type of the object or the composition of the object can also be determined based on the above multiple highlighted sub-images. And since the target object has been highlighted, the type of the object or the composition of the object can be determined more accurately.

[0227] In an exemplary embodiment, a photographing device is provided. The photographing device is used to implement the above-mentioned photographing method. Refer to Figure 12 As shown, the photographing device includes an acquisition module 101 and a determination module 102. During the implementation process,

[0228] The acquisition module 101 is configured to acquire a plurality of sub-images; wherein, each of the sub-images corresponds to a set wavelength band, and among the multiple set wavelength bands, at least two of the set wavelength bands correspond to the same set channel;

[0229] The determination module 102 is configured to determine a photographing result based on the multiple sub-images.

[0230] In an exemplary embodiment, a photographing device is provided. The device is an improvement on the above device. Specifically, refer to Figure 12 As shown, in this device, the determination module 102 is further configured to:

[0231] Based on a received first control instruction, determine at least one target sub-image corresponding to the target object; wherein, the target sub-image refers to a sub-image used to determine the contour information of the target object;

[0232] Determine the contour information of the target object according to the at least one target sub-image; wherein, the contour information refers to the information used to determine the contour of the target object;

[0233] Determine a highlighted image of the target object according to the contour information and the multiple sub-images; wherein, the highlighted image is used as the photographing result.

[0234] In an exemplary embodiment, a photographing device is provided. The device is an improvement on the above device. Specifically, refer to Figure 12 As shown, in this device, the determination module 102 is further configured to:

[0235] Based on the received first control instruction, determine the target pixel region corresponding to the target object; wherein, the target pixel region refers to the region where the pixels for displaying the target object are located in the sub-image;

[0236] Based on multiple target pixel regions and multiple sub-images, determine at least one target sub-image; wherein, the target sub-image refers to the sub-image in which the brightness information of more than a set proportion of the pixels in the target pixel region is less than or equal to the brightness threshold.

[0237] In an exemplary embodiment, a shooting device is provided. This device is an improvement on the above device. Specifically, refer to Figure 12 As shown, in this device, the determination module 102 is further configured to:

[0238] Based on the brightness information of at least one target sub-image, determine the contour information of the target object.

[0239] In an exemplary embodiment, a shooting device is provided. This device is an improvement on the above device. Specifically, refer to Figure 12 As shown, in this device, the determination module 102 is further configured to:

[0240] Determine the enhancement band;

[0241] Obtain at least one enhancement sub-image corresponding to the enhancement band;

[0242] Perform a fusion process on multiple sub-images to determine a base image;

[0243] Perform a fusion process on at least one enhancement sub-image and the base image to determine an enhanced image; wherein, the enhanced image is used as the shooting result, and the enhanced image refers to an image in which the image information of the enhancement band is improved for the base image.

[0244] In an exemplary embodiment, a shooting device is provided. This device is an improvement on the above device. Specifically, refer to Figure 12 As shown, in this device, the determination module 102 is further configured to:

[0245] Determine the spectral characteristic curve of the object to be recognized according to multiple sub-images;

[0246] Determine the type of the object to be recognized according to the spectral characteristic curve and the first pre-stored mapping table; wherein, the type of the object to be recognized is used as the shooting result, and the first pre-stored mapping table includes the correspondence between the spectral characteristic curve and the type of the object.

[0247] In an exemplary embodiment, a shooting device is provided. This device is an improvement on the above device. Specifically, refer to Figure 12As shown, in this device, the determination module 102 is further configured to:

[0248] Determine the spectral characteristic curve of the object to be determined according to multiple sub-images;

[0249] Determine the composition of the object to be determined according to the spectral characteristic curve and the second pre-stored mapping table; wherein, the composition of the object to be determined is used as the shooting result, and the second pre-stored mapping table includes the corresponding relationship between the spectral characteristic curve and the composition of the object.

[0250] In an exemplary embodiment, a terminal is provided. The terminal can be a device with a shooting function such as a mobile phone, a computer, a tablet device, a television, etc.

[0251] Reference Figure 13 As shown, the terminal 400 may include one or more of the following components: a processing component 402, a memory 404, a power component 406, a multimedia component 408, an audio component 410, an input / output (I / O) interface 412, a sensor component 414, and a communication component 416.

[0252] The processing component 402 generally controls the overall operation of the device 400, such as operations associated with display, telephone call, data communication, camera operation, and recording operation. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0253] The memory 404 is configured to store various types of data to support the operation of the device 400. Examples of these data include instructions for any application or method operating on the device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0254] The power component 406 provides power for various components of the device 400. The power component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 400.

[0255] The multimedia component 408 includes a screen that provides an output interface between the device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the terminal 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0256] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC) that is configured to receive external audio signals when the device 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 404 or transmitted via the communication component 416. In some embodiments, the audio component 410 further includes a speaker for outputting audio signals.

[0257] The I / O interface 412 provides an interface between the processing component 402 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0258] The sensor component 414 includes one or more sensors for providing an assessment of the various aspects of the status of the terminal 400. For example, the sensor component 414 can detect the open / closed state of the terminal 400, the relative positioning of components, such as the display and the keypad of the terminal 400. The sensor component 414 can also detect a change in the position of a component of the device 400 or the terminal 400, the presence or absence of user contact with the device 400, the orientation or acceleration / deceleration of the device 400, and the temperature change of the device 400. The sensor component 414 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 414 can also include a light sensor, such as a CMOS or a CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 414 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0259] The communication component 416 is configured to facilitate communication, in a wired or wireless manner, between the device 400 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0260] In an exemplary embodiment, the terminal 400 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0261] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 404 including instructions, is also provided. The above instructions can be executed by the processor 420 of the device 400 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the above method.

[0262] Those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include known common knowledge or conventional technical means in the technical field not disclosed in this disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present invention are pointed out by the claims.

[0263] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A filter structure for an image sensor, characterized in that, the filter structure includes a plurality of filter unit groups, and each filter unit group includes a plurality of filters; in each filter unit group, the plurality of filters are used to transmit light of a plurality of set wavelength bands, and the plurality of filters correspond to the plurality of set wavelength bands one by one; among the plurality of set wavelength bands, at least two of the set wavelength bands correspond to the same set channel, and the set channel includes one of the following: near-infrared light channel, red light channel, green light channel, and blue light channel; each filter unit group includes a plurality of filter units, the plurality of filter units are used to transmit light of the plurality of set channels, and each filter unit includes a plurality of the filters; in each filter unit group, all the filters in at least some of the filter units are used to transmit light of the same set channel, and in at least some of the filter units, the wavelength ranges of the set wavelength bands corresponding to the filters in each filter unit are all smaller than the total wavelength range of the light of the set channel, and the wavelength ranges of the set wavelength bands corresponding to the filters in each filter unit are different from each other; the plurality of filters in each filter unit are arranged in an N*N array, where N is an integer greater than or equal to 2; the plurality of filter units in each filter unit group are arranged in an M*M array, where M is an integer greater than or equal to 2.

2. The filter structure for an image sensor according to claim 1, characterized in that, in each filter unit group, at least one of the set channels corresponds to a plurality of the filters.

3. The filter structure for an image sensor according to claim 2, characterized in that, in at least some of the filter units, the plurality of filters correspond to at least two of the set channels.

4. The filter structure for an image sensor according to claim 2, characterized in that, in each filter unit group, the set wavelength bands corresponding to the set channel corresponding to the plurality of filters are linearly distributed.

5. The filter structure for an image sensor according to claim 1, characterized in that, in each filter unit group, the plurality of set wavelength bands are linearly distributed.

6. An image sensor, characterized in that, the image sensor includes the filter structure for an image sensor according to any one of claims 1-5.

7. A terminal, characterized in that, the terminal includes the image sensor according to claim 6.

8. A shooting method, applied to a terminal including an image sensor having the filter structure according to claim 1, characterized in that, the method includes: acquiring a plurality of sub-images; wherein, each sub-image corresponds to a set wavelength band, and among the plurality of set wavelength bands, at least two of the set wavelength bands correspond to the same set channel; determining a shooting result according to the plurality of sub-images.

9. The shooting method according to claim 8, characterized in that, the determining a shooting result according to the plurality of sub-images includes: Based on the received first control instruction, determine at least one target sub-image corresponding to the target object; wherein, the target sub-image refers to the sub-image used to determine the contour information of the target object; Determine the contour information of the target object according to the at least one target sub-image; wherein, the contour information refers to the information used to determine the contour of the target object; Determine the highlighted image of the target object according to the contour information and the multiple sub-images; wherein, the highlighted image is used as the shooting result.

10. The shooting method according to claim 9, wherein, The determining, based on the received first control instruction, at least one target sub-image corresponding to the target object includes: Based on the received first control instruction, determine the target pixel region corresponding to the target object; wherein, the target pixel region refers to the region where the pixels for displaying the target object are located in the sub-image; Determine at least one target sub-image according to the multiple target pixel regions and the multiple sub-images; wherein, the target sub-image refers to the sub-image in which the brightness information of more than a set proportion of the pixels in the target pixel region is less than or equal to the brightness threshold.

11. The shooting method according to claim 9 or 10, wherein, The determining the contour information of the object according to the at least one target sub-image includes: Determine the contour information of the target object according to the brightness information of the at least one target sub-image.

12. The shooting method according to claim 9 or 10, wherein, The determining the highlighted image of the target object according to the contour information and the multiple sub-images includes: Perform a fusion process on the multiple sub-images to determine a base image; Determine the highlighted image of the target object according to the contour information and the base image.

13. The shooting method according to claim 8, wherein, The determining the shooting result according to the multiple sub-images includes: Determine the enhancement band; Obtain at least one enhancement sub-image corresponding to the enhancement band; Perform a fusion process on the multiple sub-images to determine a base image; Perform a fusion process on the at least one enhancement sub-image and the base image to determine an enhanced image; wherein, the enhanced image is used as the shooting result, and the enhanced image refers to the image in which the image information of the enhancement band of the base image is improved.

14. The shooting method according to claim 8, wherein, The determining the shooting result according to the multiple sub-images includes: Determine the spectral characteristic curve of the object to be recognized according to the multiple sub-images; Determine the type of the object to be recognized according to the spectral characteristic curve and the first pre-stored mapping table; wherein, the type of the object to be recognized is used as the shooting result, and the first pre-stored mapping table includes the corresponding relationship between the spectral characteristic curve and the type of the object.

15. The shooting method according to claim 8, wherein, The determining the shooting result according to the multiple sub-images includes: Determine the spectral characteristic curve of the object to be determined according to the multiple sub-images; Determine the composition of the object to be determined according to the spectral characteristic curve and the second pre-stored mapping table; wherein, the composition of the object to be determined is used as the shooting result, and the second pre-stored mapping table includes the corresponding relationship between the spectral characteristic curve and the composition of the object.

16. A shooting device, applied to a terminal including an image sensor with the filter structure as described in claim 1 Characterized in that The device includes: An acquisition module, configured to acquire a plurality of sub-images; wherein, each of the sub-images corresponds to a set wavelength band, and among the plurality of set wavelength bands, at least two of the set wavelength bands correspond to the same set channel; A determination module, configured to determine a shooting result according to the plurality of sub-images.

17. The shooting device according to claim 16 Characterized in that The determination module is further configured to: Based on a received first control instruction, determine at least one target sub-image corresponding to the target object; wherein, the target sub-image refers to a sub-image used to determine the contour information of the target object; Determine the contour information of the target object according to the at least one target sub-image; wherein, the contour information refers to information used to determine the contour of the target object; Determine a highlighted image of the target object according to the contour information and the plurality of sub-images; wherein, the highlighted image is used as the shooting result.

18. The shooting device according to claim 17 Characterized in that The determination module is further configured to: Based on a received first control instruction, determine a target pixel region corresponding to the target object; wherein, the target pixel region refers to a region in the sub-image where the pixels for displaying the target object are located; Determine at least one target sub-image according to the plurality of target pixel regions and the plurality of sub-images; wherein, the target sub-image refers to a sub-image in which the brightness information of more than a set proportion of the pixels in the target pixel region is less than or equal to a brightness threshold.

19. The shooting device according to claim 17 or 18 Characterized in that The determination module is further configured to: Determine the contour information of the target object according to the brightness information of the at least one target sub-image.

20. The shooting device according to claim 17 or 18 Characterized in that The determination module is further configured to: perform a fusion process on the plurality of sub-images to determine a base image; Determine a highlighted image of the target object according to the contour information of the target object and the base image.

21. The shooting device according to claim 16 Characterized in that The determination module is further configured to: Determine an enhancement wavelength band; Acquire at least one enhancement sub-image corresponding to the enhancement wavelength band; Perform a fusion process on the plurality of sub-images to determine a base image; Perform a fusion process on the at least one enhancement sub-image and the base image to determine an enhanced image; wherein, the enhanced image is used as the shooting result, and the enhanced image refers to an image in which the image information of the enhancement wavelength band is improved for the base image.

22. The shooting device according to claim 16 Characterized in that The determination module is further configured to: Determine the spectral characteristic curve of the object to be recognized according to the multiple sub-images; Determine the type of the object to be recognized according to the spectral characteristic curve and the first pre-stored mapping table; wherein, the type of the object to be recognized is used as the shooting result, and the first pre-stored mapping table includes the correspondence between the spectral characteristic curve and the type of the object.

23. The shooting device according to claim 16, characterized in that, the determining module is further configured to: Determine the spectral characteristic curve of the object to be determined according to the multiple sub-images; Determine the composition of the object to be determined according to the spectral characteristic curve and the second pre-stored mapping table; wherein, the composition of the object to be determined is used as the shooting result, and the second pre-stored mapping table includes the correspondence between the spectral characteristic curve and the composition of the object.

24. A terminal, characterized in that, the terminal includes: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to execute the shooting method according to any one of claims 8 to 15.

25. A non-transitory computer-readable storage medium, characterized in that, when the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the shooting method according to any one of claims 8 to 15.

Images