Sensor arrangement, method for calculating a color image and a hyperspectral image, method for performing a white balance and use of the sensor arrangement in medical imaging

Abstract

Sensor arrangement (1) with - an image sensor (2) consisting of several photocells (3), - wherein at least partially a color filter (4) is assigned to each of the photocells (3), - wherein the photocells (3) are grouped into subcells (5) and the subcells (5) into supercells (6), - wherein each subcell (5) has at least one color filter (4) of a color channel (7,R,G,B,A,C), - where all color channels (7,R,G,B,A,C) together cover the entire visible spectrum (8), - wherein the characteristic wavelengths (9) of the color filters (4) belonging to a color channel (7,R,G,B,A,C) differ at least partially from each other between the subcells (5) of a supercell (6) and - wherein the color channels (7,R,G,B,A,C) of the subcells (5) are distributed to the photocells (3) according to a uniform pattern.

Description

[0001] The invention relates to a sensor arrangement with an image sensor comprising several photocells, wherein each photocell is at least partially assigned a color filter. In particular, the sensor arrangement can be configured to capture a color image in the visible spectrum and a hyperspectral image.

[0002] Furthermore, the invention relates to a method for calculating a color image and a hyperspectral image.

[0003] The invention further relates to a method for performing a white balance for a sensor arrangement.

[0004] Finally, the invention also relates to the use of a sensor arrangement in medical imaging.

[0005] Sensor arrangements are already known in which color filters for individual color channels (for example, red, green, and blue (RGB) or cyan, magenta, and yellow (CMYK)) are combined with a monochromatic image sensor. The image sensor has a multitude of small light-sensitive areas, so-called photocells, in front of which a color filter is positioned in the optical path. Thus, the photocell can only detect a brightness value for light within a specific wavelength range that passes through the preceding color filter. Such sensor arrangements are known, for example, as "Bayer sensors."

[0006] For example, the scientific article “CFA Based Simultaneous Multispectral Imaging and Illuminant Estimation” by Shrestha and Hardeberg (published in Tominaga, Shoji; Schettini, Raimondo; TrÉmeau, Alain [Eds.]: Computational Color Imaging. CCIW 2013, LNCS 7786, ISBN 978-3-642-36699-4, Berlin; Heidelberg: Springer, 2013, pages 158–170) reveals a sensor array designed for hyperspectral imaging. In addition to rearranging pixels, the article proposes the pixel-by-pixel use of an additional filter layer with three transmission windows.

[0007] A similar multispectral sensor arrangement is known from the scientific article “Multispectral color image capture using a liquid crystal tunable filter.” by the authors HARDEBERG, SCHMITT, and BRETTEL (published in: Opt. Eng. 41(10) 2532-2548, on October 1, 2002), which is based on a monochrome CCD sensor and a tunable optical filter with which wavelengths can be selected.

[0008] DE 10 2010 030 108 B4 discloses a color image sensor with a plurality of sensor elements arranged in a 2D array, each having an optical filter. Here too, the transmission behavior of the optical filters is tuned, based on the electro-optic effect.

[0009] WO 2015 / 077 493 A1 describes how a standard smartphone can be used as a spectrometer for hyperspectral imaging. It proposes using LED light sources with different emission wavelengths for synchronized and pulsed illumination. This allows for the acquisition of spectrally resolved image information without the need for special color filters.

[0010] Further relevant technological background for the invention is provided by publications US 2011 / 0 228 097 A1 and US 2016 / 0 088 265 A1.

[0011] Based on this technical background, the invention aims to expand the application possibilities of previously known sensor arrangements of the type mentioned above.

[0012] According to the invention, the solution to this problem is provided by a sensor arrangement of the type mentioned above having the features according to claim 1.

[0013] In particular, according to the invention, a sensor arrangement of the type mentioned above is proposed to solve the problem, characterized in that the photocells are grouped into subcells and the subcells into supercells, that each subcell has at least one color filter of a color channel, wherein all color channels together cover the entire visible spectrum, and that the characteristic wavelengths of the color filters belonging to a color channel differ at least partially from one another between the subcells of a supercell. It can thus be provided, for example, that all subcells have different characteristic wavelengths in pairs, or that individual color filters have matching characteristic wavelengths between two subcells.For example, the characteristic wavelengths may differ from subcell to subcell with respect to only one color channel or a selection among all color channels; or individual characteristic wavelengths may be the same within a color channel, while other characteristic wavelengths within that color channel differ across the supercell. Combinations of these individual cases are also possible.

[0014] For example, the subcells of a supercell can have color filters according to the pattern RGBX (R=red, G=green, B=blue, X=further spectral range, for example IR), where R, G, and B are color filters with a uniform characteristic wavelength across the subcells, while the color filter X has a characteristic wavelength that is individual from subcell to subcell.

[0015] One embodiment of the invention provides that the characteristic wavelengths of the color filters belonging to a color channel differ between the subcells of a supercell. Thus, a different characteristic wavelength can be selected for each pixel within at least one color channel, thereby avoiding redundancies.

[0016] One embodiment of the invention provides that the characteristic wavelengths of the color filters belonging to all color channels differ between the subcells of a supercell. Thus, a different characteristic wavelength can be selected for each pixel, thereby avoiding redundancies.

[0017] Preferably, each subcell may have a color filter from a specific spectral range, and all subcells within a supercell together may cover at least the spectral range of the primary colors of the visible spectrum. More preferably, the wavelength ranges of the color filters belonging to a given spectral range may differ between the subcells of a supercell in such a way that the ranges covered by the color filters may overlap, but are not identical.

[0018] The sensor arrangement according to the invention thus makes it possible to capture both a relatively high-resolution color image in a spectrum visible to the human eye and, in addition, a hyperspectral image using only one image sensor. The color image can therefore be reconstructed from the different spectral ranges of the individual subcells, while the spectral information is obtained from the individual subcells themselves. The term "hyperspectral image" as used in the application can be understood to include a multispectral image and / or multispectral information, depending on how many different color filters are used per supercell and / or how large the filter widths of the filters used and / or the distance between the filters are.In other words, this can refer to additional information, spatially linked to the information of the color image, in the form of a spectroscopic measurement.

[0019] The additional information opens up a multitude of possibilities, particularly in the medical field, for example, to assist users of medical imaging procedures in evaluating and interpreting the image scene. A particular advantage of using only a single image sensor with the inventive design is the ability to detect additional hyperspectral information in a captured image scene without the risk of incorrect spatial assignment, e.g., due to an incorrect alignment of two image sensors relative to each other. The inventive solution thus eliminates the need for spatial alignment (registration) required by previously known sensor arrangements with two image sensors—one for the color image and one for the hyperspectral image.The use of only one image sensor is made possible by providing a structural arrangement of the individual filters in subcells and an arrangement of the subcells in supercells, whereby a tuning of the individual filters of a supercell is required, so that in particular the characteristic wavelengths of the color filters belonging to a color channel differ from each other between the subcells of a supercell.

[0020] Other solutions that rely on the use of an optical grating in combination with microlens arrays to evaluate spectral information suffer from insufficient color image resolution. Furthermore, the entire setup is complex and requires significant assembly effort. Here, too, the solution according to the invention offers the advantage of a simpler setup and easier assembly compared to the aforementioned previously known solution.

[0021] Advantageous embodiments of the invention are described below, which can be combined alone or in combination with the features of other embodiments, optionally together with the features according to claim 1.

[0022] According to an advantageous embodiment, the characteristic wavelengths can be transmission maxima of the color filters. With respect to the transmission of light, the filter's amplitude can be greatest at the wavelength of the transmission maximum. Alternatively or additionally, the characteristic wavelength can also be defined by a mean wavelength or a wavelength that describes a parameter of a mathematical approximation of a transmission curve.

[0023] For example, it can be provided that the characteristic wavelengths of at least one color channel, preferably all color channels, are spaced apart from each other in such a way that the associated transmission maxima of the color filters are distinguishable from each other.

[0024] According to an alternative or supplementary embodiment to the aforementioned solutions, it can be provided that at least one supercell has at least two subcells that have the same characteristic wavelength for at least one color channel and / or for at least one color filter. Preferably, it can alternatively or supplementarily be provided that there are at least two subcells within a supercell that differ in the characteristic wavelength of at least one color filter.

[0025] According to a further advantageous embodiment, the supercells can each have at least four subcells. This allows a spatially high-resolution color image and, in addition, spectroscopic supplementary information to be detected using the sensor arrangement.

[0026] To obtain a good resolution of the captured image, the supercells can each have a rectangular or square pattern of subcells. In particular, the supercells can have a pattern with the same number of subcells per row (Z). n ) and subcells per column (S n ) or a different number of subcells per row (Z n+1 ; Z n+2 ; ... Z n+x ; Z n-1 , Z n-2 ; ... Z n-x ) and subcells per column (S n). Preferably, it can be a pattern or a combination of two or more patterns selected from 2x2 subcells, 3x3 subcells, 4x4 subcells, 5x5 subcells, 6x6 subcells, 2x3 subcells, 3x2 subcells, 3x4 subcells, 4x3 subcells, 4x5 subcells, 5x4 subcells, 5x6 subcells, 6x5 subcells, 2x4 subcells, 4x2 subcells, 3x5 subcells, 5x3 subcells, 4x6 subcells, 6x4 subcells.

[0027] According to a further advantageous embodiment, it can be provided that the subcells each have at least four photocells.

[0028] To obtain good resolution of the hyperspectral image, the subcells can each have a rectangular or square pattern of photocells. Preferably, the subcells can have a pattern with the same number of photocells per row (R). n ) and photocells per column (C n ) or a different number of photocells per row (R n+1 ; R n+2; ... R n+x ; R n-1 , R n-2 ; ... R n-x ) and photocells per column (C n ). It may be particularly preferred if it is a pattern or a combination of two or more patterns selected from 2x2 photocells, 3x3 photocells, 4x4 photocells, 5x5 photocells, 6x6 photocells, 2x3 photocells, 3x2 photocells, 3x4 photocells, 4x3 photocells, 4x5 photocells, 5x4 photocells, 5x6 photocells, 6x5 photocells, 2x4 photocells, 4x2 photocells, 3x5 photocells, 5x3 photocells, 4x6 photocells, 6x4 photocells.

[0029] To capture a color image of sufficient quality, particularly good color temperature, an advantageous embodiment provides that each subcell has at least three color filters. These at least three color filters can each be assigned to different color channels. This ensures that each subcell has at least one detecting photocell in each of the color channels spanning the visible spectrum. In other words, each subcell has at least one color filter for each color channel of a set of color channels spanning the visible spectrum.

[0030] Preferably, the color channels of the subcells are distributed across the photocells according to a uniform pattern. The assignment of the color channels to the photocells can also be chosen such that a subcell shifted horizontally or vertically by one photocell is again a subcell in this sense. This enables color imaging with a higher resolution than would be expected from subdivision into subcells alone.

[0031] Alternatively or additionally, the subcells may be provided with at least one additional filter whose characteristic wavelength lies outside the visible spectrum. In particular, the transmission maximum of the at least one additional filter may lie at a wavelength in the UV and / or IR range.

[0032] According to a further advantageous embodiment of the sensor arrangement, the characteristic wavelengths of adjacent color filters of different subcells of a supercell, which are assigned to the same color channel, can be equidistant or unequal from each other. In particular, it can be provided that the characteristic wavelengths of all adjacent color filters of a supercell, which are assigned to the same color channel, are equidistant and / or unequal from each other.

[0033] According to a further advantageous embodiment of the sensor arrangement, it can be provided that the spacing of the characteristic wavelengths between two adjacent color filters of one color channel in a supercell is equal to or different from the spacing of the characteristic wavelengths between two adjacent color filters of another color channel in the supercell. In particular, it can be provided that the spacing of the characteristic wavelengths between all adjacent color filters of one color channel in a supercell is equal to or different from the spacing of the characteristic wavelengths between all adjacent color filters of another color channel in the supercell.

[0034] Alternatively or additionally, according to a further embodiment, it may be provided that the characteristic wavelengths of the color filters of different color channels of the subcells of a supercell have an equidistant distance and / or an unequal distance to each other.

[0035] According to an advantageous embodiment, it can be provided that the arrangement of the color filters belonging to a color channel in the subcells of a supercell is the same or different. In particular, it can be provided that the arrangement of the color filters belonging to a color channel in the subcells of all supercells is the same or different.

[0036] Alternatively or additionally, further training may stipulate that the occupancy arrangements of the subcells within a supercell may be the same or different. In particular, it may stipulate that the occupancy arrangements of the subcells within all supercells may be the same or different.

[0037] To achieve the most space-saving design possible, an advantageous embodiment of the sensor arrangement allows the individual filters (color filters and / or the additional filters) of the sensor arrangement to be arranged in a single filter plane, particularly in only one filter plane. Compared to sensor arrangements with multiple filter planes or a single filter plane and at least one additional filter arranged in the optical beam path, this allows for a significantly more compact sensor arrangement. Preferably, the filter plane of the sensor arrangement can be integrated onto the image sensor, i.e., permanently connected to it. This has the advantage that a spatial arrangement can be defined, for example, during a lithography process, and no subsequent adjustment of the filters relative to the image sensor is required.

[0038] Single-stage color filters can therefore be used for the invention. This does not preclude the possibility of further filters being connected upstream or downstream for other purposes. In this case, the characteristic wavelengths can be referenced to these single-stage color filters.

[0039] According to an advantageous embodiment of the sensor arrangement, a supercell structure and / or a subcell structure of the sensor arrangement can be repeated periodically, particularly in an identical pixel configuration. This allows for simple evaluation, since pixel arrangements extending beyond a single subcell can be treated as averaged values ​​in one evaluation step.

[0040] It is particularly advantageous if the supercells have an identical structure. This allows the measurements from the photocells of one supercell to be compared with those from the photocells of a neighboring supercell. Consequently, a spatially resolved measurement distribution for the different characteristic wavelengths can be obtained.

[0041] To capture as many bands as possible with the sensor array and generate a high-resolution hyperspectral image, the filter width in which at least one color filter and / or at least one additional filter allows transmission can be from 8 nm to 25 nm, particularly from 10 nm to 20 nm. Specifically, all color filters of one color channel and / or all color channels and / or all additional filters can have the same filter width. Alternatively or additionally, the color filters and / or additional filters can be at least partially interference filters. This makes it possible to establish particularly narrow transmission ranges for the individual filters.

[0042] According to a further advantageous embodiment, it can be provided that the bandwidths of two adjacent color filters of a color channel in a supercell and / or of two adjacent additional filters in a supercell overlap. In particular, it can be provided that the bandwidths of all adjacent color filters of a color channel in a supercell and / or of all adjacent additional filters in a supercell overlap.

[0043] Alternatively or additionally, according to a further advantageous embodiment, a gap may be provided between the bandwidths of two adjacent color filters of a color channel in a supercell and / or between the bandwidths of two adjacent auxiliary filters in a supercell. In particular, a gap may be provided between the bandwidths of all adjacent color filters of a color channel in a supercell and / or between the bandwidths of all adjacent auxiliary filters in a supercell. It may be particularly advantageous if the overlap area is narrower than half the filter width of the color filters and / or the auxiliary filters. It may also be more preferred if the gap is narrower than half the filter width of the color filters and / or the auxiliary filters.

[0044] To achieve nearly identical sensitivity, the amplitudes of the color filters in a color channel of a supercell can be designed to be equal or nearly equal. In particular, the amplitudes of the color filters in a color channel of all supercells can be designed to be equal or nearly equal. It can also be designed to match the amplitudes of the color filters to a sensitivity function or spectral sensitivity of the sensor, for example, to achieve uniform sensitivity. This is particularly advantageous when the sensor's sensitivity is highly wavelength-dependent.

[0045] In order to be able to record as many closely spaced, especially consecutive, wavelengths as possible with a supercell, an advantageous further development of the sensor arrangement may provide that no matching characteristic wavelengths of the color filters belonging to a color channel are present in the subcells of a supercell.

[0046] According to an advantageous further development, it can be provided that an individual tuning of the color filters of a color channel in a supercell is achieved by changing the characteristic wavelength.

[0047] The invention further relates to a method for calculating a color image and a hyperspectral image using a sensor arrangement as described and / or claimed herein. In particular, it can be provided that spectroscopic measurements in the characteristic wavelengths of the color filters and / or additional filters associated with the photocells are output and / or used as a basis for calculating an image. The method according to the invention thus makes it possible to acquire a high-resolution color image and spatially related spectroscopic measurements associated with the color image, in particular with the image scene of the color image, using an image sensor, in particular a single image sensor.

[0048] It is particularly advantageous if the measured values ​​are output and / or used in an order corresponding to their characteristic wavelengths, for example, by using a specific data format for the measured values. Alternatively or additionally, it can be provided that the measured values ​​are output and / or used together with their assigned characteristic wavelengths. This allows for immediate correlation between the measured values ​​and the wavelength for which they were acquired.

[0049] The invention further relates to a method for performing a white balance for a sensor arrangement for capturing a color image. In particular, the method is suitable for performing a white balance for a sensor arrangement as described and / or claimed herein. It can be provided that a location-dependent white point, in particular a sub-cell-dependent white point, is determined and a location-specific white balance, in particular a sub-cell-specific white balance, is performed.

[0050] According to an advantageous further development of the method, it can be provided that the white points of different subcells in a supercell differ from each other and / or that the white points of identical subcells, in particular of different supercells, are the same.

[0051] The invention further relates to the use of a sensor arrangement as described and / or claimed herein in medical imaging. In particular, the sensor arrangement can be used in an endoscope. As explained above, this sensor arrangement has the advantage over previously known sensor arrangements that it can be designed to be particularly compact. Thus, the space requirement of the sensor arrangement is lower compared to previously known sensor arrangements. Especially with endoscopes used in the medical field, it is important to ensure that certain dimensions are maintained and that heat generation is kept to a minimum. By reducing the space requirement of the sensor arrangement, it is therefore possible to provide particularly narrow endoscopes that can also be inserted into difficult-to-access, especially narrow, cavities.Furthermore, the use of only one image sensor can reduce heat generation compared to sensor arrangements with two or more image sensors.

[0052] The invention will now be described in more detail with reference to several exemplary embodiments, but is not limited to these. Further exemplary embodiments result from combining the features of one or more claims with each other and / or with one or more features of the exemplary embodiments.

[0053] It shows: Fig. 1 A schematic representation of a possible embodiment of a sensor arrangement according to the invention with subcells (1 = top left, 2 = top right, 3 = bottom left, 4 = bottom right), each consisting of 2x2 photocells (with the filter properties R1, R2, R3, R4, G1, G2, G3, G4, B1, B2, B3, B4, IR1, IR2, IR3, IR4), wherein the lower half of the figure shows a detailed view of the circled area of ​​the entire sensor arrangement, which has a supercell of 2x2 subcells (corresponding to 4x4 photocells), in which the filter configuration of the individual photocells can be seen, wherein R stands for red, G for green and B for blue in the three color channels, and wherein the sensor arrangement has an additional filter (IR) for each subcell, for example in the infrared range, and wherein further supercells, in particular of the same size and / or with the same pattern of subcells, are connected to the supercell shown in detail below. Fig. 2 A further schematic detail view of a supercell of a possible embodiment of a sensor arrangement according to the invention with four color channels (1xR, 2xG, 1xB), preferably with four color filters which have a characteristic wavelength within the spectrum visible to the human eye and have been selected such that the primary colors of the visible range are included (1xR, 2xG, 1xB), wherein the subcells are composed of 2x2 photocells and the supercells of 2x2 subcells (corresponding to 4x4 photocells), wherein the characteristic wavelengths of the individual color filters correspond to the specified numerical values, wherein a matching of the characteristic wavelength between adjacent color filters of a color channel is carried out by a difference of 20 nm, which remains constant with respect to the color filters of a color channel, wherein further supercells are connected to the supercell shown. Fig. 3 A further schematic detail view of a supercell of a possible embodiment of a sensor arrangement according to the invention with four color channels (1xR, 2xG, 1xB), preferably with four color filters which have a characteristic wavelength within the spectrum visible to the human eye and have been selected such that the primary colors of the visible range are included (1xR, 2xG, 1xB), wherein the subcells are composed of 2x2 photocells and the supercells of 3x3 subcells (corresponding to 6x6 photocells), wherein the characteristic wavelengths of the individual color filters correspond to the specified numerical values, wherein a matching of the characteristic wavelength between adjacent color filters of a color channel is achieved by a varying difference (distance), wherein further supercells are connected to the supercell shown. Fig. 4 a through the in Fig. The spectrum covered by the three subcells of a supercell (consisting of 3x3 subcells) shown in Figure 5, wherein channels A, B, and C are color channels in the visible range (color range) and channel D is a channel outside the visible range (in this case, in the infrared range), where n, m, x, and y indicate the differences (distances) between the characteristic wavelengths of the individual filters of a channel between the adjacent subcells, wherein the color filters (A, A+n, A+2n) of the blue color channel have no overlapping bandwidths, wherein the color filters (B, B+m, B+2m) of the green color channel have overlapping bandwidths, and wherein the color filters (C, C+x, C+2x) of the red color channel have even wider overlapping bandwidths, and wherein the filter widths of the additional filters (D, D+y, D+2y) are significantly narrower than the filter widths of the color filters and therefore overlap only very narrowly. Fig. 5 the schematic detail view of the occupancy of the three subcells of the supercell, whose filter properties in Fig. 4 are shown, Fig. 6 alternative filter properties of the in Fig. 5 subcells shown, wherein a gap is provided between each of the individual filters of a channel, wherein the IR channel has been replaced by another green color channel, wherein the filter widths (bandwidths; passbands) of the individual color filters are the same, and wherein the difference amounts (distances) between the characteristic wavelengths of two adjacent individual filters are each the same size, Fig. 7 more alternative filter properties of the in Fig. 5 subcells shown, where, in contrast to the filter properties, Fig. 6, an overlap of the transmission ranges is provided between the individual filters of a channel, wherein the IR channel has also been replaced by another green color channel, wherein the filter widths (bandwidths; transmission ranges) of the individual color filters are the same, and wherein the difference amounts (distances) between the characteristic wavelengths of two adjacent filters are each the same.

[0054] In the Fig. 1, Fig. 2 and Fig. Figure 3 shows a sensor arrangement, which is collectively referred to as 1.

[0055] Sensor arrangements 1 are generally semiconductor-based devices that enable the acquisition of two-dimensional images of a scene. Sensor arrangements 1 based on monochromatic image sensors 2, equipped with multiple color filters 4 to direct light of a specific wavelength onto the underlying photocells 3, are already known.

[0056] The sensor arrangement 1 is configured to capture a color image and, additionally, a hyperspectral image by acquiring spectroscopic information using an image sensor 2, in particular a monochromatic image sensor 2. Crucially, this requires the correct spatial mapping of the spectroscopic information to the image scene of the color image.

[0057] In Fig. Figure 1 shows a rough schematic diagram of a possible embodiment of a sensor arrangement 1 in its upper half. This arrangement comprises an image sensor 2, which is equipped with a filter plane 15, preferably integrated on the image sensor 2, containing several individual filters (e.g., color filters 4 and / or additional filters 11). The individual filters are fixedly assigned to the photocells 3 of the image sensor 2, such that each individual filter preferably covers only exactly one photocell 3 below it. The photocells 3 detect a brightness value of the light of a characteristic wavelength 9 transmitted through the individual filters. The sensor arrangement 1 has the advantage that, due to the single filter plane, it requires a particularly small amount of space. Furthermore, the alignment of the single-stage individual filters relative to the photocells 3 is simpler than with two or more filter planes 15.

[0058] In the lower half of Fig. Figure 1 shows a schematic representation of the arrangement of individual filters on the image sensor 2. The photocells 3, each equipped with individual filters, are grouped into subcells 5, each consisting of 2x2 photocells 3. In the illustrated configuration, four subcells 5 together form a supercell 6, which is arranged in a 2x2 subcell pattern. However, other patterns of subcells 5 and / or supercells 6 are also possible, as described in detail below.

[0059] Each photocell 3 is assigned to only one subcell 5. Each subcell 5, in turn, is assigned to only one supercell 6.

[0060] Each subcell 5 covers a color channel 7,R,G,B,A,C,D via a color filter 4, in particular a sub-range (band) of a color channel 7,R,G,B,A,C,D. All color channels 7,R,G,B,A,C together cover at least the spectral range of the primary colors of the visible range – hereinafter also referred to as the visible spectrum 8 (for the human eye), in particular exactly the visible spectrum 8. The visible spectrum 8 can be understood as the range from 380 nm to 780 nm.

[0061] Each individual filter (i.e., each color filter 4 and / or each additional filter 11) has a characteristic wavelength 9. The individual filters of different subcells 5 of a supercell 6 are detuned / tuned relative to each other. Thus, the characteristic wavelengths 9 of the individual filters of a supercell belonging to a channel 7,R,G,B,A,C,D (for example, to a color channel 7,R,G,B,A,C and / or to a channel D in a region outside the visible spectrum 8) differ from each other. Adjacent individual filters of a channel 7,R,G,B,A,C,D, whose distances 12, 13, 14, 22 are smallest in the characteristic wavelengths 9, can overlap with their transmission ranges in an overlap region 17, or a gap 18 can be formed between them. Gap 18 can be understood in particular as a region of the respective channel spectrum whose wavelength is not transmitted through any single filter of the channel.

[0062] The characteristic wavelength 9 can refer to a respective transmission maximum of the individual filters.

[0063] In the exemplary embodiment, R1, R2, R3, R4, G1, G2, G3, G4, B1, B2, B3, B4, IR1, IR2, IR3, IR4 are different in pairs.

[0064] In another embodiment of the invention, the characteristic wavelengths differ only partially. For example, R1=R2=R3=R4 and / or G1=G2=G3=G4 and / or B1=B2=B3=B3 may apply, while IR1, IR2, IR3, IR4 are different in pairs. Other combinations of similarity and differences are implemented in further embodiments.

[0065] Supercell 6 from Fig. 1 covers a total of three color channels 7,R,G,B,A,C, where R corresponds to the red channel, G to the green channel, and B to the blue channel. Each subcell 5 also has an additional filter 11 assigned to a channel D that covers a range outside the visible spectrum 8 (such as infrared and / or ultraviolet radiation). Each of the four subcells 5 of the supercell 6 has four photocells 3, each with a fixed, assigned single filter (with the filter properties R1, R2, R3, R4, G1, G2, G3, G4, B1, B2, B3, B4, IR1, IR2, IR3, IR4), wherein the nine color filters 4 of the supercell 6 are each assigned to one of the color channels 7,R,G,B,A,C, so that there are three color filters 4 for each color channel 7,R,G,B,A,C.

[0066] The photocells 3 can have a square or rectangular shape. The supercells 6 can also have a rectangular or square pattern of subcells 5. The subcells 5, in turn, can also have a rectangular or square pattern of photocells 3.

[0067] The selected pattern can define the maximum achievable image resolution of the sensor arrangement 1. Some examples of possible patterns are given below, although the invention is not limited to these patterns.

[0068] Basically, a trade-off must be made between spatial resolution and the number of hyperspectral measurements.

[0069] The supercells 6 can, for example, form a pattern with the same number of subcells 5 per row (Z). n ) and subcells per column (S n ) exhibit, as in the Fig. 1, Fig. 2, Fig. 3 and Fig. 5 is shown. Patterns with a different number of subcells (5 per row) are not shown. n+1 ; Z n+2 ; ... Z n+x ; Z n-1 , Z n-2 ; ... Z n - x ) and subcells 5 per column (S n ), which are also covered by the invention.

[0070] Preferred patterns of Supercell 6 can be selected from, for example, 2x2 subcells, 3x3 subcells, 4x4 subcells, 5x5 subcells, 6x6 subcells, 2x3 subcells, 3x2 subcells, 3x4 subcells, 4x3 subcells, 4x5 subcells, 5x4 subcells, 5x6 subcells, 6x5 subcells, 2x4 subcells, 4x2 subcells, 3x5 subcells, 5x3 subcells, 4x6 subcells, 6x4 subcells.

[0071] The subcells 5 can form a pattern with the same number of photocells 3 per row (R n ) and photocells 3 per column (C n ) exhibit, as in the Fig. 1, Fig. 2, Fig. 3 and Fig. 5 is shown. Alternatively or additionally, the subcells can have 5 patterns with a different number of photocells 3 per row (R). n+1 ; R n+2 ;... R n+x ; R n-1 , R n-2 ;... R n-x ) and photocells 3 per column (C n exhibit.

[0072] Preferred patterns of subcells 5 can be, for example, patterns selected from 2x2 photocells, 3x3 photocells, 4x4 photocells, 5x5 photocells, 6x6 photocells, 2x3 photocells, 3x2 photocells, 3x4 photocells, 4x3 photocells, 4x5 photocells, 5x4 photocells, 5x6 photocells, 6x5 photocells, 2x4 photocells, 4x2 photocells, 3x5 photocells, 5x3 photocells, 4x6 photocells, 6x4 photocells.

[0073] The detuning of the individual filters (color filter 4 and / or additional filter 11) assigned to a channel (color channel 7,R,G,B,A,C and / or additional channel D outside the visible range) can be achieved by offsetting the characteristic wavelengths 9 of the individual filters belonging to a channel, particularly adjacent individual filters, from each other by an equidistant distance 12,13,14,22 and / or an unequal distance 12,13,14,22. Each individual filter of a channel thus covers a different sub-range of the entire channel spectrum. Preferably, all individual filters of a channel 7,R,G,B,A,C,D (with their passbands) together cover the entire channel spectrum, i.e., in particular, all wavelengths of the entire channel spectrum.

[0074] In the Fig. 2 and Fig. Figure 3 shows different embodiments of a sensor arrangement 1, each depicting a supercell 6 consisting of subcells, each with 2x2 photocells 3, to which further, in particular identical, supercells 6 are connected. In contrast to the supercell 6 made of Fig. 1. No channel 7,R,G,B,A,C outside the visible range 8 is provided. Instead, the supercells each have an additional green color channel G (wavelengths from 477 nm to 623 nm), so that each subcell 5 of the supercells 6 has two green color filters 4 that differ in their characteristic wavelength 9.

[0075] Supercell 6 from Fig. 2 has 2x2 subcells 5 (Nos. 1-4), i.e., two subcells 5 per row and two subcells 5 per column. As can be seen from the characteristic wavelength 9 indicated in the center of each photocell 3, the individual color filters 4 of a color channel 7,R,G,B,A,C each have an offset of a constant difference (distance 12,13,14). In this case, the distances 12,13,14 are each exactly 20 nm. However, it is also conceivable that the distances 12,13,14 differ between the at least three (here four) color channels 7,R,G,B,A,C. Alternatively or additionally, it is possible that the distances 12, 13, 14, 22 between the individual filters (such as the color filters 4 and / or the additional filter 11) within a channel 7, R, G, B, A, C, D remain the same or differ from each other at least partially.

[0076] An example of different distances 12, 13, 14, 22 between the characteristic wavelengths 9 of the individual filters is shown in Fig. 3 shown. In Fig. Figure 3 shows a supercell 6 consisting of 3x3 subcells 5 (Nos. 1-9). The distances 12, 13, 14, 22 between the individual filters (in this case, color filters 4) of a channel 7,R,G,B,A,C,D (in this case, a color channel 7,R,G,B,A,C) are not uniform. Thus, the distance 12, 13, 14, 22 between adjacent individual filters of a channel 7,R,G,B,A,C,D is alternately 9 nm and 8 nm.

[0077] In general, it can therefore be said that the distances 12, 13, 14, 22 can be freely chosen, depending on how closely the recorded spectroscopic measurements should be spaced together. It has proven advantageous if the distances 12, 13, 14, 22 of the characteristic wavelengths 9 between adjacent individual filters (color filters 4 and / or additional filters 11) are from 5 nm to 40 nm, in particular from 5 nm to 30 nm, in particular from 5 nm to 25 nm, in particular from 7 nm to 20 nm, and in particular from 8 nm to 15 nm.

[0078] In the Fig. Figures 4 to 7 show further examples of possible embodiments of a sensor arrangement 1 according to the invention.

[0079] The in Fig. The spectrum shown in 4 is the one in Fig. The 5 shown assignments are to be assigned. A here stands for the blue color channel, B for the green color channel, C for the red color channel, and D for the additional channel outside the visible range 8 (here in the IR range). The distances 12, 13, 14, 22 between the individual filters of a channel (color channel 7, R, G, B, A, C and / or additional channel D) are indicated by the letters n, m, x, and y.

[0080] As demonstrated by Fig. As can be seen in Figure 4, the filter widths 16 and the characteristic wavelengths 9 (A, A+n, A+2n) of the color filters 4 of the blue color channel A are chosen such that there is no overlap region 17 between adjacent color filters 4 of the blue color channel A. Rather, gaps 18 can be said to exist between the transmission ranges covered by the individual color filters 4 of the blue color channel A.

[0081] In contrast, for the individual filters of the other channels (Green-B, Red-C, IR-D) an overlap area 17 can be seen between the passbands of adjacent individual filters of a channel 7,R,G,B,A,C,D.

[0082] However, the invention is not limited to the foregoing embodiment. The characteristic wavelengths 9 and / or the filter widths 16 of the individual filters of the various channels 7,R,G,B,A,C,D can be selected as required, so that it can be decided on an application-specific basis whether an overlap of the transmission ranges is useful or not.

[0083] Furthermore, the width of an overlap area 17 between two individual filters can be set, particularly depending on a selected characteristic wavelength 9 and a filter width 16.

[0084] In Fig. Figure 4 shows an exemplary selection of individual filters with different filter widths 16.

[0085] A filter width of 16 can refer to a transmission range from a lower wavelength to an upper wavelength, within which the individual filter transmits light at specific wavelengths. Radiation with a wavelength outside the filter width of 16, however, cannot pass through or can only do so in a very weakened form.

[0086] The filter widths 16 of the color filters 4 and the additional filter 11 are each chosen differently here, so that only the individual filters of one channel have the same filter widths 16, but the filter widths 16 of the individual filters of different channels differ.

[0087] In the Fig. 6 and Fig. Figure 7 shows two further examples of possible filter properties, where equal distances 12, 13, 14 were chosen between the transmission ranges of adjacent color filters 4 of a color channel 7,R,G,B,A,C. The implementation variants of the Fig. 6 and Fig. The 7 differ in the choice of filter properties of the color filters 4. For example, the filter widths 16 of the color filters 4 are of the design made of Fig. 6 narrower than the color filters 4 out Fig. 7. Furthermore, the design results from Fig. 7 wider overlap areas 17 between color filters 4 of a color channel 7,R,G,B,A,C.

[0088] The filter widths 16 of the individual filters can, for example, allow transmission in a passband between a lower wavelength and an upper wavelength, wherein the passband has a width of 8 nm to 25 nm, in particular from 10 nm to 20 nm.

[0089] A supercell structure and / or a subcell structure of sensor arrangement 1 can be, as shown in the Fig. 1, Fig. 2 and Fig. 3 indicated periodically, especially in identical occupancy arrangement 21, repeat.

[0090] The individual filters (color filter 4 and / or additional filter 11) can preferably be designed as interference filters. This has the advantage that a characteristic wavelength 9 can be set with particularly precise and / or narrow limits and / or with low transmission in the stopband and with high edge steepness.

[0091] As shown by the Fig. 4, Fig. 6 and Fig. As is clearly visible in Figure 7, it is particularly advantageous if the amplitudes 19 of the individual filters (color filter 4 and / or additional filter 11) are equal or nearly equal. Alternatively or additionally, the amplitudes of the individual filters can be adjusted so that the sensitivities of the individual color channels 7,R,G,B,A,C, which result from combining the sensitivity of the underlying photocells 3 with the transmission properties of the overlying color filters 4, are equal or nearly equal. Alternatively or additionally, the amplitudes of the individual filters can be adapted to the spectral sensitivity of the sensor, for example, to achieve a uniform sensitivity. This can be implemented particularly well by using interference filters.Thus, color images and / or hyperspectral images are of particularly high quality, since the same or almost the same intensity can be achieved when recording the light through different photocells 3.

[0092] As shown by the Fig. 2 and Fig. As can be seen in Figure 3, no matching characteristic wavelengths 9 of the color filters 4 belonging to a color channel 7,R,G,B,A,C are provided for the subcells 5 of a supercell 6. Thus, a sensor arrangement 1 can be created in a relatively small space that can capture a complete spectrum of a color channel 7,R,G,B,A,C using a supercell 6.

[0093] The sensor arrangement 1 described above is therefore particularly suitable for replacing existing devices for acquiring a color image and additional hyperspectral measurements. Sensor arrangement 1 is relatively inexpensive to manufacture. Furthermore, it is particularly suitable for use in medical imaging, especially in endoscopy.

[0094] By using only one image sensor 2, less waste heat is produced compared to other technical solutions, resulting in a more compact design and lower weight. All these advantages ultimately benefit the user of an image acquisition device, such as an endoscope.

[0095] Due to its unique pattern design, sensor arrangement 1 has the special feature that a specific white balance procedure is provided to capture a color image with optimally represented color temperatures. This procedure begins by determining a location-dependent white point relative to a specific coordinate of the sensor arrangement. This white point can be referenced to a sub-cell-dependent white point. After determining the specific location-dependent white point, it can be used to perform a location-specific white balance.

[0096] The white points of different subcells 5 within a supercell 6 can differ from each other. Conversely, the white points of identical subcells 5, especially those of different supercells 6, can be the same.

[0097] The invention therefore relates in particular to a sensor arrangement 1 for recording a color image in the visible spectrum 8 and hyperspectral information that is spatially linked to the color image, wherein the sensor arrangement 1 has an image sensor 2 comprising several photocells 3, wherein at least one part of the photocells 3 is permanently assigned a color filter 4, wherein each photocell 3 is assigned to a subcell 5 and each subcell 5 to a supercell 6, wherein each subcell 5 has at least one individual filter of a channel, wherein all channels together cover at least the primary colors of the visible spectrum 8, such that the characteristic wavelengths 9 of the individual filters belonging to a channel differ from each other between the subcells 5 of a supercell 6. Reference symbol list 1 Sensor arrangement 2 image sensors 3 photocells 4 color filters 5 subcell 6 supercell 7 color channels (R,G,B,A,C) 8 visible spectrum 9 characteristic wavelengths 10 Transmission maximum 11 additional filters 12 Distance between adjacent color filters of the blue color channel 13 Distance between adjacent color filters of the green color channel 14 Distance between adjacent color filters of the red color channel 15 filter levels 16 filter width 17 Overlap area 18 gap 19 Amplitude 20 lens 21 Occupancy order 22 Distance between adjacent auxiliary filters of the auxiliary channel outside the visible range D Additional channel

Claims

[1] Sensor arrangement (1) with - an image sensor (2) consisting of several photocells (3), - wherein the photocells (3) are at least partially assigned a colour filter (4), - wherein the photocells (3) are grouped into subcells (5) and the subcells (5) are grouped into supercells (6), - wherein each subcell (5) has at least one color filter (4) of a color channel (7,R,G,B,A,C), - where all color channels (7,R,G,B,A,C) together cover the entire visible spectrum (8), - wherein the characteristic wavelengths (9) of the color filters (4) belonging to a color channel (7,R,G,B,A,C) between the subcells (5) of a supercell (6) differ from each other at least partially, and - wherein the color channels (7,R,G,B,A,C) of the subcells (5) are distributed among the photocells (3) according to a uniform pattern. [2] Sensor arrangement (1) according to claim 1, wherein the characteristic wavelengths (9) - to determine transmission maxima (10) of the color filters (4) and / or - medium wavelengths (9). [3] Sensor arrangement (1) according to claim 1 or 2, wherein at least one supercell (6) has at least two subcells (5) which have the same characteristic wavelength (9) for at least one color channel (7,R,G,B,A,C) and / or for at least one color filter (4), - preferably wherein there are at least two subcells (5) within a supercell (6) which differ in the characteristic wavelength (9) of at least one colour filter (4). [4] Sensor arrangement (1) according to one of claims 1 to 3, - wherein the supercells (6) each have at least four subcells (5). [5] Sensor arrangement (1) according to one of the preceding claims, - wherein the supercells (6) each have a rectangular or square pattern of subcells (5), - especially a pattern with - same number of subcells (5) per row (Z n ) and subcells per column (S n ) or with - a different number of subcells (5) per row (Z n+1 ; Z n+2 ; ... Z n+x ; Z n-1 , Z n-2 ; ... Z n-x ) and subcells (5) per column (S n ). [6] Sensor arrangement (1) according to one of the preceding claims, wherein the subcells (5) each have at least four photocells (3). [7] Sensor arrangement (1) according to one of the preceding claims, - wherein the subcells (5) each have a rectangular or square pattern of photocells (3), - preferably a sample with - same number of photocells (3) per row (R n ) and photocells (3) per column (Cn ) or with - a different number of photocells (3) per row (R n+1 ; R n+2 ;... R n+x ; R n-1 , R n-2 ;... R n-x ) and photocells (3) per column (C n ). [8] Sensor arrangement (1) according to one of the preceding claims, - wherein the subcells (5) each have at least three color filters (4) of different color channels (7,R,G,B,A,C). [9] Sensor arrangement (1) according to one of the preceding claims, wherein the subcells (5) have at least one additional filter (11) whose characteristic wavelength (9) lies outside the visible range (8). [10] Sensor arrangement (1) according to one of the preceding claims, wherein the characteristic wavelengths (9) of adjacent color filters (4) of different subcells (5) of a supercell (6) which are assigned to a same color channel (7,R,G,B,A,C) have an equidistant distance (12,13,14) from one another. [11] Sensor arrangement (1) according to one of the preceding claims, wherein the characteristic wavelengths (9) of adjacent color filters (4) of different subcells (5) of a supercell (6) which are assigned to a same color channel (7,R,G,B,A,C) have an unequal distance (12,13,14) from one another. [12] Sensor arrangement (1) according to one of the preceding claims, wherein a distance (12, 13, 14) of the characteristic wavelengths (9) between two adjacent color filters (4) of a color channel (7, R, G, B, A, C, D) in a supercell (6) is equal to or unequal to a distance (12, 13, 14) of the characteristic wavelengths (9) between two adjacent color filters (4) of another color channel (7, R, G, B, A, C) in the supercell (6). [13] Sensor arrangement (1) according to one of the preceding claims, wherein the characteristic wavelengths (9) of the color filters (4) of different color channels (7, R, G, B, A, C) of the subcells (5) of a supercell (6) have an equidistant distance and / or an unequal distance from one another. [14] Sensor arrangement (1) according to one of the preceding claims, wherein the occupancy arrangements of the color filters (4) belonging to a color channel (7, R, G, B, A, C) in the subcells (5) of a supercell (6), in particular in the subcells (5) of all supercells (6), are the same and / or wherein the occupancy arrangements (21) of the subcells (5) within a supercell (6) are the same. [15] Sensor arrangement (1) according to one of the preceding claims, wherein the color filters (4) of the sensor arrangement (1) are arranged in a filter plane (15) and are designed as single-stage color filters (4). [16] Sensor arrangement (1) according to one of the preceding claims, wherein a supercell structure and / or a subcell structure of the sensor arrangement (1) is repeated periodically, in particular in an identical occupancy arrangement (21). [17] Sensor arrangement (1) according to one of the preceding claims, wherein a filter width (16) in which at least one of the color filters (4) allows transmission is from 8 nm to 25 nm. [18] Sensor arrangement (1) according to one of the preceding claims, wherein the bandwidths of two adjacent color filters (4) of a color channel (7,R,G,B,A,C) in a supercell (6) overlap and / or wherein a gap (18) is provided between the bandwidths of two adjacent color filters (4) of a color channel (7,R,G,B,A,C) in a supercell (6). [19] Sensor arrangement (1) according to one of the preceding claims, wherein the amplitudes (19) of the color filters (4) of a color channel (7,R,G,B,A,C) of a supercell (6) are equal or almost equal. [20] Sensor arrangement (1) according to one of the preceding claims, wherein no matching characteristic wavelengths (9) of the color filters (4) belonging to a color channel (7,R,G,B,A,C) are present in the subcells (5) of a supercell (6). [21] Sensor arrangement (1) according to one of the preceding claims, wherein an individual tuning of the color filters (4) of a color channel (7, R, G, B, A, C) in a supercell (6) is carried out by changing the characteristic wavelength (9). [22] Method for calculating a color image and a hyperspectral image - with the aid of a sensor arrangement (1) according to one of the preceding claims, - wherein spectroscopic measured values in the characteristic wavelengths (9) of the color filters (4) and / or additional filters (11) assigned to the photocells (3) are output and used as a calculation basis for calculating an image. [23] Method for performing a white balance for a sensor arrangement (1) according to one of claims 1 to 21, wherein a subcell-dependent white point is determined and a subcell-specific white balance is performed. [24] Method according to claim 23, wherein the white points of different subcells (5) in a supercell (6) differ from each other. [25] Use of a sensor arrangement (1) according to one of claims 1 to 21 in medical imaging, in particular in an endoscope.

Images