Method and apparatus for determining wavelength deviation of an image captured by a multi-lens camera system

Abstract

This invention relates to a method for determining wavelength deviation in an image captured by a multi-lens camera system (1), comprising the steps of: - determining a color gamut (F) for a region of a filter element (4) of the multi-lens camera system (1), said region being associated with a predetermined region of an image sensor (3) of the multi-lens camera system (1); - determining a center wavelength (Z) for at least two pixels (P1, P2) of the predetermined region of the image sensor (3) based on the determined color gamut (F); - correcting the region of the image sensor (3) and / or an image (B, B1) captured by the image sensor (3) based on the determined center wavelength (Z); and / or generating a supplementary dataset for said image or for imaging control. The invention also relates to a corresponding apparatus and a multi-lens camera system.

Description

Technical Field

[0001] This invention relates to a method and an apparatus for determining, and particularly utilizing, wavelength deviations in images captured by a multi-lens camera system, especially for dispersion calibration. The multi-lens camera system is preferably a camera system for (hyper)spectral imaging. Background Technology

[0002] In many commercial and scientific fields, cameras with spectral resolution in addition to spatial resolution (spectral cameras) are used, often extending beyond the visible spectrum (multispectral cameras). For example, when measuring the Earth's surface from the air, commonly used cameras not only have normal RGB color resolution but also provide high-resolution spectra, potentially even extending into the ultraviolet or infrared range. These measurements, for instance, allow for the identification of individual planting areas within agricultural land. This enables the determination of plant growth or health status, or the distribution of different chemical elements such as chlorophyll or lignin.

[0003] In recent decades, high-resolution spectral imaging techniques known as "hyperspectral imaging" have proven suitable for these measurements. With this technology, it is possible, for example, to identify and distinguish different chemical elements based on spectra recorded in a spatially resolved manner.

[0004] In a typical (hyper)spectral imaging system, a lens array is positioned in front of the image sensor, mapping the subject onto the sensor as multiple different images (one for each lens). This type of imaging system is also called a "multi-lens imaging system." Using filtering elements, such as mosaic filters or linearly variable filters, located between the lens array and the image sensor, each image is captured in a different spectral range. This results in a large number of mappings of the subject across different spectral ranges ("channels").

[0005] However, a drawback of existing technology is that the captured images cannot be optimally compared with each other. These images also require calibration. Especially when photographing objects at different distances from the camera system (e.g., objects in front of a background, or two or more objects at different distances), the different optical paths lead to inaccurate spectral classification of the objects when photographed through the filtering elements of the camera system. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and to provide a method for determining the wavelength deviation of a multi-lens camera system, particularly for calibrating a multi-lens camera system and / or improving spectral resolution.

[0007] The solution of the present invention to achieve the above-mentioned objective is a method and an apparatus as claimed in the claims.

[0008] The present invention provides a method for determining (and advantageously utilizing) wavelength deviations in images captured by a multi-lens imaging system, particularly for achieving dispersion calibration and / or improving spectral resolution, comprising the following steps:

[0009] - Determine the spectral sensitivity (“gamut”) of a region of the filter element of the multi-lens camera system, said region corresponding to a predetermined region of the image sensor of the multi-lens camera system.

[0010] -Based on a defined spectral sensitivity (defined color gamut), determine the center wavelength for at least two pixels in a predetermined region of the image sensor.

[0011] - Based on a determined center wavelength, the region of the image sensor and / or the image captured by the image sensor (or the corresponding region of the image sensor) are corrected, and / or a supplementary dataset for the image or for imaging control is generated.

[0012] The spectral sensitivity of a region of a filter element reflects which pixel and which color (wavelength) of light is captured by the image sensor after the filter element. Ideally, this is light with a single (center) wavelength; in reality, it is a wavelength distribution with a center wavelength. Spectral sensitivity is also referred to here as the "color gamut" because it encompasses a domain (range) in which different colors dominate, depending on the filter element. For ideal filters, this color gamut is mostly extended in a linearly variable (linearly variable filter) or constant and graded (mosaic filter) manner. The area of ​​the image sensor used for imaging should have a well-defined (linearly variable / graded constant) color gamut extension. However, in practice, for many filters, the key is the path that light takes within the filter element as it propagates. In a region of a mosaic filter with angle-dependent filtering characteristics designed for a single center wavelength, the center wavelength measured under straight and oblique incidence differs. Although this difference is limited to a few nm in practice, it can still lead to systematic measurement errors and negatively impact the evaluation of spectral images. Furthermore, as described below, this effect can be used to improve spectral resolution.

[0013] It should be noted that the observations described herein primarily concern the effects caused by different light paths within the filter. However, this invention can not only control these effects but also control non-uniformity within the filter element, or wavelength deviations in the image caused by linearly variable filter elements.

[0014] Therefore, the color gamut includes information about the center wavelength of light that passes through the filtering element and is incident on pixels in a predetermined area of ​​the image sensor. Preferably, for a pixel or group of pixels, the color gamut also includes information about the angular correlation of the center wavelength of the transmitted light. This further improves accuracy because light from a closer object travels through the filtering system at a slightly different angle compared to light from a more distant object.

[0015] A color gamut is defined for specific areas of the filter element: these areas are through which light passes to capture one or more images for a predetermined area of ​​the image sensor. This method is used to correct only a single image or a corresponding area of ​​the image sensor, specifically the area used to capture the relevant image. However, the method can also be used to correct several images simultaneously, where the predetermined area on the image sensor is correspondingly larger.

[0016] Color gamut can be determined by direct measurement. For this purpose, images of the subject are captured and / or provided by a multi-lens imaging system within the spectral range. A more detailed description of this capturing or providing operation is provided below.

[0017] As an alternative or supplement, the color gamut can also be determined by calculating the filter characteristics at different incident angles when the wavelength is incident, taking into account the imaging properties of the optics of the multi-lens camera system corresponding to that region of the image sensor.

[0018] As mentioned above, it is important to note that when an image is captured through a region of the image sensor, light from different pixels in the image travels through the filter at different angles of incidence. Therefore, and depending on the situation, due to the non-uniformity of the filter thickness within the relevant region, light from different pixels in the image sensor travels through the filter element along different paths. Depending on the type of filter element (or the region of the filter element), this may result in a shift in filter properties. If z is the center wavelength of the filter for light incident at angle a, then for light passing through the filter at angle b, a center wavelength z ± Δz is produced.

[0019] The center wavelength is determined by at least two pixels, preferably all pixels of a predetermined region of interest (ROI), within a defined area of ​​the image sensor. However, it should be noted that these (at least) two pixels are preferably located within an area suitable for capturing a single image. If the area of ​​the image sensor is used to capture several images, this step should be performed for at least two pixels per image.

[0020] If the center wavelengths of two pixels are observed for each "image," a correction function ("support function") can be determined for all pixels using these pixels, assuming a linear extension of the center wavelength. If the extension is non-linear, it is preferable to determine the center wavelengths for more than two pixels and thus determine the correction function. Very accurate results are obtained by observing the center wavelengths of all pixels in the region. The advantage of the correction function is that it eliminates the need to determine the center wavelengths for all pixels; the center wavelengths can be set using the correction function itself.

[0021] Using a determined center wavelength, the region of the image sensor and / or the corresponding image captured by the image sensor is corrected. Alternatively or supplementarily, a supplementary dataset is generated for the image or for imaging control. Here, correction and dataset generation largely have the same meaning, because in a digital system, correction can be based on a dataset that is read in and used for correction of the image sensor or data.

[0022] Corrections can be for adaptation, especially calibration. However, they can also include adding image information to an image, such as adding supplementary spectral information or information about the object's radiometric properties. This supplementary information can exist directly in the image data, or as an additional dataset—that is, "outside" the image but associated with it. This dataset can be used when viewing the image. As long as the corresponding data exists and the computing unit "knows" where the data is located, it is equivalent whether the supplementary data is in the image file or in a separate dataset. This similarly applies to supplementary data for image sensors. For image sensor corrections, it is essentially irrelevant whether calibration is performed directly during shooting or on the raw shooting data (or, depending on the case, reconstructed data). Therefore, the data used to correct image sensors can also exist in the form of supplementary datasets.

[0023] For calibration, the captured images should exhibit a uniform subject, such that each pixel of the image appears identical in terms of emphasis. A preferred subject is a uniform surface, or light from a Lambert radiator or a light source incident on a Lambert diffuser. If only one image is captured, the light preferably has wavelengths approximately corresponding to the spectral channels of the image (preferably narrowband radiation); if several images are captured, the subject should encompass wavelengths from all spectral channels. However, it is also possible to capture several images sequentially, where the subject or wavelengths can be varied accordingly.

[0024] It is known to those skilled in the art that an image of a subject is captured (at least) within a spectral range using a multi-lens imaging system (i.e., in the form of a measurement in one spectral channel of the multi-lens imaging system), and is produced in a known manner using a (known) multi-lens imaging system. A multi-lens imaging system for (multi / hyper)spectral imaging includes a planar image sensor, a position-sensitive spectral filtering unit, and an imaging system. The imaging system includes a planar lens array comprising a large number of individual lenses arranged such that, at a first point in time, a large number of first maps of the subject, arranged in a grid pattern, are generated in a first region on the image sensor. These lenses are, for example, spherical lenses, cylindrical lenses, holographic lenses, or Fresnel lenses, or a lens system (e.g., an objective lens) consisting of several such lenses.

[0025] Planar image sensors are generally known to those skilled in the art. Particularly preferred here are pixel detectors that enable the electronic recording of image points (“pixels”). Preferred pixel detectors are CCD (“charge-coupled device”) or CMOS (“Complementary metal-oxide-semiconductor”) sensors. Silicon-based sensors are particularly preferred, but InGaAs sensors and lead oxide or graphene-based sensors can also be used, especially for wavelengths outside the visible range.

[0026] A spectral filter element designed to transmit different spectral proportions of incident light at different locations on its surface, while not transmitting other spectral proportions, is referred to herein as a "position-sensitive spectral filter element," or "position-dependent spectral filter element." It is used to filter the mapping generated on the image sensor by the imaging system according to different (smaller) spectral ranges.

[0027] There are several possible arrangements for the filter element. For example, the filter element can be positioned directly in front of the lens array, or between the lens array and the image sensor. Also preferably, components of the imaging system are designed as filter elements, particularly the lens array. For example, the substrate of the lens array can be designed as a filter element.

[0028] The lens array within the scope of this invention comprises a large number of lenses arranged in a grid-like, i.e., regular layout relative to each other, particularly on a carrier. The lenses are preferably arranged in regular rows and columns, or staggered from each other. A rectangular or square layout, or a hexagonal layout, is particularly preferred. The lenses may be, for example, spherical or cylindrical lenses, but in some applications, aspherical lenses are also preferred.

[0029] Typically, (multi / hyper)spectral imaging always produces similar images of the same subject. Images are captured by an image sensor at different (light) wavelengths or within different wavelength ranges using filtering elements.

[0030] Images exist in the form of digital images, and their image elements are called "pixels." These pixels are located at predetermined positions on the image sensor, so each image has a coordinate system for the pixel positions. In the range of multispectral imaging, images are often also referred to as "channels."

[0031] Since captured images are typically stored in image memory, these images can also be retrieved from there using the method. The method can also, of course, work with previously captured "old" images, which can be easily read from memory and thus provide data.

[0032] The center wavelength or correction function can exist in the x,y,λ space, where x and y are the image coordinates or the corresponding coordinates of the image sensor, and λ(x,y) is the center wavelength or the correction value associated with the corresponding coordinates.

[0033] The apparatus of the present invention for determining wavelength deviations in images captured by a multi-lens imaging system, particularly for achieving dispersion calibration and / or improving spectral resolution, comprises the following components:

[0034] A determining unit is configured to determine the color gamut of a region of the filtering element of the multi-lens camera system, the region corresponding to a predetermined region of the image sensor of the multi-lens camera system. This determining unit may include a computing system (or exist as a software module within the computing system) that determines the color gamut based on sensor information, and / or calculates based on information given for the filter, and, depending on the circumstances, also calculates based on the subject to be mapped (e.g., distance).

[0035] A determining unit, designed to determine the center wavelength for at least two pixels in a predetermined region of the image sensor based on a defined color gamut. This determining unit can also be implemented by a computation unit (or exist as a software module in a computational system) capable of evaluating sensor data from the image sensor and thereby determining the center wavelength.

[0036] A processing unit, designed to correct the settings of a region of the image sensor and / or images captured by the image sensor (or a region of the image sensor) based on a determined center wavelength, and / or generate supplementary datasets for the image or for imaging control. This processing unit can also be implemented by a computing unit (or exist as a software module in a computing system) capable of accessing the image sensor or its image storage space, or modifying image data. Where the processing unit is used solely for correcting the image or image sensor, it can also be referred to as a "correction unit."

[0037] The multi-lens camera system of the present invention includes the apparatus according to the invention and / or the method designed to carry out the invention.

[0038] Preferred embodiments of the present invention will now be described. It should be noted that preferred multi-lens imaging systems can also be constructed in a manner similar to the description of the method, and in particular, features from different embodiments can be combined with each other.

[0039] In a preferred method, the color gamut is determined by capturing and / or providing an image of the subject using a multispectral multilens camera system, and / or by calculating the filter characteristics at different incident angles when the wavelength is incident, taking into account the imaging properties of the optics of the multilens camera system corresponding to the region. That is, the color gamut is based on measurement or calculation, where a combination of both can also be used.

[0040] In a preferred method, within a defined center wavelength range, a correction function (“support function”) is determined for pixels in a predetermined region of the image sensor based on the center wavelengths of at least two determined pixels. This correction function can then be used to correct (particularly for adaptation, e.g., calibration) the region of the image sensor and / or images already captured with the image sensor (or that region of the image sensor). This also allows for correction, particularly calibration, of image regions lacking a color gamut. The correction function can also be used to adapt other images, where the color gamuts of these images should be similar.

[0041] In a preferred method, when determining the color gamut, information about the angular correlation of the center wavelength of the transmitted light is added to the color gamut. This allows for corrections even when the light path differs (e.g., the distance from the subject or object). The correction takes into account the distance of the subject's elements, and preferably, the color gamut is determined for the pixels of the image sensor based on the distance between the mapped subject area and the camera system.

[0042] In the preferred method, the parallax of the image is also determined, including the following steps:

[0043] - To capture and / or provide at least two images of an object in different spectral ranges using a multispectral, multi-lens imaging system.

[0044] - Use a digital object recognition system to identify objects in captured images and create virtual objects using the identified object mappings.

[0045] - Determine the coordinates of the virtual object in the image in the form of absolute image coordinates and / or relative coordinates to other elements in the image.

[0046] - Based on the determined coordinates and the known position of the image capture point on the image sensor of the multi-lens camera system, determine the parallax of the object.

[0047] - Preferably: Based on the determined parallax, determine the distance between the object and the multi-lens camera system and / or another object in the image, and use the distance to determine the color gamut and / or center wavelength.

[0048] According to a preferred method, the correction (adaptation) includes calibration of a region of the image sensor or an image captured using that region. That is, the aforementioned correction measures are applied to calibrate the image sensor or image to adjust the spectral information accordingly for each pixel. If the color gamut is not constant, the spectral information reflected by the pixels in the relevant region is now known. In this case, the correction function is preferably a calibration function that includes color gamut-based calibration values ​​for the pixels of the region of the image sensor or the pixels of the image captured using that region, to calibrate the relevant pixels.

[0049] According to a preferred method, the correction includes improving the spectral information of an image captured by an image sensor, wherein at least two pixels in an image are identified, their center wavelengths known, and belonging to an image region for which an approximately identical spectrum is assumed. That is, these pixels are located at different coordinates in the image and have different center wavelengths due to the aforementioned dispersion effects, filter errors, or the use of a linearly variable filter. It is assumed that these pixels all belong to a single object of the subject, wherein the object is assumed to be a region with approximately the same spectrum (where “approximately” means within the desired measurement accuracy range). In this case, the spectral information reflected by all these pixels is as expected at all pixel coordinates (due to the approximately identical spectrum). For this purpose, object recognition is preferably performed in the image, as described elsewhere herein. The spectral information of the pixels is now integrated into total spectral information for at least one pixel (preferably for a group of pixels or all pixels).

[0050] Of course, this (and subsequent) method can be implemented for different spectral channels (spectral images), thus improving spectral resolution.

[0051] According to another preferred method, the correction also includes improving the spectral information of the image captured by the image sensor. The principle of this method is the same as the aforementioned method: two pixels have individual (different) center wavelengths and originate from a region with approximately the same spectrum, these pixels complementing the total spectral information. In the former method, pixels are located in the same image but at different locations on the object, while in the following, pixels are at the same location on the object but observed in different images. Preferably, both methods can be used to further refine the spectral information. Multi-lens imaging systems incorporating linearly variable filtering elements are particularly suitable for both methods, where, in this case, the concept of "different images" can also refer to different image regions or different sub-images, since the filtering element has a linear extension of the center wavelength across the image range.

[0052] The preferred further method includes the following additional steps:

[0053] - To capture at least two images of the same subject from different viewpoints or with the subject moving, using the same area of ​​the image sensor. For example, these images can be captured at different times during relative movement between the camera system and the subject, or by moving the subject within the area of ​​the image sensor. Importantly, these images are not identical; rather, at least one object of the subject is mapped to different coordinates in the images and / or from different viewpoints. This is due to the dispersion effect, which causes different center wavelengths at the same image location. Of course, several images can be captured, each additional image contributing to spectral refinement. Viewpoint variations can also be generated by tilting the filter element or by lateral relative movement of the filter element and the lens array, as the viewpoint depends on the angle at which incident light passes through the filter element. That is, the concept of "filter-through angle" can be used instead of "viewpoint," but "viewpoint" is easier to understand.

[0054] - Identify the pixels in an image that correspond to the same area of ​​the subject. That is, these pixels are located at the same position in the subject (but not necessarily in the image). As described above, these pixels are typically located at different image coordinates, and due to different viewpoints, these pixels should have different center wavelengths at the same image coordinates. Of course, some of these pixels may have the same center wavelength, but this doesn't provide information.

[0055] - Determine the center wavelength of the corresponding pixel. Typically, a set of pixels is obtained, each with a different center wavelength.

[0056] - Integrate the spectral information of pixels into total spectral information for at least one of the pixels. In other words, the relevant pixel now contains the spectral information of the other pixels.

[0057] As an alternative or supplement, the viewing angle used to map the pixels is determined based on the center wavelength, and the intensity of the pixels is determined based on the viewing angle.

[0058] The radiation characteristics of an object (or pixel) can also be determined by viewing it from different spectral ranges, i.e., from different regions of the image sensor. This involves recording the intensity of the object (or pixel) in different channels and plotting an intensity versus viewing angle characteristic curve. Specifically, this is plotted in the form of a bidirectional reflectance distribution function (BRDF). In this case, the viewing angle is, for example, determined by the object's position on the image sensor. Particularly preferably, the object (pixel) is observed from two different viewing angles of the camera.

[0059] Although objects in an image are typically displayed in different spectral channels, in the simplest case, it can be assumed that the radiative properties are the same for all wavelengths, and that the (wavelength-dependent) intensity of each channel can be normalized by the known spectrum of the object. Alternatively, it is, of course, possible to photograph the object in the same channel from two or more perspectives, thereby achieving normalization of the intensity of the spectral channels.

[0060] Particularly preferably, after capturing the first image on the image sensor, the subject is moved, for example, by tilting the mirror or lens array of the multi-lens camera system. This tilting can also be used to capture a series of images. By this movement, the same points or regions of the subject are mapped onto different pixels of the image sensor, and the resulting shift in the center wavelength can be used to improve spectral resolution, as described above. For example, the first image can be captured first, and then, particularly in video form, a sequence of several images can be captured during the movement at a lower resolution or with a reduced exposure time to improve the spectral resolution of the first image. Alternatively or supplementarily, however, lateral relative movement of the filter element and the lens array can be performed as described above, and / or the filter element can be switched, for example, by using a filter wheel or by changing the filter.

[0061] According to a preferred embodiment, the filter element comprises a mosaic filter. Preferably, the mosaics of the mosaic filter are arranged such that larger wavelength steps are on the outside and smaller intervals are on the inside. In a preferred form of the mosaic filter, colored mosaic (especially colored glass mosaic) is applied, particularly deposited, on one side of a substrate (preferably glass). According to an advantageous embodiment, the filter element (mosaic filter or another filter) is applied to the front side of the substrate, while the lens array is applied to the back side of the substrate. Preferably, the mosaic filter transmits a different wavelength for each individual lens.

[0062] According to a preferred embodiment, the filter element comprises a linearly variable filter (“graded filter”) with filter lines, which, in terms of the orientation of the filter lines, is preferably rotated relative to the lens array at an angle between 1° and 45°. Alternatively or supplementarily, the filter element comprises a filter array, particularly preferably a mosaic filter.

[0063] According to a preferred embodiment, the multi-lens imaging system includes an aperture cover located between a lens array and an image sensor, wherein the aperture on the aperture cover is positioned according to the lenses of the lens array, and the aperture cover is positioned such that light mapped from each lens passes through the aperture of the aperture cover. That is, the aperture cover has the same pattern as the lens array, wherein the aperture replaces the lenses.

[0064] To determine wavelength deviations, particularly for improving results in dispersion calibration or spectral resolution, correction methods, especially calibration steps, can be implemented, which, like the methods described above according to the invention, demonstrate the advantages of shooting with a multi-lens camera system. The problem arising from this is that an uncalibrated multi-lens camera system does not provide optimal images. Corrections made by each of the following alternative methods, whether applied individually or in combination, improve the image quality of the multi-lens camera system.

[0065] A preferred calibration method for multispectral multilens imaging systems is used to identify regions of interest (ROIs). The method includes the following steps:

[0066] - An image is captured using a multispectral, multi-lens camera system, wherein the image preferably has a uniform brightness distribution, or its brightness causes the image sensor to be overexposed. The advantage of overexposure is that, in this case, areas in the captured image that receive less light due to occlusion effects (e.g., due to aperture edges) become visible. These areas are no longer classified as Regions of Interest (ROIs) because (due to occlusion effects) it is not guaranteed that the image information there is optimal. An image is then produced in which only the areas visible to the ROIs are illuminated.

[0067] - Cross-correlate the image with a comparison image, especially with a reference image (e.g., an ideal sensor image).

[0068] -Optional: Perform a Hough transform. The Hough transform can make the angles of the image consistent.

[0069] - Select regions from the image that have the highest similarity based on autocorrelation and / or cross-correlation. This selection preferably includes either isomorphizing the selected regions or cropping the image to limit it to the selected regions. Preferably, within this framework, corresponding predefined elements are made in the reference image, and object segmentation is performed on the captured image.

[0070] A preferred calibration method for a multispectral multilens camera system is provided for correcting lens errors. The method includes the following steps:

[0071] - An image of a pre-known optical target is captured using the lens to be tested in a multi-lens camera system. Preferably, each channel of the multi-lens camera system is imaged, allowing the entire system to be studied in light of lens errors.

[0072] Based on these images and the known size of the optical target, the optical lens error is preferably calculated using the Levenberg-Marquard algorithm with the given lens parameters.

[0073] A preferred calibration method for a multispectral multilens camera system is provided for calibrating projection errors. The method includes the following steps:

[0074] - Provides images of pre-known optical targets by means of a multi-spectral, multi-lens camera system (either by capturing images or by retrieving them from a data storage device).

[0075] - Identify targets in an image, and preferably determine at least three coordinates of a representation point of the target. A representation point may be, for example, an angle of the target.

[0076] - Inverse homography is used. This refers to the collinearity of a 2D real projection space with itself, i.e., the image projection in space, which can be implemented by an algorithm using a (digital) image. The target is adjusted based on the target that will be mapped at a specific distance without projection errors. This ideal mapping is known because the properties of the mapping optics are known, and the target is known. Preferably, in this case, only the coordinates of a representative reference point are used, and the coordinates of the corresponding representative points of the target identified in the image are recorded at the reference point. With this preferred method, translation (movement), rotation, shearing, and scaling can be corrected using homography.

[0077] A preferred calibration method for a multispectral multilens camera system is provided to improve its resolution. The method includes the following steps:

[0078] - Capture at least two images, one with higher spatial resolution than the other, and the other with higher spectral resolution. Preferably, a large number of images with lower spatial resolution are captured in different spectral ranges, and a pan image or grayscale image with higher spatial resolution is also captured.

[0079] Optionally, the multi-lens camera system can be further calibrated, particularly using the aforementioned method for calibrating projection errors. Particularly preferably, the parallax of the object in the image is predetermined, and the parallax is compensated for.

[0080] - Based on the at least two images, extrapolate the image with higher spatial resolution or spectral resolution. Specifically, the higher spatial resolution of one image is used to improve the spatial resolution of the image with higher spectral resolution, and / or the higher spectral resolution of the other image is used to improve the spectral resolution of the image with higher spatial resolution.

[0081] In particular, among several preferred single-channel (spectral images), the spatial resolution is increased based on information from images with higher spatial resolution. Preferably, the known Pan-Sharpening principle is utilized. This leverages the fact that when observing a subject, one pixel in one image of the spectral channel corresponds to a group of pixels in an image with higher spatial resolution, such as 10×10 pixels in a Pan image. Preferably, the corresponding pixel group (e.g., 10×10 spectral pixels) is generated from one pixel of the (spectral) image, specifically by using the spectral form from the original spectral pixels, but with brightness derived from the Pan image.

[0082] As an alternative or supplement, the spectral resolution of an image with high spatial resolution can be improved. This can be achieved if, as described above, there exists a first image with high spatial resolution and low spectral resolution (but must have at least three channels), and a second image with low spatial resolution and high spectral resolution. By interpolating the missing spectral channels of the first image based on the information of the second image, the spectral resolution of the first image is improved, making it (nearly) the spectral resolution of the second image. This is preferably achieved by assigning a pixel of the second image to a pixel group of the first image, and assigning the spectral information of the pixel of the second image to that pixel group. For example, if the spatial resolution of the first image is Z times larger, each block (pixel group) of Z×Z pixels in the first image can be assigned to a pixel in the second image located at the corresponding image position (taking the block into account).

[0083] However, the results can be further improved by performing object recognition within the first image (or within a group of pixels). In this case, it is assumed that the spectra within an object are approximately the same. Object recognition can also be improved using information from the second image, where portions of the object with different spectra are separated from each other and treated as independent objects. Subsequently, in the first image, for example, there may be different adjacent regions separated from each other by edges. Different regions of different objects are treated differently within a group of pixels in the first image, specifically by assigning different spectra to these regions. This is easy to implement for uniform groups of pixels: the spectral information of the corresponding pixels in the second image is assigned to these regions (see the method described above). If portions of different objects exist within a group of pixels, the information from the second image for one of the objects is used for that object, and the information from the second image for another object is used for that other object. This is done accordingly for each additional portion of the object. In this case, the spectral information of the corresponding pixels in the second image may be a convolution of the spectra of different objects; therefore, the spectra of adjacent pixels in the second image, which contain information about the objects involved, can be added here. As an alternative or supplement, additional object recognition can be performed in the second image, which corresponds to the objects in the first image, and the corresponding spectral information can be assigned to these objects in the information of the second image.

[0084] If the color channels are non-uniform, meaning that for a pixel in one channel (image), the center wavelength follows a trend that goes beyond the image level, this information can also be used to improve the spectrum. For this purpose, we again assume that the spectrum within the object is uniform, at least for two adjacent points within the object. In the case of non-uniform color channels, when photographing a uniform subject, the spectra of two adjacent pixels have slight differences. One pixel "views" the subject at wavelength w, and the second pixel at wavelength w ± Δw. If these two pixels in an image are found to represent a (assuming uniform) object, then not only can wavelength w be assigned to the object, but also wavelength w ± Δw, thus refining the object's spectrum in this respect.

[0085] In one exemplary case, an image has a spatial resolution of 50×50 pixels and a spectral resolution of 500 channels, and another image has a spatial resolution of 500×500 pixels and a spectral resolution of 10 channels. Accordingly, the spatial information of the resulting image can be adjusted to 500×500 using (partial) Pan-Sharpening, and the spectrum can be adjusted to 500 using "Spectral Sharpening". The result is a spatial resolution of 500×500 pixels and a spectral resolution of 500 channels.

[0086] The following methods can be combined with the methods described above in a particularly simple way, but do not necessarily require two images in their basic form. More precise spectral information can also be determined from a single image.

[0087] Preferably, for each pixel's spectrum, at least the spectra of its directly adjacent pixels are added. As previously mentioned, these adjacent spectra contain additional wavelength information (additional control points due to other center wavelengths). This simple approach improves spectral resolution but may lead to errors in transition regions between different objects within the subject matter.

[0088] Even under these conditions, object recognition within the image can still improve the results. Here, it is again assumed that the spectrum within the object is approximately uniform. If the spectral differences within the object are significant, object segmentation can be re-performed, where portions of the object with different spectra are separated and treated as independent objects. Subsequently, different adjacent regions may exist in the image, separated by edges. The spectra of pixels within an object (especially pixels from the object's center) are then combined. This combination can be combining the spectra of adjacent pixels or combining the spectra of all pixels. The first approach achieves acceptable improvement even with slightly non-uniform objects, while the second approach achieves very high resolution even with uniform objects. The two alternatives can be combined, where the object is divided into concentric regions, and the spectra of pixels in these regions are combined.

[0089] Other preferred calibration methods for multispectral multilens camera systems are known methods for calibrating dark current and / or white balance and / or radiometric calibration and / or photoresponse inhomogeneity (“PRNU”) calibration. Attached Figure Description

[0090] Examples of preferred embodiments of the present invention are shown in the accompanying drawings.

[0091] Figure 1 A multi-lens camera system according to the prior art is shown.

[0092] Figure 2 The shooting scene is shown.

[0093] Figure 3 The shooting scene and a multi-lens camera system including an embodiment of the device according to the invention are shown from top to bottom.

[0094] Figure 4 An example of the dispersion effect is shown.

[0095] Figure 5 Examples are shown for the center wavelength and for the correction function.

[0096] Figure 6An example of spectral improvement for captured images is shown.

[0097] Figure 7 Another example of spectral improvement for two captured images is shown.

[0098] Figure 8 An exemplary block diagram of the method according to the present invention is shown. Detailed Implementation

[0099] Figure 1 This is a schematic perspective view of a multi-lens imaging system 1 for hyperspectral imaging according to the prior art. The multi-lens imaging system 1 includes a planar image sensor 3 and a planar lens array 2 composed of uniform individual lenses 2a, which generate a large number of first maps AS arranged in a grid pattern on the image sensor 3 via a subject M (see e.g., only referenced). Figure 5 The smaller first mapping (AS) is arranged in a manner that, for clarity, only one of the individual lenses 2a is indicated by reference numerals.

[0100] To improve the quality of the first mapping AS, an aperture cover 5 is provided between the image sensor 3 and the lens array 2. Each aperture 5a of the aperture cover 5 corresponds to a single lens 2a and is arranged directly behind it. To obtain spectral information, a filter element 4 is provided between the aperture cover 5 and the image sensor 3. In other embodiments, the filter element 4 may also be arranged in front of the lens array (e.g., see [reference]). Figure 8 In the case shown here, filter element 4 is a linearly variable filter that is slightly rotated relative to the image sensor. Therefore, the center of each map lies within a different wavelength range of the filter element. Consequently, each first map AS on the image sensor provides different spectral information, and the aggregate of the first maps AS is used to create an image containing spectral information.

[0101] Figure 2 This shows a shooting scene for subject M. The subject includes a house (H) used as the background and a tree (O) used as the foreground. The subject was shot using a multi-lens camera system 1.

[0102] Figure 3 Show from top to bottom Figure 2 Subject M. The multi-lens camera system 1 herein includes one embodiment of the device 6 of the present invention. The device includes a data interface 7, a determination unit 8, a determination unit 9, and a processing unit 10.

[0103] Data interface 7 is designed to receive images captured by the multi-lens camera system 1. This data interface can, for example, directly access the image sensor, access the storage unit, or communicate with a network.

[0104] The determining unit 8 is designed to determine the color gamut F of a region of the filter element 4 of the multi-lens camera system 1, the region corresponding to a predetermined region of the image sensor 3 of the multi-lens camera system 1.

[0105] The determining unit 9 is designed to determine the center wavelength Z for at least two pixels P1 and P2 in a predetermined area of ​​the image sensor 3, based on the determined color gamut F.

[0106] Processing unit 10 is designed to correct the settings of the region of image sensor 3 and / or correct the image B captured by image sensor 3. The correction is performed using the determined center wavelength Z. Alternatively or supplementarily, the processing unit is designed to generate supplementary datasets for the image or for imaging control.

[0107] Figure 4 An example of dispersion effect is shown. Two beams of light with extended spectra (arrows) pass through filter element 6 at different angles and therefore take different paths within filter element 6. This causes a shift in the center wavelength Z. Furthermore, filter element 6 can allow different wavelengths to pass through at different locations, which similarly results in different center wavelengths (even if the beams are parallel).

[0108] Figure 5 Examples are shown for the center wavelength Z (left) and for the correction function, which is the adaptation function A (right). For the wavelength passing through filter element 6 (see, for example...) Figure 4 For a beam with a broad spectrum, a narrow spectrum with a center wavelength Z is produced after passing through it. If, for a pixel domain with coordinates x and y, all center wavelengths Z, or calibration values ​​derived from these center wavelengths Z, are input into a graph that combines coordinates x and y with wavelengths λ (e.g., center wavelength Z), a distribution is obtained, for example, as shown on the right side of the figure. This is an exemplary demonstration of the adaptor function A as an example of a correction function.

[0109] Figure 6 An example of spectral improvement for a captured image B is shown. In this example, it is assumed that the entire canopy of the tree representing object O has the same spectrum, meaning that each pixel P1, P2 reflecting the canopy should reflect the same spectral value. However, pixels P1 and P2 are located at different positions, and therefore also have different center wavelengths Z (see, for example, [reference needed]). Figure 5 (The distribution on the right). This means that the intensities of the two pixels P1 and P2 reflect information from different parts of the total spectrum. In other words, the center wavelength Z of the two pixels P1 and P2 can be assigned to each pixel P1 and P2.

[0110] Figure 7 This illustrates another example of spectral improvement for two captured images. The principle is the same as... Figure 8Very similar, the difference being that the two pixels are located at different coordinates in two different images, but at the same object coordinates in the tree (object O). Here, each pixel P1, P2 also reflects the same spectral value. However, since pixels P1, P2 are located in different regions of image sensor 3, they also have different center wavelengths Z (see, for example, [reference needed]). Figure 5 (The distribution on the right). This means that the intensities of the two pixels P1 and P2 reflect information from different parts of the total spectrum. In other words, the center wavelength Z of the two pixels P1 and P2 can be assigned to each pixel P1 and P2.

[0111] Figure 8 This is an exemplary block diagram of a method for determining the parallax of an image captured by a multi-lens camera system 1 according to the present invention.

[0112] In step I, a color gamut F is determined for a region of the filter element 4 of the multi-lens camera system 1, which corresponds to a predetermined region of the image sensor 3 of the multi-lens camera system 1.

[0113] In step II, based on the determined color gamut F, the center wavelength Z is determined for at least two pixels (P1, P2) in a predetermined area of ​​the image sensor 3.

[0114] In step III, based on the determined center wavelength Z, the region of image sensor 3 and / or the images B and B1 captured by image sensor 3 are corrected, and / or a supplementary dataset for said images or for controlling image acquisition is generated.

[0115] The preferred modifications are further subdivided in step 3.

[0116] In step IIIa, calibration is performed on the area of ​​image sensor 3 or on image B captured using that area (as an example of correction). (See also for details) Figure 6 ).

[0117] In step IIIb, the spectral information of the images captured by image sensor 3 is improved, wherein at least two pixels P1 and P2 are identified in a certain number of images B and B1, whose center wavelength Z is known and which belong to an image region, and the same spectrum is assumed for the image region, and wherein the spectral information of pixels P1 and P2 is integrated into total spectral information for at least one of pixels P1 and P2 (see also for details). Figure 7 This improvement in spectral information can be an image correction or it can exist in the form of a supplementary dataset.

[0118] Finally, it should be noted that the use of terms such as "a" or "an" does not preclude the possibility that the involved feature may exist multiple times. Accordingly, "a" can also be understood as "at least one". Concepts such as "unit" or "device" do not preclude the possibility that the involved element consists of several components that work together, which are not necessarily mounted in a common housing, although an encapsulated housing is preferred. Within the scope of optical devices, lens elements can in particular consist of a single lens, a lens system, or an objective lens, without the need for precise distinction.

[0119] Appendix Label Table

[0120] 1. Multi-lens camera system

[0121] 2 Lens Array

[0122] 2a Single lens

[0123] 3 Image Sensor

[0124] 4. Filter element

[0125] 5-aperture cover

[0126] 5a aperture

[0127] 6 devices

[0128] 7 Data Interface

[0129] 8. Determine the unit

[0130] 9. Determine the unit

[0131] 10 processing units

[0132] A adapt function

[0133] Image B

[0134] B1 image

[0135] F color gamut

[0136] H Background

[0137] M theme

[0138] O object

[0139] Z center wavelength

Claims

1. A method for determining the wavelength deviation of an image captured by a multi-lens camera system (1), comprising the steps of: - The color gamut (F) is determined by direct measurement of a region of the filter element (4) of the multi-lens camera system (1), which corresponds to a predetermined region of the image sensor (3) of the multi-lens camera system (1), wherein, The color gamut (F) represents the spectral sensitivity of the region and includes information about the center wavelength (Z) of light passing through the filter element (4) and incident on the pixels of the region of the image sensor (3), wherein the color gamut (F) is determined by capturing and / or providing an image (B, B1) of the subject (M) using a multispectral multilens camera system (1). -Based on the determined color gamut (F), determine the center wavelength (Z) for at least two pixels (P1, P2) in a predetermined area of ​​the image sensor (3). - Based on the determined center wavelength (Z), the region of the image sensor (3) and / or the images (B, B1) captured by the image sensor (3) are corrected, and / or a supplementary dataset for the images or for imaging control is generated.

2. The method according to claim 1, characterized in that, The color gamut (F) is determined by calculating the characteristics of the filter element (4) when the wavelength is incident at different incident angles, taking into account the imaging properties of the optical device (2) of the multi-lens imaging system (1) corresponding to the region.

3. The method according to any one of the preceding claims, characterized in that, Within the range of the determined center wavelength (Z), a correction function (A) is determined for the pixels (P1, P2) of a predetermined region of the image sensor (3) based on the determined center wavelength (Z) of at least two pixels (P1, P2), and the correction function (A) is used to correct the region of the image sensor (3) and / or the image (B, B1).

4. The method according to claim 1, characterized in that, When determining the color gamut (F), information about the angular correlation of the center wavelength (Z) of the transmitted light is added to the color gamut, and the distance of the elements of the subject (M) is taken into account during correction.

5. The method according to claim 1, characterized in that, in, The color gamut (F) of the pixels (P1, P2) of the image sensor (3) is determined based on the distance between the region of the mapped subject (M) and the multi-lens camera system (1).

6. The method according to claim 1, characterized in that, Also determine the parallax of the shot, including the following steps: - By using a multi-spectral multi-lens imaging system (1), at least two images (B, B1) of an object (O) are captured and / or provided in different spectral ranges. - Use a digital object recognition system to identify objects (O) in the captured images (B, B1) and create virtual objects (O) using the mapping of the identified objects (O). - Determine the coordinates of the virtual object (O) in the images (B, B1) in the form of absolute image coordinates and / or relative coordinates to other elements in the images (B, B1). - Determine the parallax of the object (O) based on the determined coordinates and the known position of the shooting point of the image (B, B1) on the image sensor (3) of the multi-lens camera system (1).

7. The method according to claim 6, characterized in that, Based on the determined parallax, the distance between the object (O) and the multi-lens imaging system (1) and / or another object (O) in the image (B, B1) is determined, and the distance is used to determine the color gamut (F) and / or the center wavelength (Z).

8. The method according to claim 1, characterized in that, The correction includes calibration of the region of the image sensor (3) or of the images (B, B1) captured by the image sensor (3).

9. The method according to claim 3, characterized in that, in, The correction function (A) is a calibration function that includes a color gamut (F)-based calibration value for pixels (P1, P2) in the region of the image sensor (3) or pixels in the image (B, B1) captured using the region, to calibrate the relevant pixels (P1, P2). And / or wherein, for calibration, the radiation characteristics of a point of the object and / or subject are determined, wherein in different images captured by the image sensor, the incident angle of the beam of light at the point of the object or subject is determined, and thereby the characteristic curve of the intensity of the corresponding image point as a function of the incident angle is determined.

10. The method according to claim 1, characterized in that, The corrected and / or generated dataset includes improved spectral information captured by the image sensor (3), wherein at least two pixels (P1, P2) are identified in the image (B, B1) with known center wavelengths (Z) and belonging to an image region, the image region is assumed to have the same spectrum, and wherein the spectral information of the pixels (P1, P2) is integrated into total spectral information for at least one of the pixels (P1, P2).

11. The method according to claim 1, characterized in that, To improve the spectral information captured by the image sensor (3), the following steps are performed: - Take at least two images (B, B1) of the same subject (M) from different perspectives or when the subject moves through the same area of ​​the image sensor (3). - Determine the pixels (P1, P2) of the image (B, B1) that correspond to the same region of the subject (M). - Determine the center wavelength (Z) of the corresponding pixels (P1, P2). a) Integrate the spectral information of the pixels (P1, P2) into total spectral information for at least one of the pixels (P1, P2). and / or b) Determine the viewing angle for mapping pixels based on the center wavelength, and determine the intensity of pixels based on the viewing angle.

12. An apparatus (6) for determining wavelength deviation of an image captured by a multi-lens camera system (1), comprising: - Determining unit (8), which is designed to determine the color gamut (F) of a region of the filter element (4) of the multi-lens camera system (1) by direct measurement, the region corresponding to a predetermined region of the image sensor (3) of the multi-lens camera system (1), wherein the color gamut (F) represents the spectral sensitivity of the region and includes information about the center wavelength (Z) of light passing through the filter element (4) and incident on the pixels of the region of the image sensor (3), wherein the color gamut (F) is determined by capturing and / or providing an image (B, B1) of the subject (M) with the multi-spectral multi-lens camera system (1). - Determining unit (9), which is designed to determine the center wavelength (Z) for at least two pixels (P1, P2) of a predetermined area of ​​the image sensor (3) based on the determined color gamut (F). - Processing unit (10), which is designed to correct the region of the image sensor (3) and / or the image (B, B1) captured by the image sensor (3) based on the determined center wavelength (Z), and / or generate a supplementary dataset for the image or for imaging control.

13. A multi-lens camera system (1) comprising the apparatus as described in claim 12 and / or designed to perform the method as described in any one of claims 1 to 11.

Images