Sensor device and method for operating a sensor device for capturing a hyperspectral image

Abstract

A sensor device (100) for capturing a hyperspectral image of an object (200) comprises a localization unit (110) that is configured to determine a position and an orientation of the sensor device (100) with respect to the object (200), a light emitting unit (120) that is configured to emit light of a predetermined wavelength band to the object (200), an image capturing unit (130) comprising an optical system (131) and an image sensor (135) having a plurality of pixels (136), which optical system (131) is configured to project reflected light that has been reflected from the object (200) onto the image sensor (135) such that specific wavelength of the reflected light are projected to specific, corresponding locations on the image sensor (135), the specific wavelength bands being part of the predetermined wavelength band, and which image sensor (135) is configured to capture a spectral image of a scene that contains the object (200) by detecting for each pixel (136) the intensity of light received at said pixel (136), and a control unit (140) that is configured to generate a hyperspectral image of the object (200), which hyperspectral image indicates for each region of the object (200) the intensity of light in the specific wavelength bands, based on a plurality of spectral images of the object (200) captured from different points in space and the positions and orientations of the sensor device (100) with respect to the object (200) during capturing of said spectral images.

Description

SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE FOR CAPTURING AHYPERSPECTRAL IMAGEFIELD OF THE INVENTIONThe present disclosure relates to a sensor device and a method for operating a sensor device. In particular, the present disclosure is related to capturing hyperspectral images of an object.BACKGROUDIn recent years techniques for measuring the chemical composition of objects via the measurement of spectra of light reflected by the object were developed, the light having a known wavelength before reflection. In particular, techniques for capturing hyperspectral images of an object were developed, i.e. for capturing images of an object, where an entire spectrum is assigned to each pixel of the image.SUMMARY OF THE INVENTIONHowever, present techniques for capturing hyperspectral images rely on static, non-portable equipment. It is desirable to allow capturing of hyperspectral images with a sensor device that is freely moveable in space.The present disclosure mitigates this shortcoming of conventional hyperspectral image capturing devices.To this end, a sensor device for capturing a hyperspectral image of an object is provided, which comprises a localization unit that is configured to determine a position and an orientation of the sensor device with respect to the object, a light emitting unit that is configured to emit light of a predetermined wavelength band to the object, and an image capturing unit comprising an optical system and an image sensor having a plurality of pixels. Here, the optical system is configured to project reflected light that has been reflected from the object onto the image sensor such that specific wavelength bands of the reflected light are projected to specific, corresponding locations on the image sensor, the specific wavelength bands being part of the predetermined wavelength band, and the image sensor is configured to capture a spectral image of a scene that contains the object by detecting for each pixel the intensity of light received at said pixel. The sensor device further comprises a control unit that is configured to generate a hyperspectral image of the object, which hyperspectral image indicates for each region of the object the intensity of light in the specific wavelength bands, based on a plurality of spectral images of the object captured from different points in space and the positions and orientations of the sensor device with respect to the object during capturing of said spectral images.Further, a method for operating an according sensor device to capture a hyperspectral image of an object comprises: by the light emitting unit, emitting light of a predetermined wavelength band to the object; by the optical system, projecting reflected light that has been reflected from the object onto the image sensor such that specific wavelength bands of the reflected light are projected to specific, corresponding locations on the image sensor, the specific wavelength bands being part of the predetermined wavelength band; by the image sensor, capturing a spectral image of a scene that contains the object by detecting for each pixel the intensity of lightreceived at said pixel; by the localization unit, determining a position and an orientation of the sensor device with respect to the object during capturing of the spectral images; and by the control unit, generating a hyperspectral image of the object, which hyperspectral image indicates for each region of the object the intensity of light in the specific wavelength bands, based on a plurality of spectral images of the object captured from different points in space and the positions and orientations of the sensor device with respect to the object during capturing of said spectral images.The hyperspectral image is thus obtained from two datasets. First, there is a series of spectral images of the object, i.e. of images showing partial spectra of partial regions of the object, which spectral images are captured from different positions in space. Second, there is the knowledge about these capturing positions. Thus, it is possible to identify in the spectral images corresponding regions of the object and to extract therefrom the entire spectrum even if its parts are only contained in different spectral images, without the need for a fixed spatial relationship between sensor device and object.BRIEF DESCRIPTION OF THE DRAWINGSFig. 1 is a simplified block diagram of a sensor device for capturing a hyperspectral image;Fig. 2 shows in a simplified manner an operation principle of the sensor device;Fig. 3 shows in a simplified manner a determination of distance and orientation between the sensor device and an object;Fig. 4 shows in a simplified manner an optical bandpass filter array;Fig. 5 shows in a simplified manner another optical bandpass filter array;Fig. 6 shows in a schematic manner an example for generation of a hyperspectral image;Fig. 7 shows in a schematic manner an example of an image capturing unit of the sensor device;Fig. 8 shows in a schematic manner another example for generation of a hyperspectral image;Fig. 9 shows in a schematic manner another example for generation of a hyperspectral image;Fig. 10 schematically shows a process flow for operating a sensor device for capturing a hyperspectral image;Figs. 11 A and 1 IB show schematically different exemplary applications of the sensor device.Fig. 1 is a schematic illustration of a sensor device 100 for capturing a hyperspectral image of an object 200 that is located in a scene.Here, a hyperspectral image is an image of the scene / the object 200, where each pixel of the image contains an electromagnetic spectmm within a predetermined range, like e.g. infrared, IR, visible, ultraviolet, UV, or combinations thereof. This means that not only an average or superposition of all intensities of captured light of different wavelengths is provided as a single value, but that light intensities for a plurality of particular wavelengths or particular wavelengths bands are provided for a single pixel. Usually such hyperspectral images are captured under a fixed relation between camera and image object, like e.g. in an analysis device in a laboratory, or movements of the camera with respect to the imaged object are small compared to the distance between camera and object, as e.g. in airborne applications in which Earth’s surface is imaged from an airplane.However, such hyperspectral imaging devices are not portable and do not allow movements of the camera that are in a comparable range as the distance between camera and object. In the following it is described how the sensor device 100 mitigates these problems.The sensor device 100 comprises a localization unit 110, a light emitting unit 120, an image capturing unit 130 and a control unit 140.The localization unit 110 is configured to determine a position and an orientation of the sensor device 100 with respect to the object 200. This means that the localization unit 110 is able to determine at least its position and orientation in space with respect to the object 200 of which a hyperspectral image is to be obtained. For example, localization might be performed by using GPS signals. To this end, the sensor device 100 can be brought to the position of the object 200 and this mutual position is determined by the sensor device 100 via GPS. Movements relative to this initial position can then be tracked by using further GPS measurements.More preferably, the localization unit 110 comprises a visual camera 112 that is configured to capture visible light images of the object 200 and / or an inertial measurement unit 114 that is configured to track motions of the sensor device 100 through space.This means that images of the object 200 are taken by the sensor device 100, which allows e.g. determining distance and orientation of the sensor device 100 with respect to the object 200 based on the appearance of the object 200 on the images. In particular, from changes of the size of the object 200 changes in distance can be deduced. Similarly, changing orientation of the object 200 in the image is a direct result of a changing orientation of the sensor device 100.Alternatively or additionally the inertial measurement unit 114 can determine linear and rotational accelerations of the sensor device 100 and can deduce therefrom movements of the senor device 100 through space.Preferably, the data of the visual camera 112 and the inertial measurement unit 114 are fused to provide an improved determination of the sensor devices 100 position and / or orientation. Also for this kind of determination of relative positions and orientations, it may be necessary to initialize the determination process at the position of the object 200 such as to determine relative distances and orientations by tracking movements of the sensor device 100 that start at the object 200.The localization unit 110 may determine the position and the orientation of the sensor device 100 with respect to the object 200 based on the captured visible light images and the tracking of motion of the sensor device 100 by using a simultaneous localization and mapping, SLAM, algorithm. Thus, the localization unit 110 (and / or the control unit 140) determines from the captured images and the measured accelerations a map of the surroundings of the sensor device 100 as well as the position of the sensor device 100 within this map. Further, by using this information, the localization unit 110 (and / or the control unit 140) determines where in the map the object 200 is located based on images of the object 200. This allows determining the distance to the object 200, while the acceleration measurements allow determining of the orientation of the sensor device 100. Here, any applicable SLAM algorithm can be used. Since SLAM algorithms are known to a skilled person, it is not necessary to explain them in further detail here.Of course, the localization unit 110 may determine the position and / or the orientation of the sensor device 100 with respect to the object 200 in any known manner. The particular way of determining the position and / orientation is arbitrary as long as the sensor device 100 is able to determine its positional relation to the object 200.The light emitting unit 120 of the sensor device 100 is configured to emit light of a predetermined wavelength band to the object 200. In particular, the light emitting unit 120 may emit light in a continuous wavelength band starting at wavelength / . I and ending at wavelength LN. Preferably, the emission spectrum is flat, i.e. all emitted wavelengths have the same intensity. However, also a varying emission spectrum may be used as long as the intensity distribution across the predetermined wavelength band is known. The predetermined wavelength band may in principle be any part of the electromagnetic spectrum that is of interest for the hyperspectral imaging. However, it is preferable that the predetermined wavelength band is from the infrared spectrum, i.e. included wavelengths between 780 nm and 1 mm. But also predetermined wavelength bands from the visible light band (wavelengths 380 nm to 750 nm), the ultraviolet light band (wavelengths 100 nm to 380 nm) or any combination of the IR light band, the visible light band and the UV light band are conceivable. The light emitting unit 120 may comprise one or a plurality of LEDs for generating the light.The particular predetermined wavelength band to use depends on the hyperspectral image that shall be captured, since the wavelengths of the predetermined wavelength band will determine which wavelengths can possibly be present in the hyperspectral image. The light emitted from the light emitting unit 120 will impinge on the object 200. A part of the light will be absorbed by the object 200, while the rest will be reflected by the object 200. Thus, what can be measured eventually by the sensor device 100 is the complement of the absorption spectra of the elements / molecules located at the surface of the object 200. Thus, by using e.g. an IR light source, (complements of) IR absorption spectra of the surface molecules of the object 200 can be obtained, the same being true for visible light sources or UV light sources. This enables in the end a classification of the surface molecules of the object 200 via measurement of the complements of their absorption spectra. Further, this classification can be improved by using light sources matching the absorption bands of the expected constituent molecules of the object 200. For example, the predetermined wavelength band could be adapted to the absorption band(s) of a particular molecule that is to be detected, as e.g. an explosive, an allergen, a drug or the like. The sensor device 100 can then be operated as a molecule detector.The image capturing unit 130 of the sensor device 100 comprises an optical system 131 and an image sensor 135 having a plurality of pixels 136. The optical system 131 is configured to project reflected light that has been reflected from the object 200 onto the image sensor 135 such that specific wavelength bands of the reflected light are projected to specific, corresponding locations on the image sensor, the specific wavelength bands being part of the predetermined wavelength band. In particular, the optical system 131 maps specific regions of spacespectrum to specific pixels.That is, the partially absorbed light of the predetermined wavelength band is reflected from the object 200 and guided by the optical system 131 onto the pixels 136 of the image sensor 135. To this end, the optical system 131 comprises standard components such as lenses or mirrors that can change the light path of the reflected light. In addition, the optical system 131 is also capable to redirect or block light according to its wavelength such that not all pixels 136 receive all possible components of the light, but that specific pixels 136 receive only light in specific wavelength bands (or even only specific single wavelengths) that can be seen as subbands of the predetermined wavelength band. Thus, the reflected light is spectrally resolved onto the pixels 136 of the image sensor 135. The manner how this is done is in principle arbitrary as long as it is known which part of the spectrum that is reflected on the object 200 is directed to which pixel 136. Moreover, it is to be understood that many different ways to achieve such a spectral resolution are known to a skilled person. The specific examples that will be discussed below in this regard are therefore to be understood as explanatory, but not as limiting.After the reflected light has been directed to the image sensor 135, a spectral image of a scene that contains the object 200 is captured therein by detecting for each pixel 136 the intensity of light received at said pixel 136. The spectral images obtained in this manner can be considered partial hyperspectral images in that different parts of the spectrum of interest are captured by different pixels 136, i.e. for different parts of the object 200. However, the spectral images do not contain all spectral information for all parts of the object 200. The pixels 136 of the image sensor 135 may comprise single photon avalanche diodes, SPADs. However, the image sensor 135 may also be a conventional CMOS image sensor. In fact, the constitution of the image sensor 135 is arbitrary as long as it is capable to detect intensities of light in the specific wavelength bands.The control unit 140 of the sensor device is configured to generate the hyperspectral image of the object 200, which hyperspectral image indicates for each region of the object 200 the intensity of light in the specific wavelength bands, based on a plurality of spectral images of the object 200 captured from different points in space, and based on the positions and orientations of the sensor device 100 with respect to the object 200 during capturing of said spectral images. Here, the control unit 140 may be a processor, like e.g. a CPU, a GPU, a FPGA or the like. In fact, the control unit 140 may be any kind of hardware, software or combination of both that allows carrying out the below described functions of the control unit 140. The control unit 140 may carry out these functions rule-based or by using an artificial intelligence model.In the process of capturing a hyperspectral image, while the image capturing unit 130 captures a spectral image, the localization unit 110 determines the distance and the orientation between the object 200 and the sensor device 100 and provides these data to the control unit 140. This process is repeated several times until a plurality of spectral images that have been captured from different viewing angles and the corresponding position / orientation information has been obtained. Due to the movement between the capturing of the spectralimages, the different spectral images will contain different distributions of intensities of the specific wavelength bands at the different pixels 136 and will show different parts of the object 200. Thus, different spectral images will contain different spectral information for the same region of the object 200.To gather this spectral information such as to provide the entire spectrum for each region of the object 200, the position information obtained by the localization unit 110 is used. In particular, from the relative distance and orientation between sensor device 100 and object 200 it is possible to determine which parts of the spectral images show the same part of the object 200. By extracting all spectral information for such a corresponding region of the object 200 from different spectral images, an entire spectrum can be obtained for this region. Carrying out this process for all regions of the object 200 allows to obtain a full hyperspectral image of the object 200. Here, it may be possible that the control unit 140 informs the user of the sensor device 100 that more spectral images need to be captured in order to obtain all spectral information for all regions of interest on the object 200.Fig. 2 illustrates how in the above process the control unit 140 is configured to determine from the determined positions and orientations of the sensor device 110 with respect to the object 200 which regions of the object 200 were captured by which spectral image, and for which of the specific wavelength bands intensities were detected for said regions, in order to generate the hyperspectral image.In Fig. 2 four spectral images SI, S2, S3, and S4 are shown. In each of these spectral images different pixels 136 receive only light in particular wavelength bands Bl, B2, B3, and B4 as illustrated at the bottom of Fig. 2, and as indicated by the different hatchings in Fig. 2. The distribution of these specific wavelength bands across the pixels 136 is the same for all spectral images SI to S4 in this example. Each spectral image contains an image of the object 200 at a different position. The spectral images SI to S4 are chosen such (either by guided capturing or by choosing among a larger plurality of spectral images) that a region of interest (black square) is imaged in each of the spectral images SI to S4 by a different pixel that is sensitive to a different one of the specific wavelength bands Bl, B2, B3, B4. Thus, combining the intensity information for these different specific wavelength bands, a hyperspectral image of the region of interest can be obtained. Of course, by the same principle, a hyperspectral image of the entire object 200 can be obtained. Moreover, the number of spectral images and the number of specific wavelength bands is limited to four only to ease the description. The number of specific wavelength bands can be arbitrary and will eventually determine the wavelength resolution of the full spectrum. Moreover, the number of spectral images necessary to generate the hyperspectral image will vary according to the properties of the object 200 and the number of specific wavelength bands.In the above manner, it is therefore possible to obtain a full hyperspectral image by capturing a plurality of spectral images containing partial information of the full spectrum for different parts of the captured object. By registering corresponding parts of the object 200 in these images based on the position information obtained during capturing, it is possible to complete the desired spectra for all the points on the object 200. In this manner, a hyperspectral image can be obtained by a handheld device, like e.g. a cell phone or a portable camera. Moreover, hyperspectral information is not limited to a two-dimensional view on the object 200. Due to the possibility to move the sensor device 100 around the object 200 a three-dimensional hyperspectral “image” can be obtained, where spectral information is determined for voxels of the object 200 / the scene instead of pixels.Thus, it is even possible to obtain more information on the object 200 by the sensor device 100 as described above than it was possible with known hyperspectral imaging devices.The control unit 140 may further be configured to normalize, before generating the hyperspectral image, the detected intensities for each of the regions of the object 200 according to the distances between the sensor device 100 and said regions of object 200 during capturing the spectral images. Alternatively or additionally, the control unit 140 may be configured to determine a light path of the reflected light through the optical system 131 from the determined positions and orientations and to normalize, before generating the hyperspectral image, the spectral images according to the respective light paths.Thus, the control unit 140 can take into account the fact that different distance to the object 200 result in different attenuations of the reflected light, since the intensity received at the sensor device 100 is proportional to 1 / d2with d being the distance between object 200 and sensor device 100. Moreover, the control unit 140 may take into account the effect that different light paths through the optical system 131 lead to different (possibly wavelength dependent) attenuation of light and / or to shifts in the wavelengths of light, e.g. by different travelling lengths through lenses or other components of the optical system 131.This is schematically illustrated in Fig. 3 that show three different position relations between sensor device 100 and object 200 during image capturing. As can be seen in Fig. 3, each image capturing is performed at different distances dl, d2, d3 and under different angles al, a2, a3. Although the light reflected from the object 200 may have the same luminosity in each of the three cases, the intensity will differ due to the dependence on the distance. Accordingly, the control unit 140 may chose one intensity value / one distance as the reference distance and normalize all other intensities to this reference. For example, if in Fig. 3 d2 is chosen as reference, the intensities obtained during image capturing at dl need to be normalized by (dl / d2)2, and the intensities obtained during image capturing at d3 need to be normalized by (d3 / d2)2.In the same manner, a standard optical path through the optical system 131 can be defined, like e.g. the center line in a geometric optics scheme of the optical system 131, for which standard optical path the least attenuations or the like occur. Consequences of deviations from this standard optical path can be simulated and / or measured in a calibration phase of the sensor device 100. Further, it can be determined in advance which optical paths result from which viewing angles. Based on this information intensity values of all viewing angles can be transformed into intensity values that would have been obtained for light traveling along the standard optical path.In this manner intensity values obtained at different distances and / or orientations to the object 200 can be brought into a comparable format, which allows an easy combination into the resulting hyperspectral image. It should be noted that the above relates to an analytic adaption of spectra captured at different distances and / or under different viewing angles. However, it might also be possible to feed all spectral images together with their positional information into an artificial intelligence model at the control unit 140 that has been trained to generate the desired hyperspectral image therefrom. Although the normalization processes described above will then be part of the training process, they will not be mandatory for the fully trained artificial intelligence model used by the control unit 140. Thus, in such a scenario carrying out the normalization processes at the sensordevice 100 is not necessary for the generation of hyperspectral images by the sensor device 100. Moreover, normalization might also not be necessary, if the distance and the viewing angle change only by small amounts during the capturing of spectral images, since then the effects of normalization can be neglected.Examples of how to achieve a mapping of different specific wavelength subbands of the predetermined wavelength band are illustrated in Figs. 4 and 5. Here, the optical system 131 comprises an optical bandpass filter array 132 that is aligned with the pixels 136 of the image sensor 135. Each optical bandpass filter 133 only allows light within one specific wavelength band to pass such that only light within this specific wavelength band reaches the corresponding pixel 136. That is, the reflected light from the object 200 travels through the optical system 131 without spectral separation until it hits the optical bandpass filter array 132. This array consists of a plurality of optical bandpass filters 133. Each optical bandpass filter 133 is only transparent for a specific wavelength band and blocks light having a different wavelength. The optical bandpass filters 133 are preferably arranged directly above the pixels 136 in order to guarantee that only light of the filtered wavelengths is reaching the pixel 136. In this manner, one image of the object 200 is generated where different pixels 136 contain information on different specific wavelength bands.Here, the optical bandpass filters 133 for different specific wavelength bands can in principle be distributed arbitrarily over the optical bandpass filter array 132, as e.g. shown in Fig. 4 where a random distribution of optical bandpass filters 133 is used. Here, the pattern of Fig. 4 will lead to a distribution of specific wavelength bands as shown in Fig. 2. It should be noted in this context that although the figures refer to two-dimensional arrays of pixels 136 and optical bandpass filters 133, in principle, it is not necessary to arrange the pixels 136 in such a manner. In particular, assigning optical bandpass filters 133 to pixels 136 will also work for sparsely and irregularly arranged pixels 136 as long as it is known which pixel 136 is allowed to receive light in which specific wavelength band.While a random distribution of optical bandpass filters 133 and / or pixels 136 may be advantageous for some applications, it is usually preferable that the image sensor comprises a two-dimensional array of pixels 136 and that the optical bandpass fdter array 132 is formed such that all pixels in one line of the array receive light within the same specific wavelength band, and that the receivable specific wavelength band changes along the column direction of the optical bandpass filter array 132. This is exemplary and schematically shown in Fig. 5, where in one line of the optical bandpass filter array 132 the same type of optical bandpass filter 133 is used such that the entire line is transparent for the same specific wavelength band. This means that also the resulting spectral images will contain spectral information ordered by lines. Preferably, between lines the wavelengths increase monotonically in the column direction such as to cover the entire predetermined wavelength band emitted from the light source 120. This can help to reduce the computational complexity of gathering spectral information from the spectral images.For an IR light emitting light source 120, the optical bandpass fdter array 132 may be formed by a plastic plate of varying thickness or varying composition that will let pass wavelengths on a line by line basis. Such an optical bandpass fdter 132 is particularly easy to manufacture.In general, it should be noted that instead of using optical bandpass filters 133 and pixels 136 that are sensitive to the entire range of the predetermined wavelength band, it might also be possible to use pixels 136 with photodiodes that are only sensitive to specific wavelengths bands. In this manner, provision of the optical bandpass filters 133 can be omitted, however, at the cost of an increased manufacturing complexity of the image sensor 135.In this example, where different pixels 136 receive different specific wavelength bands, the control unit 140 is configured to determine from the positions and orientations of the sensor device with respect to the object 200 during capturing of the different spectral images where in the different spectral images corresponding regions of the object 200 are located and light of which wavelength band was detected at the corresponding pixels 136. The control unit 140 generates the hyperspectral image then by gathering intensity values of different specific wavelength bands corresponding to the same region of the object 200.This is schematically illustrated in Fig. 6. There, it is shown that the light emitting unit 120 emits light of the predetermined wavelength band, i.e. light between wavelength / . I and N. This light is partially absorbed at the object 200, while the non-absorbed part is reflected. Thus, each region of the object 200 reflects a complementary absorption spectrum. Fig. 6 shows an example for such a complementary spectrum reflected from the region indicated by the black square.The object 200 is then captured from different points of view, i.e. with varying position and orientation to obtain different spectral images. In these spectral images the same region of the object 200 will be located at different pixels 136, i.e. the same region of the object 200 will be captured for different specific wavelength bands. In the examples of Fig. 6 four different spectral images are shown, where the black square is located in different lines of the image sensor 135. Thus, for each of these spectral images a different part of the reflection spectrum can be obtained as schematically illustrated at the bottom of Fig. 6. Since it is known from the data of the localization unit 110 to which pixel 136 the region of interest, i.e. the black square in Fig. 6, is mapped, the control unit 140 is able to put together the corresponding spectral measurement values of the single spectral images to obtain the full reflection spectrum for this region of interest. By doing so (parallelly) for all regions / points of the object 200, it is possible to generate the hyperspectral image. This manner of obtaining the hyperspectral image corresponds to the spatio-spectral scanning of conventional hyperspectral imaging devices, however, with the advantage that it can be carried out by portable and freely moveable sensor device 100.A different example for obtaining spectral information of regions on the object 200 at the image sensor 135 is schematically illustrated in Fig. 7. Here, the image sensor 135 comprises a two-dimensional array of pixels 136 and the optical system 131 comprises (amongst other standard components like lenses or mirrors) a slit 134a that allows only strip-shaped regions of the object 200 to be imaged by the image sensor 135 and a dispersive element 134b like a grating or a prism that spectrally resolves the light coming through the slit 134a in a direction perpendicular to the longitudinal direction of the slit 134a such that different wavelength bands within the light coming through the slit 134a are projected to different lines of the array.Thus, as shown in Fig. 7, the light from a strip shaped region of the object 200 is spectrally resolved and projected onto the image sensor 135 such that different lines of the image sensor 135 receive light of differentwavelengths. Here, it should be noted that although the spectral distribution across the image sensor 135 is the same as in the example of Fig. 5, there is a difference in that in the example of Fig. 5 the entire image obtainable through the optical system 131 was mapped onto the image sensor 135, while in the example of Fig. 7 only a part of the in principle obtainable image is provided through the slit and is spectrally split such as to fill the image sensor 135. Thus, while in the example of Fig. 5 pixels 136 in the same column of the pixel array were related to different points on the object 200, in Fig. 7 all pixels 136 in one column of the pixel array contain spectral information of the same point / region on the object 200.This is exemplified in Fig. 8, where an object 200 is shown, for which a plurality of spectral images based on different strip-shaped regions ST have been captured. The light coming from each of these strip shaped regions ST is dispersed onto the image sensor 135 such that each column of the pixel array contains a full reflection spectrum for the corresponding point on the object 200. This is schematically illustrated in Fig. 8 by showing five reflection spectra for the five shown columns of the image sensor 135.The hyperspectral image is then generated by the control unit 140 by matching the strip-shaped regions ST imaged in the different spectral images based on the positions and orientations of the sensor device 100 with respect to the object 200 that were obtained by the localization unit 110 during capturing of the different spectral images. In this manner a hyperspectral image can be obtained by a portable sensor device 100 according to the principles of spatial scanning.A further approach relates to non-scanning methods of conventional hyperspectral and is exemplified with respect to Fig. 9. Here, the image sensor 135 comprises again a two-dimensional array of pixels 136. The optical system 131 comprises again a dispersive element 134b that spectrally resolves the light coming from the object 200 such that different wavelength bands within the light coming from one region of the object 200 are projected to different lines of the array as in the example of Fig. 7. However, the optical system 131 does not comprise the slit 134a that restrict the image of the object 200 to a strip-shaped region. Thus, the image sensor 135 will receive a superposition of the spectrally dispersed light reflected from different regions of the object 200.This is schematically shown in Fig. 9, where light reflected from the object 200 is considered to travel through an aperture 13 la of the optical system 131, which aperture 131a defines the field of view of the image capturing unit 130. This aperture 131a can be virtually split into a plurality of slits as shown by the dotted lines in Fig. 9. Light coming from each of these virtual slits will be dispersed by the dispersive element 134b.However, since the virtual slits are shifted with respect to each other, also the resulting dispersed light beams will be shifted with respect to each other and will hit the image sensor 135 at different height positions, i.e. at different pixel lines. This is shown in Fig. 9 by the thick black rectangles above the dispersed spectra. It can be seen that each line of the image sensor 135 receives light of a different wavelength stemming from different virtual slits in the aperture 131a. These different wavelengths are superposed at the respective pixels 136 of the lines of the pixel array. Moreover, the different wavelengths originate from different positions of the object 200. Thus, although the spectral image contains the spectral information of a plurality of regions of the object 200, this information is provided in a highly convoluted manner.Nevertheless, by capturing different spectral images from different viewpoints, the composition of the light received at the different lines of the pixel array changes. In particular, light of a different specific wavelength band but stemming from the same region of the object will reach the same line of the pixel array after changing the capturing viewpoint. For example, by “moving” the object 200 across one or more virtual slits in the aperture 131a the wavelengths reaching the same line of the image sensor 135 from the same region of the object 200 can be changed. Since the imaging properties of the optical system 131 are known, it is possible to calculate the effect of such a shift on the mapping from the object 200 to the image sensor 135. It is then possible to deconvolute the information in the spectral images by analyzing the changes in the spectral images after a change of the viewpoint based on the in principle known imaging properties of the optical system 131 and based on the knowledge of the positional changes from the localization unit 110.Thus, also in this case the control unit is configured to generate a hyperspectral image, in particular by extracting for each region of the object 200 information on the intensity of light in the specific wavelength bands based on the positions and orientations of the sensor device 100 with respect to the object 200 during capturing of the different spectral images.This could also be explained mathematically according to the following principles. The optical system 131 is formed such that to a pixel 136 at row position u of the pixel array a horizontal coordinate x of the observed scene is mapped according to the formula x = a-u +c, and such that to a pixel at column position v of the pixel array a vertical coordinate y of the observed scene is mapped according to the formula y = a-v + b- + c, with a, b, and c being parameters of the optical system and / . being the wavelength of the received light.This merely indicates the fact that the horizontal coordinate x of the scene is linearly mapped to the row position u of the pixel array and that the vertical coordinate y of the scene is linearly mapped to the column position v of the pixel array, together with the fact that for the vertical coordinate / the column position there is a wavelength dependent shift due to the dispersive element 134b.The entire spectral image captured at the image sensor will have an intensity distribution F(u,v) for pixel position (u,v). Here F(u,v) can be defined as the superposition of the intensity coming from different coordinates of the scene, i.e. asHere, F(u,v)x.yindicates the intensity from coordinates (x,y) in the observed scene. The single F(u,v)x.ycan be considered itself a superposition of intensities I(x,y, ) that are contributed from the coordinates (x,y) of the scene at wavelength , i.e. asF(u, v)xy= XA KX= a ■ u + c, y = a ■ v + b + c,A).The control unit is then configured to extract I(x,y, ) for each coordinate (x,y) of the scene and for each wavelength / . in the specific wavelength bands by deconvoluting intensity distributions F(u,v) captured at different positions and orientations of the sensor device with respect to the object based on the knowledge of said different positions and orientations.Thus, it is even possible to obtain a hyperspectral image by nothing more than a dispersive element 134b.A method for operating a sensor device 100 as described above to capture a hyperspectral image of an object is schematically illustrated in Fig. 10.The method comprises at SI 10 emitting by the light emitting unit 120 light of a predetermined wavelength band to the object 200, and at S 120 projecting by the optical system 131 reflected light that has been reflected from the object 200 onto the image sensor 135 such that specific wavelength bands of the reflected light are projected to specific, corresponding locations on the image sensor 135, the specific wavelength bands being part of the predetermined wavelength band.At S130 a spectral image of a scene is captured by the image sensor 135, which scene contains the object 200, by detecting for each pixel 136 the intensity of light received at said pixel 136. At S140 a position and an orientation of the sensor device 100 with respect to the object 200 is determined by the localization unit 110 during capturing of the spectral images.At S150 the control unit 140 generates a hyperspectral image of the object 200, which hyperspectral image indicates for each region of the object 200 the intensity of light in the specific wavelength bands, based on a plurality of spectral images of the object 200 captured from different points in space and the positions and orientations of the sensor device 100 with respect to the object 200 during capturing of said spectral images.In this manner it is possible to obtain a hyperspectral image with a portable, preferably handheld device.The sensor device 100 and the method for operating it can be used in any technical area that is in need of generation of a hyperspectral image. Some examples for uses are indicated in Figs. 11 A and 1 IB.For example, as shown in Fig. 11 A sensor device 100 may be integrated into smart glasses to allow generation of hyperspectral images during an inspection process, e.g. during a quality check, during a material detection process, during a security check at an airport, or the like. By using the hyperspectral image the (complementary) absorption spectra of analyzed objects can be obtained, which allows to determine the composition of the analyzed object. In this manner, it is e.g. possible to check the composition of food for health endangering or allergen components. Further, it is possible to detect explosives, drugs, or the like in a security check. Of course, such checks need not be carried out by using the sensor device 100 on smart glasses or a head mounted display. The sensor device 100 may also be a provided as a separate checking device.Another possible application is the integration of the sensor device 100 into a mobile terminal as shown in Fig. 1 IB to allow mobile analysis of materials. For example, the mobile terminal might be used via the sensor device 100 to assist in the determination of recycling materials, like e.g. different types of plastic. The mobile terminal might also by used for allergen detection, like traces of milk, gluten, or the like. Also, an accordingly equipped mobile terminal might help to determine the contents of food such as e.g. detection of vegan diets or the estimation of how healthy a meal is.The sensor device 100 may also be mounted in a fixed system, for example over a conveyer belt of a production chain of a factory (e.g. a food factory) to monitor the quality of the product. In such a configuration, no active scanning device is needed, as the linear motion of the object through the conveyor belt will provide the required spatial information. In this case, position information from the conveyor positioning system would be fed into the processing pipeline instead of using position and orientation information of the sensor..In general, the present sensor device 100 may be applied to any field of material detection as implemented in various technical areas.The present technology can also be configured as described below:[1] A sensor device (100) for capturing a hyperspectral image of an object (200), the sensor device (100) comprising: a localization unit (110) that is configured to determine a position and an orientation of the sensor device (100) with respect to the object (200); a light emitting unit (120) that is configured to emit light of a predetermined wavelength band to the object (200); an image capturing unit (130) comprising an optical system (131) and an image sensor (135) having a plurality of pixels (136), which optical system (131) is configured to project reflected light that has been reflected from the object (200) onto the image sensor (135) such that specific wavelength of the reflected light are projected to specific, corresponding locations on the image sensor (135), the specific wavelength bands being part of the predetermined wavelength band, and which image sensor (135) is configured to capture a spectral image of a scene that contains the object (200) by detecting for each pixel (136) the intensity of light received at said pixel (136); and a control unit (140) that is configured to generate a hyperspectral image of the object (200), which hyperspectral image indicates for each region of the object (200) the intensity of light in the specific wavelength bands, based on a plurality of spectral images of the object (200) captured from different points in space and the positions and orientations of the sensor device (100) with respect to the object (200) during capturing of said spectral images.[2] The sensor device (100) according to [1], wherein the localization unit (110) comprises a visual camera (112) that is configured to capture visible light images of the object (200) and an inertial measurement unit (114) that is configured to track motions of the sensor device (100) through space; and the localization unit (110) is configured to determine the position and the orientation of the sensor device (100) with respect to the object (200) based on the captured visible light images and the tracking of motion of the sensor device (100) by using a simultaneous localization and mapping, SLAM, algorithm.[3] The sensor device (100) according to [1] or [2], wherein the control unit (140) is configured to determine from the determined positions and orientations of the sensor device with respect to the object (200) which regions of the object (200) were captured by which spectralimage, and for which of the specific wavelength bands intensities were detected for said regions, and to generate the hyperspectral image based on this determination.[4] The sensor device (100) according to any one of [1] to [3], wherein the control unit (140) is configured to normalize, before generating the hyperspectral image, the detected intensities for each of the regions of the object (200) according to the distances between the sensor device (100) and said regions of object (200) during capturing the spectral images.[5] The sensor device (100) according to any one of [1] to [4], wherein the control unit (140) is configured to determine a light path of the reflected light through the optical system (131) from the determined positions and orientations and to normalize, before generating the hyperspectral image, the spectral images according to the respective light paths.[6] The sensor device (100) according to any one of [1] to [5], wherein the optical system (131) comprises an optical bandpass filter array (132) that is aligned with the pixels (136) of the image sensor (135) and wherein each optical bandpass filter (133) only allows light within one specific wavelength band to pass such that only light within this specific wavelength band reaches the corresponding pixel (136); and the control unit (140) is configured to determine from the positions and orientations of the sensor device (100) with respect to the object (200) during capturing of the different spectral images where in the different spectral images corresponding regions of the object (200) are located and light of which wavelength band was detected at the corresponding pixels (136), and to generate the hyperspectral image by gathering intensity values of different specific wavelength bands corresponding to the same region of the object (200).[7] The sensor device (100) according to [6], wherein the image sensor (135) comprises a two-dimensional array of pixels (136) and the optical bandpass filter array (132) is formed such that all pixels (136) in one line of the array receive light within the same specific wavelength band, and that the receivable specific wavelength band changes along the column direction of the array.[8] The sensor device (100) according to any one of [1] to [5], wherein the image sensor (135) comprises a two-dimensional array of pixels (136); the optical system (131) comprises a slit (134a) that allows only strip-shaped regions of the object (200) to be imaged by the image sensor (135) and a dispersive element (134b) that spectrally resolves the light coming through the slit (134a) in a direction perpendicular to the longitudinal direction of the slit (134a) such that different wavelength bands within the light coming through the slit (134a) are projected to different lines of the array; and the control unit (140) is configured to generate the hyperspectral image by matching the strip-shaped regions imaged in the different spectral images based on the positions and orientations of the sensor device (100) with respect to the object (200) during capturing of the different spectral images.[9] The sensor device (100) according to any one of [1] to [5], whereinthe image sensor (135) comprises a two-dimensional array of pixels (136); the optical system (131) comprises a dispersive element (134b) that spectrally resolves the light coming from the object such that different wavelength bands within the light coming from one region of the object (200) are projected to different lines of the array; and the control unit (140) is configured to generate the hyperspectral image by extracting for each region of the object (200) information on the intensity of light in the specific wavelength bands based on the positions and orientations of the sensor device (100) with respect to the object (200) during capturing of the different spectral images.

[0010] The sensor device (100) according to [9], wherein the optical system (131) is formed such that to a pixel (136) at row position u of the array a horizontal coordinate x of the observed scene is mapped according to the formula x = a-u +c, and to a pixel (136) at column position v of the array a vertical coordinate y of the observed scene is mapped according to the formula y = a-v + b- + c, with a, b, and c being parameters of the optical system (131) and / . being the wavelength of the received light; the image sensor (135) is configured to capture a spectral image with intensity distribution F(u,v) at different pixel position (u,v), where F(u,v) is defined aswith F(u,v)x,y the intensity from coordinates (x,y) in the observed scene that is defined asF(u, v)xy= A KX = A’U+ c, y = a ■ v + bA + c,A), where I(x,y, ) is the intensity contributed from the coordinates (x,y) of the scene at wavelength / .. and the control unit (140) is configured to extract I(x,y, ) for each coordinate (x,y) of the scene and for each wavelength / . in the specific wavelength bands by deconvoluting intensity distributions F(u,v) captured at different positions and orientations of the sensor device (100) with respect to the object based on the knowledge of said different positions and orientations.

[0011] A method for operating the sensor device (100) according to any one of [1] to

[0010] to capture a hyperspectral image of an object (200), the method comprising: by the light emitting unit (120), emitting light of a predetermined wavelength band to the object (200); by the optical system (131), projecting reflected light that has been reflected from the object (200) onto the image sensor (135) such that specific wavelength bands of the reflected light are projected to specific, corresponding locations on the image sensor (135), the specific wavelength bands being part of the predetermined wavelength band; by the image sensor (135), capturing a spectral image of a scene that contains the object (200) by detecting for each pixel (136) the intensity of light received at said pixel (136); by the localization unit (110), determining a position and an orientation of the sensor device (100) with respect to the object (200) during capturing of the spectral images; and by the control unit (140), generating a hyperspectral image of the object (200), which hyperspectral image indicates for each region of the object (200) the intensity of light in the specific wavelength bands, based on a plurality of spectral images of the object (200) captured from different points in space and the positions and orientations of the sensor device (100) with respect to the object during capturing of said spectral images.

Claims

Claims1. A sensor device for capturing a hyperspectral image of an object, the sensor device comprising: a localization unit that is configured to determine a position and an orientation of the sensor device with respect to the object; a light emitting unit that is configured to emit light of a predetermined wavelength band to the object; an image capturing unit comprising an optical system and an image sensor having a plurality of pixels, which optical system is configured to project reflected light that has been reflected from the object onto the image sensor such that specific wavelength bands of the reflected light are projected to specific, corresponding locations on the image sensor, the specific wavelength bands being part of the predetermined wavelength band, and which image sensor is configured to capture a spectral image of a scene that contains the object by detecting for each pixel the intensity of light received at said pixel; and a control unit that is configured to generate a hyperspectral image of the object, which hyperspectral image indicates for each region of the object the intensity of light in the specific wavelength bands, based on a plurality of spectral images of the object captured from different points in space and the positions and orientations of the sensor device with respect to the object during capturing of said spectral images.

2. The sensor device according to claim 1, wherein the localization unit comprises a visual camera that is configured to capture visible light images of the object and an inertial measurement unit that is configured to track motions of the sensor device through space; and the localization unit is configured to determine the position and the orientation of the sensor device with respect to the object based on the captured visible light images and the tracking of motion of the sensor device by using a simultaneous localization and mapping, SLAM, algorithm.

3. The sensor device according to claim 1, wherein the control unit is configured to determine from the determined positions and orientations of the sensor device with respect to the object which regions of the object were captured by which spectral image, and for which of the specific wavelength bands intensities were detected for said regions, and to generate the hyperspectral image based on this determination.

4. The sensor device according to claim 1, wherein the control unit is configured to normalize, before generating the hyperspectral image, the detected intensities for each of the regions of the object according to the distances between the sensor device and said regions of object during capturing the spectral images.

5. The sensor device according to claim 1, wherein the control unit is configured to determine a light path of the reflected light through the optical system from the determined positions and orientations and to normalize, before generating the hyperspectral image, the spectral images according to the respective light paths.

6. The sensor device according to claim 1, whereinthe optical system comprises an optical bandpass filter array that is aligned with the pixels of the image sensor and wherein each optical bandpass filter only allows light within one specific wavelength band to pass such that only light within this specific wavelength band reaches the corresponding pixel; and the control unit is configured to determine from the positions and orientations of the sensor device with respect to the object during capturing of the different spectral images where in the different spectral images corresponding regions of the object are located and light of which wavelength band was detected at the corresponding pixels, and to generate the hyperspectral image by gathering intensity values of different specific wavelength bands corresponding to the same region of the object.

7. The sensor device according to claim 6, wherein the image sensor comprises a two-dimensional array of pixels and the optical bandpass filter array is formed such that all pixels in one line of the array receive light within the same specific wavelength band, and that the receivable specific wavelength band changes along the column direction of the array.

8. The sensor device according to claim 1, wherein the image sensor comprises a two-dimensional array of pixels; the optical system comprises a slit that allows only strip-shaped regions of the object to be imaged by the image sensor and a dispersive element that spectrally resolves the light coming through the slit in a direction perpendicular to the longitudinal direction of the slit such that different wavelength bands within the light coming through the slit are projected to different lines of the array; and the control unit is configured to generate the hyperspectral image by matching the strip-shaped regions imaged in the different spectral images based on the positions and orientations of the sensor device with respect to the object during capturing of the different spectral images.

9. The sensor device according to claim 1, wherein the image sensor comprises a two-dimensional array of pixels; the optical system comprises a dispersive element that spectrally resolves the light coming from the object such that different wavelength bands within the light coming from one region of the object are projected to different lines of the array; and the control unit is configured to generate the hyperspectral image by extracting for each region of the object information on the intensity of light in the specific wavelength bands based on the positions and orientations of the sensor device with respect to the object during capturing of the different spectral images.

10. The sensor device according to claim 9, wherein the optical system is formed such that to a pixel at row position u of the array a horizontal coordinate x of the observed scene is mapped according to the formula x = a-u +c, and to a pixel at column position v of the array a vertical coordinate y of the observed scene is mapped according to the formula y = a-v + b- + c, with a, b, and c being parameters of the optical system and / . being the wavelength of the received light; the image sensor is configured to capture a spectral image with intensity distribution F(u,v) at different pixel position (u,v), where F(u,v) is defined aswith F(u,v)x,y the intensity from coordinates (x,y) in the observed scene that is defined asF(u, v)xy= KX = A■U+c> y= a■v+ b . + c,A), where I(x,y, ) is the intensity contributed from the coordinates (x,y) of the scene at wavelength / .. and the control unit is configured to extract I(x,y, ) for each coordinate (x,y) of the scene and for each wavelength / . in the specific wavelength bands by deconvoluting intensity distributions F(u,v) captured at different positions and orientations of the sensor device with respect to the object based on the knowledge of said different positions and orientations.

11. A method for operating the sensor device according to claim 1 to capture a hyperspectral image of an object, the method comprising: by the light emitting unit, emitting light of a predetermined wavelength band to the object; by the optical system, projecting reflected light that has been reflected from the object onto the image sensor such that specific wavelength bands of the reflected light are projected to specific, corresponding locations on the image sensor, the specific wavelength bands being part of the predetermined wavelength band; by the image sensor, capturing a spectral image of a scene that contains the object by detecting for each pixel the intensity of light received at said pixel; by the localization unit, determining a position and an orientation of the sensor device with respect to the object during capturing of the spectral images; and by the control unit, generating a hyperspectral image of the object, which hyperspectral image indicates for each region of the object the intensity of light in the specific wavelength bands, based on a plurality of spectral images of the object captured from different points in space and the positions and orientations of the sensor device with respect to the object during capturing of said spectral images.

Images