Method for generating a hyperspectral image

Abstract

The invention relates to a method for generating a hyperspectral image, wherein the method comprises receiving first data related to a predetermined first focal length of a color image acquisition device of an underwater device, receiving second data related to a sea floor wherein the second data are acquired by a hyperspectral image acquisition device of the underwater device, characterized in that a correction factor for a focal length of the hyperspectral image acquisition device is determined on the basis of the predetermined first focal length of the color image acquisition device and a determined second focal length of the color image acquisition device, at least one light ray is determined on the basis of the correction factor, and that the hyperspectral image is generated on the basis of the determined light ray and the received second data.

Description

[0001] Method for generating a hyperspectral image

[0002] The invention relates to a method for generating a hyperspectral image. Additionally, the invention relates to a data processing device for carrying out said method and to a system and an underwater device comprising such a data processing device, respectively. Further, the invention relates to a computer program product, a computer readable medium and a data carrying signal.

[0003] There exists a need to monitor marine and freshwater ecosystems from both an economic and ecological perspective. One challenge in monitoring ecosystems is that, on the one hand, a fine spatial resolution of the sea floor is needed to capture the high diversity of organisms with sufficient position accuracy and, on the other hand, a large area must be examined.

[0004] Underwater devices are known from the prior art which have a color image acquisition device by means of which colored images of the sea floor are captured. Additionally, said underwater devices have a hyperspectral image acquisition device by means of which hyperspectral images of the sea floor are taken. Such an underwater device is disclosed in EP 3 977 204 A1.

[0005] One of the challenges of underwater imaging is that light travels slower in a denser medium like water. The light undergoes refraction, and the amount is determined by the density of the medium and all the subsequent components that the light must interact with in the optical path. The effect of this refraction is that things appear smaller than usual. That means, the scale of the features in the water is not the same as in the air. This makes the tasks where the objective is to estimate the physical conditions using imaging difficult, because the perceived scale of the features is variable.

[0006] The push broom or line scan sensor is the most widely used choice when it comes to hyperspectral imaging because of the spatial and spectral resolution. However, the downside from using such hyperspectral image acquisition devices is that the line scan cameras are susceptible to motion distortion. The data gathered from a hyperspectral line scan sensor must undergo this motion undistortion to make sense of features in the data. However, there does not exist a simple and easy method to undistort the gathered data. This is not an issue with the color image acquisition device, which acquires images representing features in two dimensions per image. However, color image acquisition devices are not preferred because they have low spectral resolution.

[0007] The object of the invention is to provide a method by means of which the challenge of motion distortion is overcome, and hyperspectral images can de generated in an easy and simple manner.

[0008] The object is solved by a method for generating a hyperspectral image, wherein the method comprises receiving first data related to a predetermined first focal length of a color image acquisition device of an underwater device, receiving second data related to a sea floor wherein the second data are acquired by a hyperspectral image acquisition device of the underwater device, characterized in that a correction factor for a focal length of the hyperspectral image acquisition device is determined on the basis of the predetermined first focal length of the color image acquisition device and a determined second focal length of the color image acquisition device, at least one light ray is determined on the basis of the correction factor, and that the hyperspectral image is generated on the basis of the determined light ray and the received second data.

[0009] In the invention it was realized that the effect of motion can be solved by using the focal length of the color image acquisition device for calculating a correction factor for the hyperspectral image acquisition device and by using said correction factor for determining at least one light ray. In other words, the data provided by the color image acquisition device can be used to enable undistorted hyperspectral images generated by the hyperspectral image acquisition device. Thus, it was realized that the negative effect on motion on the hyperspectral image can be avoided by considering the image acquisition device for which the motion does not have such a negative effect.

[0010] Specifically, the inventive approach enables determining a light ray path in which the different media, namely, air, glass and water, through which the light passes are considered. Thus, said different media do also not have a negative effect on the hyperspectral image anymore. As is explained below more in detail, said light ray is used to determine the sea floor area that is considered by the hyperspectral image acquisition device.

[0011] A hyperspectral image acquisition device is a camera that captures multispectral data in a wide spectral band of light. The user can select the hyperspectral image acquisition device that covers that lightwave band that is of interest for the specific application. Hyperspectral image acquisition devices thus allow high spectral resolution of object-specific signatures in more than 15, but generally in 30- 200 contiguous channels, enabling documentation of a nearly continuous spectrum for each image element. The hyperspectral image is an image that is generated by the hyperspectral image acquisition device. Specifically, the hyperspectral image acquisition device can acquire a two- dimensional image in an acquiring step. Alternatively, the hyperspectral image acquisition device that is a line scanner can acquire a line wherein the image is created by stacking a plurality of determined lines. The hyperspectral image acquisition device acquires images of the sea floor.

[0012] The color image acquisition device is not a hyperspectral image acquisition device and / or can be an RGB image acquisition device. Said color image acquisition device generates a colored image and / or a two-dimensional image. The color image acquisition device differs from the hyperspectral image acquisition device in that it has fewer color channels than the hyperspectral image acquisition device, in particular exactly three color channels. Accordingly, with a color image acquisition device, the viewed object cannot be spectrally resolved as high as with a hyperspectral image acquisition device.

[0013] The use of a color image acquisition device and a hyperspectral image acquisition device offers the advantage that several images are captured of the sea floor portion, which is beneficial for image evaluation.

[0014] The color image acquisition device and the hyperspectral image acquisition device can be arranged in a cavity of a housing of the underwater device. Specifically, the color image acquisition device and the hyperspectral image acquisition device can be arranged in the same cavity section and / or can be directed such that they receive light rays passing through the same housing opening.

[0015] The first data, namely the predetermined first focal length of the color image acquisition device is a predetermined value and is usually provided by the manufacturer of the color image acquisition device. The color image acquisition device is used to acquire color images of the sea floor. With "images of the sea floor" it is meant that a portion of the sea floor is acquired by the color image acquisition device.

[0016] The term "hyperspectral image" means that a portion of the sea floor image is acquired by the hyperspectral image acquisition device. As the hyperspectral image acquisition device can be a line scanner, the hyperspectral image can consist of one or more lines comprising information about the sea floor. The second data that is acquired by the hyperspectral image acquisition device comprises said information. The second data can comprise information that is used to determine the physical and / or biological condition of the sea floor. Specifically, the second data comprises spectral information related to the sea floor, in particular a spectral fingerprint, wherein the second data is processed to determine the physical and / or biological condition of the sea floor.

[0017] The determined second focal length of the image acquisition device can be determined by a data processing device in which the inventive method is executed. Below, the model is explained more in detail that is used to determine said second focal length.

[0018] In operation, the underwater device can be arranged partially or completely in the water. For the examination of a sea floor, the underwater device can be completely immersed in the water. The underwater device can be used to study marine and freshwater ecosystems. The underwater device can be a diver-operated underwater device. Diver-operated underwater device means a device that can be operated by a diver in or under water. This means that the diver can move the underwater device in or under the water and thus move it to the desired position to capture images of the desired area of the sea floor. In addition, the diver can operate the underwater device in or under water, in particular enter corresponding commands to capture images of the sea floor.

[0019] Alternatively, the underwater device can be an autonomous underwater vehicle. With this design, the underwater device does not need to be controlled by the diver to capture images of the sea floor. The underwater device can be controlled autonomously. Alternatively, the underwater device can be a cable-guided underwater vehicle (remotely operated vehicle). With this design, the underwater device does not need to be moved by the diver to capture images of the sea floor. The underwater device can be controlled by a person on the ship.

[0020] The "light ray" comprises the light that is received by the hyperspectral image acquisition device. As mentioned before the light comprises information about the sea floor, about the physical and / or biological condition of the sea floor. The light can be visible light. However, the light can have a wavelength that is not visible to the human eye.

[0021] A data processing device can be provided that comprises means for carrying out the inventive method. The data processing device can comprise at least one processor. Additionally or alternatively, the data processing device can comprise a circuit board or be a circuit board. The acquired first and second data are transmitted to the processor of the data processing device. Thus, the data processing device, in particular processor, receives the first and second data from the color image acquisition device and the hyperspectral image acquisition device, wherein the received first and second data are processed by the data processing device. The data processing device can receive the first data each time the underwater device is operated. Alternatively, the data processing device can receive the first data at the configuration of the underwater device. In said case the first data can be stored in a memory of the data processing device.

[0022] The output of the data processing device can be the generated hyperspectral image. The data processing device can be arranged outside the underwater device. In particular, the data processing device can be a server or part of a server that is arranged outside the underwater device.

[0023] According to an embodiment the correction factor can be determined by dividing the determined second focal length of the image acquisition device by the received predetermined first focal length of the image acquisition device. Specifically, the correction factor can be calculated as follows:

[0024] CF = f2 / fl

[0025] "CF" corresponds to the correction factor, "f2" corresponds to determined second focal length of the color image acquisition device and "f1" corresponds to the predetermined first focal length of the color image acquisition device.

[0026] The advantage of using the correction factor is that it considers the entire optical chain that the light ray starting from the sea floor has to pass before it is detected by the hyperspectral image acquisition device and / or the color image acquisition device. The optical chain comprises air within the housing of the underwater device, a glass element arranged in the housing opening of the underwater device and water.

[0027] A corrected focal length of the hyperspectral image acquisition device can be determined dependent of the correction factor and the predetermined focal length of the hyperspectral image acquisition device. In particular, the corrected focal length of the hyperspectral image acquisition device can be determined by multiplying the correction factor with the predetermined first focal length of the hyperspectral image acquisition device. The predetermined focal length of the hyperspectral image acquisition device can be received by the data processing device and / or can be provided by the manufacturer of the hyperspectral image acquisition device. Specifically, the corrected focal length can be determined as follows: CFL = CF * f_HS

[0028] "CFL" is the corrected focal length of the hyperspectral image acquisition device, CF is the correction factor and f_HS is the predetermined focal length of the hyperspectral image acquisition device.

[0029] By using the correction factor it is possible to determine the corrected focal length of the hyperspectral acquisition device in an easy manner. Thus, it is not necessary to directly estimate said corrected focal length but it is possible to indirectly determine it by using parameters of the color image acquisition device as discussed above.

[0030] The data processing device can receive the predetermined focal length of the hyperspectral image acquisition device each time the underwater device is operated. Alternatively, the data processing device can receive the predetermined focal length of the hyperspectral image acquisition device at the configuration of the underwater device. In said case the predetermined focal length of the hyperspectral image acquisition device can be stored in a memory of the data processing device.

[0031] In the invention it was realized that the correction factor that is determined for the color image acquisition device can be used for the hyperspectral image acquisition device to determine the corrected focal length of the hyperspectral image acquisition device. This is possible because the hyperspectral image acquisition device is in the same housing of the underwater device as the color image acquisition device, behind the same glass element as the color image acquisition device and immersed in the same medium like the color image acquisition device.

[0032] The correction factor can be determined for a set of acquired images. The images relate to a determined sea floor portion viewed by the color image acquisition device and / or hyperspectral image acquisition device. The correction factor can be determined again, when images of another sea floor portion are acquired. This is the case when the underwater device is moved to acquire image of said other sea floor portion. Additionally or alternatively, the correction factor can be determined when images of the same sea floor portion are determined at another time. Thus, the determination of the correction factor depends on the location of the underwater device and / or is time dependent.

[0033] This is necessary as water conditions can change from a sea floor portion to another sea floor portion and / or can change with time.. Thus, a new determination of the correction factor and thus the corrected focal length of the hyperspectral image acquisition device ensures that an accurate hyperspectral image is output. The determination of the correction factor and the corrected focal length of the hyperspectral image acquisition device can be done after the images are acquired by the color image acquisition device and / or the hyperspectral image acquisition device.

[0034] In other words, the correction factor can be estimated for a given set of images acquired during the scanning of the seafloor regardless of the location or time of collection. Because the medium conditions change not only with respect to location but also time. So, to allow for precise estimation of the scaling the correction factor is always estimated for any set of images that are processed together to create a large-scale (more than 1 image, typically hundreds or thousands of images combined) representation of the sea floor by stitching multiple images together. Thereby new focal length for the hyperspectral image acquisition device is calculated using the new correction factor each time to get precise focal length and subsequently accurately scaled hyper spectral features on the seafloor.

[0035] According to an embodiment third data related to the sea floor can be received, wherein the third data are acquired by the color image acquisition device of the underwater device. The third data comprises of the rgb color and two-dimensional (morphology) information about the sea floor. The determined focal length of the image acquisition device can be determined by using the received first data and third data.. The new focal length can be determined by comparing an estimated image with an actual image and iterating. This optimization can be part of a structure-from motion process described below.

[0036] It is in particular advantageous to input the received first data and / or the received third data to a structure-from motion algorithm to determine the second focal length of the color image acquisition device. Additionally, the data processing device can receive further first data, which are inputted to the structure-from-motion algorithm. The further first data refer to other calibration properties of the image acquisition device than the first focal length of the image acquisition device. The further calibration properties of the image acquisition device can include other intrinsic parameters of the image acquisition device like principal point and / or lens distortion parameters and / or a position central point of the objective lens through which the optical axis passes. In the invention it is realized that the advantages of structure from motion algorithms can be used for creating a hyperspectral image even though line scanner data cannot be inputted to the structure from motion algorithm. Specifically, it is realized that data acquired from the color image acquisition device can be inputted to the structure from motion algorithm and that its output can be used for generating hyperspectral images.

[0037] Structure from motion algorithm is known and is used for reconstructing three-dimensional structure from its projections into a series of images taken from different viewpoints. The reconstructing process begins with the detection of features (keypoints) in the input images. Common feature detectors include SIFT, ORB, or Harris corners. These detected features are matched between multiple images to establish correspondence. Given the feature correspondences, the relative pose (rotation and translation) between pairs of images using techniques like the Essential Matrix or the Fundamental Matrix can be estimated. These methods allow for establishing the epipolar geometry between images.

[0038] Thereafter a bundle adjustment is made, in which simultaneously the camera pose and 3D structure of points are refined. In this process, the camera calibration parameters along with the 3D point positions are optimized to minimize the reprojection error. The optimization process iteratively adjusts the camera parameters and 3D points until the error is minimized. In the bundle adjustment step, the intrinsic camera parameters, including focal length, principal point, and lens distortion parameters, can be included as optimization variables. The optimization process seeks to find the values of these parameters that minimize the difference between the observed image points and their corresponding projected 3D points. This is possible because the distortion parameters affect how the 3D points are projected onto the 2D image plane. Bundle adjustment is typically a non-linear optimization problem, and various optimization algorithms, such as Levenberg-Marquardt, Gauss- Newton, or gradient descent, can be used to refine the parameters.

[0039] Afterwards, to start the optimization, initial specification of the camera calibration parameters is required. These can come from manufacturer specifications (if available), camera calibration procedures, or by using some initial estimates. In practice, calibration procedures like Zhang's camera calibration method or calibration grids can be used to estimate the initial intrinsic camera parameters. The bundle adjustment is often performed iteratively with refinement, and the optimization process may run multiple times to improve parameter estimates.

[0040] The output of the structure-from motion algorithm can include 3D reconstructions of the sea floor and / or refined camera calibration parameters. Specifically, the determined second focal length of the image acquisition device can be an output of a structure-from-motion model. Thus, said factor can be easily determined by using an existing solution, i.e. a structure from motion algorithm. Further, a refined position and / or attitude of the image acquisition device can be the output of a structure-from- motion model. Additionally, an orthophoto and / or digital elevation model can be the output of the structure-from motion model.

[0041] The data processing device can receive fourth data related to the location of the color image acquisition device. "Location" means data about the position and / or attitude of the color image acquisition device. Specifically, it is referred to the position of the center position of the lens of the color image acquisition device, in particular in a world coordinate system. The attitude is stated with respect to NED (North, East, Down) in the body frame reference (Under water device) and an attitude order around x-y-z axes (roll, pitch and yaw) convention. The fourth data can be inputted to the structure from motion algorithm and an output of the structure from motion algorithm can be an optimized or refined determined location of the image acquisition device.

[0042] The data processing device can determine the location of the hyperspectral image acquisition device on the basis of the optimized, determined location of the image acquisition device being the output of a structure-from-motion model and a predetermined location offset between the image acquisition device and the hyperspectral image acquisition device. The offset is known as the location of the hyperspectral image acquisition device relative to the location of the color image acquisition device is known and does not change during the operation of the underwater device. The offset can likewise to the first data be received by the data processing device or can be stored in a memory of the data processing device. Thus, the data processing device can determine the location, in particular position and attitude (i.e. the angular position), of the hyperspectral image acquisition device using a rigid body transformation. Specifically, the known location, in particular position and attitude, of the image acquisition device, and the offset between the color image acquisition device and the hyperspectral image acquisition device is used to determine the location, in particular position and attitude, of the hyperspectral image acquisition device. Regarding the offset it is referred to the center position of the lens of the image acquisition device and to the center position of the lens of the hyperspectral image acquisition device.

[0043] According to an embodiment at least one light ray can be determined dependent of the determined location of the hyperspectral image acquisition device and the corrected focal length of the hyperspectral image acquisition device. The location of the hyperspectral image acquisition device can refer to the location of the objective lens, in particular a central point of the objective lens through which an optical axis passes, of the hyperspectral image acquisition device.

[0044] The at least one light ray can be determined by using a virtual image line of the hyperspectral image acquisition device. It is referred to a line because the hyperspectral image is a line scanner, so that a scanner of the hyperspectral image acquisition device detects a line. The virtual image line has the same extension, in particular width, as an image line of the hyperspectral image acquisition device. The image line is a line in an image plane of the hyperspectral image acquisition device in which the hyperspectral image is displayed. The image plane can comprise the focal point of the objective lens. The image line is the line that is detected by the hyperspectral image acquisition device.

[0045] Even though the hyperspectral image acquisition device's line in reality, taking into account the height of the slit line is a two-dimensional object comprising four corner points, in the invention the line is considered as a one-dimensional object. That means, the height of the line is neglected. This is possible as a spatial interpolation, which is described below, is performed between the acquired lines so that the height of the line can be ignored. Thus, the terms "left corner point" and "right corner point" of the line used below refer to a mean corner point between two corner points being adjacent to each other along a line height extension, respectively. The left corner point and right corner point limit the lines width extension.

[0046] The virtual image line can be arranged in the corrected focal length of the hyperspectral image acquisition device. That means, the distance of the virtual image line to the objective lens, in particular the center point, of the hyperspectral image acquisition device corresponds to the corrected focal length of the hyperspectral image acquisition device. The distance refers to an extension along the optical axis of the hyperspectral image acquisition device. In other words, by knowing the corrected focal length, the optimized, determined location of the hyperspectral image acquisition device, i.e. the location of the objective lens and thus, the center point, the position of the virtual image line is known. As the size, in particular width, of the virtual image line is also known, it is possible to determine the light ray in an easy manner.

[0047] Thereto a corner point of the virtual image line can be determined. A corner point is the end point of the virtual image line, in its width direction. The light ray can be determined by using the determined corner point and the central point of the hyperspectral image acquisition device. In particular, the light ray corresponds to a line that passes through the corner point and central point of the hyperspectral image acquisition device. Thus, by knowing the central point and the corner point, the light ray can be easily determined by using a line that passes through both points. Such a modeled light ray has the advantage that it mitigates the issue of needing to model the accurate light path which involves multiple bends which is not trivial. .

[0048] It is possible to determine two light rays by using the two corner points of the virtual image line. The two corner points are arranged on opposite ends of the virtual image line. Both light rays pass through the center point of the objective lens and the respective corner point of the virtual image line. The light ray is a ray that has a constant slope and / or straight course between the sea floor to the image plane in which it is detected by the sensor of the hyperspectral image acquisition device.

[0049] According to an embodiment at least one intersection point between the at least one light ray and the sea floor is determined. As two light rays can be determined, two intersection points between the respective light ray and the sea floor can be determined. The intersection points determine a region of the sea floor that is detected by the hyperspectral image acquisition device. If the hyperspectral image acquisition device comprises a 1 D scanner the region between the two intersection point is a line region.

[0050] The intersection point can correspond to a point in which the light ray intersects a three-dimensional model of the sea floor being the output of the structure-from-motion algorithm. Thus, after the light ray path is determined, the intersection point with the sea floor can be easily determined. The determination of the intersection points is precise as it uses a true terrain model.

[0051] Another way to determine the intersection point is to use a distance between the underwater device and the sea floor. Said determination can be performed alternatively or additionally to the other determination method described above. The distance can be determined by a distance sensor and can be transmitted to the data processing device as fifth data. The intersection point can correspond to an intersection between the light ray and a plane being arranged at the determined distance. Additionally the plane can be orthogonal to the optical axis of the hyperspectral image acquisition device. Said method has the advantage that it is less processing intensive than the aforementioned method and more robust against large holes where the intersections may fail by using a 3D model of the sea floor is used for determining the intersection point or points. According to an embodiment the line region between the two intersection points can be subdivided into a predetermined number of subregions. Said subregions can be one or more pixels, respectively. The received second data from the hyperspectral camera can be assigned to the subregions. Thus, a value is assigned to each pixel. The physical and / or biological property of the sea floor in the line region is known. As the location of the hyperspectral image acquisition device is known, the location of the line region is also known. For generating further line regions the aforementioned steps have to be repeated. In particular, it is possible to assign each point of the 3D model of the sea floor region a point in a line region. It is clear that the underwater device has to be moved to achieve said goal.

[0052] As mentioned before subregions of a line region are determined and second data values are assigned to the subregions. As there can be a spatial distance between the subregions, in particular pixels, an interpolation can be performed to determine values for said distance area between the subregions. This interpolation can be done between subregions of the same line regions. Additionally or alternatively it is possible to perform said interpolation between adjacent subregions that belong to adjacent line regions. Thus, a hyperspectral image is generated that comprises information relating to physical and / or biological properties of the sea floor along the extension of the hyperspectral image.

[0053] According to an aspect of the invention a system is provided that comprises the data processing and an underwater device comprising a color image acquisition device for acquiring data referring to a sea floor, a hyperspectral image acquisition device for acquiring data referring to the sea floor, wherein the data processing device and the underwater device are data connected to each other. In said case the data processing device is arranged outside the underwater device. The data connection means that the data processing device and the underwater device can communicate with each other. In particular, it is possible to exchange data between the data processing device and the underwater device.

[0054] According to another aspect of the invention an underwater device is provided. The underwater device comprises an image acquisition device for acquiring data referring to a sea floor, a hyperspectral image acquisition device for second data referring to the sea floor and a data processing device comprising means for carrying out the inventive method. In said embodiment, the data processing device can be arranged within an inner space of the underwater device. As mentioned above, in an alternative embodiment, the data processing device is arranged outside the underwater device. The data that are acquired by the hyperspectral image acquisition device correspond to the second data discussed above. The data that are acquired by the color image acquisition device correspond to the third data discussed above.

[0055] The underwater device can comprise an inertial navigation system (INS) for determining the location of the underwater device. The inertial navigation system can comprise at least one of inertial measurement unit (IMU), a GPS sensor, a doppler velocity logger (DVL) and other sensors. The inertial navigation system enables to provide the underwater device's position in at least two coordinates, in particular three coordinates, and / or its attitude. The underwater device can also comprise a distance sensor for measuring the distance between the underwater device to the sea floor.

[0056] According to an aspect of the invention a computer program product is provided. Said computer program product comprises instructions, which, when the program is executed by a data processing device, in particular a computer, cause the data processing device, in particular the computer, to carry out the inventive method. As mentioned before the data processing device can be arranged in an inner space of the underwater device or outside the underwater device. Additionally, a computer readable medium is provided, which has stored thereon the inventive computer program product of claim 23. Further a data carrier signal is provided for carrying the computer program product.

[0057] In the figures, the subject matter of the invention is shown schematically, with identical or similarly acting elements being mostly provided with the same reference signs. Therein shows:

[0058] Fig. 1 an underwater device according to a first embodiment.

[0059] Fig. 2 an underwater device according to a second embodiment.

[0060] Fig. 3 a data processing device.

[0061] Fig. 4 a light ray projection model showing the influence of refraction of the light rays.

[0062] Fig. 5 a flow chart of the major steps for generating a hyperspectral image.

[0063] Fig. 6 a flow chart explaining the processing the input data.

[0064] Fig. 7 a flow chart explaining the determining the correction factor.

[0065] Fig. 8 a flow chart explaining the determining the location of the hyperspectral image acquisition device. Fig. 9 a flow chart explaining the determining the at least one light ray and the intersection with the sea floor.

[0066] Fig. 10 a light ray projection model showing how a refined light ray is determined.

[0067] Fig. 11 a diagram showing a pixel projection of the hyperspectral image.

[0068] Fig. 12 a flow chart explaining the assigning of data to the determined line region.

[0069] Fig. 13 a flow chart showing the use of hyperspectral images for determining a physical and / or biological condition of the sea floor.

[0070] A first embodiment of an underwater device 1 is shown in Fig. 1. Said underwater device 1 comprises a color image acquisition device 2 and a hyperspectral image acquisition device 4. Both image acquisition devices 2, 4 are used for acquiring images of a sea floor 3 shown in figure 4. Further, the underwater device 1 comprises a data processing device 16 for processing received data. The data processing device 16 is electrically connected to the color image acquisition device 2 and the hyperspectral image acquisition device 4 as is shown with dotted lines in fig. 1 . Additionally, the data processing device 16 is electrically connected to an inertial navigation system 17 and to a distance sensor 18. The inertial navigation system 17 comprises a plurality of non-shown sensors that are connected to the data processing device 16. The distance sensor 18 can be used to determine the distance between the underwater device 1 and the sea floor 3.

[0071] The underwater device comprises a housing 21, which delimits an inner space 20 of the underwater device 1. The color image acquisition device 2, the hyperspectral image acquisition device 4 and the data processing device 16 are arranged within the inner space 20 of the underwater device 1.

[0072] A glass element 19 is arranged in a cutout of the housing 21 . The glass element 19 ensures together with the housing 21 that no water can enter into the inner space 20 of the underwater device 1. The color image acquisition device 2 and the hyperspectral image acquisition device 4 are arranged such that they can acquire light rays that are shown in fig. 4 and that come from the outside the underwater device 1 and pass through the glass element 19. Thus, both image acquisition devices 2, 4 can acquire the same light ray.

[0073] Fig. 2 shows an underwater device 1 according to a second embodiment. The underwater device 1 differs from the underwater device 1 shown in figure 1 in that the data processing device 16 is arranged outside the underwater device 1 . Specifically, the data processing device 16 can be a server that can be arranged on a ship or in a building. The data processing device 16 and the underwater device 1 can comprise a communication means 22 like a transmitter and / or receiver by means of which they directly or indirectly communicate with each other. The data processing device 16 and the underwater device 1 can indirectly communicate with each other via a network, in particular internet, as an intermediate medium between the data processing device 16 and the underwater device 1.

[0074] Fig. 3 shows a data processing device 16. The data processing device 16 comprises a processor 23, a memory 24, a storage device 25 and an I / O interface 26. In non-shown embodiments, the data processing device 16 can include fewer or more components than those shown in FIG. 3.

[0075] Components of data processing device 16 shown in FIG. 3 are now described in additional detail. The at least one processor 23 includes hardware for executing instructions, such as those making up a computer program product. As an example, to execute instructions, the at least one processor 23 may retrieve (orfetch) the instructions from an internal register, an internal cache, memory 24, or a storage device 25 and decode and execute them.

[0076] The memory 24 is coupled to the at least one processor 24. The memory 24 may be used for storing data, metadata, and programs for execution by the processor. The memory 24 may include one or more of volatile and non-volatile memories, such as RandomAccess Memory ("RAM"), Read Only Memory ("ROM"), a solid state disk ("SSD"), Flash, Phase Change Memory ("PCM"), or other types of data storage. The memory 24 may be internal or distributed memory.

[0077] The data processing device 16 includes a storage device 25 for storing data or instructions. As an example and not by way of limitation, storage device 25 can comprise a non-transitory storage medium described above. The storage device 25 may include a hard disk drive (HDD), flash memory, a Universal Serial Bus (USB) drive or a combination these or other storage devices.

[0078] The data processing device 16 also includes one or more input or output ("I / O") devices / interfaces 26, which are provided to allow a user to provide input to, receive output from, and otherwise transfer data to and from the data processing device 16. These I / O devices / interfaces 26 may include a mouse, keypad or a keyboard, a touch screen, camera, optical scanner, network interface, modem, other known I / O devices or a combination of such I / O devices / interfaces 26.

[0079] The data processing device 16 can further include a bus 27. The bus 27 can comprise hardware, software, or both that couples components of data processing device 16 to each other. Fig. 4 shows a light ray projection model showing the influence of refraction of the light rays, namely on a first light ray 33 and a second light ray 34. Specifically, fig. 4 shows a part of the underwater devices 1 as shown in fig 1 or 2. The following explanation is done for the hyperspectral image acquisition device 4. However, the same situation applies for the color image acquisition device 2.

[0080] The hyperspectral image acquisition device 4 is not fully shown in fig. 4. Fig. 4 shows an image plane 28 in which the image of the sea floor 3 is displayed. The image plane 28 is arranged at a distance from a central point 10 of the objective lens 32 wherein the distance corresponds to the focal length f_HS of the objective lens 32. The central point 10 is arranged on an optical axis 11 of the hyperspectral image acquisition device 4.ln the present case the hyperspectral image acquisition device 4 comprises a line scanner so that a scanner projects a line on the image plane 28 that is arranged in the distance f_HS from the central point 10.

[0081] The first light ray 33 and the second light ray 34 delimit the width W of an image line that is detected by the scanner. Fig. 4 also shows a virtual image line 7. The virtual image line 7 is part of a virtual image plane and is also arranged in a distance f_HS from the central point 10. Likewise, to the image line virtual image line is delimited by the light rays 33, 34. The width W of the virtual image line 7 corresponds the width W of the image line.

[0082] Both light rays 33, 34 extend from the non-shown sensor detecting the image line to the sea floor 3. Additionally, both light rays 33, 34 pass through the glass element 19 and through different media. Specifically, the light rays 33, 34 pass through air, namely the air of the inner space 22 of the underwater device 1, the glass element 19 and water being outside the underwater device 1. Said different media result in a refraction at the interface between the different media. In the following, the refraction on the left light ray 33 is explained. However, the same situation applies for the right light ray 34.

[0083] Passing a first interface between air and the glass element 19 results in a distracted left light ray 33a and passing through a second interface between the glass element 19 and water 29 results in a further distracted left light ray 33b. Due to said refraction and due to the inaccurate location of the hyperspectral image acquisition device no accurate hyperspectral image can be generated. Fig. 5 shows a flow chart of the major steps for generating accurate hyperspectral images. In a first step S1 data is inputted and processed. Said input data is received from the color image acquisition device 2 and from the hyperspectral image acquisition device 4. Additionally, data can be inputted by the user.

[0084] In a second step S2 a correction factor is determined. The correction factor considers the distortions of the light ray. In a third step S3 the location of the hyperspectral camera is detected and in fourth step S4 a light ray path is determined using the correction factor determined in the second step S2 and the location of the hyperspectral image acquisition device 4 determined in the third step S3. In a fifth step S5, the hyperspectral image is generated. In fig. 5 the second step S2 and the third step S3 are performed parallel to each other. In a non-shown embodiment, the two steps can be performed one after the other.

[0085] Fig. 6 shows a flow chart explaining the processing of data. Specifically, fig. 6 explains the first step S1 shown in fig. 5 more in detail.

[0086] In a first step S11 data is inputted to the data processing device 16. Said data can be first data referring to the calibration factors of the image acquisition device 2, third data referring to at least one sea floor image acquired by the color image acquisition device 2 and fourth data referring to the location of the color image acquisition device. The calibration factors can be inputted by the user via the I / O interface 26 of the data processing device 16. Alternatively, the calibration factors can be stored in memory 24 after they are received in a configuration phase of the underwater device. Then, it is not necessary to input each time said data.

[0087] Additionally, second data related to the sea floor acquired by the hyperspectral image acquisition device, data related to an offset of the hyperspectral image acquisition device and data related to a timestamp when the hyperspectral image acquisition device acquires the image can be received. In figure the hyperspectral image acquisition device is abbreviated by "HS". Further, transformation data can be received by the data processing device.

[0088] In a second step S12 the received data is processed by a structure in motion algorithm. Specifically, the received first data, namely the calibration factors, in particular the first focal length, of the color image acquisition device, the third data related to the sea floor acquired by the color image acquisition device and the fourth data related to the location of the color image acquisition device is inputted to the structure in motion algorithm.

[0089] The output of said algorithm are refined calibration factors of the color image acquisition device. Specifically, a refined second focal length as calibration factor of the color image acquisition device 2 is output by the structure from motion algorithm. The structure from motion algorithm among others also outputs a refined location of the color image acquisition device 2 and a three dimensional model of the sea floor 3. The outputs from the structure from motion algorithm are used for different kinds of determinations, in particular for the determination of the correction factor.

[0090] Fig. 7 shows a flow chart explaining the determination of the correction factor. As is evident from fig. 7 the determined second focal length of the color image acquisition device, being the output of step S12 shown in fig. 6 is used for determining the correction factor. Additionally, the predetermined first focal length of the color image acquisition device is used for determining the correction factor. Said parameter is received in the first step S11 shown in fig. 6 as one of the calibration factors of the color image acquisition device 2.

[0091] The correction factor is determined in a first substep S31. Specifically, the correction factor can be determined as follows:

[0092] CF = / 2 / fl

[0093] "CF" corresponds to the correction factor, "f2" corresponds to determined second focal length of the color image acquisition device and "f1" corresponds to the predetermined first focal length of the color image acquisition device.

[0094] The correction factor that is determined in the first substep S31 is used for determining the corrected focal length of the hyperspectral image acquisition device 4. Specifically, the corrected focal length can be determined as follows in a second substep S32:

[0095] CFL = CF * f_HS

[0096] "CFL" is the corrected focal length of the hyperspectral image acquisition device, CF is the correction factor and f_HS is the predetermined focal length of the hyperspectral image acquisition device. Said predetermined focal length is provided by the manufacturer of the hyperspectral image acquisition device 4.

[0097] Fig. 8 shows a flow chart showing the steps to determine the location of the hyperspectral image acquisition device 4, in particular the location of the central point 10 of the hyperspectral image acquisition device 4. The location of the hyperspectral image acquisition device 4 is determined by using a rigid body transformation.

[0098] In a first substep S41 the location of the hyperspectral image acquisition device 4 is determined using the location of the of the color image acquisition device 2 and the offset value between the color image acquisition device and the hyperspectral image acquisition device. Said location is the output of the processing by the structure from motion model of step S12 shown in fig. 6. The offset value is received in step S11 shown in fig. 6 and corresponds to the distance between the color image acquisition device 2 and the hyperspectral image acquisition device 4, in particular the center position of the lens of the color image acquisition device and the center position of the lens of the hyperspectral image acquisition device. Said offset is known and does not change during the operation as the two image acquisition devices 2, 4 are arranged in the inner space 20 of the underwater device 1 . After the first substep S41 is performed the location of the hyperspectral image acquisition device 4, in particular the position of the central point 10 and the orientation of the optical axis 11 , is known at the time when the image acquisition device acquired the data.

[0099] For synchronized image acquisition devices, that means, the color image acquisition device and the hyperspectral image acquisition device 4 acquire the image at the same time, the process is stopped after first substep S41. In cases in which the image acquisition devices are unsynchronized a second substep S42 is performed. In the second substep S42 the interpolated location, in particular position and attitude, are used from the known image acquisition device to query the unknown location, in particular position and attitude, of the hyperspectral image acquisition at the query timestep.

[0100] Fig. 9 shows a flow chart comprising the steps to determine at least one adapted light ray 5, 6 and Fig. 10 shows a light ray projection model showing how the at least one refined light rays are determined. For the sake of better understanding, the left light ray 33, the right light ray 34 and the virtual image line 7 of the virtual image plane that are shown in fig. 4 are also shown in figure 10. As is evident from fig. 10 an adapted virtual image line 30 is determined. Said refined virtual image line 30 has a distance to the central point 10 that corresponds to the determined corrected focal length of the hyperspectral image acquisition device in the second substep S32. In other words, the virtual image line 7 is shifted along the central axis 11 to become the adapted virtual image line 30. However, the width of the adapted virtual image line 30 corresponds to the width of the virtual image line 7. The location of the central point 10 corresponds to the determined location of the hyperspectral image acquisition device.

[0101] By knowing the corrected focal length of the hyperspectral image acquisition device, the location of the hyperspectral image acquisition device and the width W of the virtual image line 30, the adapted light rays 5, 6 can be easily determined. The corrected focal length of the hyperspectral image acquisition device is the output of step S32 shown in fig. 7 and the location of the hyperspectral image acquisition device is the output of step S41 or S42 shown in fig. 8.

[0102] In a first substep S51 shown in fig. 9 the corner points 8, 9 of the virtual image line 30 are determined. Specifically, a left corner point 8 and a right corner point 9 are determined. This is possible as the width W of the virtual image line 30 is known. Afterwards, the light ray paths are determined in a second substep S52. Thereto, an adapted left light ray 5 is determined by connecting the determined first corner 8 with the central point 10. Likewise, an adapted right light ray 6 is determined by connecting the determined second corner 9 with the central point 10. Thus, after the second substep S52 the light ray paths of the two determined adapted light rays 5, 6 are known.

[0103] The determined light ray paths of the adapted light rays 5, 6 and the 3D model of the sea floor being the output from the processing by the structure from motion model in the second step S12 shown in fig. 6 can be used in the third substep S53 to determine intersection points 12, 13 with the sea floor 3. In fig. 10 the sea floor 3 is illustrated as a plane. A left intersection point 12 corresponds to the point in which the adapted left light ray 5 intersects the sea floor 3. The right intersection point 13 corresponds to the point in which the adapted right light ray 6 intersects the sea floor 3. It is to be noted that a line region 35 of the sea floor 3 is displayed in the image line 28 detected by the scanner of the hyperspectral image acquisition device 4. The line 35 corresponds to the portion of the sea floor 3 that is arranged between the left and right intersection points 12, 13.

[0104] Fig. 11 shows a diagram showing pixel positions of the hyperspectral image 31 after the line scanner of the hyperspectral image acquisition device 4 scanned a sea floor portion. As is evident from fig. 11 , the hyperspectral image 31 consists of several line regions 35. Each of the line regions 35 is determined in the way as described above. That means, for each line region a left intersection point 12 and a right intersection point 13 is determined.

[0105] In a next step the second data related to the sea floor that is acquired by the hyperspectral image acquisition device 4 has to be assigned to the determined line region 35. The flow chart shown in fig 12 describes the steps of assigning the second data to the determined line region 35 so that the final hyperspectral image 31 is generated. In a first substep S61 subregions 36 are generated between the two intersection points 12, 13 that are determined in step 553 shown in fig. 9. This is done for each of the line regions 35 shown in fig. 11 , wherein for a better understanding one line region 35 is surrounded by a dotted line.

[0106] In second substep 562 values are assigned to the intersection points 12, 13 and the generated subregions 35. The values correspond to data acquired by the hyperspectral image acquisition device and received in step S11 shown in fig. 6 (Sea floor data from hyperspectral image acquisition device). This is done for all line regions 35. As the subregions 35 are arranged spaced apart from each other, a two dimensional spatial interpolation is performed for the area between all the subregions on a specified uniform grid resolution. Thus, at the end of the second substep S62 the hyperspectral image 31 is created in which a data value is assigned is an interpolation of the hyperspectral pixel data in the sub-regions. .

[0107] So far, the location of the detected sea floor is only known relative to the location of the underwater device. Specifically, the intersection points 12, 13 and the location of the image acquisition devices are in the cartesian coordinate system. Said position shall be converted to a WGS (World Geodetic Standard) standard or other standard. This step is done in the third substep S63 by using transformation data that are received in step S11 shown in fig. 6.

[0108] In other words, in the third substep S63 the geographical location of the sea floor is determined. Thus, after performing the third substep S63, the location of the hyperspectral image 31 is known. In other words, the intersection points 12, 13 and subregions can be assigned to real geographical coordinates. Thereto, georeferencing can be used wherein georeferencing is a type of coordinate transformation that binds a digital raster image that represents a geographic space to a spatial reference system, thus locating the digital data in the real world. Thus, an undistorted geo-referenced hyperspectral image 31 is generated. Fig. 13 shows a flow chart howto use the hyperspectral image 31 for determining a physical condition of the sea floor. In a first step TI the hyperspectral image 31 is generated. Said image is generated as it is explained before. That means, the hyperspectral image 31 corresponds to the output of the third substep S63.

[0109] In a second step T2, since the hyperspectral image at this stage consists of hundreds of georeferenced spectral layers, each of these layers highlight spectral features of targets which can be combined to generate a unique spectral fingerprint of the three dimensional features on the sea floor. Using the morphology and elevation from other datasets like orthophotos and digital elevation maps, a rich dataset can be created which can uniquely fingerprint targets on the seafloor.

[0110] Finally in the third step T3, by processing the rich dataset generated in the second step T2, the physical and / or biological condition of the seafloor is determined. This can be done by computing spectral indices or classification of (spectral) features. These yielded feature maps relate to the taxonomy of species, pigment presence, health maps of single / multiple species of interest, eutrophication, biomass, and / or coverage area and / or other proxies.

[0111] Reference Signs

[0112] 1 Underwater device

[0113] 2 Color image acquisition device

[0114] 3 Sea floor

[0115] 4 Hyperspectral image acquisition device

[0116] 5 Adapted left light ray

[0117] 6 Adapted right light ray

[0118] 7 Virtual image line

[0119] 8 Left corner point

[0120] 9 Right corner point

[0121] 10 Focal point of hyperspectral image acquisition device

[0122] 11 Optical axis

[0123] 12 Left intersection point

[0124] 13 Rightintersection point

[0125] 14 Plane

[0126] 15 Line region

[0127] 16 Data processing device

[0128] 17 Inertial navigation system

[0129] 18 Distance sensor

[0130] 19 Glass element

[0131] 20 Inner space

[0132] 21 Housing

[0133] 22 communication means

[0134] 23 Processor

[0135] 24 Memory

[0136] 25 Storage

[0137] 26 I / O interface

[0138] 27 Bus

[0139] 28 Image area

[0140] 29 Water

[0141] 30 Adapted virtual image area

[0142] 31 Hyperspectral image

[0143] 32 Objective lens

[0144] 33 Left signal ray

[0145] 33a distracted left light ray

[0146] 33b further distracted right light ray

[0147] 34 Right light ray

[0148] 35 Line region

[0149] 36 Subregion

[0150] W Width

[0151] CF Correction factor

[0152] CFL Corrected focal length of the hyperspectral image acquisition device f_HS focal length of hyperspectral image acquisition device

Claims

Patent Claims1 . Method for generating a hyperspectral image (31), wherein the method comprises receiving first data related to a predetermined first focal length of a color image acquisition device (2) of an underwater device (1 ), receiving second data related to a sea floor (3) wherein the second data are acquired by a hyperspectral image acquisition device (4) of the underwater device (1), characterized in that a correction factor (CF) for a focal length of the hyperspectral image acquisition device (4) is determined on the basis of the predetermined first focal length of the color image acquisition device (2) and a determined second focal length of the color image acquisition device (2), at least one light ray (5, 6) is determined on the basis of the correction factor (CF), and that the hyperspectral image (31) is generated on the basis of the determined light ray (5, 6) and the received second data.

2. Method according to claim 1, characterized in that the correction factor (CF) is determined by dividing the determined second focal length of the color image acquisition device (2) by the received predetermined first focal length of the color image acquisition device (2).

3. Method according to claim 1 or 2, characterized in that that a. a corrected focal length (CFL) of the hyperspectral image acquisition device (4) is determined dependent on the correction factor (CF) and a predetermined focal length of the hyperspectral image acquisition device (4) and / or b. a corrected focal length (CFL) of the hyperspectral image acquisition device (4) is determined by multiplying the correction factor (CF) with a predetermined focal length of the hyperspectral image acquisition device (4).

4. Method according to at least one of the claims 1 to 3, characterized in that the correction factor (CF) is determined again, wherein the determination is time-dependent and / or dependent on a sea floor portion viewed by the color image acquisition device (2) and / or hyperspectral image acquisition device (4).

5. Method according to at least one of the claims 1 to 4, characterized in that third data related to the sea floor (3) are received, wherein the third data related to the sea floor (3) are acquired by the color image acquisition device (2) of the underwater device (1 ), and that the determined second focallength of the color image acquisition device (2) is determined by using the received first data and the received third data.

6. Method according to at least one of the claims 1 to 5, characterized in that a. the received first data is inputted to a structure-from motion algorithm and / or in that b. the determined second focal length of the color image acquisition device (2) is an output of a structure-from-motion algorithm.

7. Method according to claim 6, characterized in that fourth data referring to a location of the color image acquisition device (2) is received, wherein the fourth data is inputted to the structure- from-motion algorithm.

8. Method according to at least one of the claims 1 to 7, characterized in that a location of the hyperspectral image acquisition device (4) is determined on the basis of a determined location of the color image acquisition device (2) being the output of a structure-from-motion algorithm and a predetermined location offset between the color image acquisition device (2) and the hyperspectral image acquisition device (4).

9. Method according to claim 8, characterized in that the at least one light ray (5, 6) is determined dependent of the determined location of the hyperspectral image acquisition device (4) and the corrected focal length (CFL) of the hyperspectral image acquisition device (4).

10. Method according to at least one of the claims 1 to 9, characterized in that the at least one light ray (5, 6) is determined by using a virtual image line (7) of the hyperspectral image acquisition device (4), wherein the virtual image line (7) is arranged in the corrected focal length (CFL) of the hyperspectral image acquisition device (4).

11. Method according to claim 10, characterized in that at least one corner point (8, 9) of the virtual image line (7) is determined and a. the at least one light ray (5, 6) is determined by using the determined at least one corner point (8, 9) and a center point (10) of an objective lens of the hyperspectral image acquisition device (4) and / orb. the light ray (5, 6) corresponds to a line passing through the determined corner point (8, 9) and a center point (10) of the objective lens of the hyperspectral image acquisition device (4).

12. Method according to at least one of the claims 1 to 11, characterized in that at least one intersection point (12, 13) of the at least one light ray (5, 6) and the sea floor (3) is determined.

13. Method according to claim 12, characterized in thatthe intersection point (12, 13) corresponds to a point in which the light ray (5, 6) intersects a three-dimensional model of the sea floor (3) being the output of the structure-from-motion algorithm.1 . Method according to at least one of the claims 1 to 13, characterized in that fifth data relating to a distance between the underwater device (1 ) and the sea floor (3) are received.

15. Method according to claim 14, characterized in thatthe intersection point (12, 13) corresponds to an intersection between the light ray (5, 6) and a plane being arranged at the determined distance.

16. Method according to at least one of the claims 12 to 15, characterized in that a line region (15) between two intersection points (12, 13) is subdivided into a predetermined number of subregions.

17. Method according to claim 16, characterized in that the received second data is assigned to the predetermined number of subregions.

18. Method according to claim 16 or 17, characterized in that the hyperspectral image is generated from a plurality of line regions (15).

19. Method according to claim 17 or 18, characterized in that an interpolation is performed between subregions being arranged in the same line region (15) or in adjacent line regions (15).

20. Data processing device (16) comprising means for carrying out the method according to at least one of the claims 1 to 19.21 . System comprising the data processing device (16) and an underwater device (1 ) comprising a color image acquisition device (2) for acquiring data referring to a sea floor (3), a hyperspectral imageacquisition device (4) for acquiring data referring to the sea floor (3), wherein the data processing device and the underwater device are data connected to each other.

22. Underwater device (1) comprising a color image acquisition device (2) for acquiring data referring to a sea floor (3), a hyperspectral image acquisition device (4) for acquiring data referring to the sea floor (3) and a data processing device (16) comprising means for carrying out the method according to at least one of the claims 1 to 19.

23. Underwater device (1) according to claim 22, characterized in that the underwater device (1) comprises a inertial navigation system (17) for determining the location of the underwater device (1 ) and / or and distance sensor (18) for measuring the distance to the sea floor.

24. Computer program product comprising instructions, which, when the program is executed by a data processing device (1 ), in particular a computer, cause the data processing device (1), in particular the computer, to carry out the method according to at least one of the claims 1 to 19.

25. Computer readable medium having stored thereon the computer program product of claim 24.

26. Data carrier signal carrying the computer program product of claim 19.

Images