Method for labeling a water surface in an image, method for providing a training data set for training, validating, and / or testing a machine learning algorithm, machine learning algorithm for detecting a water surface in an image, and water surface detection system

Abstract

A method for labeling a water surface (22) in an image (20) is provided. The method includes receiving image data of an image (20) generated by a camera, the image (20) comprising at least one water surface (22), receiving water surface extension data, the water surface extension data representing an area (28) where the water surface (22) extends in the real world, matching the water surface extension data with the image data according to a spatial relationship between the camera and the area (28) of the water surface (22), and labeling the water surface (22) in the image (20) according to the matched water surface extension data.

Description

Technical Field

[0001] The present invention relates to the field of water surface detection and machine learning for water surface detection. In particular, the present invention relates to a method for labeling water surfaces in images, a method for providing a training dataset for training, validating, and / or testing a machine learning algorithm, a machine learning algorithm for detecting water surfaces in images, a water surface detection system, a computer program, and a computer-readable medium.

Background Art

[0002] Water surface detection is an essential ability for many marine systems. In particular, knowing the presence or absence of water is clearly very important for many marine applications. Some applications are: shoreline / water surface detection, and / or detection of the water gap between a dock and a vessel for the captain of the vessel or as an aid for autonomous transportation, where knowing the position of the water surface is important, and / or where docking operations and navigation through locks may require clearly distinguishing the water surface from the land and any infrastructure on land; detection of floating objects, where floating objects such as containers or another vessel may pose a danger to the navigation of the vessel in order to prevent collisions with the corresponding objects; detection of a person in the water, where a slight "absence" of water may be a person in distress. Thus, water surface detection makes it possible to find obstacles to safe navigation, assist in docking operations, and understand where navigation is impeded.

[0003] An accurate and inexpensive water detection system may use one or more cameras in conjunction with an image segmentation algorithm that labels pixels in an image corresponding to water. In particular, the algorithm for image segmentation may be capable of identifying the presence of water for each pixel of an image of a water surface. Regarding image segmentation, deep learning solutions are state of the art, and the corresponding image segmentation algorithm may be implemented using a deep neural network. Unfortunately, they require training using a large amount of labeled images. Conventionally, the amount of labeled images may be generated by manually labeling the water surface in images showing the water surface. However, manual image labeling is time-consuming, expensive, and error-prone. Therefore, a method for automatic image labeling is needed.

SUMMARY OF THE INVENTION

[0004] An object of the present invention is to provide a method for labeling a water surface in an image, which enables automatic image labeling of the water surface in the image and / or contributes to the inexpensive and / or fast generation of a large dataset of labeled images for training a corresponding machine learning algorithm.

[0005] An object of the present invention is to provide a method for providing a training dataset for training, validating, and / or testing a machine learning algorithm, which enables automatic image labeling of the water surface in the image and / or contributes to the inexpensive and / or fast generation of a large dataset of labeled images for training a corresponding machine learning algorithm.

[0006] An object of the present invention is to provide a water detection system, which enables automatic image labeling of the water surface in the image and / or contributes to the inexpensive and / or fast generation of a large dataset of labeled images for training a corresponding machine learning algorithm.

[0007] Another object of the present invention is to provide a computer program for executing one of the above methods and / or a computer-readable medium storing the corresponding computer program.

[0008] These objects are achieved by the subject matter of the independent claims. Further exemplary embodiments are apparent from the dependent claims and the following description.

[0009] The object is achieved by a method for labeling a water surface in an image. The method includes receiving image data of an image generated by a camera, the image including at least one water surface, receiving water surface extension data, the water surface extension data representing an area where the water surface extends in the real world, matching the water surface extension data to the image data according to a spatial relationship between the camera and the area of the water surface, and labeling the water surface in the image according to the matched water surface extension data.

[0010] This method enables automatic image labeling of water surfaces in images and contributes to the inexpensive and / or fast generation of a large dataset of labeled images for training corresponding machine learning algorithms.

[0011] The image data may be transferred from the camera to a computer executing the above method and stored in the memory of the computer. When the method is executed by the computer, the processing unit of the computer may receive the image data from the memory. The water surface extension data may be collected by a device different from the camera. The device may be a sensor, for example, another imaging device. The water surface extension data may be transferred from the corresponding device to a computer executing the above method and stored in the memory of the computer. When the method is executed by the computer, the processing unit of the computer may receive the water surface extension data from the memory.

[0012] The method may include an additional step of receiving image data of one or more images that do not include a water surface. In other words, in the step of receiving image data, there may be image data of received images that do not show any water surface. In that case, the corresponding water surface extension data may be zero or may not exist at all. In this case, the water surface extension data cannot be matched to the image data. Then, the corresponding image, particularly the corresponding image data, may be labeled as not showing any water surface.

[0013] A camera is an imaging device that has sensitivity to the frequency spectrum of the received electromagnetic radiation. For example, the camera may be a "normal" photographic camera that has sensitivity to the visible light spectrum, or an IR, SWIR, or thermal camera that has sensitivity to the thermal energy spectrum. The imaging device may include a corresponding imaging sensor, such as a CCD array.

[0014] The area where the water surface extends in the real world can be given by absolute values such as coordinates, for example, longitude and latitude, length, and / or the width of the water surface, or by relative values such as the spatial relationship between the device that collects the water surface extension data and the area where the water surface extends.

[0015] There may be two or more separate water surfaces in the image. In this case, the water surface extension data may represent the area where the corresponding water surface extends in the real world, and all the water surfaces shown in the image may be automatically labeled by the above method, for example, one after another or two or more at the same time.

[0016] Labeling the water surface in an image may mean that the area where the water surface expands in the image is marked in the image. For example, the boundary or boundary line of the water surface may be drawn around the water surface, and / or the entire water surface may be marked with a given color. In particular, when the labeled image is shown on a display, the image data may be modified by the labeling so that the water surface in the image is emphasized by the boundary line, for example, the boundary, or the marked water surface. Alternatively or additionally, metadata may be added to the image data, and the metadata may represent the position and expansion of the area where the water surface expands in the image. These metadata may then be used during the training of the machine learning algorithm, and it is not necessary for the correspondingly labeled image to be shown on the display. The metadata may simply be referred to as the label of the image, in particular the label of the labeled image.

[0017] According to an embodiment, the spatial relationship between the camera and the area of the water surface is given by the absolute position and absolute orientation of the camera, and the absolute position of the area of the water surface. The absolute position may be given, for example, by coordinates, in particular by real-world coordinates, for example in terms of degrees of longitude and latitude.

[0018] The absolute position of the camera can be determined by GPS at the time the image is captured. The GPS data may refer directly to the position of the camera, or may refer to the position of the vehicle on which the camera is mounted, for example, a ship or boat. In the latter case, the position of the camera on the vehicle may be calibrated, fixed, known, and taken into account when determining or using the position of the camera. In addition, the absolute position of the camera can include the height of the camera on the vehicle and / or, in the case of a ship, the current heave of the ship, i.e., the current height of the entire ship relative to the normal sea level and / or one or more waves on the body of water on which the ship is sailing.

[0019] The absolute orientation of the camera can be given by the basic direction in which the camera is directed. The basic direction may be determined by a compass. The compass may be directly coupled to the camera, or the vehicle's compass may be used, in which case the angle between the vehicle's main axis and the camera's orientation may have to be considered. Additionally, the absolute orientation of the camera may include the tilt of the camera with respect to the vehicle. Furthermore, the vehicle may perform various movements that can affect the absolute orientation of the camera. These movements may be taken into account and, in the case of a ship as the vehicle, may include surge, sway, roll, pitch, and / or yaw of the ship.

[0020] Therefore, when the camera is calibrated, there is a one-to-one relationship between the position and orientation of the vehicle, e.g., a ship, and the position and orientation of the camera. For example, when the vehicle moves, either on its own or is moved by external factors, e.g., by waves on the water surface, the movement of the vehicle can be compensated for when matching the water surface expansion data to the image data according to the spatial relationship. The movement may be compensated for by using the vehicle's inertial measurement unit.

[0021] The absolute position of the area of the water surface may be given by at least one pair of real-world coordinates and the expansion of the area with respect to that pair of real-world coordinates, where the expansion may be given by one or more lengths and one or more widths of the area. Alternatively, the absolute position of the area of the water surface may be given by a set of pairs of real-world coordinates that define points on the boundary or border of the area of the water surface, e.g., points on the shoreline of the corresponding lake or sea. Alternatively, the absolute position of the area of the water surface may be given by a set of pairs of real-world coordinates that define all the points of the area of the water surface.

[0022] According to an embodiment, the water surface extension data is a map, particularly a digital map, the map includes an area of the water surface, particularly corresponding data, and from the map, particularly from the corresponding data, the absolute position of the area of the water surface, for example, the corresponding real-world coordinates are extracted. Thus, the map shows at least one water surface and optionally two or more separate water surfaces. After the machine learning algorithm is trained, the map is no longer required to accurately identify the water surface in the image.

[0023] According to an embodiment, matching the water surface extension data to an image according to the spatial relationship between the camera and the area of the water surface includes extracting the area of the water surface or a boundary, for example, a boundary line, from the map, and projecting the boundary from the map to the image according to the absolute position and absolute orientation of the camera. Extracting the area of the water surface or the boundary of the area from the map can include extracting a pair of coordinates representing the area or the boundary from the data provided by the map. The boundary of the area of the water surface or the boundary line may be the coastline of a lake, sea, or ocean containing the water surface. The boundary may be projected from the map to the image by any conventional projection method known in the art, as described in detail and described in, for example, a textbook on computer vision, such as "Multiple View Geometry in Computer Vision", R. Hartley, and A. Zisserman; Cambridge University Press, New York, NY, USA, 2nd edition, (2003); ISBN: 978-0-521-54051-3. For example, Chapter 6, page 154 describes the projection of world points on the image of an ideal camera, and / or Chapter 7, page 191 describes a method for correcting lens distortion. Alternatively, other conventional models of the imaging device and / or other lens distortion models may be used for the projection.

[0024] According to an embodiment, the water surface expansion data is sensor data collected by a sensor, and the spatial relationship between the camera and the water surface area is given by a fixed spatial relationship between the camera and the sensor. For example, the fixed spatial relationship may be given by the relative position and / or relative orientation of the camera with respect to the sensor. For example, the fixed spatial relationship may be given by the position and orientation of the camera with respect to the vehicle to which the camera is attached, and the position and orientation of the sensor for collecting the water surface expansion data with respect to the vehicle, and the sensor is attached to the same vehicle. After training the machine learning algorithm, the sensor is no longer required.

[0025] According to an embodiment, the fixed spatial relationship between the camera and the sensor includes the distance between the camera and the sensor, the relative orientation of the camera and the sensor with respect to each other, and / or the height level of the camera with respect to the sensor. Thus, in contrast to the previous paragraph, the fixed spatial relationship may be directly given by the relative position and / or relative orientation of the camera with respect to the sensor, without particularly referring to the corresponding vehicle.

[0026] According to an embodiment, matching the water surface expansion data to an image according to the spatial relationship between the camera and the water surface area includes extracting the water surface area or the boundary of the area from the sensor data, and projecting the area or the boundary of the area onto the image according to the fixed spatial relationship between the camera and the sensor. In particular, extracting the boundary of the water surface area from the sensor data can include extracting a pair of coordinates representing the boundary from the sensor data given by the sensor. The boundary of the water surface area may be the coastline of a lake, sea, or ocean including the water surface. Further, extracting the water surface area from the sensor data can include extracting a pair of coordinates representing the area from the sensor data given by the sensor. The area and / or the boundary may be projected from the sensor data onto the image by any conventional projection method known in the art, such as described and described in the above textbooks on computer vision.

[0027] According to an embodiment, the sensor includes an imaging device, and the other imaging device is configured to detect the water surface. The other imaging device may be a polarization-sensitive camera. In this case, the linearly polarized light reflected from the water surface can be detected using the polarization filter of the polarization-sensitive camera. Alternatively, the other imaging device may be a multi / hyper-spectral imaging device or simply another "normal" camera, and the water surface can be detected using the correspondingly collected spectral information. Alternatively, the other imaging device may be an infrared camera, such as a SWIR camera, a MIR camera, or a thermal camera. If the other imaging device is also a spectral sensitivity imaging device, such as a "normal" multi / hyper-spectral camera, a thermal camera, or an IR camera, the other imaging device may be spectrally sensitive in an energy and / or frequency spectrum different from that of the camera. In other words, when two cameras are used to collect corresponding image data and sensor data, at least two different types of cameras, particularly cameras that differ in the energy and / or frequency spectrum they sense, should be used.

[0028] If the sensor includes or constitutes the other imaging device, the sensor data may represent a sensor image that can be clearly distinguished from all other objects visible in the corresponding image when displayed on the screen or that shows only the water surface. However, to perform the above method, it is not necessary to visualize the sensor data on the screen, and the method may be performed completely automatically without showing the corresponding sensor image on the screen.

[0029] According to an embodiment, the sensor includes a lidar sensor and / or a radar sensor. The lidar and / or radar sensor makes it possible to detect the land surrounding the water surface, thereby detecting the boundary between the land and the water surface, and thereby making it possible to detect the boundary of the water surface.

[0030] According to an embodiment, the method further includes receiving object information regarding at least one object that is underwater and extends from the water surface other than land after matching the water surface expansion data to an image and before labeling the water surface in the image, determining an object area in the image according to the object information, where the object area is covered by the object, and labeling the water surface other than the object area. The object may be another ship or any other object extending from the water, such as a container, a person, a buoy, etc. The object information may be received from an Automatic Identification System (AIS) or may be generated by an object detection algorithm that analyzes the image. The object detection algorithm may include or constitute a neural network. The neural network may be any conventional artificial intelligence trained to detect objects in the image, such as containers, people, buoys, etc.

[0031] The objective is achieved by a method for providing a training dataset for training, validating, and / or testing a machine learning algorithm to enable the machine learning algorithm to detect the water surface in an image. This method includes providing, as features for training, the amount of unlabeled images, where each of the images shows at least one water surface, and providing the amount of labeled images, where each of the labeled images corresponds to an unlabeled image and each of the labeled images is labeled by the above method. Therefore, the method for providing a training dataset for training, validating, and / or testing a machine learning algorithm may include or use the method for labeling the water surface in the image, and when the method for providing a training dataset for training, validating, and / or testing a machine learning algorithm is executed once, the method for labeling the water surface in the image can be executed several times, for example, once for each image to be labeled.

[0032] It should be understood that the features of the method for labeling the water surface in an image as described above and below can be the features of the method for providing a training dataset for training, validating, and / or testing a machine learning algorithm as described above and below.

[0033] The object of the present invention is achieved by a machine learning algorithm for detecting the water surface in an image, and the machine learning algorithm is trained by at least one of the above methods. "Detect" may, in this context, mean that the machine learning algorithm can be configured to recognize whether there is any water surface in the image. Alternatively or additionally, "detect" may, in this context, mean that the machine learning algorithm determines the water surface in the image by image segmentation, such as semantic segmentation, for example, by pixel-by-pixel segmentation of the image.

[0034] It should be understood that the features of the method for labeling the water surface in an image as described above and below, and / or the features of the method for providing a training dataset for training, validating, and / or testing a machine learning algorithm as described above and below, can be the features of the machine learning algorithm for detecting the water surface in an image as described above and below.

[0035] The object is achieved by a water surface detection system for detecting the water surface in an image. The system includes a camera for generating image data of the image, a memory containing the image that includes at least one water surface and water surface extension data, where the water surface extension data represents an area where the water surface expands in the real world, and a controller configured to match the water surface extension data to the image data according to the spatial relationship between the camera and the area of the water surface, and label the water surface in the image according to the matched water surface extension data. The water surface extension data may be pre-stored in the memory, downloaded from the Internet, or generated by a sensor and stored in the memory.

[0036] It should be understood that the features of the method as described above and below can be the features of the water surface detection system as described above and below.

[0037] According to an embodiment, the water surface detection system further comprises a sensor for collecting water surface expansion data. The sensor may comprise other imaging devices, and the other imaging devices are configured to detect the water surface, lidar sensors, and / or radar sensors.

[0038] The object is achieved by a computer program comprising instructions configured to execute at least one of the above methods when the computer program is executed by a processing unit of a computer or by a controller of the above water surface detection system. It should be understood that the features of the method as described above and below can be the features of the computer program as described above and below.

[0039] The object is achieved by a trained machine learning algorithm and / or a computer-readable medium storing the above computer program. It should be understood that the features of the method as described above and below can be the features of the computer program as described above and below.

[0040] The computer-readable medium may be a floppy (registered trademark) disk, hard disk, USB (Universal Serial Bus) storage device, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), or flash memory. The computer-readable medium may be a data communication network, such as the Internet, that enables the download of program code. Generally, the computer-readable medium can be a non-transitory or transitory medium.

[0041] These and other aspects of the present invention will become apparent from the embodiments described below and will be elucidated with reference to those example embodiments.

Brief Description of the Drawings

[0042] The subject matter of the present invention will be described in more detail below with reference to exemplary embodiments illustrated in the accompanying drawings.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

[0043] The reference signs used in the drawings and their meanings are listed in summary form in the list of reference signs. In principle, in the figures, the same reference signs are assigned to the same parts.

Embodiments for Carrying Out the Invention

[0044] FIG. 1 shows a flowchart of an exemplary embodiment of a method for labeling the water surface 22 in an image 20 (see FIG. 2). This method may be executed by a general-purpose computer or by a water surface detection system as described below. The image 20, and preferably a large number of additional images 20 each labeled by this method, may be used to train a machine learning algorithm so as to be able to label the water surface 22 in an unknown image 20.

[0045] FIG. 2 shows an example of an image 20 showing a water surface 22. The image 20 can show the water surface 22, land 24, for example two pieces of land, and the sky 26. Only one continuous water surface 22 is shown in the image 20. Another image 20 may show two or more water surfaces 22 separated from each other.

[0046] In step S2, the image data of the image 20 generated by a camera (not shown) is received, for example, by a processing unit of a general-purpose computer or by a controller of a water surface detection system respectively. The image data may be transferred from the camera to a general-purpose computer executing this method and stored in the memory of the computer. When this method is executed by a general-purpose computer, the processing unit of the general-purpose computer can receive the image data from the memory.

[0047] In step S4, water surface expansion data can be received, for example, by a processing unit of a general-purpose computer or by a controller of a water surface detection system respectively. The water surface expansion data can represent an area where the water surface expands in the real world. For example, the water surface expansion data represents a map, particularly a digital map, such as the map 30 shown in FIG. 3. The area where the water surface 22 in the real world spreads can be given by absolute values such as coordinates, for example, degrees of longitude and latitude, length, and / or width of the water surface 22, or by relative values such as the spatial relationship between the device collecting the water surface expansion data and the area where the water surface 22 spreads. There may be two or more separate water surfaces 22 in the image 20. In this case, the water surface expansion data can represent the area where the corresponding water surface 22 expands in the real world. When the method is executed by a computer, the processing unit of the general-purpose computer can receive the water surface expansion data from the memory of the general-purpose computer or from the Internet.

[0048] Figure 3 shows an example of a map 30 showing the water surface 22 of Figure 2. The map 30 may include the area of the water surface 22, in particular the corresponding data also shown in the image 20. The absolute position of the area of the water surface 22, for example the corresponding real-world coordinates, may be extracted from the map 30, in particular from the corresponding data. Thus, the map 30 shows at least one water surface 22. As another example, the map 30 may show two or more separate water surfaces 22 that are separated from each other. The map may be any open-source map, for example, OpenStreetMap, or, for example, Google Maps. Optionally, a three-dimensional altitude map may be used as the map 30. This may improve the accuracy of subsequent matching steps.

[0049] In step S6, the water surface extension data is matched with the image 20, in particular the corresponding image data, according to the spatial relationship between the camera and the area of the water surface 22. The spatial relationship between the camera and the area of the water surface 22 may be given by the absolute position and absolute orientation of the camera, as well as the absolute position of the area of the water surface 22.

[0050] The absolute position may be given, for example, by coordinates, in particular real-world coordinates in terms of degrees of longitude and latitude, for example. The absolute position of the camera may be determined by GPS at the time the image 20 is captured. The GPS data may refer directly to the position of the camera, or to the position of a vehicle on which the camera is mounted, for example, a ship or boat. In the latter case, the position of the camera on the vehicle may be calibrated, fixed, known, and taken into account when determining or using the position of the camera. In addition, the absolute position of the camera may include the height of the camera on the vehicle and / or, in the case of a ship, the current heave of the ship, i.e., the current height of the entire ship relative to the normal sea level and / or one or more waves on the body of water on which the ship is sailing. Thus, the camera may be calibrated so that the orientation and / or pose of the camera is known.

[0051] The absolute orientation of the camera can be given by the basic direction in which the camera is directed. The basic direction may be determined by a compass. The compass may be directly coupled to the camera, or the vehicle's compass may be used, in which case the angle between the vehicle's main axis and the camera's orientation may have to be taken into account if there is an angle. Additionally, the absolute orientation of the camera may include the tilt of the camera with respect to the ship. Further, in the case of a ship as a vehicle, the ship can perform different movements that can affect the absolute orientation of the camera. These movements may be considered and may include the surge, sway, roll, pitch, and / or yaw of the ship.

[0052] Therefore, when the camera is calibrated, there is a one-to-one relationship between the position and orientation of the vehicle, such as a ship, and the position and orientation of the camera. For example, when the vehicle moves because it moves itself or because it is moved by external factors, such as by waves on the water surface, the movement of the vehicle can be compensated when matching the water surface expansion data to the image data according to the spatial relationship. The movement may be compensated by using the vehicle's inertial measurement unit.

[0053] The absolute position of the area of the water surface 22 may be given by at least one pair of real-world coordinates and the expansion of the area with respect to that pair of real-world coordinates, where the expansion may be given by one or more lengths and one or more widths of the area. Alternatively, the absolute position of the area of the water surface 22 may be given by a set of pairs of real-world coordinates that define points on the boundary of the area of the water surface 22, such as points on the corresponding lake or sea shoreline. Alternatively, the absolute position of the area of the water surface may be given by a set of pairs of real-world coordinates that define all the points of the area of the water surface 22.

[0054] Depending on the spatial relationship between the camera and the area of the water surface, matching the water surface expansion data to the image may include extracting from the map 30 an area or a boundary, for example, the boundary of the area of the water surface 22, and projecting the boundary from the map 30 to the image 20 according to the absolute position and absolute orientation of the camera. Extracting the area of the water surface 22 or the boundary of the area from the map may include extracting a pair of coordinates representing the area or the boundary from the data provided by the map 30. The boundary of the area of the water surface 22 may be the coastline of a lake, sea, or ocean that includes the water surface 22. The boundary may be projected from the map 30 to the image 20 by any conventional projection method known in the art, such as those described in detail and described in the above textbooks on computer vision.

[0055] In an optional step S8, object information regarding at least one object 32, 34 (see FIGS. 4 and 5), excluding the land 24 that is underwater and extends from the water surface 22, may be received by the controller or processing unit, and the object area within the image 20 may be determined according to the object information, and the object area is covered by the objects 32, 34. The object information may be received from an automatic identification system (AIS) or may be generated by an object detection algorithm that analyzes the image 20. When using AIS, the camera may preferably be calibrated so that, for example, a detected object, such as another ship, can be projected onto the real-world coordinates on the image 20. The object detection algorithm may include or consist of a neural network. The neural network may be any conventional artificial intelligence trained to detect objects 23 within the image 20, such as containers, people, buoys, etc.

[0056] FIG. 4 shows an example of an image 20 showing the water surface 22 and an object 32 extending from the water surface 22. The object 23 and its position and extent within the image 20 may be determined by the object detection algorithm.

[0057] FIG. 5 shows an example of an image 20 showing a water surface 22 and a ship 34 moving through the water, where the ship 34 can be an example of a detected object 32.

[0058] In step S10, the water surface 22 in the image 20 may be labeled according to the matched water surface extension data. In particular, all the water surfaces 22 shown in the image 20 may be automatically labeled, for example, one after another or two or more at the same time. If any step S8 is executed and at least one object 23 is found in the water, the water surface 22 can be labeled except for the corresponding object area covered by the object 23. The objects 32, 34 may be another ship or any other object extending from the water, such as a container, a person, a buoy, etc. If the objects 32, 34 are another ship 34, the object information may be received from AIS or may be generated by an object detection algorithm.

[0059] The fact that the water surface 22 is labeled in the image 20 may mean that the area where the water surface 22 extends in the image 20 is marked in the image 20. For example, the boundary of the water surface 22, such as a boundary line, may be drawn around the water surface 22 and / or the water surface 22 may be marked with a given color. In particular, the image data may be modified by the labeling so that the water surface 22 is emphasized in the image 20 labeled by the boundary or the marked water surface 22 when the labeled image 20 is shown on the display. Alternatively or additionally, metadata may be added to the image data, and the metadata may represent the area where the water surface 22 extends in the image 20. These metadata may then be used during the training of the machine learning algorithm, and it is not necessary for the correspondingly labeled image 20 to be shown on the display.

[0060] When an image 20 is labeled, another image 20 showing at least one water surface 22 may be correspondingly labeled. Thus, the above method can be executed several times to automatically provide a large number of images 20 showing the labeled images 20. Further, some images 20 that do not show water may be received from the device executing the method, and the corresponding water surface expansion data may not exist or may be set to zero. Thereafter, these images 20 may be labeled as not showing any water surface. A large number of labeled images 20 may be used to train a machine learning algorithm, whereby the water surface 22 can be labeled in any unknown image 20 showing the water surface 22. Thus, a method may be provided for providing a training data set for training, validating, and / or testing a machine learning algorithm to enable the machine learning algorithm to detect the water surface 22 in the image 20. This method includes, as features for training, providing the amount of unlabeled images 20 and providing the amount of labeled images, each of which shows at least one water surface 22, where the labeled images 20 each correspond to an unlabeled image 20 and each of the labeled images 20 is labeled by the above method. Thus, a method for providing a training data set for training, validating, and / or testing a machine learning algorithm may include or use a method for labeling the water surface 22 in the image 20, and the method for labeling the water surface 22 in the image 20 may be executed several times, for example, once for each image 20 to be labeled when the method for providing a training data set for training, validating, and / or testing a machine learning algorithm is executed once. The amount of labeled images 20 may constitute a training data set. After training, the correspondingly trained machine learning algorithm can automatically detect the water surface 22 in the image 20.

[0061] A machine learning algorithm may be used by a water surface detection system to detect a water surface 22 within an image 20. The system may include a camera for capturing an image 20 showing at least one water surface 22 and generating image data corresponding to the image 20. The system may further include a memory containing water surface extension data, where the water surface extension data represents an area where the water surface 22 expands in the real world. The system may further include a controller configured to match the water surface extension data to the image data according to a spatial relationship between the camera and the area of the water surface 22. The controller may further be configured to label the water surface 22 within the image 20 according to the matched water surface extension data. The water surface extension data may be pre-stored in the memory, downloaded from the Internet, or generated by a sensor and stored in the memory, as described with respect to FIG. 6.

[0062] FIG. 6 shows a flowchart of an exemplary embodiment of a method for labeling a water surface within an image, such as the water surface 22 within the image 20.

[0063] In step S12, the image data of the image 20 captured by a camera (not shown) is received, for example, by a processing unit of a general-purpose computer or, respectively, by a controller of the water surface detection system. The image data may be transferred from the camera to the processing unit or controller executing this method and may be stored in a memory. When this method is executed by a general-purpose computer, the processing unit of the general-purpose computer can receive the image data from the memory. When this method is executed by the water surface detection system, the controller of the water surface detection system can receive the image data from the memory or directly from the camera.

[0064] In step S14, for example, water surface expansion data can be received by a processing unit of a general-purpose computer or, respectively, by a controller of the water surface detection system. The water surface expansion data can represent an area where the water surface expands in the real world. For example, the water surface expansion data may be collected by a device different from the camera, for example, by a sensor, for example, by the above-mentioned sensor. The water surface expansion data may be transferred from the corresponding device to a general-purpose computer or a controller that executes the above method, may be stored in the memory of the computer or the controller respectively, or may be directly processed. The water surface expansion data may be sensor data collected by a sensor. For example, the water surface detection system further includes a sensor for collecting water surface expansion data. The sensor may include a lidar sensor, a radar sensor, and / or another imaging device, and the imaging device is configured to detect the water surface. The other imaging device may be a polarization-sensitive camera, a multi / hyper-spectral imaging device, or simply another "ordinary" camera, an infrared camera, for example, a SWIR camera, a MIR camera, or a thermal camera. If the other imaging device is also a camera, for example, an "ordinary" multi / hyper-spectral, thermal, or IR camera, the other imaging device may be spectrally sensitive in an energy and / or frequency spectrum different from that of the camera. In other words, if two cameras are used to collect corresponding data, in particular, at least two different types of cameras may be used with respect to the energy and / or frequency spectrum to which they are sensitive.

[0065] The sensor can utilize the physical properties of water to distinguish water, and thus the water surface 22, from other objects, land, and / or air. For example, the light reflected from the water surface 22 is partially polarized in a direction parallel to the water surface 22. The polarization can be measured by using a corresponding imaging device, namely a polarization-sensitive camera, which provides two different optical paths having a linearly polarized film attached to the front surface of the corresponding lens. One optical path may have sensitivity to horizontal polarization, and the other optical path may have sensitivity to vertical polarization. Image processing can be used to detect the water surface 22 (see Figure 7). If the sensor comprises or constitutes other imaging devices, the sensor data may represent a special image 36 (see Figure 7) that shows only the water surface 22 when displayed on the screen. However, to execute the above method, it is not necessary to visualize the sensor data on the screen, and the method may be executed completely automatically without showing the corresponding special image 36 on the screen. After training a machine learning algorithm, the sensor is no longer required.

[0066] FIG. 7 shows an example of a special image 36 captured by a camera. The scene captured in the special image 36 corresponds to the scene captured in the image 20. In other words, the real-world area shown in the special image 36 is the same as the real-world area shown in the image 20, and the illustration of the special image 36 is different from the illustration of the "normal" image 20. For example, if the other imaging device is a polarization-sensing camera, in the special image 36, the water surface 22 can be colored, for example, red, orange, or yellow, while the land 24 and the sky 26 can be black because the polarization-sensing camera can be configured to capture only polarization. In particular, the light reflected from the water surface 22 is linearly polarized and can be detected using the polarization filter of the corresponding polarization-sensing camera. An output similar to the special image 36 may be generated by a multi / hyper-spectral camera and / or an infrared camera. For example, if the other imaging device is a multi / hyper-spectral camera, the water surface 22 can be detected using the correspondingly collected spectral information. If the sensor is a lidar and / or a radar sensor, the sensor can detect the land 24 surrounding the water surface, thereby detecting the boundary between the land 24 and the water surface 22, and thereby detecting the boundary of the water surface 22.

[0067] In step S16, the water surface expansion data is matched to the image 20 according to the spatial relationship between the camera and the area of the water surface 22. The spatial relationship between the camera and the area of the water surface 22 may be given by a fixed spatial relationship between the camera and the sensor that detects the water surface 22. For example, the fixed spatial relationship may be given by the relative position and / or relative orientation of the camera with respect to the sensor. For example, the fixed spatial relationship may be given by the position and orientation of the camera with respect to the vehicle to which the camera is attached, and the position and orientation of the sensor for collecting the water surface expansion data with respect to the vehicle, and the sensor is attached to the same vehicle. Alternatively or additionally, the fixed spatial relationship between the camera and the sensor may include the distance between the camera and the sensor, the relative orientation of the camera and the sensor with respect to each other, and / or the height level of the camera with respect to the sensor. Therefore, the fixed spatial relationship may be directly given by the relative position and / or relative orientation of the camera with respect to the sensor, without referring to the corresponding vehicle in particular.

[0068] Matching the water surface expansion data to the image 20 according to the spatial relationship between the camera and the area of the water surface 22 may include extracting, from the sensor data, an area or a boundary of the area of the water surface, for example, a boundary, and projecting the boundary onto the image 20 according to the fixed spatial relationship between the camera and the sensor. In particular, the extraction of the area or the boundary of the water surface 22 from the sensor data may include extracting a pair of coordinates representing the area or the boundary from the sensor data given by the sensor. The boundary of the area of the water surface 22 may be the coastline of a lake, sea, or ocean that includes the water surface 22. The area or the boundary may be projected from the sensor data onto an image, particularly image data, by any conventional projection method known in the art, as described and detailed in the above textbooks on computer vision, for example.

[0069] In an optional step S18, object information regarding at least one object 32, 34 (see FIGS. 4 and 5), which is in water and excludes the land 24 extending from the water surface 22, may be received by the controller or processing unit, and an object area within the image 20 may be determined according to the object information, and the object area is covered by the object 24. The object information may be received from an automatic identification system (AIS) or may be generated by an object detection algorithm that analyzes the image 20. The object detection algorithm may include or consist of a neural network. The neural network may be any conventional artificial intelligence trained to detect objects 23 within the image 20, such as containers, people, buoys, etc.

[0070] In step S20, the water surface 22 in the image 20 may be labeled according to the matched water surface expansion data, as described, for example, with respect to step 10 in FIG. 1.

[0071] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative or exemplary and not restrictive, and the present invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and implemented by those skilled in the art of practicing the claimed invention, from consideration of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or control device or other unit may perform the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used advantageously. Any reference signs in the claims should not be construed as limiting the scope.

[0072] List of reference signs 20 Image 22 Water surface 24 Land 26 Air 28 Area 30 Map 32 Object 34 Ship 36 Special Image S2 - S20 Step 2 - 20

Claims

1. A method for labeling a water surface (22) in an image (20), comprising: receiving image data of the image (20) generated by a camera, the image (20) comprising at least one water surface (22); receiving water surface extension data, the water surface extension data representing an area (28) where the water surface (22) extends in the real world; matching the water surface extension data to the image data according to a spatial relationship between the camera and the area (28) of the water surface (22); labeling the water surface (22) in the image (20) according to the matched water surface extension data.

2. The method according to claim 1, wherein the spatial relationship between the camera and the area (28) of the water surface (22) is given by an absolute position and an absolute orientation of the camera and an absolute position of the area (28) of the water surface (22).

3. The extension data of the water surface (22) is represented by a map (30) including the area (28) of the water surface (22), The method according to claim 2, further comprising extracting an absolute position of the area (28) of the water surface (22) from the map (30).

4. Matching the extension data of the water surface (22) to the image (20) according to the spatial relationship between the camera and the area (28) of the water surface (22) comprises: extracting the area (28) of the water surface (22) from the map (30); projecting the area (28) or a boundary of the area (28) from the map (30) to the image (20) according to the absolute position and the absolute orientation of the camera. The method according to claim 3.

5. The extension data of the water surface (22) is sensor data collected by a sensor, The method according to claim 1, wherein the spatial relationship between the camera and the area (28) of the water surface (22) is given by a fixed spatial relationship between the camera and the sensor.

6. The method according to claim 5, wherein the fixed spatial relationship between the camera and the sensor comprises a distance between the camera and the sensor, a relative orientation of the camera and the sensor with respect to each other, and / or a height level of the camera with respect to the sensor.

7. Matching the extended data of the water surface (22) to the image according to the spatial relationship between the camera and the area (28) of the water surface (22) comprises extracting the area (28) of the water surface (22) or the boundary of the area (28) from the sensor data, and projecting the area (28) or the boundary of the area (28) onto the image (20) according to the fixed spatial relationship between the camera and the sensor, according to the method of claim 5 or 6. **Claim 8** The method according to any one of claims 5 to 7, wherein the sensor comprises another imaging device, and the other imaging device is configured to detect the water surface (22). **Claim 9** The method according to any one of claims 5 to 8, wherein the sensor comprises a lidar sensor and / or a radar sensor. **Claim 10** After matching the extended data of the water surface (22) to the image data and before labeling the water surface (22) in the image (20), receiving object information regarding at least one object (32, 34) other than the land (24) that is underwater and extends from the water surface (22), and determining an object area in the image (20) according to the object information, the object area being covered by the object (32, 34), and further comprising labeling the water surface (22) excluding the object area, according to the method of any one of claims 1 to 9. **Claim 11** A method of providing a training data set for training, validating, and / or testing a machine learning algorithm to enable the machine learning algorithm to detect a water surface (22) in an image (20), comprising providing, as features for the training, the amount of unlabeled images (20), each of the images (20) showing at least one water surface (22), and providing an amount of labeled images (20), wherein the labeled images (20) each correspond to one of the unlabeled images (20), and each of the labeled images (20) is labeled by the method of any one of claims 1 to 10. **Claim 12** A machine learning algorithm for detecting a water surface (22) in an image (20), wherein the machine learning algorithm is trained by the method according to claim 11.

13. A water surface detection system for detecting a water surface (22) in an image (20), a camera for generating image data of the image (20), the image comprising at least one water surface (22), a memory containing water surface extension data, the water surface (22) extension data representing an area (28) where the water surface (22) extends in the real world, a controller configured to match the water surface extension data to the image data according to a spatial relationship between the camera and the area (28) of the water surface (22), and label the water surface (22) in the image (20) according to the matched water surface (22) extension data.

14. The water surface detection system according to claim 13, further comprising a sensor for collecting the water surface extension data.

15. A computer program comprising instructions configured to execute the method according to any one of claims 1 to 10 and / or the method according to claim 11 when the computer program is executed by a processor of a computer.

16. A computer-readable medium storing the machine learning algorithm according to claim 12 and / or the computer program according to claim 15.

Images