Method for determining the ship-water interface, and method and system for determining the positional relationship between one's own ship and a target ship.

Abstract

A method for determining a ship-water interface (40) between at least one target ship (34) and a water surface (22) of a water area in which the target ship (34) sails from an image (20) showing the target ship (34) on the water area, the method comprising: determining, for pixels of the image (20), an interface value representing a probability of a corresponding pixel indicating the ship-water interface (40); determining, from the image data, boundary box data of a boundary box (32) surrounding the target ship (34) in the image (20); and determining ship-water interface data representing a position and an extent of the ship-water interface (40) in the image (20) according to the determined interface value and the boundary box data.

Description

Technical Field

[0001] The present invention relates to the field of ship-water interface detection and ocean range and orientation measurement. In particular, the present invention relates to a method for determining a ship-water interface and a method and system for determining the positional relationship between a self-ship and a target ship.

Background Art

[0002] Autonomous solutions are expected to transform the world of international transportation. One or more cameras as sensors for collecting real-world information can play an important role in the development of these solutions, for example, any ship, i.e., a vessel or a boat, can be operated successfully by relying on visual perception. In terms of vision, a monocular system is more attractive than a stereo system because it is less complex and less expensive, for example, compared to a stereo system. However, with respect to ship positioning, including, for example, estimating the range to a self-ship, i.e., the owned ship, and / or the orientation of a target ship, i.e., another ship, a monocular system may be limited to locating points along the ground plane, i.e., points on the water surface of the water area where the ship is navigating. Since the extraction of these points can be difficult, some methods known in the art assume that the ship-water interface is aligned with the lower edge of the bounding box generated by an object detection algorithm configured to detect ships in an image. Another common assumption in the art can be that the midpoint of the lower end of the bounding box is on the ship-water interface. However, these assumptions can introduce inaccuracies, for example, there can be applications where the determined ship-water interface needs to be more accurately determined if it is to be used to determine the positional relationship between a self-ship and a target ship, for example, the range between the self-ship and the target ship, in other words, the distance, and / or the orientation of the target ship.

[0003] Therefore, there is a need for more accurate methods to determine the ship-water interface in an image and / or the positional relationship between one's own ship and a target ship. [Overview of the Initiative]

[0004] An object of the present invention is to provide a method for determining the ship-water interface between at least one target vessel and the water surface of a body of water, which can be carried out in a highly accurate, and / or easy, simple, and / or cost-effective manner, and / or can enable the determination of the positional relationship between the target vessel and the vessel itself with great accuracy.

[0005] The object of the present invention is to provide a method for determining the relative positions of one's own vessel and a target vessel in a body of water that is highly accurate and / or can be carried out in an easy, simple, and / or cost-effective manner.

[0006] The object of the present invention is to provide a system for determining the relative positions of one's own vessel and a target vessel in a body of water that can be operated in a highly accurate, and / or easy, simple, and / or cost-effective manner.

[0007] These objectives are achieved by the subject matter of the independent claims. Further exemplary embodiments are evident from the dependent claims and the following description.

[0008] The objective is achieved by a method for determining the ship-water interface between at least one target vessel and the water surface of the area in which the target vessel is navigating, from an image showing the target vessel in the area, the method comprising: receiving image data of the image; determining an interface value for each pixel of the image, where the interface value represents the probability that the corresponding pixel represents the ship-water interface; determining bounding box data of a bounding box surrounding the target vessel in the image from the image data; and determining ship-water interface data, where the ship-water interface data represents the location and extension of the ship-water interface in the image, depending on the determined interface value and bounding box data.

[0009] Bounding box data can be determined before, after, or simultaneously with the interface value. If there are two or more target vessels in the image, more bounding boxes may be determined accordingly, for example, one for each target vessel. In this context, determining bounding boxes makes it possible to separate two different target vessels from each other, or to distinguish a buoy from a vessel, in particular when these vessels overlap each other in the image. Preferably, the bounding box data is determined so that the target vessels fit precisely into the corresponding bounding boxes, as is typical for bounding boxes determined by object detection algorithms or instance segmentation known in the art.

[0010] Determining interface values, and determining ship-water interface data based on interface values ​​and boundary box data, allows for a highly accurate determination of the ship-water interface and can be performed in an easy, simple, and / or cost-effective manner. As a result, the above method allows for a highly accurate determination of the positional relationship between the target ship and the ship itself, as described below.

[0011] The body of water may be a lake, ocean, sea, or river. Image data may be received by an entity performing the above method. The entity may be, for example, a general-purpose computer on its own vessel, or a system for determining the positional relationship between its own vessel and a target vessel in the image.

[0012] Preferably, at least one interface value is determined for each pixel of the image. The interface value may be a probability value ranging from 0 to 1 or from 0% to 100%. Alternatively, the interface value may be convertible to or from these probability values, for example, ranging from 0 to 1, 0 to 10, -5 to 5, or from negative to positive infinity. The interface value may be presented as a single interface value per pixel. For example, if the interface value ranges from 0 to 1, the threshold may be set to 0.5, where all pixels with an interface value greater than 0.5 represent corresponding pixels indicating the ship-water interface, and all pixels with an interface value less than or equal to 0.5 represent corresponding pixels that do not indicate the ship-water interface. It should be noted that the above ranges and thresholds are merely examples, and other ranges and / or thresholds may be set to distinguish pixels indicating the ship-water interface from pixels that do not.

[0013] Interface values ​​may be determined by semantic segmentation of the image and optionally by post-processing the output of the semantic segmentation. Semantic segmentation may be performed by a neural network. The neural network may be trained and / or configured to output ship-water values ​​that represent the probability of a corresponding pixel in the image showing a target ship, water, or neither, for example, an air gap, sky, land, etc. These ship-water values ​​may be modified by one or more post-processing steps such that the modified output of the neural network represents the probability of a corresponding pixel showing a ship-water interface, in this embodiment the modified ship-water values ​​correspond to interface values. This neural network may be referred to below as the first neural network. The first neural network may be trained with a training dataset containing image data of a certain amount of images showing ships in a body of water, where the ships and water surface in the corresponding body of water are labeled in the images.

[0014] Alternatively, the neural network may be trained and / or configured such that the output of the neural network directly represents the probability of the corresponding pixel, regardless of whether the output of the neural network represents the ship-water interface or not, in this embodiment, the output of the neural network directly corresponds to the interface value. Therefore, in this embodiment, no post-processing of the output of the neural network is required to obtain the interface value. This neural network may be called a second neural network. The second neural network may be trained with a training dataset containing image data of a certain amount of images representing ships on a body of water, where the ship-water interface is labeled in the images. As an alternative to using a quantity of images to train the neural network, the neural network may be trained by one-shot learning, as is known in the art.

[0015] According to the embodiment, interface values ​​are interpreted as cost or reward, and ship-water interface data is determined by minimizing cost or maximizing reward, respectively. The seam finder module may minimize cost or maximize reward, and interface values ​​may be used as input for the seam finder module. Ship-water interface data may include xy coordinates, or pairs of the number of columns and rows of pixels in an image that represent the ship-water interface in the image. The seam finder module may use, for example, dynamic programming, or one of the other methods for determining ship-water interface data, such as methods from graph theory, e.g., Dijkstra's algorithm, as is known in the art.

[0016] According to the embodiment, the interface values ​​form an array of interface values, the position of the interface values ​​in the array corresponds to the position of the corresponding pixel in the image, and the cost and reward refer to the sum of the interface values ​​of all pixels required to move pixel by pixel from one side of the array corresponding to one side of the bounding box to the other side of the array corresponding to the other side of the bounding box. Figuratively speaking, the seam finder module may be executed from one side of the bounding box to the other side in order to minimize cost or to maximize reward. One side may correspond to the left side of the bounding box, and the other side may correspond to the right side of the bounding box. In other words, when calculating cost, the seam finder module may search for the cheapest path from the left side of the bounding box to the right side of the bounding box, or when calculating reward, it may search for the most valuable path from the left side of the bounding box to the right side of the bounding box, and the result is interpreted as the ship-water interface. Alternatively, the seam finder module may be executed from the right side to the left side of the bounding box.

[0017] According to one embodiment, bounding box data is determined by an object detection algorithm configured to detect vessels in an image and determine vessel-water interface data for each determined bounding box. The object detection algorithm may be any object detection algorithm known in the art that can detect vessels in an image. In particular, the object detection algorithm may be another neural network trained to detect vessels in an image by a training dataset containing a certain amount of images showing vessels, especially vessels in the corresponding body of water. The object detection algorithm for determining the bounding box data may hereafter be referred to as the third neural network.

[0018] Alternatively, according to another embodiment, bounding box data is determined by instance segmentation. In this case, bounding box data may be extracted from the output of instance segmentation, as described later.

[0019] As an alternative to a first or second neural network using semantic segmentation to determine ship-water values ​​or interface values, respectively, in one step, and instance segmentation to determine a third neural network or bounding box data, in another step, the first and / or second neural networks may use panoptic segmentation, which enables them to determine ship-water values ​​or interface values ​​and bounding box data, respectively, in one step and / or by the corresponding neural networks alone, without requiring a third neural network or instance segmentation.

[0020] According to the embodiment, the method further includes determining ship-water values ​​for pixels in an image from the image data after receiving the image data and before determining the ship-water interface data, wherein the ship-water value represents the probability that the corresponding pixel is neither a target ship, a body of water, nor both, i.e., an air gap, air, land, etc., and determining the interface value from the ship-water value. In particular, the ship-water value may be modified by one or more post-processing steps, the input to which this post-processing is the ship-water value and the output to which this post-processing is the interface value. Alternatively, the ship-water value may be used as input to another neural network, hereafter referred to as a fourth neural network, which determines the interface value from the ship-water value. The ship-water value may be determined by semantic segmentation or panoptic segmentation of the image.

[0021] Semantic or panoptic segmentation that outputs ship-water values ​​may be performed by a first neural network. The ship-water values ​​may be presented in the form of a triplet of three ship-water values ​​per pixel, where the first ship-water value having a first index in the triplet may represent the probability that the corresponding pixel does not show water or a ship, the second ship-water value having a second index in the triplet may represent the probability that the corresponding pixel shows the water surface, and the third ship-water value having a third index in the triplet may represent the probability that the corresponding pixel shows a target ship. Preferably, at least one ship-water value is determined for each pixel of the image. The ship-water values ​​may be probability values ​​ranging from 0 to 1 or from 0% to 100%. Alternatively, the ship-water values ​​may be convertible from or to these probability values, for example, ranging from 0 to 1, 0 to 10, -5 to 5, or from negative to positive infinity.

[0022] The first neural network may be pre-trained by supervised learning using a training dataset of image data of a certain amount of images showing ships on a body of water, and by images labeled with respect to the corresponding ships and bodies of water. If the fourth neural network determines interface values ​​from ship-water values ​​output by the first neural network, the fourth neural network may be pre-trained by supervised learning using, for example, a training dataset containing a large array of ship-water values ​​determined by the first neural network from image data, and by arrays labeled with respect to the ship-water values ​​corresponding to those pixels in images showing the ship-water interface.

[0023] Alternatively, if the second neural network directly determines interface values ​​from image data, the second neural network may be pre-trained by supervised learning using a training dataset of image data of a certain amount of images showing ships on a body of water, with images labeled with respect to the corresponding ship-water interface. In particular, the second neural network may be trained so that only physical ship-water interfaces can be determined as ship-water interfaces by correspondingly labeled images. Thus, virtual ship-water interfaces shown in images but not corresponding to real-world ship-water interfaces may not be determined as ship-water interfaces by the correspondingly trained second neural network. Instead of using supervised learning with a large amount of training data, one or more machine learning techniques that do not require a large amount of training data, especially image data or ship-water values, such as one-shot learning, as known in the art, may be used to train one or more of the above neural networks.

[0024] According to the embodiment, the method further includes determining the edge values ​​of pixels in an image after receiving image data and before determining ship-water interface data, wherein the edge values ​​represent pixels corresponding to whether or not they indicate an edge in the image, and determining ship-water interface data according to the edge values. This may contribute to improving the accuracy of the ship-water interface data. The edge values ​​can be determined by any image processing algorithm configured for edge detection, as is known in the art.

[0025] According to the embodiment, ship-water interface data is determined according to the edge value by multiplying the interface value by the corresponding edge value on a pixel-by-pixel basis, and by determining the ship-water interface data according to the corresponding product.

[0026] According to an embodiment, the method further includes determining that all pixels with corresponding interface values below a predetermined threshold do not represent the ship-water interface. Thus, before outputting the position and extent of the ship-water interface in the form of ship-water interface data, the ship-water interface data determined by the seam finder module may be modified such that all pixels with corresponding interface values below the predetermined threshold are determined not to represent the ship-water interface.

[0027] According to an embodiment, the interface value and / or the ship-water value is determined by a neural network, such as a second neural network, or a first neural network and / or a fourth neural network, as described above.

[0028] According to an embodiment, the method further comprises modifying the image data according to the determined ship-water interface data such that the ship-water interface is illustrated within the image when the image is presented on a display. For example, the image data may be modified according to the ship-water interface data such that the ship-water interface is highlighted within the image after modification when the image is presented on a display. The ship-water interface can be easily displayed within the image. However, displaying the ship-water interface within the image may not be necessary for all applications that use the determined ship-water interface. For example, when the ship-water interface data is used to determine the positional relationship between the own ship and a target ship on the water surface, the ship-water interface need not be displayed on the display.

[0029] The object is achieved by a method for determining the positional relationship between a self-ship and a target ship within a water area. The self-ship is equipped with a camera for taking an image of the surroundings of the self-ship, and the method includes determining at least one ship-water interface within the image according to the above method, determining position data representing the position and orientation of the camera when the camera captured the image, determining pixel data of at least one image point on the ship-water interface within the image from the image data, determining the real-world coordinates of at least one real-world point of the ship-water interface on the water area, and determining the positional relationship between the self-ship and the target ship according to the position data and the real-world coordinates of at least one real-world point of the ship-water interface on the water surface.

[0030] It should be understood that the features of the method for determining the ship-water interface between at least one target ship and the water surface of the water area as described above and below can be the features of the method for determining the positional relationship between a self-ship and a target ship in the water area as described above and below.

[0031] At least one ship-water interface within the image can be determined by determining the corresponding ship-water interface data as described above. The camera position may correspond to the position of the center of the camera. The camera center may be given by the pinhole of the camera. Before starting one of the above methods, the camera can be calibrated. In particular, the orientation and / or position of the camera relative to the self-ship can be calibrated by any calibration method known to those skilled in the art. In particular, the intrinsic and extrinsic calibration of the camera may be performed as described, for example, in the section "Camera Models and Calibration" of "Learning OpenCV" (published by O'Reilly Media, Inc. in September 2008, ISBN: 9780596516130) by Gary Bradski and Adrian Kaehler. When the camera is calibrated, since there is a fixed spatial relationship between the camera and the self-ship, the position data also represents the position of the self-ship.

[0032] When performing the above method, a self-ship motion correction may be performed to account for the movement of the self-ship when determining the ship-water interface and / or positional relationship between the self-ship and the target ship. The self-ship motion correction may be performed by any self-ship motion correction method known in the art.

[0033] The real-world coordinates of a real-world point at the ship-water interface on a body of water can be determined by any projection method known in the art, for example, as described and explained in detail in a textbook on computer vision, e.g., "Multiple View Geometry in Computer Vision," 2nd edition, ISBN-13 978-0-521-54051-3 by Richard Hartley and Andrew Zisserman, e.g., Part I: Camera Geometry and Single View Geometry.

[0034] According to the embodiment, the real-world coordinates of a real-world point are determined by extrapolating the camera position of the camera through the image points to the water surface of the body, depending on the position data and pixel data. To extrapolate the camera position through the image points to the water surface, the image may be virtually placed in an image plane, for example, a virtual image plane, or the camera's real plane, or any other suitable plane. The camera's image plane may correspond to the camera's sensor plane, for example, the camera's CCD plane if the camera is a CCD camera. The virtual image plane may be placed between the camera and the water surface. In contrast, the image plane may be placed behind the camera when viewed from the water surface, in which case the real-world point may be determined by extrapolating the image points through the camera position to the water surface.

[0035] In some embodiments, the positional relationship refers to the range, in other words, the distance, between the vessel and the target vessel, and / or the bearing of the target vessel. The bearing of the target vessel may be a relative bearing or an absolute bearing. The relative bearing refers to the angle between the heading of the vessel and a line extending from a given point on the vessel to a determined real-world point on the vessel-water interface. The absolute bearing refers to the angle between the fundamental direction of north (hereinafter referred to as "true north") and a line extending from a given point on the vessel, such as the camera position or the center of the vessel, to a determined real-world point on the vessel-water interface. In this context, the absolute bearing may be considered as the positional relationship between the vessel and the target vessel, because the heading of the vessel relative to true north is usually known, and therefore, when the absolute bearing of the target vessel is known, the relative bearing between the vessel and the target vessel is also known.

[0036] According to one embodiment, the pixel data of at least one image point is determined such that the image point corresponds to the real-world point of the ship-water interface closest to the ship. The real-world point of the ship-water interface closest to the ship can be determined by projecting all image points of the ship-water interface in the image onto the real-world coordinate system. The point in the real-world coordinate system closest to the ship can then be determined. Once the real-world point of the ship-water interface closest to the ship is determined in the real-world coordinate system, the corresponding image point in the image coordinate system can be determined because there is a point-to-point correspondence between the image point in the image coordinate system and the corresponding real-world point in the real world.

[0037] The objective is achieved by a system for determining the relative positions of the vessel and a target vessel within a body of water, the system comprising a camera for capturing images of the vessel's surroundings, and a processing unit located on the vessel, for example, installed on the vessel, coupled to the camera, and configured to perform the above-described method for determining the relative positions of the vessel and a target vessel within a body of water. Optionally, the processing unit may be located away from the camera and / or the vessel. For example, the processing unit may be located in the vessel's computer or in a server remote from the vessel.

[0038] It should be understood that the features of a method for determining the ship-water interface between at least one target vessel and the water surface of a body of water, and / or the features of a method for determining the positional relationship between one's own vessel and a target vessel in a body of water, as described above and below, may also be features of a system for determining the positional relationship between one's own vessel and a target vessel in a body of water, as described above and below.

[0039] Furthermore, a computer program may be provided which includes instructions configured to perform at least one of the methods described above when executed by a computer processor or by a processing unit of the system described above. It should be understood that the features of the methods described above and below may be features of the control and / or power supply systems described above and below.

[0040] Furthermore, a computer-readable medium storing the above-mentioned computer program may also be provided. It should be understood that the features of the methods described above and below may be the features of the computer-readable medium described above and below. The computer-readable medium may be a floppy disk, a hard disk, a USB (Universal Serial Bus) storage device, RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), or flash memory. The computer-readable medium may also be a data communication network, such as the Internet, that enables the download of program code. The computer-readable medium may be non-temporary or temporary.

[0041] These and other aspects of the present invention will become apparent from the examples described below and will be clarified by reference to those examples. [Brief explanation of the drawing]

[0042] The subject matter of the present invention will be described in detail below with reference to exemplary embodiments illustrated in the accompanying figures. [Figure 1] Figure 1 is a block diagram illustrating an exemplary embodiment of a method for determining the ship-water interface between at least one target vessel and the water surface of a body of water. [Figure 2] Figure 2 shows an example of an image illustrating a target vessel in a body of water. [Figure 3] Figure 3 shows an example of a ship-to-water value visualization determined from the image in Figure 2. [Figure 4] Figure 4 shows an example of visualization of interface values ​​determined from the ship-water values ​​visualized in Figure 3. [Figure 5] Figure 5 shows an example image illustrating the target vessel within the water area shown in Figure 2, and the boundary box surrounding the target vessel. [Figure 6] Figure 6 shows an example of a ship-water interface visualization determined from the interface values ​​visualized in Figure 4 and the boundary box shown in Figure 5. [Figure 7] Figure 7 shows a flowchart of an exemplary embodiment of a method for determining the relative positions of one's own vessel and a target vessel within a body of water. [Figure 8] Figure 8 illustrates the principle of an exemplary embodiment of a method for determining the relative positions of one's own vessel and a target vessel within a body of water. [Figure 9] Figure 9 shows a block diagram illustrating an exemplary embodiment of a method for determining the ship-water interface between at least one ship and the water surface of a body of water. [Figure 10] Figure 10 shows an example of visualization of interface values ​​determined from the image in Figure 2.

[0043] The reference numerals used in the drawings and their meanings are listed in a summary format in the reference numeral list. As a general rule, the same reference numeral is assigned to identical parts within a drawing. [Modes for carrying out the invention]

[0044] Figure 1 shows a block diagram illustrating an exemplary embodiment of a method for determining the ship-water interface 40 (see Figure 4) between at least one target vessel 34 (see Figure 2) and the water surface 22 of a body of water in image 20 (see Figure 2). This method may be performed by a processing unit of a general-purpose computer, or by a processing unit of a system for determining the positional relationship between the vessel itself (not shown) and the target vessel 34 in the body of water, as will be described later. Hereinafter, the target vessel 34 will be referred to as "the vessel" only, while the vessel on which a camera (not shown) for capturing image 20 is positioned will be referred to as "the vessel itself." The body of water may be a lake, ocean, sea, or river.

[0045] In the first step of this method, image 20 may be captured by a camera (not shown).

[0046] Figure 2 shows an example of an image 20 showing a vessel 34 in a body of water. In particular, it can be seen from the image 20 that the vessel 34 is navigating on a body of water. The body of water may be represented in the image 20 by the surface of the water 22. The image 20 may show the surface of water 22, land 24, for example, two landmasses, the air 26, and the vessel 34. The image 20 shows only one continuous surface of water 22. Another valid image 20 may show two or more surfaces of water 22 that are separated from each other. The image 20 shows only one vessel 34. Another valid image 20 may show two or more vessels 34 that are separated from each other or overlapping each other.

[0047] The camera may be positioned on the vessel. The camera may generate image data representing image 20. The image data may be transmitted to and / or received by a processing unit that performs a method for determining the vessel-water interface 40. The image data may be stored in the memory of a general-purpose computer or system, and the memory is coupled to the processing unit. When this method is performed, the processing unit may receive the image data directly from the camera or from memory. The camera may be calibrated with respect to its position and / or orientation relative to the vessel. In particular, the camera may be calibrated by external and internal calibration. The position of the camera may include the height of the camera on the vessel. Therefore, if the position and orientation of the vessel are known, the position and orientation of the camera are also known.

[0048] The first neural network B2 may be configured to receive image data and determine at least one ship-water value for each pixel of image 20 from the image data. For example, the first neural network B2 may be configured to determine three or more ship-water values ​​for each pixel of the image. In particular, if image 20 contains an array of N × M pixels, where N and M are natural numbers, the first neural network B2 can determine 3 × N × M ship-water values ​​assigned to the corresponding pixels. The ship-water value represents the probability that the corresponding pixel is neither a target ship, the water surface 22, nor either of the above, for example, land 24, air 26, or an empty space. The ship-water values ​​may be determined by semantic segmentation of image 20. Semantic segmentation may be performed by the first neural network B2, which has been trained to distinguish between water and ships in an image by supervised learning using correspondingly labeled images, for example. For example, the first neural network B2 may be pre-trained by supervised training using a certain amount of images 20 showing a vessel 34 on a body of water, and by images 20 labeled with respect to the corresponding water surface 22 and vessel 34. In addition, images 20 that do not show any vessel 34 may also be used to train the first neural network B2. Instead of using a quantity of images 20 for training, one-shot learning, as known in the art, may be applied to train the first neural network B2.

[0049] Figure 3 shows an example of a ship-water value visualization 36 determined from image 20 of Figure 2 by the first neural network B2. In particular, Figure 3 shows a representation of the first segmentation output of the first neural network B2. The first segmentation output corresponds to the ship-water value. With respect to the output of the first neural network B2, the final layer of the first neural network B2 may be, for example, a softmax function. The softmax function may provide a pixel-by-pixel output of image 20 that can correspond to a triplet of probabilities, for example (x0, x1, x2), where the first value of the triplet with index 0 may represent the probability that the corresponding pixel does not show the water surface 22 or the ship 34, the second value of the triplet with index 1 may represent the probability that the corresponding pixel shows part of the water surface 22, and the value of the triplet with index 2 may represent the probability that the corresponding pixel shows part of the target ship 34. The ship-water value may be a probability value ranging from 0 to 1 or from 0% to 100%. Alternatively, the ship-water values ​​may be convertible to or from these probability values, for example, ranging from 0 to 1, 0 to 10, -5 to 5, or from negative to positive infinity. If these ship-water values ​​are between 0 and 1, then all ship-water values ​​in a triplet will sum to 1. Thus, for example, if the output of one of the pixels is (0.1, 0.7, 0.2), the probabilities that the pixel is neither part of the water surface 22 nor part of the ship 34 are 0.1, 0.7, and 0.2, respectively. If more classes than water, ship, or neither are required, for example for air 26 or land 24, more than three ship-water values ​​may be determined on a pixel-by-pixel basis accordingly.

[0050] For example, as shown in Figure 3, to visualize the output of semantic segmentation, the output of the first neural network B2, i.e., the ship-water value, may be assigned to different colors. For example, the index of a ship-water value in a triplet may be assigned to the corresponding pixel that represents the ship-water value representing the highest probability in the corresponding triplet. For example, if the first ship-water value in a triplet has the highest probability in the triplet, "0" may be assigned to the corresponding pixel; if the second ship-water value in a triplet has the highest probability in the triplet, "1" may be assigned to the corresponding pixel; and if the third ship-water value in a triplet has the highest probability in the triplet, "2" may be assigned to the corresponding pixel. In the above example, the ship-water value with index 1, i.e., 0.7, is the highest value in the triplet, and this ship-water value represents the probability that the corresponding pixel is part of the water surface 22. Therefore, in the above example, the value "1" may be assigned to the corresponding pixel that is part of the water surface 22. Furthermore, the numbers "0", "1", and "2" assigned to pixels may be assigned to different colors. For example, if the number assigned to one of the pixels is "1", a first color, such as green, may be used for the corresponding pixel for visualization 36, and if the number assigned to that pixel is "2", a second color, such as red, may be used for the corresponding pixel for visualization 36.

[0051] As described above, the argmax function is applied to assign the index of the ship-water value triplet to the corresponding pixel representing the ship-water value of the corresponding triplet representing the highest probability. In particular, the ship-water value may be input to the argmax function. The argmax function then outputs the index of the ship-water value representing the highest probability in the corresponding triplet for each corresponding pixel, and a color may be assigned to that output to achieve visualization 36.

[0052] The post-processing module B4 may receive the output of the first neural network B2, i.e., an array of ship-water values, and may be configured to determine at least one interface value for each ship-water value, and therefore for each pixel of the image 20. Thus, interface values ​​can be determined by the post-processing module B4. In particular, continuing the above example, if the image 20 contains an array of N × M pixels and the output of the first neural network B2 provides a corresponding array of 3 × N × M ship-water values, i.e., a 3D array, then the post-processing module B4 determines an array of N × M interface values, i.e., a 2D array, assigned to the corresponding pixels and / or ship-water values. The interface value represents the probability that the corresponding pixel indicates a ship-water interface 40 (see Figure 4) between the ship 34 and the water surface 22.

[0053] Therefore, the ship-water values ​​can form a 3D array. The first two dimensions of this 3D array may correspond to the spatial dimensions of image 20, and the last dimension may point to three channels corresponding to the ship 34 and the water surface 22, or neither. The post-processing module B4 can determine interface values ​​from the ship-water values ​​by extracting the ship-water values ​​corresponding to the ship 34 and the water surface 22 from the 3D array. The extracted values ​​form two 2D arrays, one for the ship-water values ​​corresponding to the ship 34 and the other for the ship-water values ​​corresponding to the water surface 22, and the two dimensions of the 2D arrays correspond to the spatial dimensions of image 20. These 2D arrays can then be multiplied pixel by pixel to obtain another 2D array, the resulting 2D array containing one interface value pixel by pixel, and the multiplication is defined as an Hadamard product. The two dimensions of the resulting 2D array may correspond to the spatial dimensions of image 20.

[0054] For example, suppose one of the pixels has a ship-water value of water°=°0.4, ship°=°0.5, and "neither"°=°0.1. Then, for this pixel, the ship-water value representing the probability that the pixel represents the water surface 22 is multiplied by the ship-water value representing the probability that the pixel represents the ship 34, i.e., 0.4°*°0.5°=°0.2. Thus, in the resulting 2D array, the interface value of the pixel considered is 0.2. This procedure can be repeated for all pixels in the image 22, or at least in the area of ​​interest, for example, in the bounding box 32.

[0055] However, many alternative solutions are possible, as there are many ways to define a reasonable function for converting three ship-water values ​​per pixel into a single interface value per pixel. For example, an alternative solution can be based on modeling the ship-water values ​​using a Dirichlet distribution.

[0056] The interface value may be a probability value ranging from 0 to 1 or from 0% to 100%. Alternatively, the interface value may be from or convertible to these probability values, for example, ranging from 0 to 10, -5 to 5, or from negative to positive infinity. Instead of determining the interface value from the ship-water value by the post-processing described above, the interface value may be determined by another neural network (not shown), e.g., a fourth neural network. The fourth neural network may be trained with a training dataset containing a certain amount of ship-water values ​​in an array of ship-water values. Instead of using an array of values, one-shot learning, as known in the art, may be applied when training the other neural network.

[0057] Figure 4 shows an example of a visualization 38 of interface values ​​determined from the ship-water values ​​visualized in Figure 3. In particular, Figure 4 shows the output of post-processing module B4. The output corresponds to the interface values. The interface values ​​may be binary values, where binary 1 may be assigned to all pixels indicating the ship-water interface, and binary 0 may be assigned to all pixels not indicating the ship-water interface. Then, to provide the visualization 38, a first color may be assigned to all pixels indicating the ship-water interface 40, a second color may be assigned to all pixels not indicating the ship-water interface 40, or no color may be assigned.

[0058] The ship-water interface 40 may comprise a physical ship-water interface 42 and a virtual ship-water interface 44. The physical ship-water interface 42 corresponds to a real-world ship-water interface where the water surface 22 is in direct physical contact with the ship 34. The virtual ship-water interface 44 corresponds to the ship-water interface 40 in image 20 where the pixel representing the water surface 22 is the next adjacent pixel to the pixel representing the ship 34, and in the real world, the corresponding area of ​​the surface of the ship 34 has no direct contact with the water surface 22.

[0059] Figure 5 shows an example of Image 20, which shows the target vessel 34 in the water and the boundary box 32 around the target vessel 34, according to Figure 2.

[0060] An object detection module B10 may be provided, configured to determine bounding box data for bounding boxes 32 surrounding vessels 34 in image 20 from image data. The bounding box data may be determined by an object detection algorithm represented by the object detection module B10. The bounding box data may be determined before, after, or simultaneously with the interface values. If there are two or more target vessels 34 in image 20, more bounding boxes 32 may be determined accordingly, for example, one bounding box 32 for each target vessel 34. The object detection algorithm may be any conventional object detection algorithm capable of detecting vessels 34 in image 20. The object detection algorithm may be, for example, a neural network called a third neural network. The third neural network may be pre-trained so that it can detect and mark vessels 34 in image 20. For example, the third neural network may be trained by a training dataset containing a correspondingly labeled amount of images, or by one-shot learning as known in the art. Bounding box data can be matched with image data such that the bounding box 32 surrounds the vessel 34 in the image 20 when the image 20 is displayed on a display. Preferably, the bounding box data is determined so that the corresponding vessel 34 fits precisely into the corresponding bounding box 32. This is typical for bounding boxes determined by object detection algorithms known in the art. Alternatively or additionally, the bounding box data may be matched with vessel-water interface data such that the bounding box 32 surrounds the vessel-water interface 40 in the vessel-water interface visualization 39 when the vessel-water interface visualization 39 is displayed on a display.

[0061] As an alternative to using an object detection algorithm to determine bounding box data, the bounding box data may be determined by instance segmentation. In this case, the bounding box data may be extracted from the output of instance segmentation. For example, the outermost pixels in the x and y directions of a target vessel identified by instance segmentation may be used to determine the bounding box data and thereby determine the bounding box around the target vessel. In particular, the bounding box data may be determined such that the corresponding bounding box includes these outermost pixels. The instance segmentation algorithm that performs instance segmentation may be any instance segmentation algorithm known in the art that can detect vessels in an image. In particular, the instance segmentation algorithm may be another neural network trained to detect vessels in an image by a training dataset containing a certain amount of images showing vessels, especially vessels in the corresponding waters. A suitable example of instance segmentation is described in the paper "Mask R-CNN" (2017) by He, K., Gkioxari, G., Dollar, P., and Girshick, R., in the proceedings of the IEEE International Conference on Computer Vision (pp. 2961-2969).

[0062] As an alternative to a first or second neural network using semantic segmentation to determine ship-water values ​​or interface values, respectively, in one step, and using a third neural network or instance segmentation to determine bounding box data, in another step, the first and / or second neural networks may use panoptic segmentation, which allows them to determine ship-water values ​​or interface values ​​and bounding box data, respectively, in one step and / or by the corresponding neural networks alone, without requiring a third neural network or instance segmentation. Essentially, panoptic segmentation is a combination of instance segmentation and semantic segmentation. Therefore, panoptic segmentation can be used to replace both object detection and semantic segmentation. For example, panoptic segmentation may be used as described in the paper "UPSNet: A Unified Panoptic Segmentation Network" by Yuwen Xiong et al., CVPR 2019.

[0063] The seam finder module B6 may be configured to determine ship-water interface data representing the ship-water interface 40 in response to interface values ​​provided by the post-processing module B4 and bounding box data provided by, for example, the object detection module B10. To determine the ship-water interface data from the interface values, the interface values ​​are interpreted as cost or reward, and the ship-water interface data is determined by minimizing the cost or maximizing the reward, respectively. The seam finder module B6 may minimize the cost or maximize the reward. In this embodiment, the interface values ​​may be used as input for the seam finder module B6. As described above, the interface values ​​form a 2D array of interface values, where the position of the interface values ​​in the array corresponds to the position of the corresponding pixel in the image 20. The ship-water interface data determined by the seam finder module B6 may consist of the xy coordinates of the pixels in the image 20 that indicate the ship-water interface 40 in the image 20, or pairs of column and row numbers. The ship-water interface data may represent the position and extension of the ship-water interface 40 in the image 20. The seam finder module B6 can determine ship-water interface data using, for example, seam carving, dynamic programming, or one of the other methods known in the art, such as methods from graph theory, for example, Dijkstra's algorithm.

[0064] The cost and reward refer to the sum of the interface values ​​of all pixels required to move pixel by pixel from one side of the array corresponding to one side of the bounding box 32 to the other side of the array corresponding to the other side of the bounding box 32. In other words, the seam finder module B6 may run from one column to the other of the array of interface values ​​to minimize the cost or maximize the reward. One column corresponds to the left side of the bounding box 32, and the other column corresponds to the right side of the bounding box 32. Figuratively speaking, when calculating the cost, the seam finder module B6 may search for the cheapest path from the left side of the bounding box 32 to the right side of the bounding box 32, and when calculating the reward, it may search for the most valuable path from one column of the array to the other column of the array. Alternatively, the seam finder module B6 may run from the right side to the left side of the bounding box 32. If there are two or more vessels 34 and corresponding two or more boundary boxes 32, the seam finder module B6 can run separately for each determined boundary box 32, i.e., for each determined vessel 34, and can determine vessel-water interface data for each boundary box 32, i.e., for each vessel 34.

[0065] Figure 6 shows an example of a visualization 39 of the ship-water interface 40 determined by the seam finder module B6, in particular, from the interface values ​​visualized in Figure 4 and the bounding box data visualized as a bounding box 32 in Figure 5. In other words, Figure 6 may show a visualization 39 of the output of the seam finder module B6. As can be seen from Figure 6, at least a portion of the virtual ship-water interface 44 may be removed from the ship-water interface 40 by the seam finder module B6 so that the determined ship-water interface 40 more accurately fits the real world.

[0066] The output module B8 may be configured to output the ship-water interface 40, for example, in the form of a visualization 39 of the ship-water interface 40 in image 20, or as raw data without specifically displaying the visualization 39 of the ship-water interface 40 and / or image 20 on the display.

[0067] The system may optionally be configured to modify the image data according to determined ship-water interface data so that when image 20 is shown on a display (not shown), the ship-water interface 40 is depicted in image 20. For example, the image data may be modified according to ship-water interface data so that when image 20 is shown on the display 10, the ship-water interface is highlighted in the image after modification.

[0068] Therefore, the ship-water interface 40 can be easily displayed in the image 20 and / or in the visualization 39 of the ship-water interface 40. However, displaying the ship-water interface 40 in the image 20 or in the visualization 39 of the ship-water interface 40 may not be necessary for all applications using the determined ship-water interface 40. For example, if the ship-water interface data is used to determine the positional relationship between the ship itself and the target ship 34 in a body of water, the ship-water interface 40 does not need to be displayed on the display. In this case, knowledge of the ship-water interface data may be sufficient to determine the positional relationship between the ship itself and the target ship 34.

[0069] Optionally, an edge detection module B12 may be provided and configured to determine edge values ​​from image data. The edge detection module B12 may include any software for finding edges in the image 20, as is known in the art. For example, one possible edge detection method is described in "Digital Image Processing" by Rafael Gonzalez and Richard Woods, 3rd edition, ISBN 9780131687288, where the edge value may be obtained by calculating the magnitude of the gradient, as described in section 10.2.5, "Basic Edge Detection," and may be combined with a threshold, as described in the same section. Alternatively, more advanced techniques may be used, such as those described in section 10.2.6, "More Advanced Techniques for Edge Detection," such as "The Canny edge detector." The output from the edge detection module B12 may be given by a binary value, e.g., 1 for all pixels indicating an edge and 0 for all pixels not indicating an edge. Alternatively, the magnitude of the edge may be used to represent the result of edge detection. Edge size is a measure of the edge intensity of a corresponding pixel, typically the magnitude of the gradient, and provides a non-binary value per pixel in image 20. The resulting edge value can be multiplied with the interface value, for example, by per-pixel multiplication.

[0070] Figure 7 shows a flowchart of an exemplary embodiment of a method for determining the positional relationship between the vessel and a target vessel 34 in a body of water. The positional relationship between the vessel and the target vessel 34 can refer to a range 50 (see Figure 8) between the vessel and the target vessel 34, and / or the bearings RB,AB (see Figure 8) of the target vessel 34. To carry out this method, the vessel may be equipped with a system for determining the positional relationship between the vessel and the vessel 34 in the body of water. The system includes a camera for capturing images 20 around the vessel, and the camera is positioned on the vessel. The system may further include a processing unit coupled to the camera and configured to carry out the following method. Before starting the method, the camera, in particular the orientation and / or position of the camera relative to the vessel, can be calibrated so that there is a fixed and / or known spatial relationship between the camera and the vessel. When carrying out this method, vessel motion correction may be performed to take into account the movement of the vessel when determining the positional relationship with respect to the target vessel 34.

[0071] In step S2, at least one ship-water interface 40 in image 20 is determined by the method described above with respect to, for example, Figures 1 to 6, or by the following method described with respect to Figures 9 and 10. At least one ship-water interface 40 in image 20 may be determined by determining the corresponding ship-water interface data.

[0072] In step S4, position data is determined, which may represent the camera position 28 (see Figure 7) and camera orientation of the camera when the camera captured the image 20. The camera position 28 may correspond to the position of the camera's center. The camera center may be given by the camera's pinhole. The position data may also represent the position of the vessel, i.e., the vessel's position 30, due to a fixed and / or known spatial relationship between the camera and the vessel.

[0073] Figure 8 illustrates the principle of an exemplary embodiment of a method for determining the positional relationship between the vessel and the target vessel 34 in a body of water. From Figure 8, it can be seen that a camera is positioned at camera position 28. The camera can be calibrated such that the spatial relationship between camera position 28 and the vessel's position 30 is fixed and / or known. For example, the projection of camera position 28 onto the water surface 22 can be considered the vessel's position 30. Alternatively, any other point on the vessel on the water surface 22 having a fixed and / or known spatial relationship with respect to camera position 28, such as the center of the vessel, can be considered the vessel's position 30.

[0074] The camera's image plane may correspond to the camera's sensor plane, for example, the camera's CCD plane if the camera is a CCD camera. The virtual image plane 52 may be positioned between the camera and the water body. The visualization 39 or image 20 showing the ship-water interface 40 may be positioned on the camera's virtual image plane 52.

[0075] In step S6, pixel data for at least one image point 46 on the ship-water interface 40 in the visualization 39 or in the image 20 can be determined from ship-water interface data or modified image data. The pixel data for at least one image point 46 may be determined such that the image point 46 corresponds to the real-world point 48 of the ship-water interface 40 closest to the ship. The real-world point 48 of the ship-water interface 40 closest to the ship can be determined by projecting all image points of the ship-water interface 40 in the image 20 onto a real-world coordinate system. Then, the point in the world coordinate system closest to the ship can be determined. Once the real-world point 48 of the ship-water interface 40 closest to the ship is determined in the world coordinate system, the corresponding image point in the image coordinate system can be determined because there is a point-to-point correspondence between the image point and the corresponding real-world point.

[0076] In step S8, the real-world coordinates of at least one real-world point 48 of the ship-water interface 40 on the body of water may be determined by extrapolating the camera position 28 of the camera through the image point 46 to the water surface 22 of the body of water, depending on the position data and pixel data, so that the image is in the camera's virtual image plane 52. The real-world coordinates of the real-world point 48 of the ship-water interface 40 on the body of water may be determined by any projection method known in the art, for example, as described and explained in detail in a textbook on computer vision, e.g., "Multiple View Geometry in Computer Vision", 2nd edition, ISBN-13 978-0-521-54051-3, e.g., Part I: Camera Geometry and Single View Geometry by Richard Hartley and Andrew Zisserman.

[0077] In step S10, the positional relationship between the vessel and the target vessel 34, i.e., the range 50 and / or bearings RB,AB between the vessel and the target vessel 34, can be determined according to the real-world coordinates of at least one real-world point 48 at the vessel-water interface 40 on the water surface 22 and the vessel's position 30.

[0078] The bearing of the target vessel 34 may be a relative bearing RB or an absolute bearing AB. The relative bearing RB may refer to, or correspond to, the angle between the vessel's heading 56, i.e., the vessel's orientation relative to true north 58, and a predetermined point on the vessel, for example, a line 54 extending from the vessel's position 30 or camera position 28 to a determined real-world point 48 on the vessel-water interface 40. The absolute bearing AB can refer to the angle between true north 58 and line 54. In this context, the absolute bearing AB can also be considered as the positional relationship between the vessel and the target vessel 34. This is because, typically, the heading 56 of the vessel's bow relative to true north 58 is known, and therefore, when the absolute bearing AB of the target vessel 34 is known, the relative bearing RB between the vessel and the target vessel 34 is also known.

[0079] Figure 9 shows a block diagram illustrating an exemplary embodiment of a method for determining the ship-water interface 40 between at least one ship 34 and a body of water. Some of the method steps, and some of the modules that perform the corresponding method steps, may correspond to the method steps and modules described with respect to Figures 1 to 6. Therefore, only the method steps and modules that differ between the methods described with respect to Figures 9 and 10 and those described with respect to Figures 1 to 6 are described below.

[0080] In this embodiment, the neural network, i.e., the second neural network B14, may receive image data of image 20 and may be configured to generate interface values ​​directly dependent on the image data. In this context, the second neural network B14 may be configured to determine at least one interface value for each pixel of image 20, where the interface value represents the probability that the corresponding pixel indicates a ship-water interface 40. For example, the second neural network B14 may be pre-trained by supervised learning using a certain amount of images 20 showing a ship 34 on a body of water, and by images 20 labeled with respect to the corresponding ship-water interface 40. In particular, the second neural network B14 may be trained to detect only physical ship-water interfaces 42 and not virtual ship-water interfaces 44. Instead of using a quantity of labeled images 20 to train the second neural network B14, the second neural network B14 may be trained by one-shot learning, as is known in the art.

[0081] Figure 10 shows an example of a diagram illustrating the ship-water interface 40 determined by the second neural network B14, where the virtual ship-water interface 44 is shown in Figure 10 by a dashed line for informational purposes only. The generated ship-water interface data does not necessarily have to include information corresponding to the virtual ship-water interface 44.

[0082] Optionally, a threshold module B16 may be included. The threshold module B16 may be configured to determine that all pixels whose corresponding interface value is below a predetermined threshold do not represent the ship-water interface. In other words, the threshold module B16 may be configured to set all interface values ​​below a predetermined threshold to an interface value representing the probability that the corresponding pixel does not represent the ship-water interface 40, for example, zero. Thus, before outputting the location and extension of the ship-water interface 40 in the form of ship-water interface data, the ship-water interface data determined by the second neural network B14 may be modified by the threshold module B16 so that all pixels whose corresponding interface value is below a predetermined threshold are determined not to represent the ship-water interface 40.

[0083] The aforementioned neural networks and modules, such as the first neural network B2, the second neural network B14, the third and / or fourth neural network, the seam finder module B6, the output module B8, the object detection module B10, the edge detection module B12, and / or the threshold module B16, can each be implemented by software, hardware, or a combination of software and hardware.

[0084] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustrations and descriptions should be considered illustrative or exemplary and not limiting, and the present invention is not limited to the disclosed embodiments. Other modifications of the disclosed embodiments can be understood and practiced by those skilled in the art practicing the claimed invention, by reference to the drawings, disclosure and the appended claims. In the claims, the word “equipped with” does not exclude other elements or processes, and the indefinite article “a” or “an” does not exclude plurals. A single processor or other unit may perform the functions of several items described in the claims. The mere fact that certain means are described in different dependent claims does not mean that combinations of these means cannot be used advantageously. None of the reference numerals in the claims should be construed as limiting the scope.

[0085] List of reference symbols 20 images 22 Water surface 24 Land 26 sky 28 Camera position 30 Own ship position 32 Boundary Box 34 Target Vessels 36. Visualization of Ship-Water Values 38 Visualization of interface values 39. Visualization of the ship-water interface 40 Ship-water interface 42 Physical ship-water interface 44. Virtual ship - water interface 46 points 48 Real-world points 50 range 52 Virtual Image Plane 54 lines 56 First own ship 58 Due North AB Absolute Direction RB Relative bearing B2 First Neural Network B4 Post-processing module B6 Seam Finder Module B8 Output Module B10 Object Detection Module B12 Edge Detection Module B14 The Second Neural Network B16 Threshold Module The invention described in the original claims of this application is listed below. [C1] A method for determining the ship-water interface (40) between at least one target vessel (34) and the water surface (22) of the water area in which the target vessel (34) is navigating, from an image (20) showing the target vessel (34) in the water area, Receiving the image data of the aforementioned image (20), The interface value for each pixel in the image (20) is determined, and the interface value represents the probability that the corresponding pixel represents the ship-water interface (40). The boundary box data of the boundary box (32) surrounding the target vessel (34) in the image (20) is determined from the image data, Based on the determined interface value and the bounding box data, ship-water interface data is determined, and the ship-water interface data represents the position and extension of the ship-water interface (40) in the image (20), A method for providing this. [C2] The method according to C1, wherein the interface value is interpreted as cost or reward, and the ship-water interface data is determined by minimizing the cost or maximizing the reward, respectively. [C3] The interface values ​​form an array of interface values, and the position of the interface value in the array corresponds to the position of the corresponding pixel in the image. The method according to C2, wherein the cost and the reward refer to the sum or product of the interface values ​​of all pixels necessary to move pixel by pixel from one side of the array corresponding to one side of the bounding box (32) to the other side of the array corresponding to the other side of the bounding box (32). [C4] The method according to any one of C1 to C3, wherein the bounding box data is determined by an object detection algorithm configured to detect vessels in the image (20), or by instance segmentation. [C5] After receiving the aforementioned image data, and before determining the aforementioned ship-water interface data, The ship-water value of the pixel in the image (20) is determined from the image data, and the ship-water value represents the probability that the corresponding pixel indicates the target ship (34), the water surface (22), or neither. Determining the interface value from the aforementioned ship-water value, The method described in any one of C1 to C4, further comprising: [C6] After receiving the aforementioned image data, and before determining the aforementioned ship-water interface data, The edge value of the pixel in the image (20) is determined, and the edge value indicates whether the corresponding pixel represents an edge in the image (20). The ship-water interface data is determined according to the edge value, The method described in any one of C1 to C5, further comprising: [C7] The aforementioned ship-water interface data is obtained according to the edge value, By multiplying the interface value by the corresponding edge value on a pixel-by-pixel basis, and determining the ship-water interface data according to the corresponding product, or By adding the interface value to the corresponding edge value on a pixel-by-pixel basis, and determining the ship-water interface data according to the corresponding sum, The method used to determine C6. [C8] All pixels whose corresponding interface value falls below a predetermined threshold are determined not to represent the ship-water interface (40). The method described in any one of C1 to C7, further comprising: [C9] The interface value and / or the ship-water value are determined by a neural network. The method described in any one of items C1 to C8. [C10] The ship-water interface data includes the position and / or coordinates of the pixels indicating the ship-water interface (40) in the image (20). The method described in any one of items C1 to C9. [C11] When the image (20) is displayed on the display, the image data is modified according to the determined ship-water interface data so that the ship-water interface (40) is shown in the image (20). The method described in any one of C1 to C10, further comprising: [C12] A method for determining the positional relationship between the vessel itself and a target vessel (34) in the water area, wherein the vessel itself is equipped with a camera for capturing images (20) of the area around the vessel, and the method is Determine at least one ship-water interface (40) in the image (20) according to any one of the items C1 to C10, The camera determines position data representing the camera position (28) and orientation of the camera when the camera captures the image (20), From the image data, determine the pixel data of at least one image point (46) on the ship-water interface (40) in the image (20), Determining the real-world coordinates of at least one real-world point (48) of the ship-water interface (40) on the water body, The positional relationship between the vessel and the target vessel (34) is determined based on the positional data and the real-world coordinates of at least one real-world point (48) on the vessel-water interface (40) on the water surface (22), A method for providing this. [C13] The method according to C12, wherein the real-world coordinates of the real-world point (48) are determined by extrapolating the camera position (28) of the camera to the water surface (22) of the body through the image point (46), according to the position data and the pixel data. [C14] The positional relationship refers to the range (50) between the self-vessel and the target vessel (34), and / or the bearing (AB, RB) of the target vessel (34). The method described in C12 or C13. [C15] The pixel data of the at least one image point (46) is determined such that the image point (46) corresponds to the real-world point (48) of the ship-water interface (40) closest to the ship. The method described in any one of items C12-C14. [C16] A system for determining the positional relationship between the vessel itself and a target vessel (34) within the water area, A camera for capturing images (20) of the surroundings of the vessel, the camera being positioned on the vessel, A processing unit coupled to the camera and configured to perform the method described in any one of C12 to C15, A system equipped with these features.

Claims

1. A method for determining a ship-to-water interface (40) between at least one target vessel (34) and the water surface (22) of a body of water in which the target vessel (34) is navigating, from an image (20) showing the target vessel (34) in the body of water, Receiving the image data of the aforementioned image (20), The interface value for each pixel in the image (20) is determined, and the interface value represents the probability that the corresponding pixel indicates the ship-water interface (40). The boundary box data of the boundary box (32) surrounding the target vessel (34) in the image (20) is determined from the image data, Based on the determined interface value and the boundary box data, ship-water interface data is determined, and the ship-water interface data represents the position and extent of the ship-water interface (40) in the image (20). A method for providing this.

2. The method according to claim 1, wherein the interface value is interpreted as cost or reward, and the ship-water interface data is determined by minimizing the cost or maximizing the reward, respectively.

3. The interface values ​​form an array of interface values, and the position of the interface value in the array corresponds to the position of the corresponding pixel in the image. The method according to claim 2, wherein the cost and the reward refer to the sum or product of the interface values ​​of all pixels necessary to move pixel by pixel from one side of the array corresponding to one side of the bounding box (32) to the other side of the array corresponding to the other side of the bounding box (32).

4. The method according to any one of claims 1 to 3, wherein the bounding box data is determined by an object detection algorithm configured to detect vessels in an image (20), or by instance segmentation.

5. After receiving the aforementioned image data, and before determining the ship-water interface data, The ship-water value of the pixel in the image (20) is determined from the image data, and the ship-water value represents the probability that the corresponding pixel indicates the target ship (34), the water surface (22), or neither. Determining the interface value from the aforementioned ship-water value, The method according to claim 1, further comprising:

6. After receiving the aforementioned image data, and before determining the ship-water interface data, The edge value of the pixel in the image (20) is determined, and the edge value indicates whether the corresponding pixel represents an edge in the image (20). The ship-water interface data is determined according to the edge value, The method according to claim 1, further comprising:

7. The aforementioned ship-water interface data is obtained according to the edge value, By multiplying the interface value by the corresponding edge value on a pixel-by-pixel basis, and determining the ship-water interface data according to the corresponding product, or By adding the interface value to the corresponding edge value on a pixel-by-pixel basis, and determining the ship-water interface data according to the corresponding sum, The method according to the determined claim 6.

8. All pixels whose corresponding interface value falls below a predetermined threshold are determined not to represent the ship-water interface (40). The method according to claim 1, further comprising:

9. The interface value and / or the ship-water value are determined by a neural network. The method according to claim 1.

10. The ship-water interface data includes the position and / or coordinates of the pixels indicating the ship-water interface (40) in the image (20). The method according to claim 1.

11. When the image (20) is displayed on the display, the image data is modified according to the determined ship-water interface data so that the ship-water interface (40) is shown in the image (20). The method according to claim 1, further comprising:

12. A method for determining the positional relationship between the vessel itself and a target vessel (34) in the water area, wherein the vessel itself is equipped with a camera for capturing images (20) of the area around the vessel, and the method is In accordance with claim 1, to determine at least one ship-to-water interface (40) in the image (20), The camera determines position data representing the camera position (28) and orientation of the camera when the camera captures the image (20), From the image data, determine the pixel data of at least one image point (46) on the ship-water interface (40) in the image (20), Determining the real-world coordinates of at least one real-world point (48) of the ship-water interface (40) in the water body, The positional relationship between the vessel and the target vessel (34) is determined based on the positional data and the real-world coordinates of at least one real-world point (48) on the vessel-water interface (40) on the water surface (22). A method for providing this.

13. The method according to claim 12, wherein the real-world coordinates of the real-world point (48) are determined by extrapolating the camera position (28) of the camera to the water surface (22) of the water body through the image point (46), according to the position data and the pixel data.

14. The positional relationship refers to the range (50) between the self-vessel and the target vessel (34), and / or the bearing (AB, RB) of the target vessel (34). The method according to claim 12 or 13.

15. The pixel data of at least one image point (46) is determined such that the image point (46) corresponds to the real-world point (48) of the ship-water interface (40) closest to the ship. The method according to claim 12.

16. A system for determining the positional relationship between the vessel itself and a target vessel (34) within the aforementioned water area, A camera for capturing images (20) of the surroundings of the vessel, the camera being positioned on the vessel, A processing unit coupled to the camera and configured to perform the method according to claim 12, A system equipped with these features.

Images