Method and system for determining the motion of one's own vessel, and method for determining a map of ocean navigation.

Abstract

A method and system for determining the motion of a vessel (18) are disclosed. The method comprises receiving radar data of radar measurements around the vessel (18), receiving three-dimensional object data representing one or more three-dimensional objects within the vicinity of the vessel (18), determining dynamic object data from the three-dimensional object data, wherein the dynamic object data represents one or more dynamic objects among the three-dimensional objects encoded in the three-dimensional object data, and determining the motion of the vessel (18) depending on the radar data and the dynamic object data.

Description

Technical Field

[0001] The present invention relates to the field of marine navigation. In particular, the present invention relates to a method and system for determining the motion of a vessel, and a method for determining a marine navigation map.

Background Art

[0002] Modern marine navigation heavily relies on the Global Navigation Satellite System (GNSS). Several activities, including docking, coastal navigation, and open ocean navigation, rely on important information from GNSS signals. Since GNSS signals reach the Earth with a very small amount of electromagnetic power, they are vulnerable to both intentional and unintentional jamming. Unfortunately, GNSS jamming is becoming increasingly common.

[0003] There are already several systems that employ radar for positioning and motion determination. In fact, radar can provide detailed information about the surroundings of a vessel. For example, a radar-based positioning method can, as described in, for example, US9581695B2, enable the construction of a map of the environment of the vessel using the measured radar returns. The construction of the map may or may not be facilitated by the availability of GNSS.

[0004] Such maps may be stored on the vessel and used later for determining its own position. In this positioning procedure, the vessel's actual position may not be known, and current radar measurements may be compared to the map. From this comparison, it may be possible to establish the vessel's current location, for example, as described in “Robust naval localization using a particle filter on polar amplitude gridmaps”, H. Schiller, S. Marano, D. Maas, B. Arsenali, AJ Isaksson, and F. Gustafsson, 2021 IEEE 24th International Conference on Information Fusion (FUSION), Sun City, South Africa, 2021, pp. 1-8, doi: 10.23919 / FUSION49465.2021.9627025. In particular, this reference describes how GNSS may be used to construct radar-based maps. In this case, if GNSS is not available, radar is used to find the ship's position on the map.

[0005] As another example, in radar-based odometry methods, consecutive radar measurements can be compared to one another. From such comparisons, it may be possible to extract information about the displacement and / or motion of the vessel, as described, for example, in “Improving Marine Radar Odometry by Modeling Radar Resolution and Exploiting Additional Temporal Information”, H. Schiller, B. Arsenali, D. Maas and S. Marano (with an acute accent on the 'o'), 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022, pp. 8436-8441, doi: 10.1109 / IROS47612.2022.9981293 (this reference focuses on ships and describes the estimation of their motion, speed, and rotational speed), or in “Precise Ego-Motion Estimation with Millimeter-Wave Radar Under Diverse and Challenging Conditions”, SH Cen and P. Newman, 2018 IEEE International Conference on This is described in Robotics and Automation (ICRA), Brisbane, QLD, Australia, 2018, pp. 6045-6052, doi: 10.1109 / ICRA.2018.8460687 (this reference describes radar-based speed estimation in automotive applications), or in “An EKF Based Approach to Radar Inertial Odometry”, C. Doer and GF.This is described in Trommer, 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), Karlsruhe, Germany, 2020, pp. 152-159, doi: 10.1109 / MFI49285.2020.9235254 (this reference describes how radar and IMU are used to estimate the position and speed of a vessel). In fact, assuming the surrounding environment is static and does not change over time, radar measurements collected while the vessel is moving reveal the relative motion of the vessel to the background.

[0006] However, in a marine environment, there are several radar targets whose positions are likely to change over time, and one technical challenge is the presence of these potentially dynamic or actually dynamic objects. For example, dock areas look very different to radar with or without containers, which can gradually impair radar-based positioning systems. Alternatively or additionally, the passage of another vessel near one's own vessel can render radar-based odometry systems inoperable. Alternatively or additionally, the presence or absence of other vessels in a port can affect radar measurements. Furthermore, trucks and other vehicles, and / or other entities including containers, may also be considered dynamic objects.

[0007] The presence of these dynamic objects can degrade the performance of radar-based map construction, radar-based positioning of the vessel, and radar-based odometry related to the vessel. For example, a positioning algorithm that assumes everything in the surroundings is static will be unable to accurately determine the vessel's current position if the surroundings change after the map used for positioning has been generated. For instance, if a relatively large vessel leaves the dock after the radar measurements for which the map was constructed have been taken, it will be impossible to match the map with the current radar measurements. Similarly, an odometry algorithm that assumes radar returns are generated by a static background will be unable to correctly determine the vessel's motion if one or more dynamic objects generate radar returns that are incompatible with the static background.

[0008] These considerations motivate the development of alternative odometry and / or positioning solutions that do not rely on GNSS or external infrastructure at all. [Overview of the Initiative]

[0009] The object of the present invention is to provide a method for determining the motion of a ship that is highly accurate and / or cost-effective, can be carried out in particular automatically, and / or does not rely on GNSS and / or external infrastructure at all.

[0010] Another object of the present invention is to provide a system for determining the motion of a ship that is highly accurate and / or cost-effective, can be implemented in particular automatically, and / or does not rely on GNSS and / or external infrastructure at all.

[0011] The object of the present invention is to provide a method for determining a map of ocean navigation that contributes to highly accurate maps and / or is cost-effective, can be implemented in particular automatically, and / or does not rely on GNSS and / or does not rely on external infrastructure at all.

[0012] These objectives are achieved by the subject matter of the independent claims. Further exemplary embodiments are evident from the dependent claims and the following description.

[0013] The objective is achieved by a method for determining the motion of the vessel. This method comprises receiving radar data from radar measurements around the vessel, receiving solid object data representing one or more solid objects within the vessel's vicinity, determining dynamic object data from the solid object data, where the dynamic object data represents one or more dynamic objects among the solid objects encoded in the solid object data, and determining the motion of the vessel depending on the radar data and the dynamic object data.

[0014] Determining a ship's motion based on radar data and 3D object data can potentially be done cost-effectively and / or automatically, and / or without relying on GNSS and / or any other external infrastructure. Determining a ship's motion based on radar data and dynamic object data can contribute to determining the ship's motion with great accuracy. Determining a ship's motion based on radar data and 3D object data may fall under the field of odometry.

[0015] Radar data may be generated by a radar device and sent from the radar device to an entity performing the above method, which can receive the radar data. The radar device may be located on the ship. The entity may be, for example, a system for determining the ship's motion, as described below. The area around the ship can refer to an angular range around the ship, which may be less than or equal to 360°, and the outreach of the area may be limited by the outreach of the corresponding radar device. Object data may be generated by another device, such as a camera or AIS unit, as described below, and sent from this other device to an entity performing the above method, which can receive the object data. The object may be one or more other ships, land, such as a dock, a pier, and / or a coastline.

[0016] According to one embodiment, the 3D object data is image data of one or more images captured by a camera placed on the ship, where one or more images represent one or more 3D objects, and the dynamic object data is determined from the 3D object data by detecting moving 3D objects in one or more images, determining the moving 3D objects as dynamic objects, and encoding these dynamic objects into dynamic object data.

[0017] Image data may be generated by a camera and received by an entity performing the above method. The camera may be at least one of a group of cameras, the group comprising a single or monocular camera, a pair of cameras or a stereo camera, a time-of-flight camera, a gate camera, a pan-tilt-zoom camera, and a structured light camera, and the camera, except for the single or monocular camera, may be able to estimate the depth of objects shown in the image relative to the camera. Moving three-dimensional objects in one or more images may be detected by object detection known in the art. For example, moving three-dimensional objects in one or more images may be detected by semantic segmentation of images, object detection, instance segmentation, or panoptic segmentation. Determining moving three-dimensional objects from one or more images in this way may be performed by a neural network which may be trained to classify all moving objects in an image as dynamic objects. An exemplary technique for using semantic segmentation to determine which parts of an image are static and which are dynamic is described in “DS-SLAM: A Semantic Visual SLAM towards Dynamic”, 2018 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 2018, pp. 1168-1174, doi: 10.1109 / IROS.2018.8593691. Alternatively, moving objects may be detected, for example, by object detection performed on two or more images subsequently captured by the same camera, and by comparing the positions of detected objects in different images. Assuming that the ship, in particular the camera, is not moving, if the position of a detected object changes across images, it is clear that the corresponding object is moving and therefore a dynamic object.

[0018] Furthermore, if it is known which three-dimensional objects in the image are moving, in other words, if the dynamic three-dimensional objects in the image are known, the position of the moving objects relative to the ship can be determined from one or more images in order to identify the radar data that refers to the determined dynamic three-dimensional objects. For example, if the position and orientation of the camera on the ship are known, for example in the form of known calibration values ​​of the camera, the relative position of the dynamic object and the ship can be derived from one or more images, for example, as described on pages 22 onwards, step S4 onwards, and / or Figure 8 onwards of the unpublished patent application EP22180131.9. Once the relative position of the dynamic three-dimensional object and the ship is known, the radar data that refers to these dynamic objects can be easily determined. This is because radar measurements by the radar device on the ship also provide the relative position of the corresponding radar reflector, and the radar data of the radar reflector that matches the position of the dynamic object can be determined as the dynamic three-dimensional object data.

[0019] According to one embodiment, one or more three-dimensional objects are correspondingly one or more other vessels, particularly three-dimensional objects shown in one or more images, and one or more dynamic objects are one or more other vessels, and these other vessels are moving. Radar data pointing to other vessels that are not moving may also be tagged, for example, as "potentially" moving.

[0020] Dynamic object data can be a subset of radar data. Dynamic objects can be encoded into dynamic object data by tagging radar data that points to the dynamic object as dynamic object data. Alternatively, dynamic object data may be removed from the radar data so that the radar data represents only three-dimensional objects that are not moving. Alternatively, dynamic object data may be extracted from the radar data. In that case, the dynamic object data may be stored as a separate data file.

[0021] According to one embodiment, one or more three-dimensional objects correspond to one or more other vessels, the three-dimensional object data is AIS data, and the dynamic object data is determined from the three-dimensional object data by identifying which of the other vessels encoded in AIS data that are present around the vessel are moving, determining the other moving vessels as dynamic objects, and encoding these dynamic objects into dynamic object data.

[0022] Automatic Identification System (AIS) data may be generated by other vessels and / or broadcast by Vessel Traffic Service (VTS). AIS data may contain vessel-related information about one or more other vessels in the vicinity of the vessel, for example, those near the vessel. In particular, AIS data may relate to at least one other vessel in the same water mass, which may be one of three-dimensional objects. The water mass may be a lake, ocean, sea, or river. Vessel-related information may include the unique identifier, geographical location, course, and speed of the other vessel. Furthermore, AIS data may contain extent of the geographical location of the other vessel's GPS receiver. This makes it possible to take into account the extent of the geographical location of the other vessel's GPS receiver when determining dynamic object data. Thus, from AIS data, it may be possible to derive which other vessels are moving around the vessel and where these moving vessels are located. The positions of these dynamic objects, i.e., other moving vessels, can be matched to radar data so that radar data pointing to these dynamic objects can be recognized as such and taken into consideration when determining the motion of one's own vessel.

[0023] According to one embodiment, the motion of the vessel refers to its motion relative to the land (over ground). In this case, the motion of the vessel may be called the absolute motion of the vessel.

[0024] According to one embodiment, determining the motion of a ship, particularly its absolute motion, based on radar data and dynamic object data comprises determining data from radar data corresponding to dynamic object data, and determining the motion of the ship based on radar data excluding dynamic object data. For example, dynamic object data may be removed from the radar data, and the motion of the ship may be determined based on the remaining radar data. More precisely, dynamic objects may be removed from radar measurements, particularly from the radar data, so that the radar data can refer only to stationary three-dimensional objects around the ship. Alternatively, dynamic object data may remain in the radar data, may be tagged as dynamic object data, and may not be considered and / or ignored when determining the motion of the ship. Thus, for example, as is known in the art of odometry, and as described in the cited references of H. Schiller, SH Cen, and / or C. Doer above, the (absolute) motion of the ship may be determined only with respect to stationary objects around the ship. This contributes to the fact that the ship's motion can be determined with great precision.

[0025] According to one embodiment, radar data points to an angular range around the vessel, a circular sector of the angular range in which the corresponding dynamic object is located is determined, and data from the radar data pointing to the determined sector is determined as dynamic object data. The angular range may be 360° or less around the vessel. The sector may be determined from AIS data or image data. Therefore, all radar data pointing to the sector in which the corresponding dynamic object exists can be determined as dynamic object data. Thus, radar data pointing to any object between the corresponding dynamic object and the vessel can also be determined as dynamic object data, regardless of whether these objects are moving or not. The dynamic object data may then be tagged as such within the radar data or removed from the radar data. In either case, the motion of the vessel, in particular its absolute motion, can be determined from the radar data excluding the dynamic object data.

[0026] According to one embodiment, the radar data refers to the angular range around the own ship, and the method further comprises determining distance data representing the distance between the own ship and one of the moving objects, and determining, from the distance data, the circular segment in which the corresponding moving object within the angular range is located, and the data from the radar data indicating the circular segment is determined as moving object data. The circular segment can additionally be determined from AIS data or image data. The distance data can be determined from AIS data or image data. For example, cameras and / or camera systems that provide depth information of objects captured by corresponding cameras are known. In this context, the depth can correspond to the distance from the camera to the corresponding solid object. In this case, only the radar data indicating the circular segment in which the corresponding moving object exists can be determined as moving object data. Therefore, the radar data indicating any non-moving object between the corresponding moving object and the own ship does not have to be determined as moving object data. Then, the moving object data may be tagged as such within the radar data or removed from the radar data. In either case, the movement of the own ship, particularly the absolute movement, can be determined from the radar data excluding the moving object data.

[0027] According to one embodiment, the method comprises determining the position of at least one of the moving objects relative to the own ship, and determining the extension of the corresponding moving object, and the data from the radar data indicating the position and extension of the corresponding moving object is considered as moving object data. This contributes to the fact that the moving object data can be determined very accurately, and thereby the movement of the own ship, particularly the absolute movement, can be determined very accurately. The position and / or extension of the moving object can be determined from AIS data or image data.

[0028] According to one embodiment, the movement of the own ship refers to the movement of the own ship relative to one of the moving objects. In this case, the movement of the own ship can be called the relative movement of the own ship.

[0029] According to one embodiment, determining the movement of the own ship, particularly the relative movement, depending on radar data and dynamic object data includes determining all data from the radar data that does not correspond to the dynamic object data of the corresponding dynamic object, and determining the movement of the own ship depending only on the dynamic object data. For example, all radar data except the dynamic object data can be removed from the radar data. Specifically, all data except the data of the corresponding dynamic object may be removed from the radar measurements, particularly from the radar data, so that the radar data can point only to the corresponding dynamic object. Alternatively, all of the radar data may remain within the radar data, and the dynamic object data may be tagged as dynamic object data within the radar data. Alternatively, the dynamic object data may be extracted from the radar data and stored as a separate data file. In any case, only the dynamic object data may be considered when determining the movement of the own ship, particularly the relative movement. The remaining radar data that only refers to stationary objects does not have to be considered for determining the relative movement between the own ship and another ship.

[0030] According to one embodiment, the radar data refers to an angular range around the own ship, a sector of the angular range in which the corresponding dynamic object is located is determined, and only the data from the radar data that refers to the determined sector is considered as dynamic object data. The sector can be determined from AIS data or image data. Thus, all of the radar data that refers to the sector in which the corresponding dynamic object exists can be determined as dynamic object data. Therefore, the radar data that refers to any object between the corresponding dynamic object and the own ship can also be determined as dynamic object data regardless of whether these objects are moving. Then, the dynamic object data may be tagged as such within the radar data, and all other data except the dynamic object data may be removed from the radar data, or the dynamic object data may be extracted from the radar data. In any case, the movement of the own ship, particularly the relative movement, can be determined only from the dynamic object data.

[0031] According to one embodiment, the radar data refers to an angular range around the vessel, and the method comprises determining distance data representing the distance between the vessel and one of the dynamic objects, and determining from the distance data the arc shape in which the corresponding dynamic object within the angular range is located, wherein only data from the radar data pointing to the determined arc shape is considered as dynamic object data. The arc shape may additionally be determined from AIS data or image data. The distance data may be determined from AIS data or image data. Therefore, only radar data pointing to the arc shape in which the corresponding dynamic object exists can be determined as dynamic object data. Thus, radar data pointing to any stationary object between the corresponding dynamic object and the vessel does not need to be determined as dynamic object data. The dynamic object data may then be tagged as such within the radar data, and all other data except the dynamic object data may be removed from the radar data, or the dynamic object data may be extracted from the radar data. In either case, the motion of the vessel, in particular relative motion, can be determined solely from the dynamic object data.

[0032] According to one embodiment, the method comprises determining the position of at least one of the dynamic objects relative to the vessel and determining the extension portion of the corresponding dynamic object, wherein only data from radar data pointing to the position and extension portion of the corresponding dynamic object is considered as dynamic object data. This contributes to the fact that the dynamic object data can be determined with great accuracy, thereby allowing the motion of the vessel, such as relative motion, to be determined with great accuracy. The position and / or extension portion of the dynamic object can be determined from AIS data or image data.

[0033] The objective is achieved by a system for determining the motion of the vessel. This system comprises a memory configured to store radar data, image data, 3D object data, and / or dynamic object data, and a processor configured to perform the methods described above and / or below. Furthermore, this system may include a camera, or be part of a camera. It should be understood that the features of the methods for determining the motion of the vessel described above and below may also be features of the system for determining the motion of the vessel described above and below.

[0034] The objective is achieved by a method for determining a map of ocean navigation. This method can be considered a second aspect of the present invention. This method comprises receiving radar data of radar measurements around the vessel and receiving image data of one or more images captured by cameras positioned on the vessel, wherein one or more images show one or more three-dimensional objects around the vessel, and determining dynamic object data from the image data, wherein the dynamic object data represents one or more dynamic objects among the three-dimensional objects shown in the images, and determining a map depending on the radar data and the dynamic object data. It should be understood that the features of the method and / or system for determining the motion of the vessel described above and below may be features of the method for determining a map of ocean navigation described above and below.

[0035] According to one embodiment of the second aspect, dynamic object data is determined from image data by detecting moving three-dimensional objects in one or more images, determining the moving three-dimensional objects as dynamic objects, and encoding these dynamic objects into dynamic object data.

[0036] According to one embodiment of the second aspect, one or more three-dimensional objects are correspondingly one or more other vessels, and one or more dynamic objects are one or more of the other vessels, and these other vessels are moving.

[0037] According to one embodiment of the second aspect, determining a map based on radar data and dynamic object data may comprise determining data from radar data corresponding to dynamic object data and determining a map based on radar data excluding dynamic object data. For example, data pointing to dynamic objects may be removed from radar measurements, in particular from radar data, so that the radar data can point only to stationary objects around the vessel. Alternatively, data pointing to dynamic objects, i.e., dynamic object data, may be tagged as such within the radar data, and the map may be generated based solely on the remaining untagged radar data, as described, for example, in US9581695B2 above. Optionally, radar data pointing to objects that are not currently moving but can or will be moved, such as another vessel or container in a dock, may be tagged as "potentially" moving within the radar data. In this case, radar data tagged as pointing to potentially moving objects may be excluded when generating the map, or may be considered as potentially having moved since the last radar measurement when locating the vessel with the help of the corresponding map.

[0038] According to one embodiment of the second aspect, radar data is used to determine an angular range around the ship, a sector of the angular range in which the corresponding dynamic object is located, and data from the radar data pointing to the sector is determined as dynamic object data.

[0039] According to one embodiment of the second aspect, the radar data refers to an angular range around the vessel, the method comprises determining distance data representing the distance between the vessel and one of the dynamic objects, and determining from the distance data the arc shape in which the corresponding dynamic object within the angular range is located, wherein the data from the radar data referring to the arc shape is considered as dynamic object data. The arc shape may additionally be determined from image data or AIS data. The distance data may be determined from image data or AIS data.

[0040] According to one embodiment of the second aspect, the method comprises determining the position of at least one of the dynamic objects relative to the vessel and determining the extension portion of the corresponding dynamic object, wherein data from radar data indicating the position and extension portion of the corresponding dynamic object is considered as dynamic object data.

[0041] These and other aspects of the present invention will become apparent from the embodiments described below and will be made apparent by reference to those embodiments.

[0042] The subject matter of the present invention is described in detail below with reference to exemplary embodiments illustrated in the accompanying drawings. [Brief explanation of the drawing]

[0043] [Figure 1] An example of a map for ocean navigation is shown. [Figure 2] An example of an image showing the area around the ship is shown. [Figure 3] A flowchart illustrating an exemplary embodiment of a method for determining the motion of one's own vessel is shown. [Figure 4] A flowchart illustrating an exemplary embodiment of a method for determining dynamic object data from three-dimensional object data is shown. [Figure 5] A flowchart illustrating an exemplary embodiment of a method for determining dynamic object data from three-dimensional object data is shown. [Figure 6] A flowchart illustrating an exemplary embodiment of a method for determining a map for ocean navigation is shown. [Modes for carrying out the invention]

[0044] The reference symbols used in the drawings and their meanings are listed in a reference symbol list. As a general rule, the same reference symbol is used for the same part in the drawing.

[0045] Figure 1 shows an example of a map 20 for ocean navigation. Map 20 shows the vessel 18 on the water surface 22 of a water mass, land 24, and several other vessels 26 around the vessel 18. In particular, Figure 1 shows the position and extent of the vessel 18 and the other vessels 26, especially their length and width. The water mass may be a lake, ocean, sea, or river. Land 24 may include a pier, dock, harbor, one or more islands, and / or any other part of the mainland.

[0046] Furthermore, Figure 1 shows several sectors 30 within an angular range around the vessel 18. The angular range may be 360° in this example. Each sector 30 comprises one of the other vessels 26. In fact, the sectors 30 are selected such that one of the sectors 30 is determined for each other vessel 26 around the vessel 18, and as a result, the corresponding other vessel 26 is, for example, entirely and / or strictly within the corresponding sector 30.

[0047] A certain distance 34 extends from the own ship 18 to one of the other ships 26, and is shown as a dashed line in Figure 1. Another line 36, in particular the dotted line in Figure 1, is perpendicular to the radius of the angular range around distance 34 and / or the own ship 18. Line 36 separates the arc 32 from the rest of the corresponding sector 30. Line 36 and the corresponding arc 32 are defined such that the other ships 26 in the corresponding sector 30 are located within the arc 32. The corresponding distance, line, and / or arc may be constructed for each of the other sectors 30 and the corresponding other ships 26.

[0048] Figure 1 further shows several radar reflectors 28 from radar measurements. The radar reflectors 28 are present across the entire angular range around the ship 18, except for the sector 30 and the arc 32. Radar measurements may be performed by a radar system located on the ship 18.

[0049] Figure 2 shows an example of an image 40 showing at least a portion of the area around the ship 18. Image 40 may show at least one imaged three-dimensional object. For example, the imaged three-dimensional object could be an imaged other ship 46 or an imaged land area 44. In particular, image 40 may show an imaged other ship 46 that can be seen from the ship 18, can also move through the water mass, and may correspond to one of the other ships 26 shown in Figure 1. In addition, image 40 may further show an imaged land area 44, for example, two sections of imaged land, which may correspond to a portion of the land area 24 shown in Figure 1. Furthermore, the water mass may be represented in image 40 by an imaged water surface 42 of the water mass that corresponds to the water surface 22 of the water mass in the real world.

[0050] The imaged water surface 42 is marked with several dashed hatches. The imaged other vessels 46 are marked with several dotted hatches. The markings of the imaged water surface 42 and the imaged other vessels 46 may correspond to a visualization of the output of an algorithm for determining the water surface and vessels in the image. The algorithm may constitute a neural network. The neural network may be pre-trained to detect the water surface and vessels in the image, as is known in the field of object detection. Training may be carried out with a certain amount of labeled images showing the water surface and vessels. The markings of image 40 may be generated by semantic segmentation, object detection, instance segmentation, or panoptic segmentation of image 40 based on the corresponding image data, as described below with respect to Figure 3.

[0051] Figure 3 shows a flowchart of an exemplary embodiment of a method for determining the motion of the vessel 18. This method may be carried out by a system for determining the motion of the vessel 18, as described below. The system may be at least partially installed on the vessel 18.

[0052] In step S2, radar data of radar measurements around the ship 18 may be received. The radar data may be generated by the ship 18's radar device and sent from the radar device to a system for determining the ship 18's motion, which may receive the radar data. The area around the ship 18 can refer to an angular range around the ship 18, as shown in Figure 1, for example, and the angular range may be less than or equal to 360°, and the reach of the area may be limited by the reach of the corresponding radar device. The radar data represents radar reflectors 28, thereby three-dimensional objects around the ship 18, and in particular, the relative positions of these objects with respect to the ship 18. In particular, the radar data may represent the direction in which the radar reflection was received relative to the ship 18 and the distance from the corresponding radar reflector to the ship 18. For example, the radar data may include the coordinates of the radar reflector in the ship coordinate system of the ship 18.

[0053] In step S4, object data representing one or more three-dimensional objects within the vicinity of the vessel 18 may be received. The object data may be generated by another device, such as a camera (not shown) or an AIS unit (not shown), as described below, and may be sent from this other device to the system, which may receive the object data. The three-dimensional objects may be one or more other vessels 26, land 24, such as docks, piers, and / or coastlines.

[0054] Camera 24 may be configured to capture one or more images 40 around the vessel 18 and generate corresponding image data. For example, camera 24 may be configured to generate a video stream comprising several subsequent images 40 around the vessel 18. The corresponding image data may be generated by the camera and received by the system. The camera may be at least one of a group of cameras, the group comprising a single or monocular camera, a pair of cameras or a stereo camera, a time-of-flight camera, a gate camera, a pan-tilt-zoom camera, and a structured light camera, and the camera, except for the single or monocular camera, may be able to estimate the depth of an object shown in the image relative to the camera.

[0055] In step S6, the dynamic object data may be determined from the 3D object data, and the dynamic object data may represent one or more dynamic objects from the 3D objects encoded in the 3D object data. For example, one or more 3D objects may correspond to one or more of the other ships 26. In this case, one or more dynamic objects may be one or more of the other ships 26, and these other ships 26 are moving.

[0056] In one embodiment, the 3D object data may be image data of one or more images 40 captured by a camera positioned on the vessel 18, where one or more images 40 show one or more 3D objects, for example, other vessels 46 that have been imaged, and the dynamic object data may be determined according to the method described below with respect to Figure 4.

[0057] In another embodiment, one or more of the three-dimensional objects may correspond to one or more of the other vessels 26, the three-dimensional object data may be Automatic Identification System (AIS) data, and the dynamic object data may be determined from the three-dimensional object data by the method described below with respect to Figure 5. The AIS data may be generated by the other vessels 26 and / or broadcast by the Vessel Traffic Service (VTS). The AIS data may contain vessel-related information about one or more of the other vessels 26 in the vicinity of the vessel 18, for example, near the vessel 18. In particular, the AIS data may be related to at least one of the other vessels 26 in the same water mass, which may be one of the three-dimensional objects. The vessel-related information may include a unique identifier, geographical location, course over the ground, and speed over the ground of the corresponding other vessel 26. Furthermore, the AIS data may contain the size of the geographical location of the corresponding other vessel 26's GPS receiver. This makes it possible to take into account the size of the geographical location of the corresponding other vessel 26's GPS receiver when determining the dynamic object data.

[0058] In further embodiments, the methods of the latter two embodiments, namely using image data as 3D object data and using AIS data as 3D object data, can be combined to, for example, verify each other and / or provide further details about the corresponding dynamic objects.

[0059] In step S8, the motion of the vessel 18 may be determined depending on radar data and dynamic object data. Dynamic object data may be a subset of the radar data. With regard to determining the motion of the vessel 18, it is necessary to distinguish between determining the motion of the vessel 18 relative to land, i.e., absolute motion, and determining the motion of the vessel 18 relative to one of the other vessels 26, as will be explained below.

[0060] [Absolute Movement] Generally, a common method for estimating the motion of a moving vessel 18 can be described in roughly three steps: 1) estimating landmarks or other key points from at least two radar measurements, 2) matching or relating these landmarks between the two radar measurements, and 3) estimating the motion from the matched landmarks, as described, for example, in “Improving Marine Radar Odometry by Modeling Radar Resolution and Exploiting Additional Temporal Information”, CH Schiller, B. Arsenali, D. Maas and S. Marano (o with acute accent), 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022, pp. 8436-8441, doi: 10.1109 / IROS47612.2022.9981293, or “Precise Ego-Motion Estimation with Millimeter-Wave Radar This is described in "Under Diverse and Challenging Conditions" by Cen and P. Newman, 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia, 2018, pp. 6045-6052, doi: 10.1109 / ICRA.2018.8460687.

[0061] The motion of the vessel may refer to the motion of the vessel 18 relative to land. In this case, the motion of the vessel 18 may be called the absolute motion of the vessel 18. Determining the absolute motion of the vessel 18 based on radar data and dynamic object data may involve determining data from radar data corresponding to dynamic object data and determining the motion of the vessel based on radar data excluding dynamic object data. For example, dynamic object data may be removed from the radar data, and the motion of the vessel may be determined based on the remaining radar data. To be clear, dynamic objects may be removed from radar measurements, in particular from radar data, and as a result, the radar data may refer only to stationary three-dimensional objects around the vessel 18, as visualized in Figure 1, for example, where no radar reflectors 28 are shown within a sector 30 containing other moving vessels 26. Alternatively, dynamic object data may remain in the radar data, may be tagged as dynamic object data, and may not be considered and / or ignored when determining the absolute motion of the vessel 18. Thus, for example, as is known in the art of odometry, the absolute motion of the vessel 18 may be determined only with respect to stationary objects around the vessel 18.

[0062] For example, if the relative positions of the vessel 18 and at least two stationary three-dimensional objects around the vessel 18 are determined in successive time steps, the corresponding changes in relative positions represent the absolute motion of the vessel. These relative positions may be derived from image data and / or AIS data. In the case of image data, the relative positions of the three-dimensional objects and the vessel 18 may be derived from one or more of the images 40, for example, as described on page 22 onwards, step S4 onwards, and / or Figure 8 onwards of the unpublished patent application EP22180131.9.

[0063] The radar data may point to an angular range around the vessel 18. A sector 30 of the angular range in which the corresponding dynamic object is located may be determined, and data from the radar data pointing to the determined sector 30 may be determined as dynamic object data. The angular range may be 360° or less around the vessel. The sector 30 may be determined from AIS data or image data. Therefore, all radar data pointing to the sector 30 in which the corresponding dynamic object exists may be determined as dynamic object data. Thus, radar data pointing to any object between the corresponding dynamic object and the vessel 18 may also be determined as dynamic object data, regardless of whether these objects are moving or not. The dynamic object data may then be tagged as such within the radar data, or removed from the radar data, as described above. In either case, the absolute motion of the vessel 18 may be determined from the radar data excluding the dynamic object data.

[0064] Alternatively, distance data representing the distance 34 between the vessel 18 and one of the dynamic objects, for example, one of the other vessels 26, may be determined, and the arc shape 32 in which the corresponding dynamic object is located within an angular range may be determined from the distance data. Then, data from radar data pointing to the corresponding arc shape 32 may be determined as dynamic object data. The arc shape 32 may be determined from AIS data or image data. The distance data may be determined from AIS data or image data. Therefore, only radar data pointing to the arc shape 32 in which the corresponding dynamic object exists may be determined as dynamic object data. Thus, radar data pointing to any stationary object between the corresponding dynamic object and the vessel does not need to be determined as dynamic object data. Then, the dynamic object data may be tagged as such within the radar data, or it may be removed from the radar data. In either case, the absolute motion of the vessel 18 may be determined from the radar data excluding the dynamic object data.

[0065] Alternatively, the position of at least one of the dynamic objects relative to the vessel 18 and the corresponding extension of the dynamic object may be determined, and data from radar data pointing to the position and extension of the corresponding dynamic object may be considered as dynamic object data. The position and / or extension of the dynamic object may be determined from AIS data or image data. The dynamic object data may then be tagged as such in the radar data or removed from the radar data. In either case, the absolute motion of the vessel 18 may be determined from the radar data excluding the dynamic object data.

[0066] [Relative motion] The motion of the ship 18 may refer to the motion of the ship 18 with respect to one of the moving objects, for example, one of the other ships 26. In this case, the motion of the ship 18 may be called the relative motion of the ship 18.

[0067] Determining the relative motion of the vessel 18 based on radar data and dynamic object data may involve determining all data from the radar data that does not correspond to the dynamic object data of the corresponding dynamic object, or determining the motion of the vessel 18 based solely on the dynamic object data. For example, all radar data except for the dynamic object data may be removed from the radar data. More precisely, all data except for the data of the corresponding dynamic object may be removed from the radar measurements, in particular from the radar data, so that the radar data can point only to the corresponding dynamic object. Alternatively, all radar data may remain in the radar data, and the dynamic object data may be tagged as dynamic object data within the radar data. Alternatively, the dynamic object data may be extracted from the radar data and stored as a separate data file. In either case, when determining the relative motion of the vessel, only the dynamic object data may be considered.

[0068] For example, if the relative position between the vessel 18 and the dynamic object is determined in a series of time steps, the corresponding change in relative position represents the relative motion between the vessel 18 and the corresponding dynamic object. These relative positions can be derived from image data and / or AIS data. In the case of image data, the relative position between the three-dimensional object and the vessel 18 can be derived from one or more of the images 40, for example, as described on page 22 onwards, step S4 onwards, and / or Figure 8 onwards of the unpublished patent application EP22180131.9.

[0069] The radar data may point to an angular range around the ship 18, and a sector 30 of the angular range in which the corresponding dynamic object is located may be determined, and only data from the radar data pointing to the determined sector 30 may be considered as dynamic object data. The sector 30 may be determined from AIS data or image data. Therefore, all radar data pointing to the sector 30 in which the corresponding dynamic object exists may be determined as dynamic object data. Thus, radar data pointing to any three-dimensional object between the corresponding dynamic object and the ship 18 may also be determined as dynamic object data, regardless of whether these three-dimensional objects are moving or not. The dynamic object data may then be tagged as such within the radar data, and all other data except the dynamic object data may be removed from the radar data, or the dynamic object data may be extracted from the radar data. In either case, the relative motion of the ship 18 may be determined solely from the dynamic object data.

[0070] Alternatively, distance data representing the distance 34 between the vessel 18 and one of the dynamic objects may be determined, and the arc shape 32 in which the corresponding dynamic object is located within an angular range may be determined from the distance data. Then, only the data from radar data pointing to the determined arc shape 32 may be considered as dynamic object data. The arc shape 32 may be determined from AIS data or image data. The distance data may be determined from AIS data or image data. Therefore, only the radar data pointing to the arc shape 32 in which the corresponding dynamic object exists may be determined as dynamic object data. Thus, radar data pointing to any stationary object between the corresponding dynamic object and the vessel does not need to be determined as dynamic object data. Then, the dynamic object data may be tagged as such within the radar data, and all other data except the dynamic object data may be removed from the radar data, or the dynamic object data may be extracted from the radar data. In either case, the relative motion of the vessel 18 may be determined solely from the dynamic object data.

[0071] Alternatively, the position of at least one of the dynamic objects relative to the ship 18 and the corresponding extension of the dynamic object may be determined. Then, only the data from the radar data pointing to the position and extension of the corresponding dynamic object is considered as dynamic object data. The position and / or extension of the dynamic object may be determined from AIS data or image data. The determined dynamic object may be encoded into dynamic object data by tagging the radar data pointing to the dynamic object as dynamic object data. Alternatively, the dynamic object data may be extracted from the radar data and stored as a separate data file. Alternatively, all data other than the dynamic object data may be removed from the radar data so that the radar data represents only the dynamic object data. In any case, the relative motion of the ship 18 may be determined from the dynamic object data alone.

[0072] Figure 4 shows a flowchart of an exemplary embodiment of a method for determining dynamic object data from three-dimensional object data. This method may be performed when carrying out step S6 of the method described above with respect to Figure 3.

[0073] In step S12, dynamic object data may be determined from 3D object data by detecting moving 3D objects in one or more of the images 40. Moving 3D objects in one or more of the images 40 may be detected as known in the art. For example, moving 3D objects in one or more of the images 40 may be detected by semantic segmentation, object detection, instance segmentation, or panoptic segmentation of the images 40 using a correspondingly trained neural network. For example, when using a known object detection algorithm, the corresponding algorithm may provide information about the detected object and / or the class to which the corresponding object belongs. Classes that can be recognized by the algorithm may include ships, trucks, and / or containers. The neural network may be trained to classify all moving objects in the images 40 as dynamic objects. An exemplary method for using semantic segmentation to determine which parts of an image are static and which are dynamic is described in “DS-SLAM: A Semantic Visual SLAM towards Dynamic”, 2018 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 2018, pp. 1168-1174, doi: 10.1109 / IROS.2018.8593691.

[0074] Alternatively, moving three-dimensional objects in one or more of the images 40 may be detected, for example, by comparing two or more images 40 subsequently captured by the same camera, and by comparing the positions of corresponding three-dimensional objects in different images 40.

[0075] In step S14, the moving three-dimensional object is determined to be a dynamic object and can be identified, for example.

[0076] In step S16, these dynamic objects can be encoded into dynamic object data, for example, as described above.

[0077] Figure 5 shows a flowchart of an exemplary embodiment of a method for determining dynamic object data from three-dimensional object data. This method may be performed when carrying out step S6 of the method described above with respect to Figure 3.

[0078] In step S22, dynamic object data is determined from the three-dimensional object data by identifying which of the other vessels encoded in the AIS data that are in motion are in the vicinity of the vessel. This identification can be easily performed from the AIS data, since the AIS data directly provides the geographical location, course, and speed of the corresponding other vessel 26.

[0079] In step S24, the other moving vessel 26 may be determined to be a dynamic object.

[0080] In step S26, these dynamic objects can be encoded into dynamic object data.

[0081] Figure 6 shows a flowchart of an exemplary embodiment of a method for determining a map for ocean navigation, such as map 20 in Figure 1.

[0082] In step S32, radar data from radar measurements around the ship 18 is received, for example, as described in relation to step S2 above.

[0083] In step S34, image data of one or more images 40 captured by cameras positioned on the ship 18 may be received, and one or more images 40 show one or more three-dimensional objects around the ship 18, for example, as described above with respect to steps S4 and S6. One or more three-dimensional objects may correspond to one or more other ships 26.

[0084] In step S36, dynamic object data may be determined from image data, and the dynamic object data represents, for example, one or more dynamic objects among the three-dimensional objects shown in image 40, as described above with respect to step S6. The one or more dynamic objects may be one or more of the other vessels 26, and these other vessels 26 are moving. The dynamic object data may be determined from image data by detecting moving three-dimensional objects in one or more images 40, determining the moving three-dimensional objects as dynamic objects, and encoding these dynamic objects into dynamic object data.

[0085] Optionally, radar data pointing to one or more of the other stationary vessels 26 may also be tagged, for example, as "potentially" moving.

[0086] In step S38, map 20 may be determined depending on radar data and dynamic object data. For example, radar data pointing to dynamic object data may be identified, and map 20 may be generated based on radar data that does not point to dynamic object data, as described, for example, in US9581695B2.

[0087] The neural networks and components described above can be implemented using software, hardware, or a combination of software and hardware.

[0088] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustrations and descriptions should be considered illustrative or exemplary and not limiting, and the present invention is not limited to the disclosed embodiments. Other modifications of the disclosed embodiments can be understood and achieved by those skilled in the art in carrying out the claimed invention, by reference to the drawings, disclosure and appended claims. In the claims, the term “equipped with” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude plurals. A single processing unit or other units may perform the functions of several items described in the claims. The mere fact that certain means are described in different dependent claims does not indicate that combinations of these means cannot be used advantageously. None of the reference numerals in the claims should be construed as limiting the scope. [Explanation of Symbols]

[0089] 18. Our ship 20 Maps 22 Water surface 24 Land 26 Other ships 28 Radar reflectors 30 sector 32 Arcuate 34 distance 36 lines 40 images 42 Imaged water surface 44. Photographed land 46 Other ships that have been photographed S2-S38 Steps 1-38

Claims

1. A method for determining the motion of the ship (18), Receiving radar data from radar measurements around the aforementioned vessel (18), The vessel (18) receives three-dimensional object data representing one or more three-dimensional objects located in its vicinity, The process involves determining dynamic object data from the aforementioned three-dimensional object data, where the dynamic object data represents one or more dynamic objects from among the three-dimensional objects encoded in the three-dimensional object data. The motion of the ship (18) is determined based on the radar data and the dynamic object data, A method that includes [a certain feature].

2. The aforementioned three-dimensional object data is image data of one or more images (40) captured by a camera positioned on the ship (18), The one or more images (40) mentioned above show one or more three-dimensional objects. The dynamic object data is determined from the three-dimensional object data by detecting moving three-dimensional objects in one or more images (40), determining the moving three-dimensional objects as the dynamic objects, and encoding these dynamic objects into the dynamic object data. The method according to claim 1.

3. The one or more three-dimensional objects are corresponding to one or more other ships (26), The one or more moving objects are one or more of the other vessels (26), and these other vessels (26) are moving. The method according to claim 2.

4. The one or more three-dimensional objects are corresponding to one or more other ships (26), The aforementioned 3D object data is AIS data, The dynamic object data is determined from the three-dimensional object data by identifying which of the other vessels (26) encoded in the AIS data is moving, determining the moving other vessels (26) as the dynamic object, and encoding these dynamic objects into the dynamic object data. The method according to claim 1.

5. The movement of the vessel (18) refers to the movement of the vessel (18) relative to the land. The method according to any one of claims 1 to 4.

6. The method according to claim 5, wherein determining the motion of the vessel (18) based on the radar data and the dynamic object data comprises determining data from the radar data corresponding to the dynamic object data and determining the motion of the vessel (18) based on the radar data excluding the dynamic object data.

7. The radar data refers to the angular range around the vessel (18), The sector (30) of the angular range in which the corresponding dynamic object is located is determined. The method according to claim 6, wherein data from the radar data pointing to the determined sector (30) is determined as the dynamic object data.

8. The radar data refers to the angular range around the vessel (18), and the method is Determining distance data representing the distance (34) between the vessel (18) and one of the dynamic objects, From the distance data, determine the arc shape (32) in which the corresponding dynamic object within the angular range is positioned, Equipped with, The method according to claim 6, wherein the data from the radar data referring to the arc shape (32) is determined as the dynamic object data.

9. Determine the position of at least one of the aforementioned dynamic objects relative to the ship (18), Determining the corresponding extended portion of the dynamic object, Equipped with, The method according to claim 6, wherein data from radar data indicating the position and the extending portion of the corresponding dynamic object is considered as the dynamic object data.

10. The motion of the vessel (18) refers to the motion of the vessel (18) relative to one of the dynamic objects. The method according to any one of claims 1 to 4.

11. Determining the motion of the vessel (18) based on the radar data and the dynamic object data comprises determining all data from the radar data that does not correspond to the dynamic object data of the corresponding dynamic object, and determining the motion of the vessel (18) based solely on the dynamic object data. The method according to claim 10.

12. The radar data refers to the angular range around the vessel (18), The sector (30) of the angular range in which the corresponding dynamic object is located is determined. Only data from the radar data pointing to the determined sector (30) is considered as the dynamic object data. The method according to claim 11.

13. The radar data refers to the angular range around the vessel (18), and the method is Determining distance (34) data representing the distance (34) between the vessel (18) and one of the dynamic objects, From the distance (34) data, the arc shape (32) in which the corresponding dynamic object within the angular range is positioned is determined, Equipped with, Only data from the radar data that points to the determined arc shape (32) is considered as the dynamic object data. The method according to claim 11.

14. Determine the position of at least one of the aforementioned dynamic objects relative to the ship (18), Determining the corresponding extended portion of the dynamic object, Equipped with, The method according to claim 11, wherein only data from the radar data indicating the position and extent of the corresponding dynamic object is considered as the dynamic object data.

15. A system for determining the motion of the ship (18), A memory configured to store radar data, image data, 3D object data, and / or dynamic object data, A processor configured to carry out the method described in any one of claims 1 to 14, A system equipped with these features.

16. A method for determining a map of ocean navigation (20), To receive radar data from radar measurements around the vessel (18), The system receives image data of one or more images (40) captured by a camera positioned on the vessel (18), wherein the one or more images (40) show one or more three-dimensional objects in the vicinity of the vessel (18). The process involves determining dynamic object data from the aforementioned image data, where the dynamic object data represents one or more dynamic objects among the three-dimensional objects shown in the image (40). The map (20) is determined based on the radar data and the dynamic object data, A method that includes [a certain feature].

Images