Marine environment sensing system

Abstract

The present disclosure relates to a marine environment sensing system for controlling a marine vessel (100), where the marine environment sensing system comprises: a light detection and ranging LiDAR sensor (201) mounted around the marine vessel (100) for recording the surroundings of the marine vessel (100), where the surroundings comprise obstacles (303, 500, 503, 504, 505, 506); the present invention relates to a system (100) for monitoring an ambient environment, comprising:-a LiDAR sensor (601, 601) and water (305),-a control unit (205) having a neural network for processing information about the recorded ambient environment that has been recorded by the LiDAR sensor, where the control unit (205) is programmed to visualize the recorded ambient environment based on LiDAR data enriched by the neural network, wherein the recorded information has been classified as category objects in order to distinguish between different types of objects (303, 500, 503, 601) in the surroundings.

Description

Technical Field

[0001] The present disclosure relates to a marine environment sensing system for controlling marine vessels. The present disclosure relates to a marine environment sensing system that enriches the sensed surrounding environment information with the help of a neural network. It also focuses on a complete solution for visualizing the surrounding environment and assisting in maneuvering marine vessels. The present disclosure can be implemented in various marine vessels. Background Art

[0002] The marine environment is turbulent and chaotic due to the influence of wind, drift and waves. In such an environment, it is crucial to fully understand the environment around the marine vessel to prevent collisions and avoid danger. A system that knows the distance to surrounding obstacles and water can use this information to assist the captain. However, since the water is constantly changing and the marine vessel moves in all three directions, it is difficult to interpret the information as useful assistance. If the system also knows what kind of obstacles or water there are, the assistance and help functions can have much higher accuracy. Human-machine interface (HMI) support for known objects in free space and markings is an example. Another is to prevent manipulation that will cause a collision with an object such as a dock or another marine vessel. Finally, this can be further developed into advanced driving / advanced driver assistance systems (AD / ADAS).

[0003] Marine vessels with control systems can benefit greatly from this, making it easier to dock marine vessels and avoid collisions.

[0004] Existing solutions

[0005] An existing solution is a surround view camera system that provides camera-based visualization of the immediate area around a marine vessel. Such systems are suitable for barges. However, the systems cannot see very far and the projection makes the stitched images difficult to interpret.

[0006] Another existing solution is a radar sensor system for interpreting and classifying objects and free space. Radar sensor systems are mainly used in autonomous vehicles rather than marine vessels. However, they have low-resolution measurements with noise, and they require a camera to perform the classification.

[0007] Another existing solution is a system that senses the surroundings with stereo cameras and integrates it into the auxiliary barge. This system can sense the surroundings closer than 10 meters, but cannot sense larger parts of the port.

[0008] US2021088667A1 only covers a surround view of the waterline. It discloses detecting structures spanning an area by scanning the area vertically using a LIDAR sensor. In paragraph

[0028] , it is mentioned that "embodiments of the technology are applicable to the field of marine vessel navigation systems using LIDAR sensors. The use of LIDAR sensors that utilize a near-infrared spectrum that is undetectable to the human eye enables navigation systems in marine environments to identify docks, ships, and coastlines by distinguishing objects from water, because water typically has a detectable LIDAR signature." Therefore, this refers to the intensity values ​​from the LiDAR, which are not always sufficient to accurately distinguish different objects. In this prior art, only objects close to the water surface are detectable, but this is still useful, for example, in the case of detecting swimmers near propellers.

[0009] US9778657B2 discloses a method for automatically moving a marine vessel by an automatic position setting system. The method includes receiving at least one optical feed from a vision ranging photography system by a central processing unit, the at least one optical feed including data providing a mapping of the marine vessel's surroundings. The method includes displaying the mapping of the environment on a touch screen monitor by the central processing unit. The method includes receiving target position data from the touch screen monitor by the central processing unit. The method includes using the mapping to instruct at least one element of the marine vessel's propulsion system by the central processing unit to move the marine vessel to a target position. A camera is used in this prior art.

[0010] KR20200027870A discloses an autonomous navigation method using image segmentation, which is capable of detecting the surrounding environment using an artificial neural network that performs image segmentation. It discloses a ship autonomous navigation method using maritime images and artificial neural networks. The method includes the following steps: obtaining label data; outputting output data; learning an artificial neural network using an error function; obtaining a maritime image; obtaining information about the type and distance of an obstacle; obtaining information about the direction of the obstacle included in the maritime image; obtaining the position of the obstacle included in the maritime image; generating an obstacle map; generating a path to follow; and generating a control signal. Again, only a camera is used here.

[0011] US2021012120A1 discloses a system and method for estimating free space, comprising applying a machine learning model to a camera image of a navigation area, wherein the navigation area is decomposed into cells. The method comprises: synchronizing point cloud data from the navigation area with the processed camera image, and associating the probability of the cell being occupied and the object classification of the object that may occupy the cell with the cell in the navigation area based on sensor data, sensor noise and the machine learning model. Again, only the camera is used here.

[0012] The scientific paper "Exploiting Structured CNNs for Semantic Segmentation of Unstructured Point Clouds from LiDAR Sensor" (https: / / www.mdpi.com / 2072-4292 / 13 / 18 / 3621 / htm) discloses a projection-based approach for semantic segmentation of 3D point clouds. The trained convolutional neural network (CNN) estimates the class value of each LiDAR point based on the depth value and the intensity value.

[0013] The disclosed aspects, examples (including any preferred examples), and / or the appended claims may be appropriately combined with each other, which is obvious to any person skilled in the art. Additional features and advantages are disclosed in the detailed description, claims, and drawings, and in part will be obvious to those skilled in the art or recognized by practicing the disclosure as described herein. Summary of the invention

[0014] An object of the present disclosure is to provide a marine environment sensing system for controlling and docking a marine vessel, which can help a user interpret the surrounding environment in a better way, or can be used to automatically control a transmission system of a marine vessel.

[0015] According to a first aspect of the present disclosure, the object is achieved by a marine environment sensing system for controlling a marine vessel, wherein the marine environment sensing system comprises: - a LiDAR sensor installed around a marine vessel for recording the surroundings of the marine vessel, wherein the surroundings include obstacles and water, - a control unit having a neural network for processing information about the recorded surroundings which has been recorded by the LiDAR sensor, Therein the control unit is programmed to visualize the recorded surroundings based on the LiDAR data enriched by a neural network, wherein the recorded information has been classified into categories of objects in order to distinguish between different types of objects in the surroundings.

[0016] The marine environment sensing system may include a bridge to visualize the recorded surroundings and provide input to manually control the marine vessel's powertrain.

[0017] The classified information may be disclosed in the form of 3D point cloud visualization with location and category information.

[0018] The classified information may be made public in the form of probability maps.

[0019] The probability map may be a two-dimensional (2D) point cloud visualization with location information and category information.

[0020] The probability map may be a three-dimensional (3D) point cloud visualization with location information and category information.

[0021] 3D point clouds can be semantically classified using projection-based methods.

[0022] Each point in the visualization can be colored with the color of the class object.

[0023] The control unit may make a decision suitable for the visualization object depending on the category object of the visualization object in the vicinity of the marine vessel.

[0024] The control unit may be arranged to make the decision as follows: a. If there is another marine vessel within a predetermined distance, the control unit automatically reduces the speed of the marine vessel to below a predetermined speed to avoid getting too close to the other marine vessel, while, b. If on the contrary a dock is registered, then when approaching the dock the control unit is allowed to drive faster than the predetermined speed if appropriate, since the dock is not a movable object in contrast to the other marine vessel.

[0025] According to a second aspect of the present disclosure, the object is achieved by a marine vessel comprising the marine environment sensing system according to the first aspect.

[0026] According to a third aspect of the present disclosure, the object is achieved by a computer-implemented method for controlling a marine vessel. The method comprises: - obtaining information about a recording of a surrounding environment of a marine vessel, wherein the recording is performed by a LiDAR sensor installed around the marine vessel, wherein the surrounding environment includes obstacles and water; - using a neural network to process information about the recorded surroundings that has been recorded by the LiDAR sensor; and - Visualization of the recorded surroundings based on LiDAR data enriched by a neural network, where the recorded information has been classified into categories of objects in order to distinguish between different types of objects in the surroundings.

[0027] Controlling and docking marine vessels can be difficult tasks when the marine environment is constantly changing. This can be made easier with a marine environment sensing system according to the present disclosure that helps users interpret the surrounding environment. With a LiDAR sensor, the surrounding environment can be interpreted by measuring the 3D position and intensity values ​​of light reflections. The surrounding environment can be interpreted by interpreting each LiDAR measurement result as a category object such as water, dock, marine vessel, etc. using a neural network. In addition, obstacles such as other marine vessels and docks can be distinguished. This data can then be visualized in the HMI as a probability map that presents possible surround views related to the marine vessel.

[0028] Marine vessels with control systems can benefit greatly from this, making it easier to dock marine vessels and avoid collisions.

[0029] The disclosed aspects, examples (including any preferred examples), and / or the appended claims may be appropriately combined with each other, which is obvious to any person skilled in the art. Additional features and advantages are disclosed in the detailed description, claims, and drawings, and in part will be obvious to those skilled in the art or recognized by practicing the disclosure as described herein.

[0030] Also disclosed herein are computer systems, control units, code modules, computer-implemented methods, computer-readable media, and computer program products associated with the technical benefits discussed above. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Examples are described in more detail below with reference to the accompanying drawings.

[0032] Figure 1 is an exemplary schematic diagram illustrating a marine vessel according to an example.

[0033] Figure 2 is an exemplary schematic diagram illustrating a marine vessel according to an example.

[0034] Figure 3 is an exemplary flow chart illustrating a method according to an example.

[0035] Figure 4 is an exemplary schematic diagram of a marine vessel at dock according to an example.

[0036] Figure 5 is an exemplary schematic diagram of a marine vessel in a canal according to an example.

[0037] Figure 6 is an exemplary schematic diagram of a marine vessel with various objects in the surrounding environment according to an example.

[0038] Figure 7 is a flowchart of an exemplary method according to an example.

[0039] Figure 8 is a schematic diagram of an exemplary computer system for implementing the examples disclosed herein, according to an example. DETAILED DESCRIPTION

[0040] The detailed description set forth below provides information and examples of the disclosed technology in sufficient detail to enable those skilled in the art to practice the disclosure.

[0041] The focus of the present disclosure is to enrich the surrounding environment information with the help of neural networks. It also focuses on solutions to visualize the surrounding environment of marine vessels and assist in maneuvering marine vessels. In this way, the present disclosure is more complex and advanced, so that it can better adapt to each situation.

[0042] Figure 1 1 is an exemplary block diagram illustrating a marine vessel 100. The marine vessel 100 may be any type of marine vessel, such as a boat, a marine recreational boat, a motorboat, a personal watercraft, a ship, a tanker, a cargo ship, a submarine, etc. The marine vessel 100 may be referred to as a watercraft, a water transport vessel, a water vehicle, etc. Although the present disclosure may be described with respect to a particular marine vessel, the present disclosure is not limited to any particular marine vessel.

[0043] The marine vessel 100 is arranged to be operated and controlled by a user. The user may be referred to as a captain, an operator, a controller. In the case where the marine vessel is an at least partially autonomously controlled marine vessel, the user may be a human user or a non-human user (e.g., a computer or computer system). Manipulating the marine vessel 100 may be referred to as operating the marine vessel 100, driving the marine vessel 100, etc.

[0044] The marine vessel 100 comprises a marine environment sensing system. The marine environment sensing system is arranged to sense the surroundings of the marine vessel 100. The surroundings of the marine vessel 100 may include one or more objects, also referred to as obstacles, that the marine vessel 100 may need to give way to, for example, by changing its trajectory to adapt to its speed, etc. The objects may be of different types, such as another boat, a dock, a canal boundary, a buoy, water, etc. Each object type may be associated with a class or object category, or an object type may be a class or object category.

[0045] Figure 2 is an exemplary schematic diagram illustrating a marine vessel 100 according to an example. The marine vessel 100 includes a marine environment sensing system.

[0046] The marine surrounding environment sensing system includes n LiDAR sensors 201, where n is a positive integer. The LiDAR sensor 201 is arranged to be installed around the marine vessel 100, arranged to record the surrounding environment of the marine vessel 100, that is, to sense the surrounding environment of the marine vessel 100. The LiDAR sensor 201 can be installed in the hull of the marine vessel 100, for example, integrated in the hull, or installed on the outside of the hull. The position of the LiDAR sensor 201 may depend on the size and structure of the marine vessel 100. The LiDAR sensor 203 is a sensor suitable for marine applications. Each LiDAR sensor 201 is arranged to record the reflected laser pulse as a collection of points, i.e., a point cloud. A point cloud refers to a set of data points in a 3D coordinate system. Therefore, the output of the LiDAR sensor 201 is a point cloud, i.e., a 3D point cloud. Each data point in the 3D point cloud is represented by Cartesian coordinates x, y, and z.

[0047] The LiDAR sensor 201 has a higher resolution than, for example, a radar sensor and a sonar sensor. The LiDAR sensor 201 is more accurate than a camera for measuring depth. Multiple LiDAR sensors 201 can be installed on the marine vessel 100 to cover all surrounding angles and leave no blind spots. The LiDAR sensor 201 can be arranged to sense a distance of, for example, up to 120 meters, but the marine vessel 100 can also be equipped with a LiDAR sensor 201 that can sense a longer and / or shorter distance than the indicated distance. Figure 2 An example is shown in which six LiDAR sensors 201 are installed around the marine vessel 100, but any n LiDAR sensors 201 are applicable, where n is a positive integer. Figure 2 As illustrated in FIG. 1 , three LiDAR sensors 201 may be installed on the port side of the marine vessel 100, and three LiDAR sensors 201 may be installed on the starboard side of the marine vessel 100. Figure 2 The example in FIG. 1 shows that the LiDAR sensors 201 are evenly distributed around the marine vessel 100 , but this is merely an example. The LiDAR sensors 201 may be distributed around the marine vessel 100 in any suitable manner, such as evenly or unevenly distributed.

[0048] Depending on the size and shape of the marine vessel 100, a different number of LiDAR sensors 201 may be needed to ensure that there are no blind spots. The LiDAR sensor types may also vary across the marine vessel 100, with some LIDAR sensors 201 being arranged to see farther and some LIDAR sensors seeing shorter. The same is true for the number of transmissions and thrusters, where the amount is selected depending on the size and shape of the marine vessel 100.

[0049] The marine environment sensing system may include a global positioning system (GPS) and an inertial measurement unit (IMU) 203. The GPS and IMU 203 are arranged to give position data for the marine vessel 100, which is required, for example, to compensate for the LiDAR measurements of movement when converted to an HMI map view. In this way, even if the marine vessel 100 has moved far, old LiDAR measurements can be visualized for a period of time. When the marine vessel 100 has moved a predetermined distance from the measurement point, the marine vessel has moved far away, and the old LiDAR measurements are then removed from the memory ( Figure 2 The marine environment sensing system may have a memory of the locations of detected obstacles.

[0050] The marine environment sensing system includes a control unit 205. The control unit 205 may be a computing unit, or it may include a computing unit. The control unit 205 may include a neural network (NN) that is arranged to make the previous LiDAR measurement result have richer information, so that a smarter and more accurate auxiliary function can be created. The neural network may be referred to as a semantic segmentation network or arranged to perform a semantic segmentation method. The point cloud from the LiDAR sensor 201 is projected into two 2D maps with depth values ​​and intensity values ​​for a panoramic view around the marine vessel 100. The terms map and image are used interchangeably in this article. The two 2D maps may include a first 2D map with depth information and a second 2D map with intensity information. The depth information includes the distance to each 3D point in the 3D point cloud, and the intensity information includes the reflectivity value of each (x, y, z) 3D point in the 3D point cloud, such as energy return. These two 2D maps can be processed in a neural network (e.g., a pre-trained convolutional neural network) that has an estimated 2D map of the category information for each measurement point. The estimated 2D map with class information can be projected back to a point cloud, ie a 3D point cloud, giving it a new class value feature, which can be referred to as a segmented part.

[0051] Segmentation can be described as point classification. Segmentation can include classifying each pixel or point in an image or point cloud into a class. In this way, all areas of an image or point cloud can be divided into objects belonging to different classes or types.

[0052] The estimated 2D map with category information projected back into the point cloud may be referred to as an enriched point cloud, i.e., an estimated 2D map with category information. This enriched point cloud may then be used to present more detailed views or visualizations, increased safety features for collision avoidance, and other functions that require a good interpretation of the surroundings of the marine vessel 100. This approach may be referred to as projection-based mapping. The neural network needs to be trained on labeled data to be able to predict the correct category. Before training is possible, the labeled data must be recorded and labeled as separate actions. Labeled data is or includes ground truth data of the category information of the point cloud used to train the neural network. Readings from the LiDAR sensor 201 may be removed depending on what category they belong to. For example, points of the category "water" may be removed to more easily detect objects near the waterline that would have been previously removed with an altitude filter. Objects may be referred to as obstacles, i.e., obstacles that the marine vessel 100 may need to give way to avoid collisions, etc. The above is described in detail in the accompanying drawings. Figure 3 This is schematically shown in an exemplary flow chart of FIG. Figure 3 The steps in can be performed by the control unit 205 or a processing circuit included in a computer system. In summary, Figure 3 The method shown in includes at least one of the following steps, which may be performed in any suitable order except as follows:

[0053] Step 300

[0054] LiDAR data from the LiDAR sensor 201 is obtained, for example, the LiDAR data is transmitted from the LiDAR sensor 201 to the control unit 205. LiDAR data only from the LiDAR sensor 201 may be obtained. The LiDAR data may be referred to as a 3D point cloud or included in a 3D point cloud. Before obtaining the LiDAR data, the LiDAR sensor 201 senses or records the surroundings of the marine vessel, and these records are referred to as LiDAR data.

[0055] Step 301

[0056] The 3D point cloud from step 301 is projected into one or two 2D maps. In the case of two 2D maps, there may be a first 2D map including depth information and a second 2D map including intensity information. In the case of one 2D map, the 2D map may include depth information or intensity information. In other words, one image may be created with one or two channels, wherein the two channels are associated with depth and intensity, respectively, and wherein the one channel is associated with either depth or intensity.

[0057] Step 302

[0058] Based on one or both 2D maps from step 301, a 2D map with class information for each point is estimated. A neural network can be used to perform the estimation. This step can also be referred to as performing segmentation. The output of step 302 is a rich point cloud, i.e. a segmented 2D image.

[0059] In other words, a 2D map can be created as an image with 2 channels from one of the depth map or the intensity map or using both of them. Any of these three image types can be the input to the neural network.

[0060] The neural network may classify the 2D class information map based on the first 2D map representing depth or the second 2D map representing intensity or both. Thus, the classification may use only depth information or only intensity information as input, or it may use a combination of depth information and intensity information as input.

[0061] The term segmentation used here can be described as an image or point cloud where in addition to the position / color / depth / intensity information there is information about what type of object or class each pixel or point belongs to. Thus, in step 302 the neural network can predict the class of each pixel or point from the 2D image input into the class information output as a 2D image. Then, in order to obtain a point cloud again, it may be necessary to transform it back to the 3D domain again with the known depth information, which will be described in step 303 below.

[0062] Step 302 may include determining a probability associated with each pixel or point in the 2D image. As described above, LiDAR data consists of points in a 3D point cloud with x, y, and z values. These points can be translated to a 2D plane consistent with the water surface and / or the marine vessel 100, thereby creating measurements in the 2D plane. The 2D plane can be divided into squares of any suitable size, such as 0.1m*0.13m, 0.2m*0.2m, 0.3m*0.3m, 0.4m*0.4m. For each LiDAR measurement point converted to a square on the 2D plane, the probability that something exists in the square / position increases. At the first number of measurement points in the same position, a first probability can be displayed. The second number of measurement points in the same position may correspond to the maximum probability. The first number can be, for example, 3 measurement points, and the second number can be, for example, 8 measurement points. Each time if there is no measurement point in a square, but it is determined that the square is not blocked by an object in front, the probability is reduced in the same way.

[0063] Step 303

[0064] The enriched point cloud from step 602 is projected back to a 3D point cloud. This step can be described as a projection-based mapping, i.e., a mapping from a segmented 2D image to a 3D point cloud.

[0065] Step 304

[0066] The image or map representing the 3D point cloud from step 303 may be displayed in a display, for example, included in the marine vessel 100. The display may be included in or connected to the bridge 210. Each point in the image or map representing the 3D point cloud may be colored with a color corresponding to the class object. Instead of or in addition to the color, any other suitable representation of the class object may be used, such as a point, a diagonal line, etc.

[0067] The displayed image or map may be a probability map including the probabilities determined in step 302 .

[0068] In other words, step 304 may be described as displaying the recorded surroundings based on the LiDAR data enriched by the neural network, wherein the recorded information has been classified into categories of objects in order to distinguish between different types of objects in the surroundings.

[0069] Based on the displayed image or map, for example, if the marine vessel 100 is an at least partially autonomous marine vessel, the control unit 205 may determine to control the marine vessel 100, for example, according to the category of the objects in the surrounding environment. For example, according to the category of the objects in the surrounding environment of the marine vessel 100, the speed of the marine vessel 100 may be reduced, it may be increased, etc. In another example, the displayed image or map may assist the user in operating the marine vessel 100, such as an auxiliary barge.

[0070] Now back to Figure 2 .

[0071] The marine environment sensing system may include an HMI and a steering system. These functions are particularly useful by combining LiDAR measurements and segmentation with the electrical control system and associated HMI of the marine vessel 100. The assistance function, which takes into account what obstacles are around the marine vessel 100, can make docking of the marine vessel 100 easier. The marine environment sensing system can prevent actions from users of the marine vessel 100 that could cause a collision. With the full integration of the functions from the LiDAR sensor 201 to the control unit 205, autonomous driving of the marine vessel 100 can be further developed and implemented. However, as a start, this is mainly targeted at the assistance function.

[0072] The marine environment sensing system may include an input unit 208 arranged to receive user input. The input unit 208 may be in the form of a joystick or other directional device to input the desired movement of the marine vessel 100 to the marine environment sensing system. Another example of the input unit 208 may be an HMI display arranged to obtain user input, which may be used, for example, to activate or deactivate and control functions. The HMI display may be an HMI touch screen.

[0073] The marine environment sensing system may include an actuator ( Figure 2 (not shown). There may be one or more actuators. If there are multiple actuators, they can be individually controlled to generate the same force direction as the user input. The force direction to prevent collisions and dangers can also be given autonomously by the marine environment sensing system.

[0074] The ocean environment sensing system may include a compass ( Figure 2 2 ), such as a Global Navigation Satellite System (GNSS) compass, which can be used to provide information about the heading and position of the marine vessel to the control unit 205. The position information can be used to move the map and accurately convert the LiDAR measurements to the marine vessel 100.

[0075] The marine environment sensing system may include a computing unit, which may be a stand-alone unit or included in the control unit 205. The computing unit is arranged to predict the class value of each LiDAR measurement using a pre-trained neural network. The class value is used in conjunction with the 3D position and intensity value to interpret the surrounding environment. This information can then be sent to the bridge 210, where the user can view information about the surrounding environment. User inputs that may cause collisions can be neutralized. In addition, readings from the LiDAR sensor 201 can be removed depending on what class they belong to. For example, points of the class "water" can be removed to more easily detect objects near the waterline.

[0076] The marine environment sensing system may include an HMI ( Figure 2 (not shown), the HMI is arranged to show a probability map of the surrounding environment based on the enriched LiDAR data with category information. Water, docks, other marine vessels, etc. are displayed in different ways to increase the information of what the surrounding environment looks like. The probability of obstacles can also be presented in different colors or other representations depending on how confident the marine environment sensing system is about the area. The 3D view can be switched to where the user can see a 3D point cloud with different colors or other representations based on category values. For example, marine vessels can be represented in blue, docks can be represented in red, etc. Touch gestures can be input from the user to set functions and interact with the 2D map and 3D map.

[0077] With the full ocean environment sensing system, the user is given a better understanding of the ocean vessel's surroundings. It is possible to visualize where the ocean vessel can travel freely, where the ocean vessel can fit, and where collisions may occur. The ocean environment sensing system can also assist by preventing user inputs that would cause collisions, and assist in maintaining a uniform distance to objects around the ocean vessel 100.

[0078] By adding information with the class value of the LiDAR measurements, a barge function can also be created. Since different barge locations require different barge technologies, it is advantageous that the marine environment sensing system can identify situations. Barging at a dock, next to another marine vessel, or next to a cliff are three examples of different situations where the type of obstacle plays an important role in how the marine vessel 100 is docked.

[0079] An autopilot that follows other ocean vessels and maintains distance to them can be made more robust by identifying ocean vessels, water, and other obstacles rather than just measuring the distance to them.

[0080] Figure 2 It is further shown that the marine vessel 100 includes two engines 213, two gears 215 and two rudders 218. Note that Figure 2 The number of engines 213, gears 215, and rudders 218 shown in the figure are examples only, and the marine vessel 100 may include any suitable number of engines 213, gears 215, and rudders 218. The number of engines 213, gears 215, and rudders 218 may be the same or different. The marine vessel 100 may include at least one engine 213, at least one gear 215, and at least one rudder 218.

[0081] Figure 4 is an exemplary schematic diagram of a marine vessel 100 at a dock 403 . Figure 4 The example illustrations seen in FIG. 2 may be displayed on a HIM that is included in or connected to the bridge 210 , for example.

[0082] Dock 403 is represented using diagonal lines. Instead of diagonal lines, dock 403 may be represented using a specific color, such as red. Water 405 is represented by a curve around marine vessel 100, or by blue, for example. Alignment lines 408 are located on each side of marine vessel 100 and represent the track of marine vessel 100. Figure 2 The empty space in may represent "free space" in which it is safe to travel, and the portion with the diagonal line is a possible collision area where the marine environment sensing system should prevent the marine vessel 100 from traveling. The LiDAR measurements 405 of water are removed with the aid of the water category 405 predicted by the neural network so that the marine environment sensing system can understand where it is safe to go. The dock 403 is also classified and can therefore be considered a dock by the marine environment sensing system. The marine environment sensing system can also be configured to determine the distance between the marine vessel 100 and the dock 403. Figure 4 In the example shown in FIG. 1 , the distance between the starboard side of the marine vessel 100 and the dock 403 may be 2.1 m and the angle difference index may be 2°, and the distance between the port side of the marine vessel 100 and the dock 403 may be 3.0 m and the angle difference index may be -2.5°. The distance between the stern of the marine vessel 100 and the dock 403 may be 0.8 m and the angle difference index may be -1.2°. The angle difference index indicates the angle difference between the ship side and the nearest object side when an edge is detected.

[0083] 3D point clouds of different categories of objects can be visualized, which can specify to the marine environment sensing system what is located near the marine vessel 100. The 3D point cloud can be displayed in an HMI, such as in or connected to the bridge 210. For example, a lighter blue color can indicate other marine vessels, while a red area can indicate a dock 403. In another example, a dotted object can represent other marine vessels, an object filled with a diagonal line can represent the dock 403, and a circular line can represent water 405. By interpreting the differences in these objects, the speed or other parameters of the marine vessel 100 can be adjusted. Object recognition can help avoid collisions and enable a higher level of understanding of the surrounding environment.

[0084] Figure 5 is an exemplary schematic diagram of a marine vessel 100 traveling in a canal. The marine environment sensing system may include a canal lane keeping aid that may be created using a free space algorithm that may be more robust with the aid of class information. The marine vessel 100 travels in the canal and gives way to another marine vessel 500 (e.g., an oncoming marine vessel), whereby the marine environment sensing system understands the difference between the other marine vessel 500 and a canal boundary 503. The canal boundary 503 is represented by a diagonal line, and the other marine vessel 500 is represented by a dot. The water 405 is represented by a curved line and may be ignored. The trajectory 505 of the marine vessel 100 is also shown in FIG. Figure 5 Shown in.

[0085] Figure 6 is an exemplary diagram of how the ocean environment sensing system can utilize information about different objects in the surrounding environment to adjust the maximum speed of the ocean vessel 100 allowed to have in the direction toward a particular type of object. The maximum speeds toward corresponding types of objects can be different or the same. Figure 6 It is shown that the surrounding environment may include different types of objects. Figure 6 The object types illustrated in are another marine vessel 500, dock 403, water 405, and buoy 601. Another marine vessel 500 is represented by a dot, dock 403 is represented by a diagonal line, water 405 is represented by a curved line, and the buoy is represented by a dot larger than that representing another marine vessel 500.

[0086] By using LiDAR data, the calculation speed and distance accuracy can be significantly improved. Compared to the LiDAR sensor 201, the camera has less distance accuracy and requires more computing power. Compared to the camera, the LiDAR sensor 201 also detects better in darkness or bad weather. Other sensors such as radar or sonar have lower resolution than the LiDAR sensor 201, making it difficult to retrieve enough information about the surrounding environment.

[0087] The marine environment sensing system uses semantic segmentation based on LiDAR data, such as LiDAR data alone, combined with marine environment, HMI features, and electrical control systems to create a complete high-value marine surrounding environment sensing system.

[0088] The marine environment sensing system can provide an HMI view of a probabilistic map based on LiDAR data (e.g., based only on LiDAR data) around the marine vessel 100. LiDAR data gives the possibility to present a map view of a larger area (e.g., about 100m radius), while previously known systems were limited to about 15m radius around the marine vessel 100.

[0089] The LiDAR sensor 201 used in the marine environment sensing system may have, for example, 32 layers in a 70° vertical field of view (FOV), giving more than 1 million points / second for each LiDAR sensor 201. This makes it possible to not only draw attention to the closest distance around the marine vessel 100 at certain angles, but also display all data of the surrounding environment.

[0090] The marine environment sensing system enables a 3D view with location information and category information to be presented in the HMI. Such a view is detailed enough so that a user can navigate the marine vessel 100 just by looking at the HMI. The marine environment sensing system can present a 2D probability view in which the probability of surrounding obstacles is visualized.

[0091] By using the segmented point cloud, the marine environment sensing system can make a decision suitable for nearby objects according to the category of the nearby objects. If there is another marine vessel 500 within a predetermined distance, the marine environment sensing system can automatically reduce the speed of the marine vessel 100 and avoid getting too close to the other marine vessel 500, and if the dock 403 is detected, since the dock 403 is not a movable object compared to the other marine vessel 500, the marine environment sensing system can trigger the marine vessel 100 to travel faster when approaching the dock 403 than when approaching the other marine vessel 500.

[0092] Using the marine environment sensing system, a 3D view can be presented in which each point is colored with a class value, or in which each point is represented by a specific fill pattern or any other representation. The fill pattern can be, for example, a dot, a diagonal line, a horizontal line, a vertical line, a star, etc. This gives a good overview of the surroundings of the marine vessel 100, even if it is dark outside. This feature can reduce the number of times that the user of the marine vessel 100 would otherwise have to temporarily leave the bridge 210 to go to and check the distance to the surrounding obstacles close to the marine vessel 100, such as other marine vessels 500, docks 403, etc. Therefore, the marine environment sensing system provides a good understanding of the surrounding environment, reduces the need to leave the bridge 210 and / or control the marine vessel 100, thereby making the driving of the marine vessel 100 safer.

[0093] The marine environment sensing system has classified intelligence, which is integrated with the control system and realizes dynamic HMI functions.

[0094] Figure 7 is a flow chart of an exemplary method according to an example. The method may be performed by the control unit 205 or the marine environment sensing system or a processing circuit included in a computer system. The method includes at least one of the following steps, which may be performed in any suitable order except as described below:

[0095] Step 701

[0096] This step corresponds to Figure 3 Step 300 in. Obtain recorded information about the surrounding environment of the marine vessel 100. The recorded information can be obtained from the LiDAR sensor 201. The recording is performed by the LiDAR sensor 201 installed around the marine vessel (e.g., by using only the LiDAR sensor). The surrounding environment includes obstacles 403, 500, 503, 601 and water 405.

[0097] Step 702

[0098] This step corresponds to Figure 3 Steps 301, 302, 303 in FIG. 304. Information about the recorded surroundings recorded by the LiDAR sensor is processed using a neural network. The processing may include performing the segmentation already described in detail above.

[0099] Step 702 may include enriching the recorded surroundings based on the LiDAR data neural network. Enriching the recorded surroundings based on the LiDAR data may include classifying the recorded information into category objects so as to distinguish different types of objects 403, 500, 503, 601 in the surroundings. The output of step 702 may be described as classified information, the classified information including information about which category the object has been classified into, i.e., information about which category the object belongs to. Examples of categories may be: water 405, dock 403, another marine vessel 500, canal boundary 503, buoy 601, etc. Other examples of categories may be moving objects, stationary objects, or non-moving objects, etc.

[0100] Step 702 may include determining a probability. A probability associated with each pixel or point in the 2D image may be determined. As described above, LiDAR data consists of points in a 3D point cloud having x, y, and z values. These points may be translated to a 2D plane consistent with the water surface and / or the marine vessel 100, thereby creating measurements in the 2D plane. The 2D plane may be divided into squares of any suitable size, such as 0.1m*0.13m, 0.2m*0.2m, 0.3m*0.3m, 0.4m*0.4m. For each LiDAR measurement point converted to a square on the 2D plane, the probability that something exists in the square / position increases. At a first number of measurement points in the same location, a first probability may be displayed, such as color coded from blue to red. A second number of measurement points at the same location may correspond to a maximum probability. The first number may be, for example, 3 or more measurement points, and the second number may be, for example, 8 measurement points. Each time if there is no measurement point in a square, but it is determined that the square is not blocked by an object in front, the probability is reduced in the same way. In addition to other probabilities, the first probability and the maximum probability can be displayed using any suitable type of representation, such as different colors / symbols, etc.

[0101] Step 703

[0102] This step corresponds to Figure 3 304 in step 304. The recorded surroundings based on the LiDAR data enriched by the neural network (i.e., the recorded surroundings based on the LiDAR data enriched by the neural network, wherein the recorded information has been classified into the category object) can be visualized in order to distinguish between different types of objects 403, 500, 503, 601 in the surroundings. The visualization content can be, for example, in the bridge 210. The visualization can include displaying the recorded surroundings based on the LiDAR data enriched by the neural network in a display, which is, for example, included in the bridge 210 or connected to the bridge.

[0103] The classified information may be disclosed in the form of a 3D point cloud visualization with location information and category information. The location information may indicate the location of the object in the surrounding environment. The location information may be included in the LiDAR data or determined based on the LiDAR data. The category information may indicate to which category the object belongs.

[0104] The classified information may be disclosed in the form of a probability map. The probability map may include different colors or symbols to represent different probabilities.

[0105] The probability map can be a 2D point cloud visualization with location information and category information. The location information can indicate the location of the object in the surrounding environment. The location information can be included in the LiDAR data or determined based on the LiDAR data. The category information can indicate which category the object belongs to.

[0106] The probability map can be a 3D point cloud visualization with location information and category information. The location information can indicate the location of the object in the surrounding environment. The location information can be included in the LiDAR data or determined based on the LiDAR data. The category information can indicate which category the object belongs to.

[0107] 3D point clouds can be semantically classified using projection-based methods.

[0108] Each point in the visualization can be colored with the color of the class object, or represented using any of the other types of representations mentioned in this article.

[0109] Step 704

[0110] Based on the category object of the visualized object in the vicinity of the marine vessel 100, a decision may be made appropriate for the visualized object. The decision may be such that: If there is another marine vessel 500 within a predetermined distance, the speed of the marine vessel 100 may be triggered to be reduced to below a predetermined speed to avoid being too close to the other marine vessel 500, while, b. If, on the contrary, the dock 403 is recorded, the marine vessel 100 may be allowed to travel at a faster speed than the predetermined speed when approaching the dock 403 because the dock 403 is not a movable object compared to the other marine vessel 100 .

[0111] Step 705

[0112] An input is obtained from the bridge 210, ie an input for manually controlling the transmission system of the marine vessel 100. This may trigger the marine vessel to move accordingly.

[0113] Figure 8800 is a schematic diagram of a computer system 800 for implementing the examples disclosed herein. The computer system 800 is suitable for executing instructions from a computer-readable medium to perform these and / or any functions or processes described herein. The computer system 800 can be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, or the Internet. Although only a single device is shown, the computer system 800 may include any device collection that executes an instruction set (or multiple instruction sets) individually or jointly to perform any one or more of the methods discussed herein. Therefore, any reference to a computer system, a computing system, a computer device, a computing device, a control system, a control unit, an electronic control unit (ECU), a processor device, a processing circuit, etc. in the present disclosure and / or claims includes a reference to one or more such devices to execute an instruction set (or multiple instruction sets) individually or jointly to perform any one or more of the methods discussed herein. For example, the control system may include a single control unit or multiple control units connected to each other or otherwise communicatively coupled, so that any executed functions can be distributed between the control units as needed. In addition, such devices can communicate with each other or with other devices through various system architectures such as directly or via a controller area network (CAN) bus.

[0114] The computer system 800 may include at least one computing device or electronic device that can include firmware, hardware and / or execute software instructions to implement the functionality described herein. The computer system 800 may include a processing circuit 802 (e.g., a processing circuit including one or more processor devices or control units), a memory 804, and a system bus 806. The computer system 800 may include at least one computing device having a processing circuit 802. The system bus 806 provides an interface for system components including, but not limited to, the memory 804 and the processing circuit 802. The processing circuit 802 may include any number of hardware components for performing data or signal processing or for executing computer code stored in the memory 804. The processing circuit 802 may include, for example, a general-purpose processor, a special-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a circuit containing a processing component, a group of distributed processing components, a group of distributed computers configured for processing, or other programmable logic devices designed to perform the functions described herein, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The processing circuit 802 may also include computer executable code that controls the operation of the programmable device.

[0115] The system bus 806 can be any of several types of bus structures, which can also be interconnected to a memory bus (with or without a memory controller), a peripheral bus, and / or a local bus using any of a variety of bus architectures. The memory 804 can be one or more devices for storing data and / or computer code to complete or facilitate the methods described herein. The memory 804 can include a database component, an object code component, a script component, or any type of information structure for supporting various activities herein. Any distributed or local memory device can be utilized with the systems and methods of this specification. The memory 804 can be communicatively connected to the processing circuit 802 (e.g., via a circuit or any other wired, wireless, or network connection) and can include computer code for performing one or more processes described herein. The memory 804 may include nonvolatile memory 808 (e.g., read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.) and volatile memory 810 (e.g., random access memory (RAM)), or any other medium that can be used to carry or store desired program code in the form of machine-executable instructions or data structures and that can be accessed by a computer or other machine having the processing circuit 802. A basic input / output system (BIOS) 812 may be stored in the nonvolatile memory 808 and may include the basic routines that help to transfer information between elements within the computer system 800.

[0116] The computer system 800 may also include or be coupled to non-transitory computer-readable storage media such as storage device 814, which may include, for example, an internal or external hard disk drive (HDD) (e.g., enhanced integrated drive electronics (EIDE) or serial advanced technology attachment (SATA)), a HDD for storage (e.g., EIDE or SATA), flash memory, etc. The storage device 814 and other drives associated with computer-readable and computer-usable media may provide non-volatile storage of data, data structures, computer-executable instructions, etc.

[0117] The hard-coded or soft-coded computer code can be provided in the form of one or more modules. The module can be implemented as software and / or hard-coded in the circuit to implement the functionality described herein in whole or in part. These modules can be stored in a storage device 814 and / or a volatile memory that can include an operating system 816 and / or one or more program modules 818. All or part of the examples disclosed herein can be implemented as a computer program 820 stored on a temporary or non-temporary computer-usable or computer-readable storage medium (e.g., a single medium or multiple media) such as a storage device 814, which includes complex programming instructions (e.g., complex computer-readable program code) that cause the processing circuit 802 to perform the actions described herein. Therefore, the computer-readable program code of the computer program 820 may include software instructions for implementing the functionality of the examples described herein when executed by the processing circuit 802. In some examples, storage device 814 may be a computer program product (e.g., a readable storage medium) having computer program 820 stored thereon, wherein at least a portion of computer program 820 may be loadable (e.g., loaded into a processor) for implementing the functionality of the examples described herein when executed by processing circuit 802. Processing circuit 802 may serve as a controller or control system of computer system 800 for implementing the functionality described herein.

[0118] The computer system 800 may include an input device interface 822 configured to receive input and selections to be transmitted to the computer system 800 when executing instructions, such as from a keyboard, mouse, touch-sensitive surface, etc. Such input devices can be connected to the processing circuit 802 through an input device interface 822 coupled to the system bus 806, but can be connected through other interfaces (such as a parallel port, an Institute of Electrical and Electronics Engineers (IEEE) 1394 serial port, a universal serial bus (USB) port, an IR interface, etc.). The computer system 800 may include an output device interface 824, which is configured to forward output to a display, a video display unit (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 800 may include a communication interface 826 suitable for communicating with a network as appropriate or as needed.

[0119] A computer program product comprising program code for performing the method described herein when executed by a processing circuit 802. A non-transitory computer readable storage medium comprising instructions which, when executed by a processing circuit 802, cause the processing circuit 802 to perform the method described herein.

[0120] Embodiment 1: A marine environment sensing system for controlling a marine vessel (100), wherein the marine environment sensing system comprises: a light detection and ranging LiDAR sensor (201), the light detection and ranging LiDAR sensor being installed around the marine vessel (100) for recording the surrounding environment of the marine vessel (100), wherein the surrounding environment includes obstacles (303, 500, 503, 601) and water (305), - a control unit (205) having a neural network for processing information about the recorded surroundings which has been recorded by the LiDAR sensor, The control unit (205) is programmed to visualize the recorded surroundings based on the LiDAR data enriched by the neural network, wherein the recorded information has been classified into categories of objects in order to distinguish between different types of objects (303, 500, 503, 601) in the surroundings.

[0121] Embodiment 2: The marine environment sensing system according to embodiment 1 comprises: - a bridge (210) for visualizing the recorded surroundings and providing input for manually controlling the transmission system of the marine vessel (100),

[0122] Embodiment 3: The marine environment sensing system according to any one of embodiments 1 to 2, wherein the classified information is disclosed as a three-dimensional 3D point cloud visualization content having position information and category information.

[0123] Embodiment 4: The ocean environment sensing system according to any one of embodiments 1 to 3, wherein the classified information is disclosed as a probability map.

[0124] Embodiment 5: The marine environment sensing system according to embodiment 4, wherein the probability map is a three-dimensional 3D point cloud visualization content having location information and category information.

[0125] Embodiment 6: The marine environment sensing system according to embodiment 4, wherein the probability map is a two-dimensional 2D point cloud visualization content having location information and category information.

[0126] Embodiment 7: The ocean environment sensing system according to one of the preceding embodiments, wherein classification is performed using a projection-based method for semantic classification of three-dimensional 3D point clouds.

[0127] Embodiment 8: The ocean environment sensing system according to one of the preceding embodiments, wherein each point in the visualization content is colored with the color of a category object.

[0128] Embodiment 9: A marine environment sensing system according to one of the preceding embodiments, wherein the control unit (205) is arranged to make a decision suitable for the visual object (303, 500, 503, 601) near the marine vessel (100) based on the category object of the visual object.

[0129] Embodiment 10: The marine environment sensing system according to embodiment 9, wherein the control unit (205) is arranged to make a decision in the following manner: a. If there is another marine vessel (500) within a predetermined distance, the control unit (205) automatically reduces the speed of the marine vessel (100) to below a predetermined speed to avoid getting too close to the other marine vessel (500), and at the same time, b. If on the contrary a dock (303) is recorded, then when approaching the dock (303), the marine vessel (100) is allowed to drive faster than the predetermined speed under appropriate circumstances because the dock (303) is not a movable object compared to the other marine vessel (500).

[0130] Embodiment 11: A marine vessel (100), comprising a marine environment sensing system according to one of the preceding embodiments.

[0131] Embodiment 12: A computer-implemented method for controlling a marine vessel (100), the method comprising: Obtaining (300, 701) recorded information about the surroundings of the marine vessel (100), wherein the recording is performed by a light detection and ranging LiDAR sensor (201) installed around the marine vessel (100), wherein the surroundings include obstacles (303, 500, 503, 601) and water (305); processing (301, 302, 303, 702) information about the recorded surrounding environment that has been recorded by the LiDAR sensor using a neural network; and The recorded surroundings are visualized (304, 703) based on the LiDAR data enriched by the neural network, wherein the recorded information has been classified into categories of objects in order to distinguish between different types of objects (303, 500, 503, 601) in the surroundings.

[0132] Embodiment 13: The method according to embodiment 12, wherein the classified information is disclosed as 3D point cloud visualization content having location information and category information.

[0133] Embodiment 14: The method according to any one of embodiments 12 to 13, wherein the classified information is disclosed as a probability map.

[0134] Embodiment 15: The method according to embodiment 14, wherein the probability map is a 2D point cloud visualization content having location information and category information.

[0135] Embodiment 16: The method according to embodiment 14, wherein the probability map is a 3D point cloud visualization content having location information and category information.

[0136] Embodiment 17: The method according to any one of embodiments 12 to 16, wherein the classification is performed using a projection-based method for semantic classification of 3D point clouds.

[0137] Embodiment 18: The method according to any one of embodiments 12 to 17, further comprising:

[0138] A decision is made (704) appropriate for the visualization object based on the category object of the visualization object in the vicinity of the marine vessel (100).

[0139] Embodiment 19: A computer program product comprising program code for performing the method according to any one of embodiments 12 to 18 when executed by a processing circuit (802).

[0140] Embodiment 20: A non-transitory computer-readable storage medium comprising instructions which, when executed by a processing circuit (802), cause the processing circuit (802) to perform the method according to any one of Embodiments 12 to 18.

[0141] The operational actions described in any exemplary aspects of this article are described to provide examples and discussions. These actions can be performed by hardware components, can be embodied in machine executable instructions to enable a processor to perform these actions, or can be performed by a combination of hardware and software. Although a specific order of method actions can be shown or described, the order of actions can be different. In addition, two or more actions can be performed simultaneously or partially simultaneously.

[0142] It should be understood that the present disclosure is not limited to the embodiments described above and shown in the drawings; rather, those skilled in the art will recognize that many changes and modifications may be made within the scope of the appended claims.

[0143] The terms used herein are only for the purpose of describing specific aspects and are not intended to be limitations of the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms as well. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. It should also be understood that the term "comprises / comprising / includes and / or including" when used herein indicates the presence of stated features, integers, actions, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, actions, steps, operations, elements, parts and / or their groups.

[0144] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the present disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0145] Relative terms such as "below" or "above" or "upper" or "lower" or "horizontal" or "vertical" may be used herein to describe the relationship of one element to another element, as shown in the figures. It should be understood that these terms and those discussed above are intended to cover different device orientations in addition to the orientations depicted in the figures. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, the element may be directly connected or directly coupled to the other element, or there may be intervening elements. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements.

[0146] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present disclosure belongs. It should also be understood that, unless otherwise clearly defined herein, the terms used herein should be interpreted as meanings consistent with their meanings in the context of this specification and the relevant art, and should not be interpreted in an idealized or overly formal sense.

[0147] It should be understood that the present disclosure is not limited to the aspects described above and shown in the accompanying drawings; rather, the skilled person will recognize that many changes and modifications may be made within the scope of the present disclosure and the appended claims. In the drawings and description, various aspects have been disclosed for illustrative purposes only and not for limiting purposes, and the scope of the disclosure is set forth in the appended claims.

Claims

1. A marine environment sensing system for controlling a marine vessel (100), wherein the marine environment sensing system comprises: a light detection and ranging LiDAR sensor (201), the light detection and ranging LiDAR sensor being installed around the marine vessel (100) for recording the surrounding environment of the marine vessel (100), wherein the surrounding environment includes obstacles (303, 500, 503, 601) and water (305), - a control unit (205) having a neural network for processing information about the recorded surroundings which has been recorded by the LiDAR sensor, The control unit (205) is programmed to visualize the recorded surroundings based on the LiDAR data enriched by the neural network, wherein the recorded information has been classified into categories of objects in order to distinguish between different types of objects (303, 500, 503, 601) in the surroundings.

2. The marine environment sensing system according to claim 1, comprising: - A bridge (210) for visualizing the recorded surroundings and providing input for manually controlling the transmission system of the marine vessel (100).

3. The marine environment sensing system according to any one of claims 1 to 2, wherein the classified information is disclosed as a three-dimensional 3D point cloud visualization content having position information and category information.

4. The marine environment sensing system according to any one of claims 1 to 3, wherein the classified information is disclosed as a probability map. 5 . The marine environment sensing system according to claim 4 , wherein the probability map is a two-dimensional (2D) point cloud visualization content having location information and category information. 6 . The marine environment sensing system according to claim 4 , wherein the probability map is a three-dimensional 3D point cloud visualization content having location information and category information.

7. The marine environment sensing system according to one of the preceding claims, wherein the classification is performed using a projection-based method for semantic classification of three-dimensional 3D point clouds.

8. The marine environment sensing system according to one of the preceding claims, wherein each point in the visualization is colored with the color of a category object.

9. A marine environment sensing system according to one of the preceding claims, wherein the control unit (205) is arranged to make a decision suitable for the visual object (303, 500, 503, 601) in the vicinity of the marine vessel (100) depending on the category object of the visual object.

10. The marine environment sensing system according to claim 9, wherein the control unit (205) is arranged to make a decision in the following manner: c. If there is another marine vessel (500) within a predetermined distance, the control unit (205) automatically reduces the speed of the marine vessel (100) to below a predetermined speed to avoid getting too close to the other marine vessel (500), and at the same time, d. If on the contrary a dock (303) is recorded, then when approaching the dock (303), the marine vessel (100) is allowed to drive faster than the predetermined speed under appropriate circumstances because the dock (303) is not a movable object compared to the other marine vessel (500).

11. A marine vessel (100) comprising a marine environment sensing system according to one of the preceding claims.

12. A computer-implemented method for controlling a marine vessel (100), the method comprising: Obtaining (300, 701) recorded information about the surroundings of the marine vessel (100), wherein the recording is performed by a light detection and ranging LiDAR sensor (201) installed around the marine vessel (100), wherein the surroundings include obstacles (303, 500, 503, 601) and water (305); processing (301, 302, 303, 702) information about the recorded surrounding environment that has been recorded by the LiDAR sensor using a neural network; and The recorded surroundings are visualized (304, 703) based on the LiDAR data enriched by the neural network, wherein the recorded information has been classified into categories of objects in order to distinguish between different types of objects (303, 500, 503, 601) in the surroundings.

13. The method of claim 12, wherein the classified information is disclosed as 3D point cloud visualization content having location information and category information.

14. The method according to any one of claims 12 to 13, wherein the classified information is disclosed as a probability map. The method according to claim 14 , wherein the probability map is a 2D point cloud visualization content having location information and category information. The method according to claim 14 , wherein the probability map is a 3D point cloud visualization content having location information and category information.

17. The method according to any one of claims 12 to 16, wherein the classification is performed using a projection-based method for semantic classification of 3D point clouds.

18. A method according to any one of claims 12 to 17, comprising: A decision is made (704) appropriate for the visualization object based on the category object of the visualization object in the vicinity of the marine vessel (100).

19. A computer program product comprising program code for performing the method according to any one of claims 12 to 18 when executed by a processing circuit (802).

20. A non-transitory computer-readable storage medium comprising instructions which, when executed by a processing circuit (802), cause the processing circuit (802) to perform the method of any one of claims 12 to 18.

Images