Assisting an operator during automatic control of a vessel

Abstract

The invention relates to a method for assisting an operator during automatic control of a vessel, where the vessel (V1) is controlled to move according to a present control scheme in a present passage along a route towards a destination. The method is performed by an operator assisting device and comprises displaying (S100) the progress of the vessel (V1) in the present passage, obtaining (S110) an actual or proposed change in the operation of the vessel (V1) from the present control scheme; obtaining (S120) data about the cause for the change of operation; and linking (S130) the change in operation and the cause of the change to the present passage or to contextual data of the route and presenting them (S150) to the operator in at least one display item that is linked to the present passage or to the contextual data of the route.

Description

TECHNICAL FIELD

[0001] The invention relates to a method, operator assisting device, computer program and computer program for assisting a user during the automatic control of a vessel as well as to a vessel comprising such an operator assisting device.BACKGROUND

[0002] In some automation scenarios, vessels are equipped to make autonomous decisions under the supervision of human operators. For instance, a vessel navigating system may continuously determine an optimal manoeuvring strategy, taking into account factors like traffic, environmental constraints, and the need for energy efficiency. Even though these autonomous decisions are made within a predefined operational envelope established by human operators, there are times when human operator intervenes to deviate the plan, or critical decisions are made to make automation behave outside of original operational envelope. Additionally unforeseen events such as system malfunction, maintenance requirements delivered from the engine department, or unexpected situations like oil leakage information, floating containers, or search and rescue activities received via SMS, emails or VHF calls, can also influence navigational decisions.

[0003] Such changes of the operation will thus affect the operation of the vessel.

[0004] Capturing and understanding these deviations, anomalies, and events, including the reasons behind them or their future navigational implications, becomes very important. It is not just for seamless handovers between shifts but can also serve for other purposes such as analysing operational efficiency and safety, improving qualities of the operational envelopes or machine learning performances, or enhancing training procedures for the human users.

[0005] Logging of events is known for instance from CN113806319. CN113806319 discloses a navigation log electronization method and a system, where the method comprises: inputting an obtained log event and parameters related to the log event into a constructed finite-state machine model according to a time sequence in a navigation process; adopting the finite-state machine model to process the log event to obtain log information needing to be recorded, updating the log state, and storing the log event and the log information; and displaying the recorded log information according to a specified format. The navigation log is recorded electronically, and conveniently utilized to display the chart operation on an electronic chart.

[0006] There is however a need for an improvement in the field when the automation level increases for ship operations. More, particularly, there exists a need for an improvement in the way that an operator is informed of a change in operation.SUMMARY

[0007] One object of the invention is therefore to assist an operator of a vessel during the automatic control of the vessel.

[0008] This object is according to a first aspect achieved through a method for assisting an operator during automatic control of a vessel, wherein the vessel is controlled to move according to a present control scheme in a present passage along a route towards a destination, the method being performed by an operator assisting device and comprising: displaying the progress of the vessel in the present passage, obtaining an actual or proposed change in the operation of the vessel from the present control scheme; obtaining data about the cause for the change of operation; and linking the change in operation and the cause of the change to the present passage or to contextual data of the route and presenting them to the operator in at least one display item that is linked to the present passage or to the contextual data of the route.

[0009] The object is according to a second aspect achieved through an operator assisting device for assisting an operator during automatic control of a vessel, where the vessel is controlled to move according to a present control scheme in a present passage along a route towards a destination, the operator assisting device comprising a processor operative to: display the progress of the vessel in the present passage, obtain an actual or proposed change in the operation of the vessel from the present control scheme; obtain data about the cause for the change of operation; and link the change in operation and the cause of the change to the present passage or to contextual data of the route and present them to the operator in at least one display item that is linked to the present passage or to the contextual data of the route.

[0010] The object is according to a third aspect achieved through a vessel comprising the operator assisting device according to the second aspect.

[0011] The object is accord to a fourth aspect achieved through a computer program for assisting an operator during automatic control of a vessel, wherein the vessel is controlled to move according to a present control scheme in a present passage along a route towards a destination, the computer program comprising computer program code which when run by a processor of an operator assisting device causes the processor to: display the progress of the vessel in the present passage, obtain an actual or proposed change in the operation of the vessel from the present control scheme; obtain data about the cause for the change of operation; and link the change in operation and the cause of the change to the present passage or to contextual data of the route and present them to the operator in at least one display item that is linked to the present passage or to the contextual data of the route.

[0012] The object is according to a fifth aspect achieved through a computer program product for assisting an operator during automatic control of a vessel, where the computer program product comprises a data carrier with the computer program according to the fourth aspect.

[0013] The change in operation may be a change carried out for a group of waypoints along the route. There may additionally be a control goal associated with the present control scheme.

[0014] In this case the method may further comprise determining an adjustment of the present control scheme applicable after the group of waypoints have been passed in order to fulfil the control goal.

[0015] In this case the operator assisting device may be further operable to determine an adjustment of the present control scheme in order to fulfil the control goal, which adjustment is applicable after the group of waypoints have been passed.

[0016] The change in operation may additionally comprise a change of an operational envelope, such as a speed envelope, acceleration envelope or heading envelope.

[0017] In a further variation of the first aspect, the method further comprises receiving communication data, such as radio, AIS and NAVTEX data and analysing the communication data.

[0018] In a corresponding variation of the second aspect, the operator assisting device may be further operable to receive communication data, such as radio, AIS and NAVTEX data and analyse the communication data.

[0019] The cause for change in operation may have been determined based on the communication data.

[0020] The cause for change in operation may be determined by the operator based on the communication data. It is additionally possible that the operator determines the cause for change in operation based on the analysis of the communication data.

[0021] Alternatively, the operator assisting device may determine the cause for change in operation based on the analysis of the communication data.

[0022] It is additionally possible that the change in operation is determined based on the communication data. It may additionally have been determined based on the analysis of the communication data.

[0023] The determination of the change in operation may be made by the operator.

[0024] In this case the obtaining of an actual or proposed change in operation of the method may comprise detecting an operator -determination of a change of the present control scheme and implementing the change in operation.

[0025] In this case the operator assisting device may be further operative to, when obtaining an actual or proposed change in operation, detect an operator-made determination of a change of the present control scheme and implement the change in operation.

[0026] Alternatively, the determination of the change in operation may be made by the operator assisting device.

[0027] In this case the obtaining of an actual or proposed change in operation of the method may further comprise determining at least one change in operation the vessel, providing the at least one change in operation to the operator as a proposed change, receiving a determination of a change in operation from the operator as a confirmation that the proposed change is to be implemented and implementing the confirmed change in operation.

[0028] In this case the operator assisting device may be further operative to, when obtaining an actual or proposed change in operation, determine at least one change in operation of the vessel, provide the at least one change in operation to the operator as a proposed change, receive a determination of a change in operation from the operator as a confirmation that the proposed change is to be implemented and implement the confirmed change in operation.

[0029] The at least one display item may comprise an event notification being linked to a waypoint at which the change in operation is determined.

[0030] The obtaining of an actual or prosed change in operation of the method may comprise receiving, at a first point in time, a suggestion from the operator of a future change in operation.

[0031] The operator assisting device may be further operative to, when obtaining an actual or proposed change in operation, receive, at a first point in time, a suggestion from the operator of a future change in operation.

[0032] Alternatively, the obtaining of an actual or proposed change in operation of the method may comprise providing, at a first point in time, a suggestion of a future change in operation.

[0033] In this case, the operator assisting device may be further operative to, when obtaining an actual or proposed change in operation, provide, at a first point in time, a suggestion of a future change in operation.

[0034] The future change in operation may be suggested to be made at a future point in time, i.e. at a time later than the first point in time.

[0035] The at least one display item may additionally comprise an annotation and the suggestion of the future change in operation may be presented in the annotation, where the annotation is linked to a future point of the route, which future point may or may not coincide with a waypoint at the future point in time.

[0036] It is possible to combine a determined change in operation being presented in an event notification with a proposed change in operation being presented in an annotation.

[0037] In this case the change in operation may additionally be determined at a second point in time and the annotation may be linked to the event notification.

[0038] Here, the second point in time may be after the first point in time. It may additionally or instead be before the future point in time.

[0039] In a further variation, the method may additionally comprise obtaining contextual data of the route through obtaining a determination of a zone of interest at or around at least a part of the route, where the proposed change in operation is made in order to consider the zone and the annotation is linked to the zone.

[0040] In this further variation the operator assisting device may be further operative to obtain contextual data of the route through obtaining a determination of a zone of interest at or around at least a part of the route, where the proposed change in operation is made in order to consider the zone and the annotation is linked to the zone.

[0041] The zone of interest may be determined by the operator. Alternatively, the zone of interest may be obtained through being determined by the operator assisting device. The zone of interest may also be determined based on an analysis of received communication data, for example, by analysing and extracting coordinates from VHF call languages.

[0042] The communication data may be communication data of the present passage along the route.

[0043] It should be emphasized that the term "comprises / comprising" when used in this specification is taken to specify the presence of stated features, steps or components, but does not preclude the presence or addition of one or more other features, steps, components or groups thereof.BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The invention will now be described in more detail in relation to the enclosed drawings, in which: fig. 1 schematically shows a first way of realizing an operator assisting device, fig. 2 shows a computer program product in the form of a CD ROM with computer program code used to implement the operator assisting device, fig. 3 schematically shows a first vessel comprising the operator assisting device, fig. 4 shows a flow chart of a number of method steps in a method of assisting an operator during automatic control of the first vessel, fig. 5 shows a navigational chart of a part of a route along which the first vessel travels together with a box where an operator can change an operational envelope of the first vessel and a box for input of data about the cause for change, fig. 6 shows a presentation of an operator change of the operation of the first vessel, the cause, consequences of the change and additional data in an event notification linked to a waypoint at which the operator made the change, fig. 7 shows a presentation of an event notification concerning a change of operation of the first vessel, the cause, consequences of the change and additional data linked to a waypoint at which the change was implemented, fig. 8 shows an annotation made by the operator linked to a future waypoint of the route as well as to a zone of interest, fig. 9 shows an annotation made by the operator assisting device linked to the waypoint of the route and the zone of interest, fig. 10 shows an event notification linked to another waypoint as well as to the annotation of fig. 9, and fig. 11 shows a highlighting of parts of the route where the first vessel has an operation that deviates from a normal operation. DETAILED DESCRIPTION

[0045] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, interfaces, techniques, etc. in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known devices, circuits and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0046] Fig. 1 schematically shows one realization of an operator assisting device OAD 10. In the present example, the operator assisting device 10 comprises a processor PR 12 and a data storage or memory 14 with computer program instructions or computer program code 16 that, when executed by the processor 12, implements an operator assisting function. There is also a communication interface CI 18. The communication interface 18 may have a computer communication section CCS 18A and / or a radio communication section RCS 18B, where the computer communication section 18A may be an Ethernet interface for connection to a local area network (LAN) for instance a LAN provided in a vessel and the radio communication section 18B may comprise a transceiver for transmission and reception of radio signals. It should be realized that it is possible that the communication interface 18 only comprises the computer communication section 18A and not the radio communication section 18B or vice versa.

[0047] The operator assisting device 10 may thus comprise a processor 12 with associated program memory 14 including computer program code 16 for implementing the operator assisting function.

[0048] A computer program may also be provided via a computer program product, for instance in the form of a computer readable storage medium or data carrier, like a CD ROM or a memory stick, carrying such a computer program with the computer program code, which will implement the operator assisting function when being loaded into a processor. Such a storage medium may be a non-transitory storage medium. One such computer program product in the form of a CD ROM 20 with the above-mentioned computer program code 16 is schematically shown in fig. 2.

[0049] The operator assisting device 10 may with advantage be provided in a vessel.

[0050] Fig. 3 schematically shows a first vessel V1 22 comprising the operator assisting device OAD 10. As can be seen in the figure, the first vessel 22 also comprises a first sensor S1 24, a second sensor 26, a vessel navigation control device VNCD 28 that implements a vessel navigation control function VNCF 30 and a display D 34. The computer communication section of the previously described communication interface may here be used for communicating with the vessel navigation control device 28, the first and second sensors 24, 26 and the display 34. For this reason, these may all be connected to a LAN.

[0051] The first sensor 24 may be a sensor for sensing weather conditions around the first vessel 22. Thereby the first sensor 24 may be a wind sensor that senses the speed and / or direction of the wind. The second sensor 26 may comprise a sensor sensing surrounding traffic, such as an AIS (Automatic Identification System) transceiver. Alternatively or instead, it can comprise a radar or a Lidar. It can also comprise a radio unit for receiving VHF (Very High Frequency) and / or UHF (Ultra High Frequency) radio traffic. AIS and VHF radio also allow the reception of VTS (Vessel Traffic Service) data, which data may be displayed using navigation equipment, such as an Electronic Chart Display and Information System (ECDIS).

[0052] There may also be other sensors for sensing the vessel position, heading and speed. Alternatively, vessel position, heading and speed may be obtained from the vessel navigation control device 28 together with other vessel operations data such as data about when a change of heading and speed is made and data about commands effecting such change. Also, other sensors may be provided.

[0053] Aspects of the present disclosure are directed towards assisting an operator of the vessel during automatic control of the vessel. Aspects of the present disclosure may more particularly be directed towards assisting an operator in understanding and managing the operations of the vessel, including past operations, planned future operations, and during automatic control.

[0054] In some automation scenarios, vessels are equipped to make autonomous decisions under the supervision of human operators. There are times when interventions are needed to be made both by the system and the operator.

[0055] Capturing and understanding these deviations, anomalies, and events, including the reasons behind them or their future navigational implications, may be important. It is not just for seamless handovers between shifts but can also serve for other purposes such as analysing operational efficiency and safety, improving qualities of the operational envelopes or machine learning performances, or enhancing training procedures for the human users.

[0056] Aspects of the present disclosure are directed towards providing improvements in relation to such changes of the automatic control. Aspects of the present disclosure may more particularly be directed towards enhancing the operator's understanding of deviations, anomalies, and events in automatic control, including the reasons behind them or their implications for future operations, thereby supporting informed decision-making and improving overall operational awareness.

[0057] How this may be done will now be described with reference being made to fig. 4, 5 and 6, where fig. 4 shows a flow chart of a number of method steps in a method of assisting a user during automatic control of the first vessel, fig. 5 shows a navigational chart together with a first box in which operational changes can be made and a second box in which the cause of the operational changes can be added and fig. 6 shows the navigational chart together with an event notification showing the operator change, the cause, consequences of the change and additional data linked to a waypoint at which the operator made the change.

[0058] The first vessel V1 22 may travel along a route from a point of origin to a point of destination (not shown). This travel may be made in a present passage along the route. During this present passage the vessel navigation control device 28 may control the first vessel 22 according to a present control scheme. Thus, according to this control scheme, the first vessel V1 follows a route between an origin and a destination. The control may also have operational envelopes within which the automatic control is allowed to be made, which control envelopes may be set by an operator. As an example, shown in fig. 5 and 6, one operational envelope may be speed. This means that the speed of the automatic control may be allowed to vary in a speed interval that is set by the operator. In the example in fig. 5 the interval is between 7.0 and 10.0 knots (kn). It is of course also possible with other operational envelopes, such as acceleration and heading. There may also exist control objectives or control goals associated with the present control scheme that need to be met. One control goal may be that the first vessel 22 has to arrive at the destination at an estimated time of arrival ETA. There may also be a fuel consumption or emissions control goals that need to be met. If a control goal is not met for a certain operational envelope, the operator may be prompted to make changes to it so that the control goal is met.

[0059] Thus the operation of the first vessel V1 in the present passage along the route may be controlled by the navigation control device 28 in a present control scheme, where the operator merely is in standby and observing the passage.

[0060] In order for the operator to be able to better monitor the operation, it is possible that the operator assisting device 10 presents the present passage on the display 34. The progress of the first vessel V1 in the present passage along the route may thus be displayed, S100, where the first vessel V1 is controlled for fulfilling one or more of the previously mentioned control goals.

[0061] The presenting may be made using a graphical symbol of the first vessel V1 following the route on a navigational chart, for instance in the way shown in fig. 5 and 6, which shows a part of the navigational chart with a part of the route and the first vessel V1. The route may comprise a number of waypoints, where as an example there is a first waypoint WP1, a second waypoint WP2 and a fourth waypoint WP4. There is also a third waypoint between the second and fourth waypoints WP2, WP4, which third waypoint has however been omitted from the chart.

[0062] The operator assisting device 10 may monitor the navigation control of the vessel. It may for instance monitor that the vessel navigation control function 30 controls the vessel according to the present control scheme. It may more particularly generate an event if the control scheme is not being followed or if a suggestion is made that the scheme is not to be followed. It may obtain an actual or proposed change in the operation of the vessel from the present control scheme, S110. This may be done through the operating assisting device 210 detecting an actual or proposed change in the present control scheme, S110.

[0063] The obtaining of a proposed change in operation may additionally comprise the operator assisting device 10 proposing the change in operation.

[0064] A change may be obtained in case an operational envelope of the present control scheme is adjusted or proposed to be adjusted for one or more waypoints of the route, such as a change of a speed envelope, an acceleration envelope or a heading envelope. A change may be made through the operator setting a different operational envelope for these waypoints. Alternatively, the operator assisting device 10 may suggest one or more changes to the control scheme, such as one or more changes to a number of waypoints, and a change may be made in case the operator accepts one of the suggested changes. The operator assisting device 10 may generate an event when a change is detected.

[0065] The change of operation may be made for a number of reasons. The change in operation and the reason for the change may additionally be determined based on radio communication received in the first vessel.

[0066] For this reason, the operator assisting device 10 may receive communication data, such as radio, AIS, VTS and NAVTEX (Navigational Telex) data. It may additionally analyse the communication data.

[0067] The change in operation may be determined based on the communication data. It may additionally have been determined based on the analysis of the communication data. The determination of the change in operation may be made by the operator.

[0068] In this case the obtaining of an actual or proposed change in operation may comprise detecting an operator made determination of a change of the present control scheme and implementing the change in operation.

[0069] Alternatively, the determination of the change in operation may be made by the operator assisting device.

[0070] In this case, the operator assisting device 10 may suggest a change in operation, for instance if the vessel navigation control function 30 cannot achieve certain pre-defined performance goals within existing operational envelopes / constraints, or if there is a system failure or malfunction that could potentially influence the navigation.

[0071] A change may thus be made if the operator changes a control envelope of the present control scheme and a change may be proposed if the operator assisting device suggests a change. The proposed change may also be made into an actual change if the proposed change is accepted by the operator.

[0072] The corresponding navigational decision or adopted system recommendation or system malfunction reasons may be logged. It also possible that contextual information of such events such as reasons for such an operational change, who confirms the change, time, location and environmental or traffic conditions are logged by the operator assisting device 10.

[0073] Thus, an actual or proposed change in the operation of the vessel is obtained, S110, where an actual change may be a change made by the operator and a proposed change may be a change suggested by the operator assisting device 10. A proposed change can also be made into an actual change if confirmed by the operator.

[0074] Thus, in case the operator assisting device 10 proposes the change in operation, the obtaining of an actual or proposed change in operation may comprise determining at least one change in operation of the first vessel V1, providing the at least one change in operation to the operator as a proposed change, receiving, from the operator, a determination of a change in operation as a confirmation that the proposed change is to be implemented and implementing the confirmed change in operation.

[0075] The change may be applicable for a group of waypoints along the route, such as from the second waypoint WP2 until the fourth waypoint WP4. Alternatively, it may be applicable for a number of future waypoints, such as a number of later waypoints WP12 - WP17. However, it should also be known that the change does not have to be restricted to predefined waypoints but can be applied at any point along the route. The change in operation may be a change carried out. There may additionally be a control goal associated with the present control scheme.

[0076] There is also an obtaining of data about the cause for the change of operation, S120. There may thus be an obtaining of data about the cause of the actual or proposed change in operation. If the operator has made an actual change, then he or she may be asked to enter such data via a prompt. He or she may also talk about the cause, where the speech of the operator is transcribed into text. Additionally or instead a voice recording may be made of the speech.

[0077] The cause for change in operation may have been determined based on the communication data. The cause for change in operation may be determined by the operator based on information in the communication data. It is additionally possible that the operator determines the cause for change based on the analysis of the communication data.

[0078] Alternatively, the operator assisting device may determine the cause for change based on the analysis of the communication data.

[0079] It thus also possible that the operator assisting device estimates the cause. If for instance an AIS or VTS message is received before the actual or proposed change in operations is detected, the user assisting device may assume that the cause can be found in such a message and deduce such data from the message. A VHF radio transmission being received may likewise be assumed to include the cause. Such a message may also be transcribed, analysed and the cause deduced through the analysis. The cause may be traffic crossing the path, forecasts about bad weather, search and rescue operations as well as oil spills in an area at or around at least a part of the route.

[0080] Thus, when the human operator intervenes or alters the operational envelope for reasons not known to the automation, the operator assisting device 10 can optionally prompt the operator for rationale. The reason for the change can be verbally communicated, transcribed, and then attached to the decision that the human operator makes, as a decision-making annotation, a part of an event notification, or the reasons can be linked to an existing annotation.

[0081] Human-made annotations can be further used for machine learning, training or operation optimisation purposes. If the operator accepts the recommendation for the adjustment of the automation operational envelope, the rationale should also be logged. It is possible that the cause is deduced using machine-learning.

[0082] The operator assisting device 10 may additionally determine consequences of the change of operations and especially the impact on the control goals of the change in operation is implemented. It may also determine ways to adjust the control scheme in order to fulfil the control goals. The operator assisting device 10 may thus determine an adjustment of the control scheme that fulfils the control goal, S140.

[0083] The actual or proposed change in operation, the cause of the actual or proposed changed in operation and optionally also the proposed adjustment of the control scheme may then be linked to the present passage and / or contextual data of the route, S130, which change, course and perhaps also the proposed adjustment are then presented to the operator in at least one display item that is linked to the present passage or to the contextual data of the route. The at least one display item may comprise an event notification and / or an annotation.

[0084] An event may be generated in a log of the passage if the operator decides to intervene, or if the operator assisting device proposes a change and the operator accepts. In case there is only a proposed change, this may be made as an annotation.

[0085] If the change of operation is actually made by the operator, the cause may be provided in an event item or event notification WN together with the change of operation, which event notification EN is created by the operator assisting device 10 based on the detected change in operations. Additionally or instead, the cause may be provided, possibly together with a change of operation, in an annotation, where an annotation may be provided in case the change is only proposed and not implemented or as an annex to an event item.

[0086] An event item or annotation may be linked to a present waypoint at which the first vessel V1 and operator is presently located when the event or annotation is created. It may more particularly be linked to a waypoint at which the vessel is located when the change in operation is determined. It may also be linked to another waypoint, such as a waypoint at which the actual implemented or proposed change of operations is to be started or ended.

[0087] The impact of the change in operation or the consequences of the change in operation may also be included in the event item and / or the annotation.

[0088] Also other information may be included in the annotation or event item, such as the duration of the suggested or determined change, the time when it is to be implemented, minimum closest point to approach (CPA) to surrounding vessels, collision risk and wind speed.

[0089] Additionally or instead an annotation or event item may be linked to contextual information of the route, such as an area or zone of interest close to or around a number of waypoints of the route, such a zone of interest may additionally be a hazard zone.

[0090] Thus, the operator assisting device 10 may further obtain contextual data of the route through a determination of a zone of interest at or around at least a part of the route. If such a zone is determined, the proposed change in operation may be made in order to consider the zone.

[0091] The zone may be determined by the operator. Alternatively, it is possible that the operator assisting device 10 determines the zone, for instance based on analysing the previously mentioned communication data.

[0092] As can be seen in fig. 5 and 6, the event or annotation may also be linked to an interactive timeline. The timeline may display a chronological sequence of events and annotations. The operator may click on an event or annotation indication on the timeline to access detailed information and its impact and implications. There may also be a slider SL on the timeline that can be slid forward and backwards. Normally the slider indicates the present point in time. The movement of the slider may additionally cause replay of previous positions and anticipated future positions of the first vessel V1 along the route.

[0093] The event item and / or the annotation may then be displayed to the operator, S150, where an event item may be displayed linked to the interactive timeline, to the waypoint at which the first vessel V1 is located when the event is generated and / or to the waypoint at which the change in operation is set to start. If the cause is linked to a zone, such as a hazard zone, the annotation and / or event item may additionally or instead be linked to this zone. If there is both an annotation and an event item, the annotation may additionally or instead be linked to the event item.

[0094] The display of an event item and / or annotation may also be made based on the operator moving the slider to the time at which the first vessel V1 passes the waypoint to which the annotation / event item is linked.

[0095] In case there is actually a change in operation, the vessel navigation control device 28 is then instructed to return to operating according to the present control scheme, which return to the present control scheme is made after the passing of the last waypoint for which the change was in place. Thus, there is a return to the present control scheme, S160. After the change in operation has ceased, the operator assisting device 10 may also instruct the vessel navigation control device 28 to adjust, perhaps after confirmation by the operator, the present control scheme so that the control goal is fulfilled. If the control goal is that the destination is to be reached at the estimated time of arrival, the change in operation may involve increasing the speed for one or more later waypoints so that the destination can be reached at the estimated time of arrival. The adjustment of the present control scheme may additionally be applicable after the group of waypoints have been passed in order to fulfil the control goal. It should thus be applied after the group of waypoints, like after the second to fourth waypoints WP1 - WP4.

[0096] Some more examples will now be given.

[0097] A first examples is given with reference to fig. 5 and 6.

[0098] As can be seen in fig. 5, the first vessel V1 may be located at the first waypoint WP1 of the route at a present point in time, which time as an example may be the time 14.29. The route comprises the further waypoints WP2 - WP4 after this first waypoint WP1, including the second, third and fourth waypoints. As an example, there may also be other vessels around the route or close to other waypoints of the route. In this example there is a first surrounding vessel SV1, a second surrounding vessel SV2 and a third surrounding vessel SV3 close to the route, here north of the second to fourth waypoints WP2 - WP4 of the route.

[0099] In this situation, the operator may decide to change the operational speed envelope. There may thus be an operator-made change of an operational envelope OCE at the first waypoint WP1 associated with the present point in time. The operator may for instance select a display item in which a present range of the speed envelope or speed threshold in knots (Kn) is shown, which display item may be a box that is shown on top of the navigational chart of the route. The present speed interval or the speed envelope may as an example be 7.0 - 10.0 knots, which the operator thus changes. The operator may thus change this interval, through specifying a speed envelope change. The change may additionally be made in respect of future waypoints of the route. An example the operator may specify that the speed envelope change is to be 6.0 -7.0 knots, which is to be applied between the second and fourth waypoints WP2 - WP4.

[0100] As the operator makes such a change, he or she may also be prompted to provide the reason or cause for the change. These rationales may be based on information not directly captured and analysed by the first vessel, such as a text message (SMS) received by the human operator informing him or her of certain navigational hazards that have not yet been updated in the corresponding navigational systems.

[0101] The operator may for instance enter text making clear that the surrounding vessels VS1 and VS2, which may be some leisure boats, have a behaviour that is strange.

[0102] The change in operation may then be displayed in an event item or event notification EN, which event item may also comprise the cause. One example of this is shown in fig. 6, which shows the navigational chart when the first vessel V1 has moved to the second waypoint WP2. The present point in time is then the point in time at which the first vessel V1 is at this waypoint WP2 and is in this case the time 15:01. As can be seen there is also an event notification linked to the first waypoint WP1, showing the change in the speed envelope from 7.0 -10.0 knots to 6.0- 7.0 knots, where the change in operation was decided at the time 14:29, i.e. the time the first vessel was at the first waypoint WP1, and is to be started at 15:01, i.e. the present point in time, which in this case is the time when the first vessel V1 and the operator is at the second waypoint WP2. The duration of the change is between the second and fourth waypoints WP2 - WP4. The event notification EN also comprises the cause of the change, which may have been entered as text by the operator. The operator may also have made a speech recording that is then transcribed and entered as text in the notification. It is also or instead possible that a sound file of the recorded speech with an explanation of the cause is attached to the event notification EN, which sound file can be selected in order to be replayed.

[0103] As can be seen, the consequences of the change are also shown in the event notification as an impact. The impact can as an example be that the control goal is fulfilled through a compensation being made at a later stage. In the present example the control goal of reaching the destination at the estimated time of arrival ETA is achieved through a slight speed increase between a twelfth and a seventeenth waypoint WP12 - WP17 with a consequential increase in fuel consumption fc of 2%. Also other data is included, such as minimum Closest Point of Approach (CPA) to the third surrounding vessel SV3 being 1.5 nautical miles (nm), the risk for collision being low and the wind speed being 2 m / s.

[0104] The waypoint, here the first waypoint WP1, at which the operator made the envelope change UCE may also be indicated on the timeline.

[0105] Fig. 6 thus shows an example of a logged operator-initiated control event. When the operator decided to change the existing operational envelope for speed between the second and fourth waypoints WP2 - WP4 due to reasons that are so far unknown to the operator assisting device 10 (because the operator saw some leisure boats that were doing risky manoeuvring nearby), the operator assisting device 10 would log such events with the decisions that the operator made. The operator can click the "event indicator" in the event timeline or use the slider to see the logged information, including optionally provided rationale with voice and transcribed text messages for providing the reasons for the operational envelope change, impact analysis, how the user intervened by manipulating the operational envelopes (e.g., speed) for automation, and environmental information such as traffic and wind.

[0106] Fig. 7 shows another example, when the first vessel V1 is at the second waypoint WP2 and an event notification EN was generated at the first way point WP1 at the time 14:29 with a change in operational envelope to be implemented between waypoints WP2 - WP4 at the present point in time of 15:01. In this case the operator assisting device 10 has suggested one or more changes of the speed envelope and the operator accepted a change from 7 - 10 knots to 6 - 7 knots.

[0107] It can be seen that the third surrounding vessel SV3 is expected to cross the route before the fourth waypoint WP4. This may additionally lower the CPA to 0.5 nautical miles. In this case the cause of the change of operation is to give way to the third surrounding vessel SV3 in order to increase the CPA from 0.5 nautical miles to 1.0 nautical miles.

[0108] The cause may in this case be deduced by the operator assisting device 10 from AIS messages or ARPA (Automatic radar plotting aid) information concerning the third surrounding vessel SV3. The deducing is with advantage made using machine-learning.

[0109] The impact is here the same as in the previous example as is the wind speed and the collision risk.

[0110] When the operator assisting device 10 detects that the first vessel V1 will not be able to keep safety distance to the third surrounding vessel SV3 using the present operational constraints (e.g., speed), the operator assisting device 10 may thus provide recommendations to the operator. Such moments would be logged with the system recommendation that the operator selects together with the machine generated rationale and impact analysis. This information is made accessible for human users.

[0111] It is additionally possible that operators may use voice commands to quickly annotate or tag existing events or create new incidents. These annotations could be voice memos or transcribed notes. This feature is especially useful when manual input might be time-consuming or challenging, and can be used for understanding past or reminding future events.

[0112] An annotation can be linked to a series of objects. It can be attached to specified voyage segments, waypoints, time duration, time point, zone, obstacles, target ships etc. The operator can quickly navigate to the annotation with the slider SL on the timeline or a zoom-out voyage map. The operator assisting device 10 may also take these voice-activated annotations to make an impact analysis, which include possible changes to navigational behaviours. An example of such an annotation ANN is shown in fig. 8.

[0113] There is here a SAR (Search and Rescue operation) in a zone of interest ZN north of the route between the second and fourth waypoints WP1 - WP4, in which zone ZN the, first second and third surrounding vessels SV1, Sv2, SV3 are all located.

[0114] The view also shows a window of the previously mentioned zoom-out view of the route on top of the view with the zone ZN.

[0115] As can be seen, an operator has created an annotation ANN at a point in time 08:29, which is a first point in time well before the first vessel V1 is expected to reach the first and the second waypoints WP1, WP2. The annotation ANN comprises a recommended change of operation, a cause as well as an impact if the change in the operation is implemented, i.e. how the operation can be later adjusted in order to reach a control goal, which adjusting, if needed, may be made after the possible implementation of the change of operation. Thus in this case, the operator assisting device 10 receives, at the first point in time, a suggestion from the operator of a future change in operation.

[0116] The recommended change of operation involves a recommended change of the speed envelope from 7.0 - 10.0 knots to 6.0 - 7.0 knots, which change is recommended to be performed between the second and fourth waypoints, WP2 - WP4 at the time 15:01, which is the time that the first vessel V1 is expected to reach the second waypoint WP2. The annotation also comprises the cause of the change, which may have been entered as text by the operator. The operator may also have made a recording of his or her speech that is then transcribed and entered as text in the notification. It is also or instead possible that a sound file of the recorded speech with an explanation of the cause is attached to the notification, which sound file can be selected in order to be replayed.

[0117] The consequences of the change are also shown in the event notification as an impact. The impact can as an example be that the control goal is fulfilled through a compensation being made at a later stage. In the present example the control goal of reaching the destination at the estimated time of arrival ETA is achieved with a slight speed increase between the twelfth and seventeenth waypoints WP12 - WP17. There may also in this case be a consequential increase in fuel consumption. Also other data is included in the annotation ANN, such as minimum Closest Point of Approach (CPA) to the third surrounding vessel SV3 to be 1.0 nautical miles, the risk for collision being low and the wind speed being 2 m / s.

[0118] The operator has also enclosed the three surrounding vessels SV1, SV2 and SV3. in a zone of interest ZN made up of coordinates a, b, c and d. Furthermore, the operator has also linked the annotation to the second waypoint WP2 at which the prosed change of operation could be started, as well as to the zone ZN.

[0119] The cause is in this case provided as transcribed speech uttered by the operator, which is included in the annotation ANN. As an example the text is "Captain said SAR operation zone so I mark it". A sound file may also be attached to the annotation.

[0120] This is an example of voice-activated annotations. Before the departure, the operator may have been told by the captain that there is a search and rescue mission happening inside coordinates [a, b, c, d]. Although this does not impact the work of the operator because the first vessel V1 will not reach the zone ZN before he or she finishes his or her shift, the operator may believe this is important piece of information for a following operator. The operator of a present shift can then quickly locate the area of interest in the navigational chart, create a zone with these parameters and provide a voice input attached to this tagged zone. The operator of the next shift can then easily locate such information by using the slider SL or the zoom-out map where the annotation is marked.

[0121] In addition to the operator's annotation and data captured from internal systems such as radar, AIS and GPS (Global Positioning System), the operator assisting device may also capture communication data such as the relevant information in VHF or UHF (Ultra High Frequency) radio calls, emails, satellite messages, NAVTEX messages. For instance, every VHF radio call may be transcribed, timestamped, logged, and associated with relevant events.

[0122] For example, if the first vessel V1 is informed by VTS or some shore-based maritime service organisation about a certain navigational hazard, either through VHF call or email, that key information in that communication may be extracted, potentially transcribed and logged. It may also be used to prove an annotation as shown in fig. 9.

[0123] Fig. 9, shows a detailed chart of a part of the route as well as a zoom out view. In this case an annotation is created by the operator assisting device 10 at the time 08.29 which is again the first point in time that is well before the first vessel is expected to reach the first and the second waypoints WP1 and WP2. The annotation ANN is again concerned with the zone ZN, which in this case is a hazard zone, and is likewise linked to the zone as well as to the second waypoint WP2.

[0124] In this case the first vessel V1 received a VTS call with the content "oil spill and floating container discovered stay cautious". This VTS message may then be transcribed, the text analysed, the zone ZN identified and an analysis of the impact on the first vessel made, which analysis may include a determination of the waypoints affected by the zone. The analysis may with advantage be made using machine-learning. The operator assisting device 10 may then determine an operation change that introduces caution close to the zone ZN, which as an example is a speed change within the existing speed envelope of 7.0 - 10.0 knots. The operations change is thus a change that is within the speed envelope that the vessel navigation control device is allowed to operate independently.

[0125] Thus, in this case the operator assisting device 10 provides, at the first point in time, a suggestion of a change in operation at a future point in time in relation to the first point in time, where the first point tin time in this case is the time of 15:09.

[0126] Thus, the operator assisting device 10 automatically captures communication data from the VHF call and uses it to create a hazard zone near the second waypoint WP2 and attaches an annotation ANN to this waypoint WP2. The impact analysis indicates that the ship will slow a bit but remain inside the present operational envelope.

[0127] This annotation ANN may then be displayed to the operator before or when the first vessel 22 reaches the second waypoint WP2. The operator may also become aware of the annotation via the timeline.

[0128] It is possible that the operator creates a log event in relation to such an annotation ANN created by the operator assisting device 10.

[0129] If for instance the operator subsequently makes a navigational decision, he or she has the possibility to link the decision with a logged navigational hazard information, in order to document that particular context, see fig. 10.

[0130] The operator may be very cautious regarding unforeseen events such as the oil spill and floating container incident. Though the operator assisting device 10 could automatically slow down based on this navigational hazard information, the operator in this case decides to change the operational envelope for obtaining an even large safety margin by further reducing the speed. Fig. 10 shows an event item being created at the time 14:29, which is a second point time when the first vessel V1 is at the first waypoint WP1, where the speed envelope for the waypoints WP2 - WP4 has been reduced to 6.0 - 7.0 knots. The event item or event notification EN is linked to the first waypoint WP1 as well as to the logged navigational hazard annotation which was created earlier by the operator assisting device (shown in fig. 9).

[0131] It can be seen that in this case there is a combination of an event notification EN and an annotation ANN, where a determined change in operation is being presented in the event notification and a proposed change in operation is at least initially presented in the annotation, where the annotation is linked to the event notification. Thus, the second point in time is after the first point in time. It is also before the future point in time, i.e. the time at which the first vessel 22 reaches the second waypoint WP2.

[0132] It is here also possible that the annotation ANN only comprises the cause. The rest of the relevant data originally appearing in the annotation, such as the impact, may be moved to the event notification EN.

[0133] As can be seen above, the operator may select to link the event notification to the annotation. Alternatively, the operator assisting device 12 may know that the event notification is to be linked to the annotation because the waypoint to which the event notification is linked is close to the waypoint to which the annotation is linked. It may for instance be a neighbouring waypoint or a waypoint separated from the waypoint of the annotation by a limited number of waypoints, such as two or three. It is additionally or instead possible that the decision to link the event notification to the annotation is based on the fact that the operational envelope that the operator changes is the same as that being analysed by the operator assisting device.

[0134] Another example of the use of an annotation can be when an engine department of the first vessel V1 calls the bridge regarding an emergent issue on certain parts of the propulsion system for instance via radio, and shares their maintenance plan. Radio call information for such a radio call may be transcribed, analysed, logged and linked to the waypoint at which the first vessel is located when the radio traffic was sent.

[0135] In addition to communication data, external data feeds may also be integrated and logged. The operator assisting device may overlay real-time information such as weather updates or maritime traffic news onto the logs and intelligently link it to potential subsequent navigational decision. For example, if a route deviation is made, and there was a storm alert in the original route, both pieces of information can be auto linked and cohesively presented to provide the full context. To ensure clarity and transparency, this linkage could include a mechanism to indicate that it represents the system's best guess based on the available data. Such a mechanism could involve a visual marker, such as an icon or color-coded tag, or an accompanying note explicitly stating that the connection is inferred rather than confirmed. This approach helps operators understand the rationale while maintaining trust in the system by clearly distinguishing between inferred suggestions and verified information.

[0136] As can be understood from the description above, aspects of the present disclosure provide an event impact analysis function, which event impact analysis function may be based on advanced machine learning. The event impact analysis function may make the impact of the change in operation on the voyage made accessible to the operator. The event impact analysis function may provide insights into the consequences of operator interventions, performance goal deviations, or system malfunctions on navigation. Subsequent decisions by the automation, potentially influenced by this human intervention, may also be auto-logged within the annotation. For example, if the operator chose to reduce speed due to an unforeseen reason, the operator assisting device may log how the future control strategies should be adjusted, such as accelerating between waypoints later on.

[0137] The operator assisting device may meticulously log any instances where the vessel's operation deviates from the normal operational performance range that is developed based on historical data with similar external environmental conditions. It can be highlighted on the chart, offering quick insights at a glance.

[0138] An example of this shown in fig. 11 for the first vessel V1. In this case the first vessel V1 is located at a present waypoint of the route and has passed a first and as second previous route section RS1 and RS2. The first vessel V1 may here have consumed fuel outside an expected fuel consumption range (above historical worst performance or historical best performance) in the first and second previous route sections RS1, RS2, these "outside of the norm" moments may be automatically highlighted in the user interface either in the timeline or the navigational chart, allowing for easy retrieval and analysis.

[0139] The previously described human-made annotations can be further used for machine learning, training or operation optimisation purposes.

[0140] The operator assisting device could use the large amount of data collected to offer predictive insights or recommendations. For instance, if the vessel continually breaks the operational envelope in a specific region, the system might suggest a review of set parameters for that leg of the journey.

[0141] Thus, there is provided an intelligent dynamic logbook that doesn't merely capture events but contextualizes them within the voyage's broader narrative. It integrates and correlates data from multiple sources, human input, internal systems and external communications. The operator assisting device aims to ensure that operators, both current or next shift, understand not just the 'what', but also the 'why' behind every decision and deviation, thus empowering them to make more informed decisions and allows continuous learning and adaptation. Overall it aims to build a rich data set that is context dependent.

[0142] The operator assisting device logs operator interventions and rationale, performance goal deviations, and system recommendations and malfunctions. It captures contextual information like time, location, and environmental conditions. An interactive event timeline may be used to display a chronological sequence of events for detailed information and impact analysis. Decision-making annotations may be attached to interventions. Rationales for accepting system recommendations may also be logged. Advanced machine learning may analyze event impacts and show insights into consequences and future strategies.

[0143] The operator assisting device may thus provide voice-activated annotations. This enables operators to create annotations and incidents using voice commands. It allows annotations to be linked to voyage segments, waypoints, zones, target ships and more.

[0144] The operator assisting device may provide for communication data capture. It may capture and log communication data like VHF calls, emails, and more. It also transcribes and timestamps communication data, associating it with events. Additionally, it enables users to link navigational decisions with logged annotated information.

[0145] The operator assisting device may also provide for external data integration, e.g., it can log external data feeds like weather updates, traffic news into logs. It may also allow users to link external data to navigational decisions.

[0146] The operator assisting device may provide for logging performance deviations from historical operational data and may highlight them in the user interfaces for quick insights. It may also utilize collected data to provide predictive insights and recommendations for optimization of the data driven systems used in high-level automation context.

[0147] While the invention has been described in connection with what is presently considered to be most practical and preferred embodiments, it is to be understood that the invention is not to be limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements. Therefore, the invention is only to be limited by the following claims.

Claims

1. A method for assisting an operator during automatic control of a vessel (22), wherein the vessel (22) is controlled to move according to a present control scheme in a present passage along a route towards a destination, the method being performed by an operator assisting device (10) and comprising: displaying (S100) the progress of the vessel (V1) in the present passage, obtaining (S110) an actual or proposed change in the operation of the vessel (V1) from the present control scheme; obtaining (S120) data about the cause for the change of operation; and linking (S130) the change in operation and the cause of the change to the present passage or to contextual data of the route and presenting them (S150) to the operator in at least one display item that is linked to the present passage or to the contextual data of the route.

2. The method as claimed in claim 1, wherein the change in operation is a change carried out for a group of waypoints (WP2 - WP4) along the route, there is a control goal associated with the present control scheme and further comprising determining (S140) an adjustment of the present control scheme in order to fulfil the control goal, which adjustment is applicable after the group of waypoints (WP2 - WP4) have been passed.

3. The method according to claim 1 or 2, wherein the change in operation comprises a change of an operational envelope, such as a speed envelope.

4. The method as claimed in any previous claim, further comprising receiving communication data, such as radio, AIS and NAVTEX data and analysing the communication data.

5. The method as claimed in claim 4, whereon the cause for change in operation has been determined based on the analysis of the communication data.

6. The method according to claim 4 or 5, wherein the change in operation has been determined based on the analysis of the communication data.

7. The method as claimed in any of claims 1 - 6, wherein the obtaining of an actual or proposed change in operation comprises detecting an operator-made determination of a change of the present control scheme and implementing the change in operation.

8. The method as claimed in of claims 1 - 6 6, wherein the obtaining of an actual or prosed change in operation comprises determining at least one change in operation the vessel (22), providing the at least one change in operation to an operator as a proposed change, receiving, from the operator, a determination of a change in operation as a confirmation that the proposed change is to be implemented and implementing the confirmed change in operation.

9. The method as claimed in claim 7 or 8, wherein the at least one display item comprises an event notification (EN) being linked to a waypoint (WP1) at which the change in operation is determined.

10. The method as claimed in any of claims 1 - 9, wherein the obtaining of an actual or prosed change in operation comprises receiving, at a first point in time, a suggestion from the operator of a future change in operation.

11. The method as claimed in any of claims 1 - 9, wherein the obtaining of an actual or proposed change in operation comprises providing, at a first point in time, a suggestion of a future change in operation.

12. The method as claimed in claim 10 or 11, wherein the future change in operation is suggested to be made at a future point in time, the at least one display item comprises an annotation (ANN) and the suggestion of the change in operation is presented in said annotation, where the annotation is linked to a future point of the route.

13. The method as claimed in claim 12 when depending on claim 9, wherein the change in operation is determined at a second point in time (before the future point in time) and the annotation is linked to the event notification.

14. The method according to claim 12 or 13, further comprising obtaining contextual data of the route through obtaining a determination of a zone (ZN) of interest at or around at least a part of the route, where the proposed change in operation is made in order to consider the zone and the annotation (ANN) is linked to the zone.

15. An operator assisting device (10) for assisting an operator during automatic control of a vessel (22), wherein the vessel (22) is controlled to move according to a present control scheme in a present passage along a route towards a destination, the operator assisting device (10) comprising a processor (12) operative to: display the progress of the vessel (V1) in the present passage, obtain an actual or proposed change in the operation of the vessel (V1) from the present control scheme; obtain data about the cause for the change of operation; and link the change in operation and the cause of the change to the present passage or to contextual data of the route and present them to the operator in at least one display item that is linked to the present passage or to the contextual data of the route.

16. A computer program for assisting an operator during automatic control of a vessel (22), wherein the vessel (22) is controlled to move according to a present control scheme in a present passage along a route towards a destination, the computer program (16) comprising computer program code which when run by a processor (12) of an operator assisting device (10) causes the processor (12) to: display the progress of the vessel (V1) in the present passage, obtain an actual or proposed change in the operation of the vessel (V1) from the present control scheme; obtain data about the cause for the change of operation; and link the change in operation and the cause of the change to the present passage or to contextual data of the route and present them to the operator in at least one display item that is linked to the present passage or to the contextual data of the route.

17. A computer program product for assisting an operator during automatic control of a vessel (22), the computer program product comprising a data carrier (20) with said computer program (16) according to claim 16.

Images