End-to-end multi-modal large model autonomous decision-making method and system for ship-aided navigation

Abstract

The invention, which relates to the technical field of intelligent shipping, discloses an end-to-end multi-modal large model autonomous decision-making method and system for ship-aided navigation, and the method comprises the steps: obtaining a marine collision avoidance rule based on ship autonomous level optimization; acquiring multi-modal input data of a current navigation scene of the ship; inputting the multi-modal input data into a multi-modal large model subjected to pre-training and efficient parameter fine tuning; the fine tuning is carried out by using a professional data set containing a navigation scene image and a corresponding decision mark based on an optimized rule; and directly generating navigation decision output containing current situation analysis and collision avoidance operation suggestions by the multi-modal large model according to the optimized rules and input data. The technical problems that a traditional collision avoidance algorithm and a modular autonomous system are low in decision making efficiency, insufficient in safety and prone to falling into local optimum in a complex and dynamic seaborne encounter scene are solved, and the end-to-end multi-mode large model autonomous decision making method oriented to ship auxiliary navigation is provided.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent shipping, and more particularly to an end-to-end multimodal large-scale autonomous decision-making method and system for ship assisted navigation. Background Technology

[0002] Maritime Autonomous Surface Ships (MAS) are an important development direction in the shipping industry, with core technologies involving autonomous navigation, environmental perception, and intelligent decision-making. With advancements in artificial intelligence, the autonomy level of MASSes is continuously improving, from partial autonomy (Level 1) to full autonomy (Level 4), gradually reducing or even eliminating the need for human intervention. The International Maritime Organization (IMO) has clearly defined the autonomy levels of MASSes, but existing maritime collision avoidance rules (such as COLREGs) are primarily designed for traditional manned vessels and have many limitations in their application to high-level MASSes. Specifically, clauses regarding responsible parties and yielding obligations are difficult to directly apply to unmanned or minimally manned MASSes, potentially leading to ambiguity in liability definition and uncertainty in collision avoidance decisions during actual navigation.

[0003] Currently, the collision avoidance decision-making system of MASS mainly adopts a modular architecture of "perception-prediction-planning-control". Although the technologies of each sub-module are relatively mature, the existence of multi-level interfaces makes it easy for the system to accumulate errors and suffer from latency problems during operation, making it difficult to meet the real-time safety requirements of highly dynamic, multi-ship interaction scenarios. Especially when encountering abnormal ship behavior or complex intersection situations, traditional rule-based planning methods are prone to getting stuck in local optima and cannot guarantee global navigation safety. In addition, the patented solutions disclosed in the existing technology also have obvious shortcomings: for example, the ship collision avoidance test scenario generation method proposed in CN119088684A can improve test coverage, but it does not solve the applicability problem of COLREGs rules to high-level MASS; the trajectory planning method disclosed in CN118938932A encounters overly static situation judgment, has a single collision avoidance target selection strategy, and the path planning algorithm is prone to getting stuck in local optima. These technical defects together lead to the insufficient reliability of the existing MASS collision avoidance system in complex sea conditions, and there is an urgent need for a new solution that can take into account real-time performance, safety, and rule adaptability. Summary of the Invention

[0004] In view of this, the present invention provides an end-to-end multimodal large model autonomous decision-making method for ship assisted navigation, which solves the technical problems of low decision-making efficiency, insufficient safety, and easy getting trapped in local optima in complex and dynamic maritime encounter scenarios proposed above.

[0005] The technical means employed in this invention are as follows: On the one hand, this invention provides an end-to-end multimodal large-scale autonomous decision-making method for ship assisted navigation, including: Obtain maritime collision avoidance rules optimized based on the ship's autonomy level; Acquire multimodal input data of the ship's current navigation scenario; the multimodal input data includes image information of the navigation scenario, the ship's current position information, AIS information, and ship radar image information; The multimodal input data is fed into a pre-trained and fine-tuned multimodal large model; the parameter fine-tuning is based on a dataset including navigation scene images and corresponding decision labels based on optimized rules; the multimodal large model is trained with multimodal data from different sources and provides answers based on the pre-training results; The multimodal large model directly generates a navigation decision text output, including current situation analysis and collision avoidance operation suggestions, based on the optimized maritime collision avoidance rules and input data. The current situation analysis determines the encounter situation of the ship so as to generate collision avoidance decisions according to the rules corresponding to different situations.

[0006] Furthermore, obtain maritime collision avoidance rules optimized based on the ship's autonomy level, including: For vessels of different MASS levels, their operating procedures during encounters are specified according to their level of autonomy. Vessels with higher autonomy levels shall take collision avoidance measures to actively avoid collisions with vessels of lower autonomy levels.

[0007] Furthermore, the optimized maritime collision avoidance rules include provisions for the division of responsibilities and collision avoidance actions for vessels of different levels of autonomy in specific encounter situations.

[0008] Furthermore, the parameter fine-tuning employs the low-rank adaptation LoRA method.

[0009] Furthermore, the dataset employs radar images and image-text multimodal data for scene description and decision-making; the image input displays the actual navigation scene of the ship, as well as ship information; the text input is a textual description of the navigation situation shown in the image input, and combined with the optimized maritime collision avoidance rules, reasonable suggestions are made for the current state as a decision to guide the decision-maker to take appropriate actions.

[0010] Furthermore, the multimodal large model adopts Qwen2.5-VL.

[0011] Furthermore, the multimodal large model includes: a ViT visual encoder extracting image features, a language encoder processing text, and a visual-language decoder performing multimodal fusion and generation.

[0012] In another aspect, the present invention also provides an end-to-end multimodal large-scale autonomous decision-making system for ship assisted navigation, comprising: a storage unit, an input interface unit, a processing unit, and an output interface unit; The storage unit stores maritime collision avoidance rules optimized based on the ship's autonomy level and a multimodal large model that has been pre-trained and fine-tuned. The input interface unit is used to receive multimodal input data of the ship's current navigation scenario; The processing unit is used to call the multimodal large model according to the optimized rules and input data, and directly generate navigation decision text containing current situation analysis and collision avoidance operation suggestions; The output interface unit is used to output the navigation decision text.

[0013] Compared with the prior art, the present invention has the following advantages: The present invention provides an end-to-end multimodal large model autonomous decision-making method for ship assisted navigation. By deeply integrating optimized maritime collision avoidance rule knowledge with the powerful perception and reasoning capabilities of the multimodal large model, it achieves accurate understanding of complex navigation scenarios and rapid, safe and compliant autonomous collision avoidance decisions, thereby significantly improving the navigation safety and autonomous decision-making level of high-level MASS. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of an end-to-end multimodal large model autonomous decision-making method for ship assisted navigation, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the end-to-end autonomous decision-assisted navigation system framework in an embodiment of the present invention; Figure 3 This is an example of image input based on a radar system and an ECDIS system in an embodiment of the present invention; wherein, (a) is a radar system image; and (b) is an ECDIS system image. Figure 4 This is a schematic diagram illustrating the setting of prompt words in an embodiment of the present invention; Figure 5 This is a visualization interface for the end-to-end ship auxiliary navigation system in an embodiment of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] like Figure 1 As shown in the figure, an end-to-end multimodal large-scale autonomous decision-making method for ship assisted navigation in this embodiment of the invention specifically includes the following steps: Step 1: Obtain maritime collision avoidance rules optimized based on the ship's autonomy level; Specifically, the existing maritime collision avoidance rules will be optimized based on the ship's autonomy level. The optimization method is to stipulate the operating procedures for ships with different MASS levels during encounters according to their level of autonomy, with ships with higher autonomy levels bearing more responsibility for collision avoidance.

[0019] The Maritime Collision Avoidance Rules introduce the principle of "class distinction," stipulating that higher-class collision avoidance vehicles (MASSes) must give way to lower-class collision avoidance vehicles (MASSes) in specific encounter situations (head-on encounters, cross encounters, overtaking).

[0020] For example, when encountering or overtaking, higher-class MASS vessels will actively give way; when crossing paths, they will act according to whether they are the vessels giving way.

[0021] A "proactive avoidance" responsibility mechanism is introduced, emphasizing that higher-level MASS vessels utilize their enhanced perception and reaction capabilities to proactively take collision avoidance measures when collision risks exist. Vessels are categorized according to their level of autonomy, and the model can identify the autonomy levels of both the vessel and other vessels. Collision risk is analyzed by examining the encounter states and distances between different vessels. Proactive avoidance means that vessels with higher autonomy levels must take collision avoidance measures to proactively avoid collisions with vessels with lower autonomy levels.

[0022] The optimized maritime collision avoidance rules include provisions for the division of responsibilities and collision avoidance maneuvers for vessels of different levels of autonomy in specific encounter situations (head-on encounter, cross encounter, overtaking).

[0023] Step 2: Obtain multimodal input data of the ship's current navigation scenario; Specifically, acquire multimodal input data of the ship's current navigation scenario, including image information of the navigation scenario, the ship's current position information, AIS information, ship radar image information, etc., including at least scene images; Step 3: Input the multimodal input data into the pre-trained and fine-tuned multimodal large model; The pre-trained large language model is developed by collecting multimodal data from different sources, enabling it to respond to and process user questions. For example, if a user asks, "What kind of encounter state does the ship in the picture belong to?" and sends a picture of a ship encounter, the multimodal large model will provide an answer based on its pre-training performance. In this embodiment, the Qwen2.5-VL pre-trained multimodal large model, with its powerful visual understanding and natural language processing capabilities, is selected as the base model. This model can simultaneously process image and text inputs and perform cross-modal reasoning and generation.

[0024] Fine-tuning was performed using a specialized dataset containing navigation scene images and corresponding decision annotations based on optimized rules. The dataset employs image-text multimodal data combining radar images, scene descriptions, and decisions. For example, the scene description might be, "This vessel is in a wide waterway, facing an oncoming vessel." The decision might be, "It is recommended that this vessel proceed to the right-hand side to avoid the oncoming vessel." In practice, a multimodal dataset based on real-world scenarios was constructed, including image inputs composed of radar and ECDIS images and corresponding text inputs. The image input displays the actual navigation scene, allowing the model to analyze the vessel's navigation status and encounter situation based on the current scenario. The image input also displays vessel information such as AIS, providing a multi-faceted interpretation of the vessel's current navigation status. The text input provides a textual description of the navigation situation shown in the image input, detailing the current state, environmental factors, and encounter conditions. Combined with the optimized COLREGs, it offers reasonable suggestions as decisions, guiding the decision-maker to take appropriate actions.

[0025] The parameter fine-tuning employs the LoRA technique, which avoids altering a large number of weights in the original model. Instead, it adds small, trainable "adapter modules" next to key layers (such as attention / feedforward layers). Only these small matrices are trained, and their outputs are added back to the original layers. This approach significantly reduces memory usage and training costs, ensuring good performance while minimizing computational load. It also allows for the storage of multiple small "adapters" for different tasks, enabling on-demand loading and rapid switching. By introducing low-rank matrices for fine-tuning, LoRA significantly reduces the number of parameters that need to be trained, thereby lowering computational and storage costs while maintaining and improving model performance and mitigating overfitting issues.

[0026] In practice, the model is fine-tuned using the aforementioned multimodal dataset. Images and corresponding text descriptions are used as input for training. The optimization goal is to enable the model to accurately understand the encounter situation based on the input navigation scene image, apply the optimized COLREGs rules, and generate corresponding high-quality decision suggestion text. Training is optimized using appropriate loss functions, learning rates, batch sizes, and training cycles. Lightweight frameworks such as ModelScope Swift can be used to support LoRA fine-tuning. The model architecture includes a ViT visual encoder to extract image features, a language encoder to process text, and a visual-language decoder for multimodal fusion and generation.

[0027] The specific implementation process is as follows: First, the model is fine-tuned using the constructed multimodal dataset. The fine-tuning code and parameters are set by the user, and multiple fine-tunings are performed to select the best result. In this way, the model can understand issues related to the ship's navigation status. The constructed multimodal dataset includes navigation image information, a detailed description of the image information, such as 'What is the current state of the ship?', and navigation suggestions for the ship, such as 'The ship should move to the right front to avoid a collision'.

[0028] Step 4: The multimodal large model directly generates a navigation decision text output, including current situation analysis and collision avoidance operation suggestions, based on the optimized maritime collision avoidance rules and input data. The current situation analysis determines the encounter situation of the ship so as to generate collision avoidance decisions according to the rules corresponding to different situations.

[0029] In practice, the system includes the MASS assisted navigation interface module, which is a visual human-computer interaction interface. Users can call upon a finely tuned multimodal large model to analyze the current navigation scene image. The model outputs its understanding of the current encounter situation, responsibility allocation based on optimized COLREGs, and specific collision avoidance operation suggestions, which are displayed to the user on the interface to assist in navigation decision-making.

[0030] like Figure 2As shown, this embodiment of the invention also provides an end-to-end multimodal large model autonomous decision-making system for ship assisted navigation, including: a storage unit, an input interface unit, a processing unit, and an output interface unit; The storage unit stores maritime collision avoidance rules optimized based on the ship's autonomy level and a pre-trained, parameter-fine-tuned multimodal large model; The input interface unit is used to receive multimodal input data of the ship's current navigation scenario; The processing unit is used to call the multimodal large model based on the optimized rules and input data to directly generate navigation decision text containing current situation analysis and collision avoidance operation suggestions; The output interface unit is used to output navigation decision text.

[0031] The following example illustrates the end-to-end multimodal large model autonomous decision-making method for ship-assisted navigation.

[0032] Example 1 Rules 16, 17, and 18 of COLREGs are refined in conjunction with the characteristics of MASS Classes 1-4. Rule 16: Vessels required to give way to other vessels should take significant action as early as possible to allow sufficient time for the other vessel to pass; Rule 17: Vessels traveling in a straight course should maintain their course and speed; Rule 18: The determination of the giving way vessel, including the requirement that motorized vessels give way to vessels that have lost control, etc. For example, it is clearly stipulated that when a Class 4 unmanned vessel encounters a Class 1-3 vessel, it bears primary collision avoidance responsibility and takes specific avoidance actions when encountering, overtaking, or acting as the giving way vessel in a cross-encounter. The specific navigation situation and responsibility division for encounters between MASS Class 4 vessels and MASS Class 1-3 vessels are shown in Table 2. The navigation situation and responsibility division for encounters between two Class 4 unmanned vessels are shown in Table 3.

[0033] Table 2

[0034] Table 3

[0035] Example 2: Based on radar systems and ECDIS systems (such as Figure 3 As shown in the image, screenshots are automatically taken every 6 seconds. The screenshots are then manually or semi-automatically labeled to generate corresponding JSON data files containing image paths and detailed text descriptions. The training and validation sets are divided in a 7:3 ratio, and approximately 1000 image and text data points are prepared to form an independent dataset. Dataset processing includes setting prompts and creating question-and-answer dialogues, such as... Figure 4 As shown.

[0036] The Qwen2.5-VL-7B model was selected as the base model. LoRA fine-tuning was performed using the ModelScopeSwift framework. Configuration parameters: learning rate 5e. -5 Batch size 32, training 10 epochs, loss function MSE. Training environment: Linux operating system, CPU E5-2698v4, GPU Tesla V100-32GB, CUDA 12.4. Monitor loss reduction and token-level accuracy improvement during training to ensure model convergence.

[0037] like Figure 5 As shown, this is a web-based visualization interface. After a user uploads an image, the backend calls a finely tuned Qwen2.5-VL model to perform inference and displays the generated text decision on the frontend. The model's performance is evaluated using a test set, including scene understanding accuracy, response time, and simulated collision rate.

[0038] Compared with the prior art, the present invention has the following advantages: (1) Improved decision-making accuracy and safety: The optimized rules of this invention are more applicable to MASS, clarify the division of responsibilities, and reduce ambiguity. Combined with the powerful understanding capabilities of the large model, it can more accurately identify complex encounter situations and generate compliant and safe collision avoidance decisions. Experiments show that the scene understanding accuracy is significantly improved (to 92.5% in normal environments and 76.8% in harsh environments), and the simulated collision rate is greatly reduced (from 6.8% to 2.2%).

[0039] (2) Improved decision-making efficiency: This invention adopts an end-to-end model that goes directly from input to output, reducing intermediate steps. Combined with the model's efficient reasoning, it can quickly generate decision recommendations. Experiments show that the average response time is shortened to 0.8 seconds, which is much faster than traditional rule-driven methods. This is crucial for dealing with the rapidly changing marine environment.

[0040] (3) Enhanced robustness: The model of this invention is trained on datasets containing different environmental conditions, and exhibits adaptability and robustness to different environments. It can still maintain high accuracy and low collision rate under harsh conditions.

[0041] (4) Good human-computer interaction: The visual interface of this invention provides intuitive decision support, which makes it easier for human operators to understand the current situation and model suggestions, and achieve effective human-computer collaboration.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An end-to-end multimodal large-scale autonomous decision-making method for ship assisted navigation, characterized in that, include: Obtain maritime collision avoidance rules optimized based on the ship's autonomy level; Acquire multimodal input data of the ship's current navigation scenario; The multimodal input data includes image information of the navigation scene, current ship position information, AIS information, and ship radar image information; The multimodal input data is then input into a pre-trained and fine-tuned multimodal large model. The parameter fine-tuning is based on a dataset that includes navigation scene images and corresponding decision labels based on optimized rules; the multimodal large model is trained with multimodal data from different sources and provides answers based on the pre-training results; The multimodal large model directly generates a navigation decision text output, including current situation analysis and collision avoidance operation suggestions, based on the optimized maritime collision avoidance rules and input data. The current situation analysis determines the encounter situation of the ship so as to generate collision avoidance decisions according to the rules corresponding to different situations.

2. The end-to-end multimodal large-scale autonomous decision-making method for ship-assisted navigation according to claim 1, characterized in that, Obtain maritime collision avoidance rules optimized based on the ship's autonomy level, including: For vessels of different MASS levels, their operating procedures during encounters are specified according to their level of autonomy. Vessels with higher autonomy levels shall take collision avoidance measures to actively avoid collisions with vessels of lower autonomy levels.

3. The end-to-end multimodal large-scale autonomous decision-making method for ship-assisted navigation according to claim 2, characterized in that, The optimized maritime collision avoidance rules include provisions for the division of responsibilities and collision avoidance maneuvers for vessels of different levels of autonomy in specific encounter situations.

4. The end-to-end multimodal large-scale autonomous decision-making method for ship-assisted navigation according to claim 1, characterized in that, The parameter fine-tuning employs the low-rank adaptation LoRA method.

5. The end-to-end multimodal large-scale autonomous decision-making method for ship-assisted navigation according to claim 1, characterized in that, The dataset uses radar images and image-text multimodal data for scene description and decision-making. The image input displays the actual navigation scene of the ship and the ship information. The text input is a textual description of the navigation situation shown in the image input, and combined with the optimized maritime collision avoidance rules, reasonable suggestions are made for the current state as a decision to guide the decision-maker to take appropriate actions.

6. The end-to-end multimodal large-scale autonomous decision-making method for ship-assisted navigation according to claim 1, characterized in that, The multimodal large model adopts Qwen2.5-VL.

7. The end-to-end multimodal large-scale autonomous decision-making method for ship-assisted navigation according to claim 1, characterized in that, The multimodal large model includes: a ViT visual encoder to extract image features, a language encoder to process text, and a visual-language decoder to perform multimodal fusion and generation.

8. An end-to-end multimodal large-scale autonomous decision-making system for ship assisted navigation, characterized in that, include: Storage unit, input interface unit, processing unit, and output interface unit; The storage unit stores maritime collision avoidance rules optimized based on the ship's autonomy level and a multimodal large model that has been pre-trained and fine-tuned. The input interface unit is used to receive multimodal input data of the ship's current navigation scenario; The processing unit is used to call the multimodal large model according to the optimized rules and input data, and directly generate navigation decision text containing current situation analysis and collision avoidance operation suggestions; The output interface unit is used to output the navigation decision text.

Images