A Ship Behavior Decision-Making Method Based on Heterogeneous Information Fusion and Integrated Reasoning

Abstract

This invention discloses a ship behavior decision-making method based on heterogeneous information fusion and ensemble reasoning. First, it integrates multiple data sources to preprocess different types of data. Then, it extracts features from different data using deep learning technology and converts them into vector form. Next, it fuses different types of features through a cross-attention mechanism to generate a unified comprehensive vector for behavior prediction. Finally, it fine-tunes a pre-trained decision model using the fused information, enabling the model to generate natural language navigation strategies based on the prediction results. The fine-tuned decision model is then tested offline, deployed in the ship system, and continuously optimized through a feedback mechanism. Finally, it introduces multiple decision models and calculates the Shapley value for each model to generate a weighted voting decision result. This invention improves navigation safety and efficiency.

Description

Technical Field

[0001] This invention relates to the field of ship technology, and specifically to a ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning. Background Technology

[0002] With the rapid development of the global shipping industry, maritime traffic has become increasingly busy and complex, posing unprecedented challenges to the safe and efficient navigation of ships. Ships are subject to various uncertainties during navigation, such as weather changes, complex channel conditions, and the dynamic behavior of other vessels, placing immense pressure on the decision-making of captains and crew. In recent years, automated navigation technologies have made some progress; for example, Automatic Identification Systems (AIS), radar, and sensors have played a crucial role in navigation safety. However, these systems primarily rely on hard-coded rules and limited sensor data, making it difficult to cope with the complex and ever-changing maritime environment. Traditional systems have significant shortcomings in processing unstructured data (such as text, logbooks, and incident reports) and in information fusion, thus limiting their reaction speed and decision-making flexibility in response to emergencies. Summary of the Invention

[0003] Purpose of the Invention: The purpose of this invention is to provide a ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning. It aims to integrate multiple data sources (such as AIS data, meteorological information, radar detection data, navigation logs, and maritime rules) and use a multi-model weighted voting method to predict potential navigation risks and generate interpretable avoidance strategies, thereby solving the problems existing in the background technology.

[0004] Technical solution: The present invention provides a ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning, comprising the following steps:

[0005] (1) Collect multi-source heterogeneous navigation data, including AIS data, radar images and text data, and preprocess each data type;

[0006] (2) Feature extraction is performed on the text data, AIS time series data and radar image respectively to obtain the text embedding vector, time series feature vector and image feature vector;

[0007] (3) Map the text embedding vector, temporal feature vector and image feature vector to the same semantic space, and realize the dynamic fusion of multimodal features through the cross attention mechanism to generate a unified comprehensive feature vector;

[0008] (4) Fine-tune the pre-trained decision model using comprehensive feature vectors so that it can identify navigation risks based on multimodal inputs and generate decision suggestions and explanations in natural language form;

[0009] (5) Deploy the fine-tuned decision-making model to the ship system, conduct offline and online tests, and continuously optimize the model performance through feedback mechanisms;

[0010] (6) Introduce multiple heterogeneous decision-making models, use Shapley values ​​to quantify the marginal contribution of each model in historical decision-making, assign weights according to the marginal contribution and conduct weighted voting to generate the final behavioral decision.

[0011] Furthermore, in step (1), the preprocessing includes: denoising, time alignment and normalization of the time series data, extracting navigation rules and event information from the text data, and having experts annotate the ship decision-making behavior based on multi-source data.

[0012] Further, step (2) is as follows: use a pre-trained large language model to encode the text information and generate text embedding vectors; extract the temporal features of ship speed, heading and position changes from AIS data; use a convolutional neural network to process radar images, identify the environment and obstacles and generate image feature vectors.

[0013] Furthermore, in step (3), the heterogeneous data fusion adopts a cross-attention mechanism, which uses the query vector of one modality to focus on the key and value vector of another modality, thereby realizing dynamic semantic association and fusion between multimodal information.

[0014] Furthermore, the fusion formula is as follows:

[0015] ;

[0016] in, A query vector representing data from a modality. and A key-value vector representing data from another modality. This represents the dimension of the key vector, used for scaling. The function is used to convert scores into a probability distribution, thereby assigning weights to different values.

[0017] Furthermore, in step (4), the decision model generates navigation strategies through natural language and triggers alarms when an emergency is detected, pushing decision suggestions and explanations to the crew through voice or visual interfaces.

[0018] Furthermore, in step (5), the feedback mechanism specifically involves continuously optimizing the accuracy and reliability of the decision-making model through comparison of manual decisions in the actual deployment environment and user feedback.

[0019] Furthermore, step (6) includes the following steps:

[0020] (61) Use the Shapley value to quantify the contribution of different models in each decision task, assuming that there are There are 1 model, and the Shapley value for each model is calculated as follows:

[0021] ;

[0022] in, It is a model Shapley value, It is a subset Size, It is a subset The corresponding decision value is based on The model generates prediction results; by traversing all possible model combinations, the model is calculated. The incremental contribution after adding a subset is used to evaluate its marginal contribution throughout the decision-making process;

[0023] (62) Based on the Shapley value calculation results, assign corresponding weights to each model, and the final decision result is obtained. Generated through weighted voting:

[0024] ;

[0025] in, It is the first The output of each model.

[0026] (63) Update the contribution values ​​of each model to achieve dynamic adaptation of decision-making strategies under different environmental conditions.

[0027] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: By comprehensively processing structured and unstructured data related to ship navigation, this invention intelligently fuses and analyzes various data sources such as text, time series, images, and meteorological data, thereby providing accurate ship behavior prediction and decision support. It can comprehensively process multiple data sources, improve the system's situational awareness capabilities, and generate robust and interpretable decision outputs through inter-model collaboration and voting mechanisms. Simultaneously, this invention generates explanations using natural language, increasing crew trust in the system. This technology will provide strong technical support for future intelligent navigation and autonomous ships. Attached Figure Description

[0028] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0029] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0030] like Figure 1As shown, this invention provides a ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning. First, this invention integrates multiple data sources (including AIS data, radar images, meteorological information, and navigation log text, etc.) and preprocesses different types of data. Next, this invention extracts features from different data using deep learning technology and converts them into vector form. Then, it fuses different types of features through a cross-attention mechanism to generate a unified comprehensive vector for behavior prediction. The pre-trained decision model is fine-tuned using the fused information, enabling the model to generate natural language navigation strategies based on the prediction results. Next, the fine-tuned decision model is tested offline, deployed in a ship system, and continuously optimized through a feedback mechanism. Finally, multiple decision models are introduced, and the Shapley value for each model is calculated to generate a weighted voting decision result; including the following steps:

[0031] Step 1: Data Acquisition and Preprocessing. In ship behavior decision-making technology based on heterogeneous information fusion and integrated reasoning, the first step is to comprehensively collect various types of data related to ship navigation, providing foundational information for subsequent behavior prediction and decision-making. This step involves acquiring real-time information from multiple data sources, including AIS data, radar images, meteorological data, and text information. Preprocessing is performed according to data type, removing noise from time-series data, performing time synchronization and normalization, and extracting key navigation rules and event information from text data. Finally, experts annotate ship decision-making behaviors based on the multi-type data.

[0032] Step 2: Feature Extraction. The core objective of this step is to extract valuable features from different types of input data for subsequent data fusion and behavior prediction. Data from multiple sensory sources is converted into high-dimensional embedding vectors that the model can understand, thereby achieving a unified representation of data from different modalities. For AIS data, time-series features are extracted using a time-series model; for image information, spatial features are extracted from images using a convolutional neural network to detect other vessels, obstacles, or maritime facilities; for text data, deep semantic features are extracted using a pre-trained language model.

[0033] Step 3: Data Fusion. The goal is to effectively integrate features from different data sources to provide more accurate decision support. Since ships need to consider multiple factors during navigation (such as navigation data, radar imagery, meteorological information, and textual rules), single-modal data is often insufficient to fully reflect the ship's current status and potential risks. Multimodal data fusion can effectively improve the system's understanding and response capabilities to complex navigation environments.

[0034] In this step, the feature vectors extracted in step 2 are concatenated to form a composite feature vector. The main application here is the cross-attention mechanism, an important tool in multimodal data fusion. This mechanism allows the system to dynamically adjust its dependence on different modalities by having a query from one modality focus on the keys and values ​​of another modality. The formula is as follows:

[0035] ;

[0036] in, A query vector representing data from a modality. and A key-value vector representing data from another modality. This represents the dimension of the key vector, used for scaling. The function is used to convert scores into a probability distribution, thereby assigning weights to different values.

[0037] In data fusion, the system can adjust the sources of Q, K, and V to allow one type of data (e.g., time series data) to focus on the features of another type of data (e.g., image data). This mechanism helps the system establish semantic relationships between different types of data, enabling dynamic information fusion.

[0038] Step 4: Behavior Prediction and Decision Generation. The input to the decision model can be represented in triplet form. ,in Indicates instructions, Indicates multimodal input, This represents the ground truth (GT) response. The decision model predicts the answer given the instructions and multimodal inputs:

[0039] ;

[0040] here, Indicates the predicted answer. These are the parameters of the model. The training objective is usually the original autoregressive objective used to train the decision model, and the objective can be represented as:

[0041] ;

[0042] in It is the corresponding length of GT.

[0043] In this step, the decision-making model is fine-tuned using the fused data from step 3. The fine-tuned model can analyze and model existing data to predict the system's possible behavior over a future period, including course prediction, collision avoidance prediction, and environmental risk assessment. The decision-making model generates decision information based on the input data, guiding the system to make corresponding decisions using natural language and explaining the reasons for those decisions. When an emergency is detected, the model generates an alarm notification and conveys advice to the crew through voice broadcast or a visual interface.

[0044] Step 5: System Deployment and Real-Time Inference. This step begins with offline testing of the decision-making model. Historical data or pre-collected simulated data are used to evaluate the system's performance in various possible scenarios, ensuring the accuracy and robustness of its predictions and decisions. After the model passes offline testing, the trained decision-making model is deployed to the ship's autopilot system or edge devices. The system can quickly perform inference and decision-making upon receiving real-time data. Based on this, online testing is conducted, and adjustments are made through the input and feedback of real-time data streams. The system can continuously optimize its prediction and decision-making capabilities.

[0045] Step 6: Multi-model voting decision based on Shapley values. Multiple decision models of different types are introduced and optimized for different data types. Shapley values ​​are used to quantify the contribution of different models in each decision task. Specifically, assume there are... There are 1 model, and the Shapley value for each model is calculated as follows:

[0046] ;

[0047] in, It is a model Shapley value, It is a subset Size, It is a subset The corresponding decision value (i.e., based on) The formula calculates the prediction results generated by the model in the model (by iterating through all possible model combinations). The incremental contribution after adding a subset is used to evaluate its marginal contribution throughout the decision-making process.

[0048] Next, based on the Shapley score calculation, appropriate weights are assigned to each model. The final decision result... It can be generated through weighted voting:

[0049] ;

[0050] in, It is the first The output of each model.

[0051] Finally, to enable the system to adapt to different environmental conditions, the ensemble method continuously updates the Shapley value of each model, thereby dynamically adjusting the model weights to generate more accurate decisions in complex navigation environments.

Claims

1. A ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning, characterized in that, Includes the following steps: (1) Collect multi-source heterogeneous navigation data, including AIS data, radar images and text data, and preprocess each data type; (2) Feature extraction is performed on the text data, AIS time series data and radar image respectively to obtain the text embedding vector, time series feature vector and image feature vector; (3) Map the text embedding vector, temporal feature vector and image feature vector to the same semantic space, and realize the dynamic fusion of multimodal features through the cross attention mechanism to generate a unified comprehensive feature vector; (4) Fine-tune the pre-trained decision model using comprehensive feature vectors so that it can identify navigation risks based on multimodal inputs and generate decision suggestions and explanations in natural language form; (5) Deploy the fine-tuned decision-making model to the ship system, conduct offline and online tests, and continuously optimize the model performance through feedback mechanisms; (6) Introduce multiple heterogeneous decision-making models, use Shapley values ​​to quantify the marginal contribution of each model in historical decision-making, assign weights according to the marginal contribution and conduct weighted voting to generate the final behavioral decision.

2. The ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning according to claim 1, characterized in that, In step (1), the preprocessing includes: denoising, time alignment and normalization of the time series data, extracting navigation rules and event information from the text data, and having experts annotate the ship decision-making behavior based on multi-source data.

3. The ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning according to claim 1, characterized in that, Step (2) is as follows: Encode the text information using a pre-trained large language model to generate text embedding vectors; extract the temporal features of ship speed, heading and position changes from AIS data; use a convolutional neural network to process radar images, identify the environment and obstacles and generate image feature vectors.

4. The ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning according to claim 1, characterized in that, In step (3), the heterogeneous data fusion adopts a cross-attention mechanism, which uses the query vector of one modality to focus on the key and value vector of another modality, thereby realizing dynamic semantic association and fusion between multimodal information.

5. The ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning according to claim 4, characterized in that, The fusion formula is as follows: ； in, A query vector representing data from a modality. and A key-value vector representing data from another modality. This represents the dimension of the key vector, used for scaling. The function is used to convert scores into a probability distribution, thereby assigning weights to different values.

6. The ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning according to claim 1, characterized in that, In step (4), the decision model generates navigation strategies through natural language and triggers alarms when an emergency is detected, pushing decision suggestions and explanations to the crew through voice or visual interfaces.

7. The ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning according to claim 1, characterized in that, In step (5), the feedback mechanism specifically involves continuously optimizing the accuracy and reliability of the decision-making model by comparing human decision-making in the actual deployment environment with user feedback.

8. The ship behavior decision-making method based on heterogeneous information fusion and integrated reasoning according to claim 1, characterized in that, Step (6) includes the following steps: (61) Use the Shapley value to quantify the contribution of different models in each decision task, assuming that there are There are 1 model, and the Shapley value for each model is calculated as follows: ； in, It is a model Shapley value, It is a subset Size, It is a subset The corresponding decision value is based on The model generates prediction results; by traversing all possible model combinations, the model is calculated. The incremental contribution after adding a subset is used to evaluate its marginal contribution throughout the decision-making process; (62) Based on the Shapley value calculation results, assign corresponding weights to each model, and the final decision result is obtained. Generated through weighted voting: ； in, It is the first The output of each model; (63) Update the contribution values ​​of each model to achieve dynamic adaptation to the optimization of decision-making strategies under different environmental conditions.

Images