Ship monitoring system based on multi-agent cooperation

Abstract

The invention discloses a ship monitoring system based on multi-agent collaboration, which comprises an MCP server, a ship operation state analysis module, a visual analysis module, a knowledge question and answer module and a central collaborative decision module, and is characterized in that the MCP server is used for being connected with each sensor and each camera preset on a ship to obtain original sensing data and image data; the ship running state analysis module is connected with the MCP server and determines the running state of the current ship according to the sensing data; the visual analysis module is connected with the MCP server and determines the working condition of the crew on the current ship according to the image data; the knowledge question and answer module is used as a knowledge base for storing ship knowledge and offshore work knowledge; the central collaborative decision-making module is connected with the ship operation state analysis module, the visual analysis module and the knowledge question and answer module and is provided with an early warning unit and an interaction unit interacting with a user, collaborative work of all the modules is achieved, unified processing is conducted on the ship operation state and violation behaviors of sailors on a ship, the management process is simplified, and the management efficiency is improved. The management efficiency is improved.

Description

[0001] Technology Neighborhood

[0002] This invention relates to the field of artificial intelligence technology, and in particular to a ship monitoring system based on multi-agent collaboration. Background Technology

[0003] The safety situation is severe: According to a 2024 report by the International Maritime Organization (IMO), approximately 68% of maritime accidents worldwide are caused by crew misconduct or untimely equipment malfunctions, resulting in economic losses exceeding US$9 billion annually. Operational efficiency is low: Traditional ships rely on human experience for decision-making, with equipment repair response times averaging 48 hours, leading to an unplanned downtime rate as high as 22%. Regulatory pressure is increasing: International maritime conventions have added mandatory provisions for smart ship data recording, requiring the establishment of real-time risk warning systems.

[0004] Current ship management systems suffer from coordination deficiencies. Single-type intelligent systems can only solve problems in their specific scenarios, and there are also information silos, such as video streams and sensor data belonging to independent platforms, which makes it impossible to retrieve the corresponding safety provisions in a timely manner when violations are detected. Summary of the Invention

[0005] The purpose of this invention is to address the technical problems existing in the background art by proposing a ship monitoring system based on multi-agent collaboration.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention provides a ship monitoring system based on multi-agent collaboration, comprising an MCP server, a ship operation status analysis module, a visual analysis module, a knowledge question answering module, and a central collaborative decision-making module. The MCP server connects to pre-installed sensors and cameras on the ship to acquire raw sensor data and image data. The ship operation status analysis module connects to the MCP server and determines the current ship operation status based on the sensor data. The visual analysis module connects to the MCP server and determines the current crew work status based on the image data. The knowledge question answering module serves as a knowledge base for storing ship knowledge and maritime work knowledge. The central collaborative decision-making module connects to the ship operation status analysis module, the visual analysis module, and the knowledge question answering module, and includes an early warning unit and an interactive unit for user interaction.

[0008] Preferably, each sensor includes an electrical sensor, a vibration sensor, a temperature sensor, a pressure sensor, and an attitude sensor.

[0009] Preferably, the visual analysis module has a built-in target recognition unit and a violation judgment unit. The target recognition unit is used to identify and track each crew member in the image data using the deep learning target detection algorithm YOLOv11 and the multi-target tracking algorithm ByteSort. The violation judgment unit is used to send the image data into a preset large visual model to determine whether each crew member in the image data has violated any rules.

[0010] Preferably, the large visual model includes a convolutional neural network and a transformer encoder layer.

[0011] Preferably, the methods for visual large-scale model verification of violations include:

[0012] Convolutional neural networks are used to extract features from image data to obtain human feature sequences.

[0013] A transformer encoder is used to receive sequences of human features and output action categories.

[0014] Preferably, the large visual model further includes a linear transformation layer, which is placed between the convolutional neural network and the transformer encoder layer. The large visual model also verifies violations through the following methods:

[0015] The feature token is obtained by transforming the human feature sequence through a linear transformation layer.

[0016] The location code and category token are added to the feature token and then input into the transformer encoder.

[0017] Preferably, the knowledge question answering module includes interconnected vectorized database construction units and user query and retrieval units;

[0018] The vectorized database construction unit is used to acquire texts of various types of knowledge, extract all text information from the texts, slice and clean each text information to obtain a text slice set formed by multiple text fragments; all text fragments are vectorized and embedded to obtain a text slice vector set, and the text slice vector set is stored in a preset vector database.

[0019] The user query retrieval unit is used to obtain the user's query text and perform vector embedding processing on the query text using a pre-built text vector embedding model to obtain vector Q. Vector Q is compared with the text slice vector set and a recall process is performed to obtain multiple recall texts. The recall texts are sorted according to their relevance to the user's query text. Multiple recall texts with high relevance are selected and concatenated with the user's original query text, and then fed into a pre-built large language model. After the large language model summarizes and sorts the data, it outputs a response index.

[0020] Preferably, the MCP server includes a ship sensor network MCP service unit and a shipborne camera MCP service unit; the ship sensor network MCP service unit is used to provide raw data from each sensor; the shipborne camera MCP service unit is used to provide raw video streams from each camera.

[0021] Compared with existing technologies, this invention has the following beneficial technical effects: It includes an MCP server, a ship operation status analysis module, a visual analysis module, a knowledge question and answer module, and a central collaborative decision-making module. The MCP server is used to connect with various sensors and cameras pre-installed on the ship to obtain raw sensor data and image data; the ship operation status analysis module is connected to the MCP server and determines the current operation status of the ship based on the sensor data; the visual analysis module is connected to the MCP server and determines the current working status of the crew on the ship based on the image data; the knowledge question and answer module serves as a knowledge base for storing ship knowledge and maritime work knowledge; the central collaborative decision-making module is connected to the ship operation status analysis module, the visual analysis module, and the knowledge question and answer module respectively, and is equipped with an early warning unit and an interactive unit for user interaction, realizing the collaborative work of each module, uniformly handling the ship operation status and crew violations on the ship, simplifying the management process, and improving management efficiency. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the structure of an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the structure of the visual analysis module in an embodiment of the present invention;

[0024] Figure 3 This is a flowchart illustrating the text vectorization process of the knowledge question-answering module in this embodiment of the invention.

[0025] Figure 4 This is a flowchart of the user query process for the knowledge question and answer module in this embodiment of the invention;

[0026] Figure 5 This is an architecture diagram of the central collaborative decision-making module in an embodiment of the present invention.

[0027] Figure label:

[0028] 100MCP server;

[0029] 200 Ship Operation Status Analysis Module;

[0030] 300 visual analysis module;

[0031] 400 knowledge Q&A module;

[0032] 500 Central Collaborative Decision-Making Module, 501 Early Warning Unit, 502 Interaction Unit. Detailed Implementation

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0034] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figures 1-5 The ship monitoring system based on multi-agent collaboration in this embodiment of the invention includes an MCP server 100, a ship operation status analysis module 200, a visual analysis module 300, a knowledge question answering module 400, and a central collaborative decision-making module 500. The MCP server 100 is used to connect with various sensors and cameras pre-installed on the ship to obtain raw sensor data and image data. The ship operation status analysis module 200 is connected to the MCP server 100 and determines the current operation status of the ship based on the sensor data. The visual analysis module 300 is connected to the MCP server 100 and determines the current working status of the crew on the ship based on the image data. The knowledge question answering module 400 is used as a knowledge base to store ship knowledge and maritime work knowledge. The central collaborative decision-making module 500 is connected to the ship operation status analysis module 200, the visual analysis module 300, and the knowledge question answering module 400, and is equipped with an early warning unit 501 and an interaction unit 502 for user interaction.

[0035] It should be noted that the Central Collaborative Decision-Making Module 500 includes three working modes: the large model active mode, the video analysis module linkage mode, and the operational status module linkage mode.

[0036] The large language model is connected to the early warning unit 501 and the interaction unit 502. The large language model is then connected to the central collaborative decision-making module 500. The central collaborative decision-making module 500 is then connected to four functional modules: the MCP server 100, the knowledge question answering module 400, the visual analysis module 300, and the ship operation status analysis module 200.

[0037] In the active mode of the large model, when the crew member inputs inquiry information in the interaction unit 502, the large language model first judges the user's intent after receiving the inquiry information. When the user's intent is not related to the central collaborative decision-making module 500, the large model responds based on its own knowledge reserves.

[0038] When a user's intent is related to topics such as "want to check the status of shipboard equipment" or "check the camera", the central collaborative decision-making module 500 will call the relevant MCP service through the MCP server 100 to obtain the raw device data information based on the device ID information entered by the user. After summarizing, it will be returned to the large language model, which will then process it and return it to the interaction unit 502.

[0039] When a user's intent is related to topics such as "nautical knowledge", "equipment maintenance", or "knowledge base", the central collaborative decision-making module 500 will send the user's query information to the knowledge question-and-answer module 400 through the interface of the knowledge question-and-answer module 400. The knowledge question-and-answer module 400 will then retrieve relevant information from the database based on the user's query information and return it to the large language model. The large language model will then process the information and return it to the interaction unit 502.

[0040] The video analysis modules work in a coordinated manner as follows: When the visual analysis module 300 detects a user's violation, it obtains the violation type and reports it to the central collaborative decision-making module 500. The central collaborative decision-making module 500 then retrieves the corresponding processing suggestions and subsequent procedures from the knowledge-based question-and-answer module 400. Additionally, the central collaborative decision-making module 500 obtains the raw video stream from the camera that captured the user's violation through the MCP server 100 for evidence preservation. Finally, the central collaborative decision-making module 500 sends the user's violation type, corresponding processing suggestions, subsequent procedures, and the raw video evidence to the large model, which then processes the data and sends it to the early warning unit 501.

[0041] The operational status linkage mode of each module is as follows: When the ship's operational status analysis module 200 detects an anomaly in a sensor within the ship's sensor network, it determines the type of the abnormal sensor, its ID number, and the type of sensor anomaly, and then sends this information to the central collaborative decision-making module 500. The central collaborative decision-making module 500 obtains the relevant data from the abnormal sensor and sends it to the knowledge-based question-and-answer module 400. The knowledge-based question-and-answer module 400 retrieves data related to the sensor anomaly (repair methods, standard repair procedures, etc.) and returns it to the central collaborative decision-making module 500. Additionally, the central collaborative decision-making module 500 retrieves the original data of the abnormal sensor through the MCP service based on its ID number. Finally, the central collaborative decision-making module 500 combines the abnormal sensor ID and its anomaly information with the repair methods and procedures returned by the knowledge-based question-and-answer module 400 and the original data of the abnormal device returned by the MCP server, and sends this data to the large language model. The large language model then processes the data and sends it to the early warning unit 501.

[0042] The large language model uses the DeepSeek-R1-Distill-Qwen-32B model.

[0043] In one embodiment of this application, each sensor includes an electrical sensor, a vibration sensor, a temperature sensor, a pressure sensor, and an attitude sensor.

[0044] It should be noted that the ship operation status analysis module 200 also includes an anomaly detector, which receives data from electrical sensors, vibration sensors, temperature sensors, pressure sensors, and attitude sensors. When the detector detects an anomaly in the signal from an connected sensor, it outputs the ID number of the device to which the anomaly sensor belongs, as well as the type of anomaly.

[0045] The anomaly detector follows these criteria to identify abnormal signals: For electrical sensors, temperature sensors, pressure sensors, and attitude sensors, a high threshold τh and a low threshold τl are set. When the sensor value is below τl, a low threshold alarm is triggered; when the sensor value is above τh, a high threshold alarm is triggered. For vibration sensor anomalies, vibration sensor values ​​need to be sampled to obtain a numerical sequence of length L. A deep learning LSTM model is then used to output whether the vibration numerical sequence is abnormal and the type of vibration anomaly.

[0046] In one embodiment of this application, the visual analysis module 300 has a built-in target recognition unit and a violation judgment unit. The target recognition unit is used to identify and track each crew member in the image data using the deep learning target detection algorithm YOLOv11 and the multi-target tracking algorithm ByteSort. The violation judgment unit is used to send the image data into a preset large visual model to determine whether each crew member in the image data has violated any rules.

[0047] In one embodiment of this application, the large visual model includes a convolutional neural network and a transformer encoder layer.

[0048] In one embodiment of this application, the method for verifying violations using a large visual model includes:

[0049] Convolutional neural networks are used to extract features from image data to obtain human feature sequences.

[0050] A transformer encoder is used to receive sequences of human features and output action categories.

[0051] In one embodiment of this application, the large visual model further includes a linear transformation layer, which is placed between the convolutional neural network and the transformer encoder layer. The large visual model also verifies violations through the following methods:

[0052] The feature token is obtained by transforming the human feature sequence through a linear transformation layer.

[0053] The location code and category token are added to the feature token and then input into the transformer encoder.

[0054] It should be noted that the process first acquires video streams from each camera, then uses the YOLOv11 deep learning object detection algorithm for human target detection, followed by the ByteSort multi-target tracking algorithm to track the human target and obtain its tracking ID. For the same tracking ID, cropped human images from the preceding and following 15 frames are used to form a human image sequence. Finally, the human image sequence is fed into a large-scale visual model, which outputs whether any violations occur and the type of violation.

[0055] Specifically, the large-scale visual model in this section adopts a convolutional neural network + transformer architecture, and its structure is as follows: Figure 4 As shown, the human images in the input image sequence are first processed by a convolutional neural network to extract features, resulting in a human feature sequence F = [f0, f1, f2, ..., f14]. Then, the feature sequence F is transformed into the input format required by the Transformer encoder after passing through a linear transformation layer. After being fed into the Transformer encoder, the action category is output.

[0056] It is important to note that the feature sequence F undergoes a linear transformation to obtain feature tokens. These tokens are then appended with positional encoding before being fed into the Transformer encoder layer. Furthermore, a category token embedding needs to be added before the token sequence before feeding it into the Transformer encoder layer. After Transformer encoding, the output category is obtained by decoding the output corresponding to the category token embedding.

[0057] In one embodiment of this application, the knowledge question answering module 400 includes an interconnected vectorized database construction unit and a user query retrieval unit;

[0058] The vectorized database construction unit is used to acquire texts of various types of knowledge, extract all text information from the texts, slice and clean each text information to obtain a text slice set formed by multiple text fragments; all text fragments are vectorized and embedded to obtain a text slice vector set, and the text slice vector set is stored in a preset vector database.

[0059] The user query retrieval unit is used to obtain the user's query text and perform vector embedding processing on the query text using a pre-built text vector embedding model to obtain vector Q. Vector Q is compared with the text slice vector set and a recall process is performed to obtain multiple recall texts. The recall texts are sorted according to their relevance to the user's query text. Multiple recall texts with high relevance are selected and concatenated with the user's original query text, and then fed into a pre-built large language model. After the large language model summarizes and sorts the data, it outputs a response index.

[0060] It should be noted that the text formats used for various knowledge materials can include Txt, Markdown, Word, Excel, PDF, etc., and these texts need to be processed by a pre-built text parser to extract the corresponding text information. Then, this text information is sliced ​​and cleaned, including the following steps:

[0061] Text cleaning: Remove redundant spaces, special characters, and formatting marks;

[0062] Text standardization: English characters should be uniformly converted to lowercase, and traditional Chinese characters should be converted to simplified Chinese.

[0063] Paragraph division: Divide the text into paragraphs according to paragraph marks;

[0064] Text Slicing: A hybrid slicing method combining fixed-length slices and semantically aware slices is used to divide text paragraphs into the smallest possible fragments.

[0065] It should be added that the user query text vector embedding also uses the bge-large-zh-v1.5 model, and the rerank model is used to rank the recalled texts according to their relevance to the user query text. The rerank model uses the bge-reranker-large model, and the large language model uses the Qwen3-8B model. In addition, the specific execution process of vector recall is as follows: after obtaining the user query vector, the cosine similarity is calculated with the vectors in the vector database. Only when the similarity exceeds a threshold is the smallest fragment of the original text corresponding to the vector recalled.

[0066] In one embodiment of this application, the MCP server 100 includes a ship sensor network MCP service unit and a shipborne camera MCP service unit.

[0067] The ship sensor network MCP service unit is used to provide raw data from each sensor;

[0068] The shipborne camera MCP service unit is used to provide raw video streams from each camera.

[0069] The above describes a method or various implementations for ship monitoring based on multi-agent collaboration, but it is not intended that the specific implementation of this invention is limited to these descriptions. Any methods or structures that are similar to or identical to those of this invention, or any technical deductions or substitutions made based on the concept of this invention, should be considered within the scope of protection of this invention.

Claims

1. A ship monitoring system based on multi-agent collaboration, characterized in that, include: The MCP server is used to connect with the various sensors and cameras pre-installed on the ship to obtain raw sensor data and image data. The ship operation status analysis module is connected to the MCP server and determines the current operation status of the ship based on the sensor data. A visual analysis module is connected to the MCP server and determines the current working status of the crew on the ship based on the image data. The knowledge question and answer module is used as a knowledge base to store ship knowledge and knowledge about maritime operations. The central collaborative decision-making module is connected to the ship operation status analysis module, the visual analysis module, and the knowledge question and answer module, and is equipped with an early warning unit and an interactive unit for user interaction.

2. The ship monitoring system based on multi-agent collaboration according to claim 1, characterized in that, Each of the aforementioned sensors includes an electrical sensor, a vibration sensor, a temperature sensor, a pressure sensor, and an attitude sensor.

3. A ship monitoring system based on multi-agent collaboration according to claim 2, characterized in that, The visual analysis module has a built-in target recognition unit and a violation judgment unit. The target recognition unit is used to identify and track each crew member in the image data using the deep learning target detection algorithm YOLOv11 and the multi-target tracking algorithm ByteSort. The violation judgment unit is used to send the image data into a preset large visual model to determine whether each crew member in the image data has violated any rules.

4. The ship monitoring method based on multi-agent collaboration according to claim 3, characterized in that, The large visual model includes a convolutional neural network and a transformer encoder layer.

5. A ship monitoring system based on multi-agent collaboration according to claim 4, characterized in that, The methods for verifying violations using the large visual model include: The convolutional neural network is used to extract features from the image data to obtain a human feature sequence. The transformer encoder is used to receive the human feature sequence to output the action category.

6. A ship monitoring system based on multi-agent collaboration according to claim 5, characterized in that, The large visual model further includes a linear transformation layer, which is placed between the convolutional neural network and the transformer encoder layer. The large visual model also verifies violations through the following methods: The human feature sequence is transformed by the linear transformation layer to obtain the feature token; The location code and category token are added to the feature token and then input into the transformer encoder.

7. A ship monitoring system based on multi-agent collaboration according to claim 6, characterized in that, The knowledge question answering module includes interconnected vectorized database construction units and user query and retrieval units; The vectorized database construction unit is used to acquire texts of various types of knowledge, extract all text information from the texts, slice and clean each text information to obtain a text slice set formed by multiple text fragments; perform vectorized embedding processing on all the text fragments to obtain a text slice vector set, and store the text slice vector set in a preset vector database. The user query retrieval unit is used to obtain the user's query text, and to perform vectorization embedding processing on the query text using a preset text vector embedding model to obtain vector Q. The vector Q is compared with the text slice vector set and a recall process is performed to obtain multiple recall texts. The recall texts are sorted according to their relevance to the user's query text. Multiple recall texts with high relevance are selected and concatenated with the user's original query text, and then sent to a preset large language model. After the large language model summarizes and sorts the data, a response index is output.

8. A ship monitoring system based on multi-agent collaboration according to claim 7, characterized in that, The MCP server includes a ship sensor network MCP service unit and a shipborne camera MCP service unit. The ship sensor network MCP service unit is used to provide raw data from each of the sensors; The shipborne camera MCP service unit is used to provide the raw video streams of each of the cameras.

Images