Ship intelligent driving control system and method based on multi-modal interaction and cloud collaboration and ship

Abstract

The invention discloses a ship intelligent driving control system and method based on multi-modal interaction and cloud cooperation and a ship, and belongs to the field of intelligent ships. The system comprises a central processing unit, a multi-mode interaction module, a cloud collaborative battery replacement management module, a graded alarm and intelligent broadcast module, an electronic navigation log module, a multi-source data fusion display module, a digital electronic control integration module and a cloud platform. Wherein the multi-modal interaction module, the cloud collaborative battery replacement management module, the graded alarm and intelligent broadcast module, the electronic navigation log module, the multi-source data fusion display module, the digital electronic control integration module and the cloud platform are respectively connected with the central processing unit, and interaction of control instructions and data is realized through the central processing unit. According to the invention, all-around and integrated intelligent management and control of ship navigation, electromechanics, battery replacement, logs and the like are realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent ships. Specifically, this invention relates to an intelligent ship control system, method, and ship based on multimodal interaction and cloud collaboration. Background Technology

[0002] With the continuous development of the shipping industry, the level of intelligence and automation of ships is increasing. Traditional ship control consoles typically consist of separate, single-function devices, resulting in fragmented information displays, cumbersome operating procedures, and high demands on the experience and attention of the pilots. For example, pilots need to simultaneously monitor multiple screens, including radar, AIS (Automatic Identification System), electronic charts, and engine room monitoring, leading to severe information silos and hindering the formation of a unified navigational situational awareness, thus affecting decision-making efficiency and accuracy. Furthermore, with the application of new energy technologies in the shipping field, especially the emergence of battery swapping, how to efficiently and intelligently manage the ship's refueling process has become a new challenge.

[0003] While some ships have adopted centralized displays and touch controls, the human-machine interaction methods remain relatively simplistic. Advanced technologies such as voice control and AI visual monitoring are not yet widely applied in ship navigation, and a system that organically integrates multiple interaction methods is lacking. Furthermore, traditional alarm methods primarily rely on uniform audible and visual alarms, lacking the ability to categorize alarms based on urgency and provide intelligent broadcasting, which can easily lead to pilots overlooking critical alarm information in complex situations. Navigation log entries largely depend on manual entry, which is not only inefficient and prone to errors and omissions but also hinders subsequent data analysis and traceability.

[0004] Therefore, the market urgently needs a brand-new intelligent ship control system that can deeply integrate internal and external ship data, simplify operations through multimodal interaction, realize intelligent scheduling of the battery swapping process, provide hierarchical and intelligent alarm prompts, and automatically generate compliant electronic navigation logs, thereby comprehensively improving the safety, economy, and convenience of ship navigation.

[0005] Therefore, this invention proposes a ship intelligent driving control system, method, and ship based on multimodal interaction and cloud collaboration. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of existing technologies and proposes a ship intelligent navigation and control system, method and ship based on multimodal interaction and cloud collaboration, in order to achieve the following objectives: to realize comprehensive and integrated intelligent management and control of ship navigation, electromechanical, battery swapping and logs.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a ship intelligent navigation and control system based on multimodal interaction and cloud collaboration, the system comprising a central processing unit, a multimodal interaction module, a cloud-collaborative battery swapping management module, a hierarchical alarm and intelligent broadcasting module, an electronic navigation log module, a multi-source data fusion display module, a digital electronic control integration module, and a cloud platform; wherein the multimodal interaction module, the cloud-collaborative battery swapping management module, the hierarchical alarm and intelligent broadcasting module, the electronic navigation log module, the multi-source data fusion display module, the digital electronic control integration module, and the cloud platform are respectively connected to the central processing unit, and the interaction of control commands and data is realized through the central processing unit.

[0008] Preferably, the multimodal interaction module is used to realize the interaction between the system and the driver, including a voice interaction unit, a touch display unit, and an AI visual recognition unit, wherein the voice interaction unit, the touch display unit, and the AI ​​visual recognition unit are respectively connected to the central processing unit.

[0009] Preferably, the cloud-based collaborative battery swapping management module is used to make intelligent battery swapping decisions for ships through cloud collaboration, and it communicates in real time with the remote cloud platform data base through the shipborne network interface of the central processing unit.

[0010] Preferably, the graded alarm and intelligent broadcasting module is used for monitoring and alarming the entire ship. The graded alarm and intelligent broadcasting module includes an alarm rule engine, which is used to perform graded alarms based on the alarm source and parameters.

[0011] Preferably, the electronic navigation log module includes a log template automatic matching unit, a multi-source data automatic collection and filling unit, an electronic signature and revision recording unit, and an abnormal behavior automatic annotation unit; wherein, the log template automatic matching unit is used to automatically match and load log templates according to preset rules; the multi-source data automatic collection and filling unit is used to collect multi-source data of the ship and fill it into the log template; the electronic signature and revision recording unit is used to bind the log through electronic signature and record the log revision operation; the abnormal behavior automatic annotation unit is used to perform real-time analysis of the operation log based on preset rules, and automatically identify and annotate abnormal behaviors.

[0012] Preferably, the multi-source data fusion display module is used to acquire multi-source ship data and execute a data fusion algorithm to generate a unified navigation situation map and send it to the central processing unit to call the multimodal interaction module for display.

[0013] Preferably, the digital electronic control integrated module adopts a distributed control architecture, in which each electromechanical device on board is connected to one or more field controllers via an industrial bus or Ethernet, and all field controllers are then connected to the central processing unit via the ship's local area network.

[0014] This invention also provides a ship intelligent control method based on multimodal interaction and cloud collaboration, using the aforementioned ship intelligent control system based on multimodal interaction and cloud collaboration, characterized in that: the method includes:

[0015] Step S1: Start the system, each module starts running, and collects real-time data from ship sensors, cloud platform data, and cockpit environment video data;

[0016] Step S2: The multimodal interaction module continuously listens for voice commands and touch operations, and analyzes the video stream in real time; upon receiving a command, it parses its intent and sends it to the central processing unit to call the corresponding functional module to perform the operation;

[0017] Step S3: The cloud-based collaborative battery swapping management module determines in real time whether battery swapping is needed, and executes intelligent battery swapping decisions when battery swapping is required;

[0018] Step S4: The graded alarm and intelligent broadcast module polls and monitors the system data in real time. Once an abnormal parameter is detected or an alarm signal is received, the alarm level is determined and fed back to the central processing unit to invoke the multimodal interaction module to execute the corresponding alarm prompt strategy.

[0019] Step S5: During system operation, the electronic navigation log module runs silently in the background, continuously and automatically recording various navigation and operation data according to the preset template, and completing the signing and management of the log based on the pilot's interaction.

[0020] Step S6: The multi-source data fusion and display module performs fusion calculations on multi-source sensor data to generate a unified navigation situation map, and sends the unified navigation situation map to the central processing unit to call the multimodal interaction module for display.

[0021] Step S7: When the system receives the driver's equipment control command, the central processing unit sends the corresponding control message to the digital electronic control integration module, drives the corresponding actuator to complete the operation through the digital electronic control integration module, and feeds back the operation result to the central processing unit to call the multimodal interaction module for display.

[0022] Preferably, step S3 includes:

[0023] When the cloud-based collaborative battery swapping management module detects that the ship's battery level is below a preset threshold or receives a battery swapping request, it determines that a battery swapping is necessary; otherwise, a battery swapping is not necessary.

[0024] When battery swapping is needed, the cloud-based collaborative battery swapping management module requests real-time data from nearby battery swapping stations from the cloud platform;

[0025] The cloud-based collaborative battery swapping management module uses a weighted multi-attribute decision-making method to comprehensively evaluate the advantages and disadvantages of each battery swapping station and generate one or more recommended solutions.

[0026] The cloud-based collaborative battery swapping management module sends the recommended solution to the central processing unit, which then calls the multimodal interaction module to display the recommended solution to the driver; the driver's selection result is fed back to the cloud-based collaborative battery swapping management module.

[0027] After receiving the recommended route selected by the driver, the cloud-based collaborative battery swapping management module sends a reservation instruction to the cloud platform to lock the battery; at the same time, the central processing unit sets the battery swapping station to which the recommended route belongs as the current navigation target.

[0028] The present invention also provides a ship, the ship including the above-described intelligent ship control system based on multimodal interaction and cloud collaboration.

[0029] The technical effects of this invention are as follows:

[0030] This invention enhances the naturalness and convenience of human-computer interaction: by integrating three interaction methods—voice, touch, and AI vision—drivers can choose the most natural way to operate according to their habits and scenarios, reducing operational complexity and cognitive load. Especially during high-intensity operations, voice control and AI safety reminders can significantly improve operational efficiency and safety.

[0031] This invention realizes intelligent and efficient ship refueling: the innovative cloud-based collaborative battery swapping management module automates and intelligently resolves the complex battery swapping decision-making process, effectively solving the "range anxiety" of electric ships and improving the operational efficiency and economy of ships.

[0032] This invention enhances the effectiveness and relevance of the alarm system: the tiered alarm mechanism enables the driver to quickly distinguish and respond to critical alarms, avoiding information overload, and the intelligent broadcast function ensures that important information is not missed, significantly improving the navigation safety of the ship.

[0033] This invention ensures the standardization and traceability of navigation logs: the electronic and automated log recording method not only reduces the paperwork burden on pilots but also ensures the accuracy, completeness, and compliance of the logs. The application of electronic record keeping and biometric technology greatly enhances the legal validity and traceability of the logs.

[0034] This invention improves the comprehensiveness and accuracy of navigation situational awareness: by fusing multi-source data, it integrates previously scattered information into a unified and intuitive situational map. In particular, the precise dynamic data provided during critical phases such as berthing and unberthing provides unprecedented decision support for pilots and reduces the risk of accidents.

[0035] This invention realizes the intensive and integrated control of ship equipment: the digital electronic control integration module unifies the control of complex ship electrical equipment onto a single platform, simplifies the physical layout and operation process of the bridge, and improves the integration and automation level of the entire ship. Attached Figure Description

[0036] Figure 1 This is a structural block diagram of a ship intelligent driving control system based on multimodal interaction and cloud collaboration. Detailed Implementation

[0037] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. This is to help those skilled in the art to have a more complete, accurate, and in-depth understanding of the inventive concept and technical solutions of the present invention, and to facilitate its implementation. It should be noted that the terms "first," "second," etc., used in this application are only for the convenience of describing the technical solutions and to distinguish components; the corresponding component configurations may be the same or different, and are not intended to limit the scope of this application. To make the technical solutions of the present invention clearer, the present invention will be explained and illustrated through the following embodiments.

[0038] This embodiment provides a ship intelligent navigation and control system based on multimodal interaction and cloud collaboration, aiming to solve the technical problems of isolated information, simple interaction, and low level of intelligence in existing ship navigation and control systems, and to achieve comprehensive and integrated intelligent management and control of ship navigation, electromechanical systems, battery swapping, and logs. Figure 1 As shown, the system includes a central processing unit, a multimodal interaction module, a cloud-based collaborative battery swapping management module, a hierarchical alarm and intelligent broadcasting module, an electronic navigation log module, a multi-source data fusion display module, a digital electronic control integration module, and a cloud platform. The multimodal interaction module, cloud-based collaborative battery swapping management module, hierarchical alarm and intelligent broadcasting module, electronic navigation log module, multi-source data fusion display module, digital electronic control integration module, and cloud platform are all connected to the central processing unit, enabling the interaction of control commands and data through the central processing unit.

[0039] Specifically, the central processing unit (CPU) is the brain of the entire system, responsible for running core software algorithms and scheduling and coordinating the work of all other modules. It can be a high-performance marine-grade server or domain controller, possessing powerful computing capabilities and compatibility with multiple operating systems (such as Linux, HarmonyOS, Android, etc.) to ensure stable and reliable system operation.

[0040] The multimodal interaction module of this embodiment is used to realize the interaction between the system and the driver, including a voice interaction unit, a touch display unit, and an AI visual recognition unit, which are respectively connected to the central processing unit.

[0041] The voice interaction unit is responsible for receiving and parsing the driver's voice commands. It features a built-in high signal-to-noise ratio microphone array and an advanced speech recognition engine. It supports custom wake words (such as "Hello, Captain") and can accurately recognize driver voice commands in noisy cockpit environments, such as "Check remaining battery power," "Plan route to the nearest battery swapping station," and "Turn on navigation lights." The unit also integrates a TTS (Text-to-Speech) engine, capable of converting text information (such as alarm messages, battery swapping suggestions, and navigation prompts) into natural and fluent speech for playback.

[0042] The voice interaction unit employs a multi-level processing architecture to ensure a high recognition rate in complex ship environments. Specific implementation details include:

[0043] Wake-up word detection layer: Employs a lightweight keyword recognition model (such as a DNN-HMM-based or end-to-end KWS model), which runs continuously on the local embedded processor, offering low power consumption and fast response. The wake-up word recognition rate is required to reach over 99%, with a false wake-up rate controlled to less than once every 48 hours. The system supports user-defined wake-up words, and personalized training can be completed with a small number of samples (5-10 recordings).

[0044] Speech Recognition Engine: Upon activation, the system feeds subsequent audio streams into a complete Automatic Speech Recognition (ASR) engine. This engine employs acoustic and language models based on deep neural networks and has been optimized for training with specialized marine terminology (such as "battery swapping station," "berthing," and "bilge pump"). Under ambient noise conditions of 70dB, the command recognition accuracy is required to be no less than 90%.

[0045] Semantic understanding and intent recognition: The recognized text is fed into the Natural Language Understanding (NLU) module, which uses a pre-trained intent classifier and slot filling model to convert the user's natural language commands into structured control commands.

[0046] Voice Synthesis Broadcast: The system integrates a high-quality TTS voice synthesis engine, capable of converting text information into natural and fluent speech. This engine supports various broadcast scenarios, including timed broadcasts (such as broadcasting remaining battery power hourly), event-triggered broadcasts (such as when an alarm occurs), and responsive broadcasts (such as answering driver inquiries). Voice output supports dynamic adjustment of volume, speech rate, and timbre to adapt to different environments and user preferences.

[0047] The touch display unit provides a graphical, centralized information display and touch operation interface, employing at least one or more high-resolution, high-brightness industrial-grade touchscreens. A graphical user interface (GUI) runs on the screen, displaying navigation status, electromechanical status, and monitoring video information in a centralized and intuitive manner through instrument panels, electronic charts, 3D renderings, and other formats. The operator can query information and control equipment using gestures such as clicking and swiping.

[0048] The AI ​​visual recognition unit captures real-time video streams from inside the cockpit using a wide-angle camera installed within the cockpit. The video stream is fed into an existing deep learning-based AI behavior recognition model for analysis. This model can detect in real-time whether the driver is fatigued (e.g., frequent yawning, eye closure), whether they are wearing a life jacket and helmet as required, and whether they are engaging in violations such as smoking or making phone calls. Once an anomaly is detected, the system can issue alarms or reminders via the voice interaction unit and touch display unit. For example, a convolutional neural network (CNN) combined with a temporal modeling network (LSTM) is used to construct the behavior recognition model. This model can identify several key behavioral states, including:

[0049] Fatigue driving detection: By detecting features such as eye closure frequency, number of yawns, and head posture (looking down, tilting head), the driver's fatigue level is comprehensively judged. When PERCLOS exceeds a threshold (e.g., the percentage of time with eyes closed exceeds 20% within 30 consecutive seconds), a fatigue alarm is triggered.

[0050] Safety equipment detection: The system uses a target detection algorithm (YOLO) to identify whether the driver is wearing protective equipment such as life jackets and helmets. The system enforces detection in specific scenarios (such as severe weather or nighttime navigation) and issues a warning if the equipment is not worn.

[0051] Violation detection: Identifies whether drivers are engaging in violations such as smoking, using mobile phones, or leaving their posts. It uses technologies such as gesture recognition and object detection to determine and record violations in real time.

[0052] Watchkeeping and lookout detection: By using face detection and head posture estimation, it determines whether the driver is on duty and maintaining a normal lookout posture, ensuring compliance with maritime regulations for regular lookout.

[0053] The cloud-based collaborative battery swapping management module in this embodiment is used for intelligent battery swapping decisions for ships via cloud collaboration. It communicates in real time with the remote cloud platform data base through the shipboard network interface of the central processing unit. Its core function is as follows: when the system detects that the battery level is lower than a preset threshold (e.g., 20%) or the driver actively initiates a battery swapping request, the cloud-based collaborative battery swapping management module immediately queries the cloud platform for detailed information on nearby battery swapping stations, including the geographical location of each station, current queuing status, number of available battery packs, and state of health (SOH). The module internally runs a multi-factor decision-making algorithm, comprehensively considering the ship's current location, range, voyage plan, distance to the battery swapping station, and load status, to calculate the optimal battery swapping plan (e.g., "Recommended to go to XX battery swapping station, expected arrival in 15 minutes, battery reserved for you"), and displays the recommendation result on the touch display unit. After the driver confirms, the system sends a command to the cloud platform to lock the reserved battery, achieving "battery ready before the ship arrives".

[0054] The multi-factor decision-making algorithm employs a weighted multi-attribute decision-making method to calculate a comprehensive score (Score_i) for each candidate battery swapping station to determine the optimal recommendation. The comprehensive score formula is as follows:

[0055] Score_i = w1 * f(d_i) + w2 * f(n_i) + w3 * f(t_i) + w4 * f(h_i) + w5* f(c_i);

[0056] Where w1~w5 are weighting coefficients; f is a normalization function used to unify parameters of different dimensions to a scoring range of 0-1; d_i (distance): the flight distance to the battery swapping station; n_i (availability): the number of available battery packs; t_i (waiting time): the estimated queuing waiting time; h_i (battery health status SOH): the battery health status; c_i (cost): the battery swapping cost.

[0057] Ultimately, the multi-factor decision-making algorithm outputs the top three battery swapping stations with the highest scores and displays them to the user via a touch screen. After the driver selects a swapping station, the cloud-based collaborative battery swapping management module sends a reservation request to the cloud platform via a RESTful API or WebSocket protocol. The request includes information such as the vessel ID, swapping station ID, and estimated arrival time. Upon receiving the request, the cloud platform immediately locks a set of batteries at that swapping station and returns reservation confirmation information. Preferably, the system calls a route planning algorithm (such as a path search algorithm based on A* or Dijkstra, combined with information such as water depth and channel restrictions on the electronic nautical chart) to generate the optimal route from the current location to the swapping station. This route is stored as a sequence of waypoints and marked with a prominent color (such as a blue dashed line) on the electronic nautical chart on the touch screen to guide the driver.

[0058]

[0059] Table 1

[0060] The hierarchical alarm and intelligent broadcasting module in this embodiment is used for monitoring and alarming the entire ship. This module includes an alarm rule engine, which is used to classify alarms based on their source and parameters. The alarm rule engine maintains an alarm rule library, where each rule defines the source, triggering conditions, level, and processing strategy of the alarm. An exemplary alarm rule library is shown in Table 1.

[0061] The alarm rule engine receives alarm signals from various monitoring systems in real time, matches and classifies them according to the rule base, and feeds them back to the central processing unit to call the corresponding output module to execute alarm actions.

[0062] In addition to passive alarm response, the tiered alarm and intelligent broadcast module also has active broadcast capabilities. Active broadcasting is event-driven or timed.

[0063] Scheduled broadcasts: such as broadcasting the remaining battery power every hour on the hour. The tiered alarm and intelligent broadcast module is triggered by a timer, reads the current SOC from the BMS, generates broadcast text (such as "Current remaining battery power 65%, estimated range of 80 kilometers"), and broadcasts it through the TTS engine.

[0064] Event-driven broadcasting: When a maritime alert is received, the cloud platform sends the maritime alert to the central processing unit via message queue or push notification. Upon receiving the alert, the central processing unit immediately calls the touch display unit to display the alert content and calls the voice interaction unit to broadcast it via TTS to ensure that the driver is informed in a timely manner.

[0065] The electronic navigation log module in this embodiment includes an automatic log template matching unit, an automatic collection and filling unit for multi-source data, an electronic signature and revision tracking unit, and an automatic annotation unit for abnormal behavior.

[0066] The automatic log template matching unit is used to automatically match and load log templates according to preset rules. During system initialization, the automatic log template matching unit automatically executes the template matching algorithm based on pre-entered basic ship information (gross tonnage, main engine power). For example, according to Article 18 of the "Rules for Recording Navigation Logs of Inland Waterway Vessels":

[0067] Vessels with a gross tonnage of less than 150 tons and an main engine power of less than 220 kW shall use the HC-I type logbook template (simplified version, fewer record items); vessels with a gross tonnage between 150 tons and 3000 tons, or a main engine power between 220 kW and 1470 kW, shall use the HC-II type logbook template (standard version); vessels with a gross tonnage greater than 3000 tons or a main engine power greater than 1470 kW shall use the HC-III type logbook template (detailed version, most record items). The logbook template automatic matching unit will automatically select and load the corresponding template according to the above rules to ensure that the logbook format complies with regulatory requirements.

[0068] The multi-source data automatic acquisition and filling unit is used to collect multi-source data of the ship and fill it into the log template. The multi-source data of the ship includes navigation data, engine room data, meteorological and hydrological data, operation logs, alarm events, etc.

[0069] The electronic signature and revision recording unit is used to bind logs via electronic signatures and record log revisions. At the end of each voyage or shift, the pilot is required to sign the log. The system provides electronic signature functionality; the pilot can sign via a handwriting pad on the touchscreen, and the signature image is encrypted, stored, and linked to the log. Optionally, the system can be equipped with facial recognition to automatically capture the pilot's face and verify their identity during signing, further enhancing the log's legal validity. The log supports post-revision revisions (e.g., if the pilot discovers an error in data entry), but all revisions are fully recorded, including the revision time, the reviser, and the content before and after the revision. Revision records are permanently stored electronically, ensuring the log's traceability and non-repudiation.

[0070] The automatic abnormal behavior labeling unit is used to perform real-time analysis of operation logs based on preset rules, automatically identifying and labeling abnormal behaviors. For example:

[0071] Frequent operation: If the same equipment is switched on and off more than 10 times in a short period of time (e.g., within 5 minutes), it may indicate equipment failure or improper operation by the driver.

[0072] Improper operation: such as turning off navigation lights during navigation, or using certain equipment in prohibited areas.

[0073] Prolonged inactivity: If there is no operation record for more than 30 minutes and the vessel is underway, it may indicate that the operator is off duty or fatigued.

[0074] After detecting an anomaly, the automatic annotation unit highlights the relevant entries in the operation log.

[0075] The multi-source data fusion display module in this embodiment is used to acquire multi-source ship data and execute a data fusion algorithm to generate a unified navigation situation map, which is then sent to the central processing unit to invoke the multimodal interaction module for display. The data fusion algorithm adopts a layered fusion architecture, including data layer fusion, feature layer fusion, and decision layer fusion.

[0076] Data layer fusion (spatiotemporal registration): Data from different sensors have different timestamps, coordinate systems, and update frequencies. Data layer fusion first performs spatiotemporal alignment on this data. Temporal alignment uses interpolation or extrapolation methods to unify all data to the same time point; spatial alignment uses coordinate transformation to unify the coordinate systems of different sensors (such as polar coordinates of radar, geodetic coordinates of GPS, and pixel coordinates of video) to the ship's coordinate system or geographic coordinate system.

[0077] Feature layer fusion (target tracking and association): For the same target (such as another ship), different sensors may provide different observation results. For example, radar provides the target's range and bearing, AIS provides the target's identity and heading, and video provides the target's appearance features. Feature layer fusion uses multi-target tracking algorithms (such as Joint Probabilistic Data Association (JPDA) or Multiple Hypothesis Tracking (MHT)) to associate these observation results with the same target and estimates the target's optimal state (position, velocity, heading, etc.) through methods such as Kalman filtering or particle filtering.

[0078] Decision-level fusion (situation assessment): After obtaining the fused tracks of all targets, a situation assessment is further conducted to calculate key parameters such as CPA (closest encounter distance), TCPA (time to closest encounter), collision risk index, etc., and targets are classified into threat levels (such as high risk, medium risk, low risk) based on these parameters to provide decision support for the driver.

[0079] Preferably, in berthing and unberthing modes, the multi-source data fusion display module switches to a high-precision display mode, rendering the relative position of the ship and the dock in real time, and displaying the lateral and longitudinal distances of the bow and stern to the dock, the approach speed, and the angle between the ship and the shoreline in digital and graphical form. All this data is sent to the touch display unit in digital and graphical form through the central processing unit for display, thereby assisting the operator in accurately controlling the ship's berthing process.

[0080] The digital electronic control integrated module in this embodiment adopts a distributed control architecture. Various electromechanical devices on board are connected to one or more field controllers via industrial bus or Ethernet. All field controllers are then connected to the central processing unit via the ship's local area network. Correspondingly, the central processing unit runs device management software, which maintains a device database recording information such as the type, address, control protocol, and current status of all controllable devices. When the operator issues a control command, the software searches for the target device according to the device database, generates the corresponding control message (such as a CAN frame or Modbus command), and sends it to the corresponding field controller via the network. After receiving the command, the field controller drives the actuators (such as relays, frequency converters, and electric valve actuators) to complete the actual control action.

[0081] This embodiment also provides a ship intelligent driving control method based on multimodal interaction and cloud collaboration. Using the above-mentioned ship intelligent driving control system based on multimodal interaction and cloud collaboration, the method includes:

[0082] Step S1: Start the system, each module starts running, and collects real-time data from ship sensors, cloud platform data, and cockpit environment video data;

[0083] Step S2: The multimodal interaction module continuously listens for voice commands and touch operations, and analyzes the video stream in real time; upon receiving a command, it parses its intent and sends it to the central processing unit to call the corresponding functional module to perform the operation;

[0084] Step S3: The cloud-based collaborative battery swapping management module determines in real time whether battery swapping is needed, and executes intelligent battery swapping decisions when battery swapping is required;

[0085] Step S4: The graded alarm and intelligent broadcast module polls and monitors the system data in real time. Once an abnormal parameter is detected or an alarm signal is received, the alarm level is determined and fed back to the central processing unit to invoke the multimodal interaction module to execute the corresponding alarm prompt strategy.

[0086] Step S5: During system operation, the electronic navigation log module runs silently in the background, continuously and automatically recording various navigation and operation data according to the preset template, and completing the signing and management of the log based on the pilot's interaction.

[0087] Step S6: The multi-source data fusion and display module performs fusion calculations on multi-source sensor data to generate a unified navigation situation map, and sends the unified navigation situation map to the central processing unit to call the multimodal interaction module for display.

[0088] Step S7: When the system receives the driver's equipment control command, the central processing unit sends the corresponding control message to the digital electronic control integration module, drives the corresponding actuator to complete the operation through the digital electronic control integration module, and feeds back the operation result to the central processing unit to call the multimodal interaction module for display.

[0089] Step S3 includes:

[0090] When the cloud-based collaborative battery swapping management module detects that the ship's battery level is below a preset threshold or receives a battery swapping request, it determines that a battery swapping is necessary; otherwise, a battery swapping is not necessary.

[0091] When battery swapping is needed, the cloud-based collaborative battery swapping management module requests real-time data from nearby battery swapping stations from the cloud platform;

[0092] The cloud-based collaborative battery swapping management module uses a weighted multi-attribute decision-making method to comprehensively evaluate the advantages and disadvantages of each battery swapping station and generate one or more recommended solutions.

[0093] The cloud-based collaborative battery swapping management module sends the recommended solution to the central processing unit, which then calls the multimodal interaction module to display the recommended solution to the driver; the driver's selection result is fed back to the cloud-based collaborative battery swapping management module.

[0094] After receiving the recommended route selected by the driver, the cloud-based collaborative battery swapping management module sends a reservation instruction to the cloud platform to lock the battery; at the same time, the central processing unit sets the battery swapping station to which the recommended route belongs as the current navigation target.

[0095] This embodiment also provides a ship, which includes the above-described intelligent ship control system based on multimodal interaction and cloud collaboration.

[0096] This invention deeply integrates various human-computer interaction technologies such as voice, touch, and AI vision, and combines cloud data with shipboard intelligence to achieve comprehensive intelligent management of ship navigation, electromechanical systems, battery swapping, and logs. The system not only significantly improves the ease of operation and decision-making efficiency for operators, but also comprehensively enhances the safety of ship navigation through innovative functions such as tiered alarms, proactive broadcasting, and behavior monitoring. Electronic and automated navigation logs ensure compliance and traceability, while multi-source data fusion provides unprecedented situational awareness capabilities. This invention provides a comprehensive and feasible technical solution for the intelligent development of ships, with broad application prospects and significant economic and social benefits.

[0097] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution; or the direct application of the inventive concept and technical solution to other situations without modification, are all within the protection scope of the present invention.

Claims

1. A ship intelligent navigation and control system based on multimodal interaction and cloud collaboration, characterized in that: The system includes a central processing unit, a multimodal interaction module, a cloud-based collaborative battery swapping management module, a hierarchical alarm and intelligent broadcasting module, an electronic navigation log module, a multi-source data fusion display module, a digital electronic control integration module, and a cloud platform. The multimodal interaction module, cloud-based collaborative battery swapping management module, hierarchical alarm and intelligent broadcasting module, electronic navigation log module, multi-source data fusion display module, digital electronic control integration module, and cloud platform are all connected to the central processing unit, enabling the interaction of control commands and data through the central processing unit.

2. The intelligent ship control system based on multimodal interaction and cloud collaboration according to claim 1, characterized in that: The multimodal interaction module is used to realize the interaction between the system and the driver, and includes a voice interaction unit, a touch display unit, and an AI visual recognition unit. The voice interaction unit, the touch display unit, and the AI ​​visual recognition unit are respectively connected to the central processing unit.

3. The intelligent ship control system based on multimodal interaction and cloud collaboration according to claim 1, characterized in that: The cloud-based collaborative battery swapping management module is used to make intelligent battery swapping decisions for ships through cloud collaboration. It communicates in real time with the remote cloud platform data base through the shipborne network interface of the central processing unit.

4. The intelligent ship control system based on multimodal interaction and cloud collaboration according to claim 1, characterized in that: The hierarchical alarm and intelligent broadcasting module is used for monitoring and alarming the entire ship. The hierarchical alarm and intelligent broadcasting module includes an alarm rule engine, which is used to perform hierarchical alarms based on the alarm source and parameters.

5. The intelligent ship control system based on multimodal interaction and cloud collaboration according to claim 1, characterized in that: The electronic navigation log module includes an automatic log template matching unit, an automatic multi-source data collection and filling unit, an electronic signature and revision recording unit, and an automatic abnormal behavior labeling unit. The automatic log template matching unit automatically matches and loads log templates according to preset rules. The automatic multi-source data collection and filling unit collects multi-source data from the vessel and fills it into the log template. The electronic signature and revision recording unit binds the log to an electronic signature and records log revision operations. The automatic abnormal behavior labeling unit performs real-time analysis of the operation log based on preset rules, automatically identifying and labeling abnormal behaviors.

6. The intelligent ship control system based on multimodal interaction and cloud collaboration according to claim 1, characterized in that: The multi-source data fusion display module is used to acquire multi-source ship data and execute data fusion algorithms to generate a unified navigation situation map and send it to the central processing unit to call the multimodal interaction module for display.

7. The intelligent ship control system based on multimodal interaction and cloud collaboration according to claim 1, characterized in that: The digital electronic control integrated module adopts a distributed control architecture. The various electromechanical devices on board are connected to one or more field controllers via industrial bus or Ethernet. All field controllers are then connected to the central processing unit via the ship's local area network.

8. A ship intelligent driving control method based on multimodal interaction and cloud collaboration, using a ship intelligent driving control system based on multimodal interaction and cloud collaboration according to any one of claims 1-7, characterized in that: The method includes: Step S1: Start the system, each module starts running, and collects real-time data from ship sensors, cloud platform data, and cockpit environment video data; Step S2: The multimodal interaction module continuously listens for voice commands and touch operations, and analyzes the video stream in real time; upon receiving a command, it parses its intent and sends it to the central processing unit to call the corresponding functional module to perform the operation; Step S3: The cloud-based collaborative battery swapping management module determines in real time whether battery swapping is needed, and executes intelligent battery swapping decisions when battery swapping is required; Step S4: The graded alarm and intelligent broadcast module polls and monitors the system data in real time. Once an abnormal parameter is detected or an alarm signal is received, the alarm level is determined and fed back to the central processing unit to invoke the multimodal interaction module to execute the corresponding alarm prompt strategy. Step S5: During system operation, the electronic navigation log module runs silently in the background, continuously and automatically recording various navigation and operation data according to the preset template, and completing the signing and management of the log based on the pilot's interaction. Step S6: The multi-source data fusion and display module performs fusion calculations on multi-source sensor data to generate a unified navigation situation map, and sends the unified navigation situation map to the central processing unit to call the multimodal interaction module for display. Step S7: When the system receives the driver's equipment control command, the central processing unit sends the corresponding control message to the digital electronic control integration module, drives the corresponding actuator to complete the operation through the digital electronic control integration module, and feeds back the operation result to the central processing unit to call the multimodal interaction module for display.

9. A ship intelligent driving control method based on multimodal interaction and cloud collaboration according to claim 8, characterized in that: Step S3 includes: When the cloud-based collaborative battery swapping management module detects that the ship's battery level is below a preset threshold or receives a battery swapping request, it determines that a battery swapping is necessary; otherwise, a battery swapping is not necessary. When battery swapping is needed, the cloud-based collaborative battery swapping management module requests real-time data from nearby battery swapping stations from the cloud platform; The cloud-based collaborative battery swapping management module uses a weighted multi-attribute decision-making method to comprehensively evaluate the advantages and disadvantages of each battery swapping station and generate one or more recommended solutions. The cloud-based collaborative battery swapping management module sends the recommended solution to the central processing unit, which then calls the multimodal interaction module to display the recommended solution to the driver; the driver's selection result is fed back to the cloud-based collaborative battery swapping management module. After receiving the recommended route selected by the driver, the cloud-based collaborative battery swapping management module sends a reservation instruction to the cloud platform to lock the battery; at the same time, the central processing unit sets the battery swapping station to which the recommended route belongs as the current navigation target.

10. A ship, characterized in that: The vessel includes a ship intelligent driving control system based on multimodal interaction and cloud collaboration as described in any one of claims 1-7.

Images