Inland river ship navigation man-machine cooperative control system and method for inland river shore-based driving control

Abstract

The invention discloses an inland river ship navigation man-machine cooperative control system and method for inland river shore-based driving control. Comprising a data acquisition module used for acquiring position information, navigation state information and environment data information of a ship, and a man-machine cooperation module used for determining a control mode of the ship intelligent navigation control system based on shore-based driving and control according to the information acquired by the data acquisition module. The control modes comprise an automatic control mode, a semi-automatic control mode and a manual control mode; when the information acquired by the data acquisition module has factors having potential safety risks on ship navigation, the control mode is switched to a semi-automatic mode, and when the information acquired by the data acquisition module has factors having direct safety risks on ship navigation, the control mode is switched to a manual mode; according to the invention, an intelligent and high-efficiency solution is provided for inland river shipping, and new possibility is brought to ship operation and management.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent technology for waterway transportation, and specifically relates to a human-machine collaborative control system and method for inland river ship navigation with inland river shore-based driving control, focusing on the research of remote driving human-machine collaborative control. Background Art

[0002] The new generation of intelligent navigation ships characterized by shore-based control and ship-side duty have changed the traditional ship control mode, and the ship navigation control system also needs to be innovated. The shore-based control ship navigation control system is essentially a hybrid intelligent system with humans in the loop. There are problems such as the real-time communication between the ship and the shore, the reliability of shore-based driving, and the smoothness of human-machine switching control. It is urgent to solve key technical problems such as the development of basic software for shore-based control ship-borne intelligent navigation control system and the integration of key system terminals. With the rapid development of inland shipping industry, the safety and efficiency of ship navigation have received more and more attention. However, inland waters usually have complex terrain, narrow channels, and changeable water flow. Traditional ship autonomous navigation systems often seem to be unable to cope with these complex environments. Especially in the process of ship operation in complex inland scenes, manual operation depends on the experience and judgment of the crew, which not only increases the workload, but also poses a great safety hazard. Therefore, there is an urgent need for a solution that can effectively improve the navigation safety and operating efficiency of inland ships. Summary of the invention

[0003] In order to improve the safety and efficiency of inland river vessel navigation control, the present invention proposes an inland river vessel navigation human-machine collaborative control system and method with inland river shore-based control.

[0004] A human-machine collaborative control system for inland river vessel navigation based on shore-based control to achieve one of the purposes of the present invention comprises:

[0005] Data acquisition module: used to collect ship location information, navigation status information and environmental data information;

[0006] Human-machine collaboration module: used to determine the control mode of the ship intelligent navigation control system based on shore-based control according to the information collected by the data acquisition module, and the control modes include: automatic control mode, semi-automatic control mode and manual control mode; when the information collected by the data acquisition module contains factors that pose potential safety risks to the ship's navigation, the control mode is switched to the semi-automatic mode; when the information collected by the data acquisition module contains factors that pose direct safety risks to the ship's navigation, the control mode is switched to the manual mode.

[0007] The system runs in automatic control mode by default. The shore-based control center monitors the ship's navigation status and environmental data in real time, uses model predictive control methods to control and send control instructions to the ship. In this mode, the crew is mainly responsible for monitoring system operation and handling emergencies, which reduces the operating burden and improves navigation efficiency.

[0008] Furthermore, it also includes a three-dimensional scene reconstruction module, which is used to convert the information collected by the data collection module into an image of the ship's surrounding environment.

[0009] Furthermore, the three-dimensional scene reconstruction module includes:

[0010] Feature extraction module: used to extract environmental information features from the environmental data information collected by the data acquisition module according to the convolutional neural network; also used to perform semantic segmentation and annotation on the video image information in the environmental data information collected by the data acquisition module;

[0011] Perception fusion module: used to fuse the semantic segmentation and annotation results of the video image information and the environmental data information collected by the data acquisition module into multi-source heterogeneous data, and to convert them into understandable three-dimensional information through coordinate transformation, time unification, target association and information fusion;

[0012] Rendering module: used to generate synthetic images based on the three-dimensional information output by the perception fusion module.

[0013] Furthermore, it also includes a task scheduling module, which is used to detect the AIS data delay through the data acquisition thread. When the AIS data delay exceeds the set threshold, the software sends a high priority interrupt signal to the task scheduler. After receiving the interrupt signal, the task scheduler increases the priority of the acquisition thread to ensure the normal transmission of AIS data.

[0014] Furthermore, it also includes a path planning module, which is used to plan the ship's navigation path using Bezier curves, set control points according to the ship's current position information, and use the control points to generate a Bezier curve trajectory for ship navigation path planning to ensure the smoothness and continuity of the trajectory.

[0015] Furthermore, it also includes an emergency takeover module: used to automatically trigger emergency takeover and determine the takeover method of the ship's intelligent navigation control system based on shore-based control when the ship's intelligent navigation control system based on shore-based control fails or the environment changes or receives human instructions.

[0016] Furthermore, in the emergency takeover module, when the shore-based control center is in normal working condition and can obtain real-time data of the ship, the shore-based control center takes over the operation of the ship; if the shore-based control center fails or cannot reliably obtain ship data, it switches to manual mode or semi-automatic mode and takes over the operation of the ship.

[0017] Furthermore, it also includes a hardware integration module: used to ensure the connection and communication between the ship's intelligent navigation control system based on shore-based driving control and various external devices.

[0018] Furthermore, the hardware integration module includes a variety of hardware interfaces for connecting and communicating with external devices, including: an analog signal interface and / or IO data interface and / or pulse data interface for exchanging data with the driving control device; an Ethernet interface for exchanging data with the shipboard local area network; an RS232 interface and / or RS458 interface and / or RS422 interface for exchanging data with the shipboard communication and navigation equipment; a CAN interface for exchanging data with the cabin equipment; and a network communication interface for communicating with the shore.

[0019] Furthermore, it also includes the ship-shore system module, including the hardware layer, data layer, platform layer, and application layer; the hardware layer is used to provide basic computing, storage, and service support; the data layer is used to integrate the shore-based and ship-side data collected by the hardware layer, and realize sharing through data exchange; the platform layer contains the Yangtze River electronic navigation chart and the three-dimensional digital twin base, which is used to process and integrate the data layer information. Based on the information of the platform layer, the application layer provides bad flow warning, optimized route recommendation, real-life three-dimensional enhancement, and follow-up navigation organization functions, realizing a complete functional chain from basic data collection to advanced applications, and providing all-round intelligent support for shipping.

[0020] Furthermore, the system structure of the ship intelligent navigation control system based on inland shore-based control includes: a hardware layer, a hardware abstraction layer, a driver layer, an intermediate interface layer and a business logic layer; the hardware layer is used to connect to external connection devices; the hardware abstraction layer is used to simplify hardware access by encapsulating the interfaces of external devices connected to the hardware layer; the driver layer is used to provide software drivers for external devices connected to the hardware layer; the intermediate interface layer is used to encapsulate the device functions of the external devices into software and provide them to the business logic layer for calling; the business logic layer realizes status collection, navigation decision-making and navigation control functions based on the interface provided by the intermediate interface layer.

[0021] A human-machine collaborative control method for inland river vessel navigation based on shore-based control to achieve the second purpose of the present invention comprises:

[0022] Collect the ship's location information, navigation status information and environmental data information;

[0023] The control mode of the ship intelligent navigation control system based on shore-based control is determined according to the information collected by the data acquisition module, and the control modes include: automatic control mode, semi-automatic control mode and manual control mode; when the information collected by the data acquisition module contains factors that have potential safety risks to the ship's navigation, the control mode is switched to the semi-automatic mode, and when the information collected by the data acquisition module contains factors that have direct safety risks to the ship's navigation, the control mode is switched to the manual mode.

[0024] The beneficial effects of the present invention include:

[0025] The human-machine collaborative design of the present invention can reduce the operating burden of the crew and improve the navigation efficiency; at the same time, when the system fails, it can be discovered in time and switched to semi-automatic or manual control mode, which effectively ensures the navigation safety of the ship while improving the navigation efficiency;

[0026] In the hardware design, the present invention adopts a powerful embedded processing controller with rich interfaces and power protection modules to ensure the stable operation of the equipment in complex working conditions.

[0027] The present invention adopts embedded compiler software for development in software design, which provides functions such as macro assembly, linker, library management and simulation debugging, provides comprehensive support for software development and adopts UCOS embedded operating system to realize multi-threaded software architecture. At the same time, environmental adaptability simulation test and electromagnetic compatibility simulation test are also carried out, covering high temperature, low temperature, damp heat, vibration, impact, waterproof and other aspects;

[0028] The invention provides an intelligent and efficient solution for inland waterway navigation and brings new possibilities for ship operation and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the hardware structure of the ship intelligent navigation control system based on inland river shore-based control;

[0030] Figure 2 This is a schematic diagram of the software structure of the ship intelligent navigation control system based on inland river shore-based control;

[0031] Figure 3 Reconstruct a schematic diagram of the three-dimensional scene of the navigation environment;

[0032] Figure 4 This is a schematic diagram of the technical architecture for ship-shore system integration;

[0033] Figure 5 Schematic diagram of the principle of human-machine collaboration in inland river bank-based driving and control. DETAILED DESCRIPTION

[0034] The following specific implementations are used to explain the technical solutions of the claims of the present invention so that those skilled in the art can understand the claims. The protection scope of the present invention is not limited to the following specific implementation structures. The technical solutions of the claims of the present invention made by those skilled in the art but different from the following specific implementations are also within the protection scope of the present invention.

[0035] Example 1

[0036] A human-machine collaborative control system for inland river vessel navigation based on shore-based driving control, comprising:

[0037] 1. Control system hardware design and integration:

[0038] The hardware of the ship intelligent navigation control system based on inland river shore-based control adopts high-performance ARM chip, Cortex-M7 core, maximum main frequency not less than 200MHz, FLASH capacity not less than 512K, RAM capacity not less than 256K, and has rich interfaces to meet the processing capability requirements of the ship intelligent navigation control algorithm based on inland river shore-based control.

[0039] Adopting highly compatible and rich data interfaces. The interfaces of the ship intelligent navigation control system based on inland river shore-based control include analog signal input, IO data interface, pulse data interface, RS232 interface, RS458 interface, RS422 interface, CAN interface, Ethernet interface and 4G / 5G communication interface with shore-based communication, so as to ensure the connection and communication requirements between the ship intelligent navigation control system based on inland river shore-based control and different types of vehicles, rudders, main engine remote control, rudder angle feedback and other equipment.

[0040] Power protection and interface protection modules are used. The power supply and other interface designs of the ship intelligent navigation control system based on inland shore control are protected by TVS, LC filters, magnetic beads and other circuits, which can meet the requirements of electromagnetic compatibility tests such as surge, static electricity, and pulse groups, and ensure the normal operation of the equipment under complex working conditions. The hardware structure of the ship intelligent navigation control system based on inland shore control is as follows: Figure 1As shown in the figure, the connection relationship of each module is shown. Each sensor and input device is connected to the embedded processing control system through analog signal input, IO data interface and pulse data interface. The control system is connected to the shipboard LAN through the Ethernet interface to realize communication with other shipboard equipment. Radar, AIS, compass, GPS and other equipment are connected to the control system through RS232, RS458 and RS422 interfaces to provide navigation data. The cabin equipment communicates with the control system through the CAN interface for real-time control. The 4G / 5G communication interface provides remote communication capabilities. The power supply and interface protection module ensures the stable operation of the entire system in a complex environment. During the operation of the ship, the power system may be impacted by sudden voltage (surge), which may damage the electronic equipment. Therefore, TVS (transient voltage suppression diode) is used in the design to protect the equipment from high voltage, magnetic beads and LC filters can effectively filter out electrostatic interference and protect sensitive electronic components, and LC filters can smooth pulses to prevent the equipment from being interfered by short-term high-frequency pulses. At the same time, the modular design makes the expansion and maintenance of the system more convenient. Different interface modules can be replaced or upgraded as needed, ensuring the reliability, scalability and ease of maintenance of the system, and meeting the requirements of intelligent navigation control for inland shore-based ships.

[0041] 2. Design and implementation of navigation control system software:

[0042] The UCOS embedded operating system is used, and a multi-threaded software architecture is adopted to achieve synchronous acquisition, decision-making, and control. The UCOS system is a real-time operating system that includes functions such as task scheduling, task management, time management, memory management, and communication and synchronization between tasks. It has the characteristics of high execution efficiency, small footprint, excellent real-time performance, and strong scalability, meeting the multi-threaded and high-efficiency development requirements of the ship intelligent navigation control system based on inland river shore-based driving control.

[0043] In order to achieve efficient and reliable operation of the ship intelligent navigation control system based on inland shore-based control, the navigation control system software is developed based on the embedded compiler software, combined with the UCOS embedded operating system and the IEC61162 / NMEA-0183 standard protocol to solve the system delay problem. The navigation control system software adopts a multi-threaded architecture to achieve the synchronization of data acquisition, decision-making and control. When the system detects that the delay of data acquisition exceeds the preset threshold, the data acquisition thread will immediately send an interrupt signal to give priority to the current task to ensure the real-time nature of the data. In the specific implementation, the data acquisition thread first detects that the AIS data delay exceeds 100ms, and immediately sends a high-priority interrupt signal. After the task scheduler receives the interrupt signal, it will increase the priority of the acquisition thread and process the AIS data. After the AIS data is processed, the decision module updates the navigation path according to the latest data and sends the instruction to the control module. The real-time scheduling function provided by the real-time operating system ensures the efficient execution and timely response of tasks. When there are critical tasks that need to be executed immediately, the UCOS system will suspend non-critical tasks to ensure the priority processing of critical tasks.

[0044] The AIS data of the ship comes from the Automatic Identification System (AIS), which is installed on the ship and is used to automatically and continuously send the ship's static information (such as ship name, call sign, MMSI code, ship length, ship width), dynamic information (such as ship position, course, speed), voyage information and safety-related information, etc. This information is automatically broadcast to other ships and shore-based facilities in nearby waters to achieve automatic identification and collision avoidance between ships, and also helps maritime administration departments to manage ship traffic.

[0045] For abnormal situations that may be detected by different modules, the inter-task communication and synchronization functions of the UCOS system are used to facilitate efficient collaboration and data sharing between different modules. When a module detects an abnormal situation, it promptly notifies the relevant modules for processing to ensure the overall reliability of the system. If the monitoring module detects that the GPS signal is lost, it sends an alarm signal to the control module. After receiving the alarm signal, the control module starts the backup navigation system such as the compass system to continue to provide location data. At the same time, the monitoring module records the abnormal situation and sends the alarm information to the shore-based control center through the Ethernet interface to facilitate remote intervention and processing.

[0046] The use of IEC61162 / NMEA-0183 standard protocols and data interfaces ensures the compatibility and communication reliability of the system with most ship equipment. When the system receives data from different devices, it can effectively integrate and process it to avoid data loss or misjudgment. The communication module receives the navigation data sent by the radar and AIS equipment, and parses and integrates it according to the IEC61162 / NMEA-0183 protocol. After the data parsing is completed, the communication module transmits the integrated data to the embedded processing control system for decision analysis. The decision module optimizes the navigation path based on the comprehensive data and adjusts the control instructions in real time to ensure the safe navigation of the ship. The structure of the ship intelligent navigation control system based on inland shore-based control is as follows: Figure 2 As shown in the figure, the hierarchical structure of a ship intelligent navigation control system is shown, which includes the hardware layer, hardware abstraction layer, driver layer, intermediate interface layer and business logic layer from bottom to top; the hardware layer is used to connect with external connection devices; the hardware abstraction layer is used to simplify hardware access through interface encapsulation; the driver layer is used to provide software drivers for external devices; the intermediate interface layer is used to encapsulate device functions into software and provide them to the business logic layer for calling; the business logic layer implements functions such as status collection, navigation decision-making and navigation control based on these interfaces. Such a design enhances the flexibility, maintainability and scalability of the system. Each layer focuses on specific functions, and the layers communicate through standardized interfaces, thereby simplifying the development process and improving system stability and performance.

[0047] 3. Ship navigation path planning and optimization:

[0048] The shore-based control system is used to collect the ship's location information, speed, heading, and environmental data such as wind speed, tide, and ocean current in real time. These data are collected through GPS, lidar, AIS, and meteorological sensors, and transmitted to the shore-based control center through the 4G / 5G communication network. The shore-based control center performs preliminary processing and analysis on these data. Based on the collected real-time data, the shore-based control center reconstructs the three-dimensional scene of the navigation environment, including inland waterways, geometric information of bridges and docks, the location and shape of surrounding obstacles, and the influence of water flow and wind. Through the combination of environmental perception technologies such as lidar and high-precision cameras, the accuracy of the model is ensured and it is updated in real time. Among them, the navigation environment three-dimensional scene reconstruction module has an architecture as follows: Figure 3As shown in the figure, the processing flow of human-computer interaction from the real world to the three-dimensional scene is specifically demonstrated. The process starts from the real scene. The camera captures the image of the real world and converts it into a video signal. The sensor also captures environmental information as additional input data. For example, the laser radar can provide more accurate distance and height information that the camera cannot provide, and is suitable for various lighting conditions; AIS can obtain information such as ship MMSI, ship name, ship size, draft, etc. The video signal and the environmental information data collected by the sensor are sent to the scene reconstruction module, which consists of a U-Net network model and a perception fusion module. The U-Net network model is characterized by a U-shaped symmetric structure and consists of three parts: an encoder, a jump connection, and a decoder. In the encoder stage, the environmental information features are extracted through a 3×3 convolution layer, a Re-LU activation function, and a 2×2 maximum pooling layer downsampling operation, and the input video image information is semantically segmented and annotated. The perception fusion module is to fuse the semantic segmentation annotation and the environmental information collected by various sensors for multi-source heterogeneous data, and convert the data into understandable three-dimensional information through four steps: coordinate transformation, time unification, target association, and information fusion. Finally, the fused information is sent to the 3D rendering module, displayed on the screen, and provides human-computer interaction support.

[0049] Ship-shore system integration technology architecture Figure 4 As shown in the figure, a multi-level system architecture from the hardware layer to the application layer is presented. The hardware layer provides basic computing, storage and service support, and the data layer integrates shore-based and ship-side data to achieve sharing through data exchange. The platform layer contains the Yangtze River electronic navigation chart and the three-dimensional digital twin base to process and integrate the data layer information. Based on the information of the platform layer, the application layer provides functions such as adverse flow warning, optimized route recommendation, real-life three-dimensional enhancement and follow-up navigation organization, realizing a complete functional chain from basic data collection to advanced applications, and providing all-round intelligent support for shipping.

[0050] With the assistance of the shore-based control system, the Bezier curve is used to plan the ship's navigation path, and control points are set according to the ship's current position information and intermediate key points. The initial Bezier curve trajectory is generated using the control points to ensure the smoothness and continuity of the trajectory. At the same time, the dynamic characteristics of the ship during navigation, as well as the impact of wind, current, and other ships on the navigation trajectory are fully considered to ensure that the trajectory can avoid various static and dynamic obstacles as the goal, to ensure the smoothness of the trajectory as much as possible, and to provide a smooth and safe trajectory for the control module. During the navigation process, the position and attitude of the ship are monitored in real time, and the trajectory is dynamically corrected by adjusting the control points of the Bezier curve. When the environment changes, such as new obstacles or changes in wind speed, the control points can be quickly adjusted to regenerate a new trajectory. Under the premise of ensuring safety, multi-objective trajectory optimization is input. In severe weather conditions, the safety weight is increased, and in emergency situations, the shortest time weight is given priority.

[0051] Real-time monitoring of ship status and environmental changes. If deviation from the planned trajectory or new obstacles are detected, the trajectory is adjusted immediately:

[0052] Through sensor data and algorithm analysis, the deviation between the current position of the ship and the predetermined trajectory is found, and the adjusted optimal trajectory is recalculated using Bezier curves and optimization algorithms. The adjusted trajectory instructions are sent to the ship control system through the communication network, instructing the ship to make corresponding adjustments. The ship's autonomous control system works closely with the shore-based control center to ensure the precise execution of navigation operations. The shore-based control center needs to provide macro guidance, overall trajectory planning and adjustment instructions. The ship's autonomous control system performs micro operations and instant adjustments based on real-time data. Through the collaborative work of the two, the smooth completion of the ship's navigation trajectory planning is guaranteed, human intervention is minimized, and navigation efficiency and safety are improved.

[0053] 4. Human-machine collaborative control and emergency takeover:

[0054] During the navigation of inland ships, ensuring the efficiency and reliability of human-machine collaborative control and achieving seamless emergency takeover in emergency situations are the key to ensuring safe navigation of ships. This stage focuses on the following aspects: mechanism design of human-machine collaborative control, triggering conditions and processes of emergency takeover, and multi-level redundancy design and fault handling.

[0055] In terms of the mechanism design of human-machine collaborative control, the core lies in achieving seamless cooperation between man and machine. In order to achieve this goal, the system has designed a multi-level human-machine interaction mechanism: Automatic control mode: Under normal circumstances, the system runs in automatic control mode by default. The shore-based control center monitors the navigation status and environmental data of the ship in real time, uses the model predictive control method to control and send control instructions to the ship. In this mode, the crew is mainly responsible for monitoring the operation of the system and handling emergencies, reducing the operating burden and improving navigation efficiency. Semi-automatic control mode: When the system detects a complex environment or potential risks (such as narrow waterways, bad weather, etc.), it will automatically switch to semi-automatic control mode. In this mode, the system still performs automatic navigation and control, but the crew needs to participate in the decision-making process of navigation control. For example, the system will recommend an avoidance path, and the crew can confirm or fine-tune it to ensure the safety and flexibility of the operation. Manual control mode: In emergency or special circumstances (such as sensor failure or receiving a command to switch to manual control), the crew can switch to manual control mode at any time and take full control of the ship's operations. In this mode, all automated functions are temporarily shut down, and the crew relies on their own experience and judgment to operate, ensuring that they can respond to various emergencies.

[0056] In terms of the triggering conditions and processes of emergency takeover, clear triggering conditions and response processes are designed to ensure that when a system or shore-based control center fails, the control authority can be quickly switched to ensure navigation safety. When a system failure occurs: including sensor failure, communication interruption, control system failure, etc. Environmental changes: such as sudden severe weather, sudden changes in the waterway, unexpected obstacles, etc. Human instructions: When the crew or the shore-based control center finds an abnormal situation, the emergency takeover will be triggered. When the system detects a fault or abnormality, it will immediately issue an alarm signal and display specific fault information and recommended treatment plans. Automatically switch to emergency mode: The system automatically switches to emergency control mode while issuing an alarm signal. At this time, the shore-based control center and the crew need to quickly assess the situation and decide on the takeover method. There are two main takeover methods. One is shore-based takeover: If the shore-based control center is in normal working condition and can reliably obtain real-time data of the ship, the shore-based control center takes over the ship operation, performs emergency processing and navigation adjustments. The other is ship-side takeover: If the shore-based control center fails or cannot reliably obtain data, the crew switches to manual or semi-automatic mode to take over the ship operation. The system will assist the crew in providing real-time data and suggestions to help the crew complete emergency operations. In terms of fault handling and recovery, the system will perform fault isolation and preliminary repair work during the emergency takeover, and continuously report the progress of fault repair to the crew and the shore-based control center. After the fault is handled, the system will automatically evaluate whether it can be safely restored to normal control mode, and prompt the crew or the shore-based control center to confirm to ensure that the demonstration application can be successfully demonstrated. The multi-segment demonstration application framework is as follows: Figure 5As shown in the figure, this figure describes the process and logic of how a ship switches between the autopilot system and the manual driving system in different navigation environments. Sensors are used for situational awareness and collect data on the ship's surrounding environment and its own status. The navigation brain makes intelligent decisions based on sensor data, assesses risks and selects corresponding navigation strategies. The controller is responsible for executing these decisions and automatically controlling the ship. In a low-risk environment, the ship is in an intelligent navigation state and is controlled by the automatic system. When a high-risk environment is detected, the system evaluates the reliability of the automatic control. If it is unreliable, it switches to manual driving mode, and the crew makes driving decisions and manually operates the ship by perceiving the situation information. The operating control system continuously monitors environmental changes. Once the risk is reduced or the automatic control becomes reliable, the system can switch back to the autopilot mode. This design ensures safety and operational flexibility in complex environments, and achieves seamless switching between automatic and manual driving.

[0057] In terms of multi-level redundancy design and fault handling, in order to ensure the high reliability of the human-machine collaborative control system, redundant backups are configured for key sensors, processors and communication modules. For example, dual GPS systems, dual communication links (4G / 5G and satellite communications), dual control systems, etc. Redundant backups are also designed for key control algorithms and decision logic to ensure that the backup system can quickly take over when the main system fails. The system can also detect the operating status of each module in real time through self-checking and monitoring mechanisms. Once an abnormality is found, fault isolation and alarm are immediately performed. In the event of complex faults, the system will provide detailed fault information and handling suggestions to assist the crew and the shore-based control center in making quick decisions and handling. The optimal solution for the allocation of driving rights under multiple constraints. Define the preset state of intelligent ship navigation as the benchmark behavior for switching the ship's driving rights, that is, B D ={T D , O D}, where B D Indicates the preset state of the smart ship, which is used as the benchmark behavior for switching the ship's driving rights. D Indicates the preset track of the ship, O D Indicates the preset control instructions of the ship. The control factor α is used as the driving right H and α A In order to control the variables, a ship switching behavior prediction model is established, namely:

[0058] B P =g(B D ,τ,α H ,α A )

[0059] In formula (1), B p represents the predicted behavior of the ship, including acceleration and deceleration, obstacle avoidance, steering, etc., g(·) is the ship behavior prediction function, and τ is the communication delay.

[0060] 5. Optimization and improvement of human-machine collaborative control:

[0061] This stage focuses on optimizing and improving the existing system, starting from multiple angles such as technological innovation, data analysis, and the application of artificial intelligence to ensure that the system can perform at its best in various complex situations. The latest sensor technology, communication technology, and control algorithm are introduced into the system to enhance the system's perception and response speed. High-precision lidar and visual sensors are used to improve the ship's perception accuracy of the surrounding environment, especially in complex inland waterways and port environments, and obstacles and channel changes can be more accurately identified. In addition, the communication module is upgraded and low-latency, high-bandwidth 5G technology is introduced to ensure that the data transmission between the shore-based control center and the ship is more stable and faster, reducing control errors caused by communication delays. In data analysis, the value of data is crucial. Through big data analysis and application, smarter decision-making and control can be achieved. This stage will build a ship navigation big data platform to collect and store data from various sensors in real time, including ship position, speed, heading, and environmental data (such as wind speed, tide, ocean current, etc.). Through in-depth analysis of these data, potential problems and risks can be discovered and preventive measures can be taken in advance. At the same time, historical data is used for pattern recognition and trend analysis to optimize navigation strategies. By analyzing historical weather data and navigation trajectories, challenges that may be encountered during future navigation can be predicted and navigation plans can be adjusted in advance.

[0062] A friendly and efficient human-computer interaction interface is a key component of the human-computer collaborative control system. In this stage, the operating interface of the crew and the shore-based control center will be comprehensively optimized to improve user experience and operating efficiency. The use of advanced graphical interface design technology and intuitive control panels will enable crew members to monitor and control the ship more easily. Introduce voice control and natural language processing technology to achieve more natural human-computer interaction. Crew members can operate and query through voice commands, and the system provides real-time information and suggestions through voice feedback, which reduces the operating burden of crew members and improves the convenience and safety of operation. At the same time, the optimization of the human-computer collaborative control system is a continuous process. A continuous optimization and feedback mechanism for the system will be established to ensure that the system is always in the best condition. The system will be evaluated and updated regularly, and the system functions and performance will be continuously optimized based on the feedback from operators and actual operation conditions. For example, a user feedback platform will be established to collect problems and suggestions encountered by crew members and shore-based operators during use, and respond and improve quickly. The principle of human-computer collaboration in inland shore-based driving and control is as follows Figure 5 At the same time, data analysis and artificial intelligence technology are used to evaluate the operating effect of the system, identify potential problems and optimization space, and make targeted improvements and upgrades.

[0063] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0064] Example 2

[0065] A human-machine collaborative control system for inland river vessel navigation based on shore-based driving control, comprising:

[0066] Data acquisition module: used to collect the ship's location information, navigation status information and environmental data information; the navigation status information includes speed and heading; the environmental data information includes wind speed, tide and ocean current;

[0067] Human-machine collaboration module: used to determine the control mode of the ship intelligent navigation control system based on shore-based control according to the information collected by the data acquisition module, and the control modes include: automatic control mode, semi-automatic control mode and manual control mode; when the information collected by the data acquisition module contains factors that pose potential safety risks to the ship's navigation, the control mode is switched to the semi-automatic mode; when the information collected by the data acquisition module contains factors that pose direct safety risks to the ship's navigation, the control mode is switched to the manual mode.

[0068] In the automatic control mode, the shore-based control center monitors the ship's navigation status information and environmental data information in real time, uses the model predictive control method to control and send control instructions to the ship. In this mode, the crew is mainly responsible for monitoring system operation and handling emergencies, which reduces the operating burden and improves navigation efficiency.

[0069] The potential safety risks include: narrow waterways, bad weather; in semi-automatic mode, the ship is still in automatic navigation, but the crew needs to participate in the decision-making process of navigation control; the crew can confirm or fine-tune to ensure the safety and flexibility of the operation;

[0070] The direct safety risks include: sensor failure resulting in failure to receive collected information or receiving commands to switch to manual control; in manual control mode, all automated functions are temporarily shut down, and the crew relies on their own experience and judgment to operate to ensure that they can respond to various emergencies.

[0071] In some embodiments, in order to ensure the high reliability of the human-machine collaborative control system, key sensors, processors and communication modules are configured with redundant backups; key control algorithms and decision logic are also designed with redundant backups to ensure that the backup system can quickly take over when the main system fails. The system detects the operating status of each module in real time through self-checking and monitoring mechanisms. Once an abnormality is found, fault isolation and alarm are immediately carried out.

[0072] In some embodiments, a three-dimensional scene reconstruction module is also included, which is used to convert the information collected by the data collection module into an image of the surrounding environment of the ship.

[0073] In some embodiments, the three-dimensional scene reconstruction module includes:

[0074] Feature extraction module: used to extract environmental information features from the environmental data information collected by the data acquisition module according to the convolutional neural network; also used to perform semantic segmentation and annotation on the video image information in the environmental data information collected by the data acquisition module;

[0075] Perception fusion module: used to fuse the semantic segmentation and annotation results of the video image information and the environmental data information collected by the data acquisition module into multi-source heterogeneous data, and to convert them into understandable three-dimensional information through coordinate transformation, time unification, target association and information fusion;

[0076] Rendering module: used to generate synthetic images based on the three-dimensional information output by the perception fusion module.

[0077] In some embodiments, the feature extraction module uses a U-Net network model to extract environmental information features, and the U-Net network model consists of an encoder, a jump connection, and a decoder. In the encoder stage, a 3×3 convolution layer, a Re-LU activation function, and a 2×2 maximum pooling layer downsampling operation are used to extract environmental information features and perform semantic segmentation and annotation on the input video image information.

[0078] In some embodiments, a power supply and interface protection module is also included: it is used to use a transient voltage suppression diode to protect the device from high voltage; this module uses magnetic beads and LC filters to filter out static electricity and short-term high-frequency pulse interference.

[0079] In some embodiments, a task scheduling module is also included, which is used to detect the AIS data delay through the data acquisition thread. When the AIS data delay exceeds a set threshold, the software sends a high priority interrupt signal to the task scheduler. After receiving the interrupt signal, the task scheduler increases the priority of the acquisition thread to ensure the normal transmission of the AIS data.

[0080] In some embodiments, a path planning module is also included, which is used to plan the ship's navigation path using Bezier curves, set control points according to the ship's current position information, and use the control points to generate a Bezier curve trajectory for ship navigation path planning to ensure the smoothness and continuity of the trajectory.

[0081] In some embodiments, an emergency takeover module is also included: it is used to automatically trigger the emergency takeover and determine the takeover method of the ship's intelligent navigation control system based on shore-based control when the ship's intelligent navigation control system based on shore-based control fails or the environment changes or receives human instructions.

[0082] In certain embodiments, in the emergency takeover module, when the shore-based control center is in normal working condition and can obtain real-time data of the ship, the shore-based control center takes over the operation of the ship; if the shore-based control center fails or cannot reliably obtain the ship data, it switches to manual mode or semi-automatic mode and takes over the operation of the ship.

[0083] In some embodiments, a monitoring module and a control module are also included. When the monitoring module detects that the GPS signal is lost, an alarm signal is sent to the control module; after receiving the alarm signal, the control module starts the backup navigation system to continue to provide location data.

[0084] In some embodiments, a hardware integration module is also included: used to ensure the connection and communication between the ship intelligent navigation control system based on shore-based control and various external devices. The hardware integration module includes a variety of hardware interfaces for connecting and communicating with external devices, including: an analog signal interface and / or an IO data interface and / or a pulse data interface for data exchange with the control device; an Ethernet interface for data exchange with the ship-borne local area network; an RS232 interface and / or an RS458 interface and / or an RS422 interface for data exchange with the ship-borne communication and navigation equipment; a CAN interface for data exchange with the cabin equipment; and a network communication interface for communication with the shore.

[0085] In some embodiments, it also includes a ship-shore system module, including a hardware layer, a data layer, a platform layer, and an application layer; the hardware layer is used to provide basic computing, storage, and service support; the data layer is used to integrate the shore-based and ship-side data collected by the hardware layer, and realize sharing through data exchange; the platform layer contains the Yangtze River electronic navigation chart and a three-dimensional digital twin base, which is used to process and integrate data layer information. Based on the information of the platform layer, the application layer provides adverse flow warnings, optimized route recommendations, real-life three-dimensional enhancements, and follow-up navigation organization functions, realizing a complete functional chain from basic data collection to advanced applications, and providing all-round intelligent support for shipping.

[0086] In certain embodiments, the system structure of a ship intelligent navigation control system based on inland shore-based control includes: a hardware layer, a hardware abstraction layer, a driver layer, an intermediate interface layer and a business logic layer; the hardware layer is used to connect to external connection devices; the hardware abstraction layer is used to simplify hardware access by encapsulating interfaces of external devices connected to the hardware layer; the driver layer is used to provide software drivers for external devices connected to the hardware layer; the intermediate interface layer is used to encapsulate the device functions of the external devices into software and provide them to the business logic layer for calling; the business logic layer implements status collection, navigation decision-making and navigation control functions based on the interface provided by the intermediate interface layer.

[0087] Example 3

[0088] A human-machine collaborative control method for inland river vessel navigation based on shore-based driving control, comprising:

[0089] Collect the ship's location information, navigation status information and environmental data information;

[0090] The control mode of the ship intelligent navigation control system based on shore-based control is determined according to the information collected by the data acquisition module, and the control modes include: automatic control mode, semi-automatic control mode and manual control mode; when the information collected by the data acquisition module contains factors that have potential safety risks to the ship's navigation, the control mode is switched to the semi-automatic mode, and when the information collected by the data acquisition module contains factors that have direct safety risks to the ship's navigation, the control mode is switched to the manual mode.

[0091] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. A human-machine collaborative control system for inland river ship navigation based on shore-based control, characterized in that: include: Data acquisition module: used to collect ship location information, navigation status information and environmental data information; Human-machine collaboration module: used to determine the control mode of the ship intelligent navigation control system based on shore-based control according to the information collected by the data acquisition module, and the control mode includes: automatic control mode, semi-automatic control mode and manual control mode; When the information collected by the data acquisition module contains factors that pose potential safety risks to ship navigation, the control mode is switched to semi-automatic mode. When the information collected by the data acquisition module contains factors that pose direct safety risks to ship navigation, the control mode is switched to manual mode.

2. The human-machine collaborative control system for inland river vessel navigation based on shore-based control as claimed in claim 1, characterized in that: It also includes an emergency takeover module: it is used to automatically trigger the emergency takeover and determine the takeover method of the ship's intelligent navigation control system based on shore-based control when the ship's intelligent navigation control system based on shore-based control fails or the environment changes or receives human instructions.

3. The human-machine collaborative control system for inland river vessel navigation based on shore-based control as claimed in claim 2 is characterized in that: In the emergency takeover module, when the shore-based control center is in normal working condition and can obtain real-time data of the ship, the shore-based control center takes over the operation of the ship; if the shore-based control center fails or cannot reliably obtain ship data, it switches to manual mode or semi-automatic mode and takes over the operation of the ship.

4. The human-machine collaborative control system for inland river vessel navigation based on shore-based control as claimed in claim 1, characterized in that: It also includes a three-dimensional scene reconstruction module, which is used to convert the information collected by the data collection module into an image of the ship's surrounding environment.

5. The human-machine collaborative control system for inland river vessel navigation based on shore-based control as claimed in claim 4, characterized in that: The three-dimensional scene reconstruction module includes: Feature extraction module: used to extract environmental information features from the environmental data information collected by the data acquisition module according to the convolutional neural network; also used to perform semantic segmentation and annotation on the video image information in the environmental data information collected by the data acquisition module; Perception fusion module: used to fuse the semantic segmentation and annotation results of the video image information and the environmental data information collected by the data acquisition module into multi-source heterogeneous data, and to convert them into understandable three-dimensional information through coordinate transformation, time unification, target association and information fusion; Rendering module: used to generate synthetic images based on the three-dimensional information output by the perception fusion module.

6. The human-machine collaborative control system for inland river vessel navigation based on shore-based control as claimed in claim 1, characterized in that: It also includes a task scheduling module, which is used to detect the AIS data delay through the data acquisition thread. When the AIS data delay exceeds the set threshold, the software sends a high-priority interrupt signal to the task scheduler. After receiving the interrupt signal, the task scheduler increases the priority of the acquisition thread to ensure the normal transmission of AIS data.

7. The human-machine collaborative control system for inland river vessel navigation based on shore-based control as claimed in claim 1, characterized in that: It also includes a path planning module, which is used to plan the ship's navigation path using Bezier curves, set control points according to the ship's current position information, and use the control points to generate a Bezier curve trajectory for ship navigation path planning to ensure the smoothness and continuity of the trajectory.

8. The human-machine collaborative control system for inland river vessel navigation based on shore-based control as claimed in claim 1, characterized in that: It also includes a hardware integration module: used to ensure the connection and communication between the ship's intelligent navigation control system based on shore-based control and various external devices.

9. The human-machine collaborative control system for inland river vessel navigation based on shore-based control as claimed in claim 8, characterized in that: The hardware integration module includes a variety of hardware interfaces for connecting and communicating with external devices, including: an analog signal interface and / or an IO data interface and / or a pulse data interface for exchanging data with the driving control device; an Ethernet interface for exchanging data with the shipboard local area network; an RS232 interface and / or an RS458 interface and / or an RS422 interface for exchanging data with the shipboard communication and navigation equipment; a CAN interface for exchanging data with the cabin equipment; and a network communication interface for communicating with the shore.

10. A human-machine collaborative control method for inland river vessel navigation based on shore-based control based on the system of claim 1, characterized in that: include: Collect the ship's location information, navigation status information and environmental data information; Determine the control mode of the ship intelligent navigation control system based on shore-based control according to the information collected by the data acquisition module, wherein the control mode includes: automatic control mode, semi-automatic control mode and manual control mode; When the collected data contains factors that pose potential safety risks to the ship's navigation, the control mode is switched to the semi-automatic mode; when the collected data contains factors that pose direct safety risks to the ship's navigation, the control mode is switched to the manual mode.

Images