Unmanned ship intelligent control system and method based on multi-modal semantic analysis

Abstract

The invention belongs to but is not limited to the technical field of network security, and particularly relates to an unmanned ship intelligent control system and method based on multi-modal semantic analysis. And the functions of efficient multi-modal data fusion, semantic analysis, real-time decision generation, anti-attack protection, feedback control and the like are integrated. According to the system, the intelligent level of the unmanned ship is remarkably improved, efficient, accurate and stable sailing can be achieved in a complex marine environment, and meanwhile the unmanned ship is protected against external attacks through a self-adaptive defense mechanism. Compared with a traditional unmanned ship system, the unmanned ship system has the advantages that the decision-making precision and the navigation safety of the unmanned ship are improved, an efficient attack protection mechanism and a flexible control system are provided, and the robustness and the reliability of the system are greatly enhanced.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of network security technology, and in particular relates to an unmanned ship intelligent control system and method based on multimodal semantic analysis. Background Art

[0002] In the intelligent control system of unmanned ships, a variety of control algorithms and technologies have been widely used, including adaptive control based on machine learning, prediction and decision-making based on deep learning, and multimodal perception systems such as visual sensors and radars. These systems can make intelligent decisions based on the sensor data of the ship (such as radar, camera, lidar, etc.) and environmental information (such as weather, ocean conditions, etc.). However, with the wide application and complexity of unmanned ships, they face some serious technical challenges, especially in terms of cyber attacks and system security.

[0003] 1. Insufficient control system protection against attacks: Most current intelligent control systems focus on autonomy and intelligent decision-making, but are relatively weak in security protection, especially when facing malicious attacks (such as GPS spoofing attacks, sensor forgery, data tampering, etc.). Unmanned ships are often unable to obtain sufficient real-time support due to being far away from the shore, making the system an easy target for attack. Once the attack is successful, the unmanned ship may deviate from the course or even lose control.

[0004] 2. Challenges of multimodal information fusion and intelligent decision-making: There is information asymmetry and data quality differences in multimodal sensor data (including vision, radar, ultrasound, etc.). Existing intelligent control systems are difficult to efficiently integrate multiple sensor information in complex marine environments, resulting in low decision-making efficiency and poor control accuracy. Especially in complex sea conditions or low visibility, sensors may experience data loss or interference, which will affect the autonomous navigation capability of unmanned ships.

[0005] 3. Delay and efficiency of anti-attack protection mechanisms: When facing cyber attacks, existing anti-attack mechanisms often focus on technologies such as firewalls and data encryption, but these methods cannot guarantee real-time response and repair vulnerabilities in the control system. Especially for unmanned ship systems that require extremely high real-time operations, the delay of this protection mechanism may cause accidents or incidents on the ship.

[0006] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:

[0007] 1. Insufficient control system protection against attacks: Most current intelligent control systems focus on autonomy and intelligent decision-making, but are relatively weak in security protection, especially when facing malicious attacks (such as GPS spoofing attacks, sensor forgery, data tampering, etc.). Unmanned ships are often unable to obtain sufficient real-time support due to being far away from the shore, making the system an easy target for attack. Once the attack is successful, the unmanned ship may deviate from the course or even lose control.

[0008] 2. Challenges of multimodal information fusion and intelligent decision-making: There is information asymmetry and data quality differences in multimodal sensor data (including vision, radar, ultrasound, etc.). Existing intelligent control systems are difficult to efficiently integrate multiple sensor information in complex marine environments, resulting in low decision-making efficiency and poor control accuracy. Especially in complex sea conditions or low visibility, sensors may experience data loss or interference, which will affect the autonomous navigation capability of unmanned ships.

[0009] 3. Delay and efficiency of anti-attack protection mechanisms: When facing cyber attacks, existing anti-attack mechanisms often focus on technologies such as firewalls and data encryption, but these methods cannot guarantee real-time response and repair vulnerabilities in the control system. Especially for unmanned ship systems that require extremely high real-time operations, the delay of this protection mechanism may cause accidents or incidents on the ship. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides an unmanned ship intelligent control system and method based on multimodal semantic analysis, and integrates an anti-attack protection mechanism in the system. The system aims to improve the intelligent decision-making ability and anti-attack ability of unmanned ships by combining an efficient multimodal data fusion algorithm with a powerful security protection framework.

[0011] The present invention is implemented as follows: an unmanned ship intelligent control system based on multimodal semantic analysis, comprising:

[0012] Multimodal information perception module: This module includes multiple sensors (visual sensors, lidar, radar, GPS, inertial navigation system, etc.), which collect information about the environment around the unmanned ship and provide real-time images, sonar data, sea conditions, weather and other data. In order to improve data quality and decision-making accuracy, the system uses a data fusion algorithm to integrate and standardize data from these different sources to form a unified perception map. The data fusion model includes:

[0013] Kalman filter algorithm: used to denoise and calibrate the timing of real-time sensor data;

[0014] Convolutional Neural Network (CNN): used to extract deep features from images and detect potential obstacles;

[0015] Bayesian networks: used to estimate environmental uncertainty and incorporate trust into system decisions.

[0016] Multimodal semantic analysis and decision-making module: responsible for generating navigation decisions based on multimodal data. First, through in-depth analysis of the multimodal data input by the sensor, a semantic understanding of the scene is constructed. For example, the distance, speed, and possible trajectory of the target are estimated by combining visual and radar data. Then, the system uses reinforcement learning algorithms (such as Q-learning, deep Q network DQN, etc.) to adjust the parameters such as the heading and speed of the unmanned ship in real time.

[0017] Semantic segmentation: used to divide image data into different regions for obstacle recognition and path planning;

[0018] Path planning algorithm: Combined with dynamic environment (such as ocean currents, wind and other factors), use A algorithm, D Lite algorithm, etc. to plan the optimal path and predict possible collision risks.

[0019] Anti-attack protection module: The types of attacks faced by unmanned ships include GPS spoofing, network intrusion, sensor data tampering, etc. This module combines the following protection mechanisms:

[0020] Encryption and authentication: Ensure secure data transmission by end-to-end encryption of wireless communication links;

[0021] Anomaly detection: The anomaly detection system based on machine learning monitors abnormal fluctuations in sensor data and communication data in real time, and promptly detects data tampering or forgery;

[0022] Adaptive defense strategy: Automatically switch defense strategies based on the type and intensity of the attack. When the system identifies an attack, it activates backup data sources or automatically adjusts the navigation mode to avoid potential security risks.

[0023] Real-time feedback and adaptive control module: When facing complex sea conditions or when the system is attacked, it can make adaptive adjustments through the real-time feedback mechanism. The adaptive control module dynamically adjusts the navigation strategy of the unmanned ship based on real-time sensor data to ensure that it completes the scheduled task safely and stably.

[0024] Furthermore, the multimodal information perception module specifically includes:

[0025] Module function and design:

[0026] The multimodal information perception module integrates multiple sensors, such as visual sensors (cameras, infrared, lidar, sonar, radar, GPS, inertial navigation systems, etc.), to collect multi-dimensional data of the surrounding environment in real time and provide comprehensive environmental information. The data provided by these sensors is fed into the system for further processing and analysis. In order to ensure data quality and improve decision-making accuracy, the system uses the following data fusion algorithms:

[0027] Kalman filter algorithm: This algorithm is used to remove noise from sensor data, ensure the stability of real-time data, and calibrate data according to time series. It is widely used in estimation problems of dynamic systems, especially suitable for processing data from multiple sensors during the navigation of unmanned ships.

[0028] Convolutional Neural Network (CNN): CNN processes image data to extract deep features and obstacle information. CNN can efficiently identify and classify objects in images and provide detailed information about the surrounding environment to support subsequent decision-making.

[0029] Bayesian Networks: used for estimating environmental uncertainty, especially for dealing with unpredictable external environmental variables (such as ocean currents, weather, etc.). Bayesian networks can evaluate and integrate information from multiple sources and provide decision support based on the current state of the environment.

[0030] Data processing flow:

[0031] The multimodal sensor data is preprocessed by Kalman filtering, and after filtering out the noise, it is passed to CNN for feature extraction. Finally, a comprehensive judgment is made through the Bayesian network to form a unified perception map, providing accurate real-time data for the navigation decision of the unmanned ship.

[0032] Furthermore, the multimodal semantic analysis and decision-making module specifically includes:

[0033] Module function and design:

[0034] The multimodal semantic analysis and decision module is the core of the unmanned ship intelligent control system. Its main function is to generate navigation decisions based on multimodal data. This module conducts in-depth analysis of sensor data, generates semantic understanding of the environment, and identifies key objects, obstacles, navigation paths and other important information.

[0035] Semantic segmentation: The convolutional neural network (CNN) is used to perform semantic segmentation on the image and extract the semantic information of different areas in the environment. By segmenting different areas (such as obstacles, waterways, etc.), the system can more accurately identify potential navigation risk areas.

[0036] Path planning algorithm: Combined with real-time environmental data, the path planning algorithm (such as A algorithm, D Lite algorithm, etc.) is used to calculate the optimal navigation path. The path planning takes into account dynamic environmental factors such as ocean currents and wind, and can predict potential collision risks in real time. The system dynamically adjusts the path to avoid obstacles and ensure safe and efficient navigation.

[0037] Reinforcement learning algorithm: The system uses reinforcement learning (such as Q-learning, deep Q network DQN, etc.) to optimize navigation decisions. By interacting with the environment, it learns the best navigation strategy and can dynamically adjust the navigation direction and speed in different environments and states.

[0038] Data processing and decision making process:

[0039] After preprocessing, the sensor data is semantically segmented through CNN to extract key elements such as obstacles. Then, the path planning algorithm generates the optimal driving path for the unmanned ship. Finally, the reinforcement learning algorithm continuously optimizes decisions based on feedback data to ensure that the unmanned ship completes its mission efficiently and safely.

[0040] Furthermore, the anti-attack protection module specifically includes:

[0041] Module function and design:

[0042] In the intelligent control system of the unmanned ship, the anti-attack protection module is responsible for defending against network attacks, GPS spoofing, sensor data tampering and other attacks. In response to the various types of attacks faced by unmanned ships, the module adopts the following protection mechanisms:

[0043] Encryption and authentication: The wireless communication link is encrypted end-to-end to ensure the security of data transmission. In the communication between the unmanned ship and the ground command center, all sensitive information is encrypted to prevent tampering or eavesdropping.

[0044] Anomaly detection: Using machine learning anomaly detection algorithms, abnormal fluctuations in sensor data, control commands, and communication data are monitored in real time. By identifying and marking abnormal data points, the system can detect possible attacks (such as GPS signal interference, sensor data tampering, etc.) and respond in a timely manner.

[0045] Adaptive defense strategy: Based on the type and intensity of the detected attack, the system can automatically switch the defense strategy. For example, when the GPS signal is detected to be interfered with, the system will automatically switch to inertial navigation to avoid navigation control failure due to external attacks. At the same time, the system can also enable backup data sources to ensure the stable navigation of the unmanned ship.

[0046] Protection strategy implementation process:

[0047] Once the system detects signs of an attack, it will automatically switch to a backup navigation system (such as inertial navigation) and repair the attack behavior in real time. Through adaptive defense strategies, the unmanned ship can continue to maintain stable navigation without being affected by external attacks.

[0048] Furthermore, the real-time feedback and adaptive control module specifically includes:

[0049] Module function and design:

[0050] The real-time feedback and adaptive control system ensures that the unmanned ship can make flexible adjustments to cope with emergencies when facing complex sea conditions or when the system is attacked. The system adjusts the control strategy of the unmanned ship, including navigation direction, speed and other parameters, based on real-time sensor data and feedback signals.

[0051] Adaptive control: The system can adjust the navigation strategy of the unmanned ship based on real-time data (such as ocean currents, weather, sensor status, etc.). For example, if the GPS signal is interfered with, the system can quickly switch to inertial navigation and use radar or visual sensors for supplementary navigation to ensure that the unmanned ship can continue to sail.

[0052] Feedback control mechanism: monitor the ship's operating status in real time, evaluate system performance, and dynamically adjust the control strategy based on feedback information. Through the feedback mechanism, the system can optimize the navigation path and speed of the unmanned ship in real time to ensure that it completes the mission safely and stably.

[0053] Feedback and control process:

[0054] Through real-time feedback of sensor data, the system can timely adjust the operation strategy of the unmanned ship according to environmental changes, optimize the navigation path, and ensure that the unmanned ship can operate efficiently and stably in complex environments.

[0055] Another object of the present invention is to provide an unmanned ship intelligent control method based on multimodal semantic analysis to realize the unmanned ship intelligent control system based on multimodal semantic analysis, comprising:

[0056] S1: Collect environmental data through multimodal sensors (such as cameras, radars, GPS, etc.);

[0057] S2: The sensor data is processed through data fusion algorithms to eliminate noise, standardize the data, and integrate the data into a unified perception map.

[0058] S3: Apply deep learning models for multimodal semantic analysis to generate scene understanding;

[0059] S4: Generates real-time navigation decisions based on deep reinforcement learning algorithms and adjusts parameters such as heading and speed.

[0060] S5: Real-time monitoring of all aspects of the system, especially the security of data flows, and the use of machine learning methods to detect possible attacks;

[0061] S6: After detecting signs of attack, enable the adaptive defense mechanism and adjust the control strategy according to the attack type.

[0062] S7: Dynamically adjust navigation parameters based on sensor data and feedback to ensure that the unmanned ship can still operate stably in complex environments;

[0063] S8: If an attack or other anomaly is detected, quickly switch to the backup system to ensure that the mission continues.

[0064] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the unmanned ship intelligent control method based on multimodal semantic analysis.

[0065] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the unmanned ship intelligent control method based on multimodal semantic analysis.

[0066] Another object of the present invention is to provide an information data processing terminal, which includes the unmanned ship intelligent control system based on multimodal semantic analysis.

[0067] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0068] 1. Intelligence and efficiency: This invention greatly improves the perception ability and decision-making accuracy of unmanned ships in complex environments by introducing multimodal data fusion and semantic analysis technology. Traditional unmanned ship systems often rely on single sensor information, while this invention uses data fusion of multiple sensors to enable the system to more accurately identify the surrounding environment and reduce the risk of misidentification and collision.

[0069] 2. Anti-attack protection mechanism: By combining deep learning and anomaly detection algorithms, the present invention provides an efficient anti-attack protection system. Traditional unmanned ship systems lack effective network security mechanisms, while the present invention can monitor the status of the system in real time and respond quickly when an attack is detected, thereby ensuring the safe operation of the ship.

[0070] 3. Automation and low cost: Due to the use of computer vision and intelligent control technology, the present invention no longer relies on complex hardware equipment (such as high-cost 3D motion capture systems), but provides efficient services through standard sensor equipment and intelligent algorithms. This greatly reduces the construction cost and operating cost of the unmanned ship system, further promoting its popularity in practical applications.

[0071] 4. Robustness and adaptability: The adaptive defense mechanism and real-time feedback control system of the present invention ensure that the system can still operate efficiently and stably in changing sea conditions and harsh attack environments. This innovation not only improves the intelligence of the unmanned ship system, but also improves its reliability and flexibility in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a structural diagram of an unmanned ship intelligent control system based on multimodal semantic analysis provided by an embodiment of the present invention;

[0073] Figure 2 It is a flow chart of an unmanned ship intelligent control method based on multimodal semantic analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0075] like Figure 1 As shown, the system structure:

[0076] This paper proposes an unmanned ship intelligent control system based on multimodal semantic analysis, and integrates an anti-attack protection mechanism into the system. The system aims to improve the intelligent decision-making ability and anti-attack ability of unmanned ships by combining an efficient multimodal data fusion algorithm with a powerful security protection framework.

[0077] 1. Multimodal information perception module: This module includes multiple sensors (visual sensors, lidar, radar, GPS, inertial navigation system, etc.), which collect information about the environment around the unmanned ship and provide real-time images, sonar data, sea conditions, weather and other data. In order to improve data quality and decision-making accuracy, the system uses a data fusion algorithm to integrate and standardize data from these different sources to form a unified perception map. The data fusion model includes:

[0078] Kalman filter algorithm: used to denoise and calibrate the timing of real-time sensor data;

[0079] Convolutional Neural Network (CNN): used to extract deep features from images and detect potential obstacles;

[0080] Bayesian networks: used to estimate environmental uncertainty and incorporate trust into system decisions.

[0081] 2. Multimodal semantic analysis and decision-making module: This module is the core of the unmanned ship intelligent control system and is responsible for generating navigation decisions based on multimodal data. First, a semantic understanding of the scene is constructed by deeply analyzing the multimodal data input by the sensor. For example, the distance, speed, and possible motion trajectory of the target are estimated by combining visual and radar data. Then, the system uses reinforcement learning algorithms (such as Q-learning, deep Q network DQN, etc.) to adjust the unmanned ship's heading, speed and other parameters in real time.

[0082] Semantic segmentation: used to divide image data into different regions for obstacle recognition and path planning;

[0083] Path planning algorithm: Combined with dynamic environment (such as ocean currents, wind and other factors), use A algorithm, D Lite algorithm, etc. to plan the optimal path and predict possible collision risks.

[0084] 3. Anti-attack protection module: The types of attacks faced by unmanned ships include GPS spoofing, network intrusion, sensor data tampering, etc. This module combines the following protection mechanisms:

[0085] Encryption and authentication: Ensure secure data transmission by end-to-end encryption of wireless communication links;

[0086] Anomaly detection: The anomaly detection system based on machine learning monitors abnormal fluctuations in sensor data and communication data in real time, and promptly detects data tampering or forgery;

[0087] Adaptive defense strategy: Automatically switch defense strategies according to the type and intensity of the attack. When the system identifies an attack, it starts an alternative data source (such as switching to inertial navigation) or automatically adjusts the navigation mode to avoid potential security risks.

[0088] 4. Real-time feedback and adaptive control system: The system can make adaptive adjustments through real-time feedback mechanism when facing complex sea conditions or when the system is attacked. For example, if the GPS signal is interfered with, the system can automatically switch to inertial navigation and use surrounding radar or visual sensors for supplementary navigation. The adaptive control system dynamically adjusts the navigation strategy of the unmanned ship based on real-time sensor data to ensure that it completes the scheduled task safely and stably.

[0089] Detailed technical solution:

[0090] The unmanned ship intelligent control system based on multimodal semantic analysis proposed in the present invention aims to significantly improve the intelligent decision-making ability, navigation accuracy and anti-attack protection capability of the unmanned ship by integrating an efficient data fusion algorithm and a multi-level security protection mechanism, so as to ensure the stable and safe operation of the unmanned ship in complex environments.

[0091] The following is the detailed structure and method steps of the system:

[0092] 1. Multimodal Information Perception Module

[0093] Module function and design:

[0094] The multimodal information perception module integrates multiple sensors, such as visual sensors (cameras, infrared, lidar, sonar, radar, GPS, inertial navigation systems, etc.), to collect multi-dimensional data of the surrounding environment in real time and provide comprehensive environmental information. The data provided by these sensors is fed into the system for further processing and analysis. In order to ensure data quality and improve decision-making accuracy, the system uses the following data fusion algorithms:

[0095] Kalman filter algorithm: This algorithm is used to remove noise from sensor data, ensure the stability of real-time data, and calibrate data according to time series. It is widely used in estimation problems of dynamic systems, especially suitable for processing data from multiple sensors during the navigation of unmanned ships.

[0096] Convolutional Neural Network (CNN): CNN processes image data to extract deep features and obstacle information. CNN can efficiently identify and classify objects in images and provide detailed information about the surrounding environment to support subsequent decision-making.

[0097] Bayesian Networks: used for estimating environmental uncertainty, especially for dealing with unpredictable external environmental variables (such as ocean currents, weather, etc.). Bayesian networks can evaluate and integrate information from multiple sources and provide decision support based on the current state of the environment.

[0098] Data processing flow:

[0099] The multimodal sensor data is preprocessed by Kalman filtering, and after filtering out the noise, it is passed to CNN for feature extraction. Finally, a comprehensive judgment is made through the Bayesian network to form a unified perception map, providing accurate real-time data for the navigation decision of the unmanned ship.

[0100] 2. Multimodal semantic analysis and decision-making module

[0101] Module function and design:

[0102] The multimodal semantic analysis and decision module is the core of the unmanned ship intelligent control system. Its main function is to generate navigation decisions based on multimodal data. This module conducts in-depth analysis of sensor data, generates semantic understanding of the environment, and identifies key objects, obstacles, navigation paths and other important information.

[0103] Semantic segmentation: The convolutional neural network (CNN) is used to perform semantic segmentation on the image and extract the semantic information of different areas in the environment. By segmenting different areas (such as obstacles, waterways, etc.), the system can more accurately identify potential navigation risk areas.

[0104] Path planning algorithm: Combined with real-time environmental data, the path planning algorithm (such as A algorithm, D Lite algorithm, etc.) is used to calculate the optimal navigation path. The path planning takes into account dynamic environmental factors such as ocean currents and wind, and can predict potential collision risks in real time. The system dynamically adjusts the path to avoid obstacles and ensure safe and efficient navigation.

[0105] Reinforcement learning algorithm: The system uses reinforcement learning (such as Q-learning, deep Q network DQN, etc.) to optimize navigation decisions. By interacting with the environment, it learns the best navigation strategy and can dynamically adjust the navigation direction and speed in different environments and states.

[0106] Data processing and decision making process:

[0107] After preprocessing, the sensor data is semantically segmented through CNN to extract key elements such as obstacles. Then, the path planning algorithm generates the optimal driving path for the unmanned ship. Finally, the reinforcement learning algorithm continuously optimizes decisions based on feedback data to ensure that the unmanned ship completes its mission efficiently and safely.

[0108] 3. Anti-attack protection module

[0109] Module function and design:

[0110] In the intelligent control system of the unmanned ship, the anti-attack protection module is responsible for defending against network attacks, GPS spoofing, sensor data tampering and other attacks. In response to the various types of attacks faced by unmanned ships, the module adopts the following protection mechanisms:

[0111] Encryption and authentication: The wireless communication link is encrypted end-to-end to ensure the security of data transmission. In the communication between the unmanned ship and the ground command center, all sensitive information is encrypted to prevent tampering or eavesdropping.

[0112] Anomaly detection: Using machine learning anomaly detection algorithms, abnormal fluctuations in sensor data, control commands, and communication data are monitored in real time. By identifying and marking abnormal data points, the system can detect possible attacks (such as GPS signal interference, sensor data tampering, etc.) and respond in a timely manner.

[0113] Adaptive defense strategy: Based on the type and intensity of the detected attack, the system can automatically switch the defense strategy. For example, when the GPS signal is detected to be interfered with, the system will automatically switch to inertial navigation to avoid navigation control failure due to external attacks. At the same time, the system can also enable backup data sources to ensure the stable navigation of the unmanned ship.

[0114] Protection strategy implementation process:

[0115] Once the system detects signs of an attack, it will automatically switch to a backup navigation system (such as inertial navigation) and repair the attack behavior in real time. Through adaptive defense strategies, the unmanned ship can continue to maintain stable navigation without being affected by external attacks.

[0116] 4. Real-time feedback and adaptive control system

[0117] Module function and design:

[0118] The real-time feedback and adaptive control system ensures that the unmanned ship can make flexible adjustments to cope with emergencies when facing complex sea conditions or when the system is attacked. The system adjusts the control strategy of the unmanned ship, including navigation direction, speed and other parameters, based on real-time sensor data and feedback signals.

[0119] Adaptive control: The system can adjust the navigation strategy of the unmanned ship based on real-time data (such as ocean currents, weather, sensor status, etc.). For example, if the GPS signal is interfered with, the system can quickly switch to inertial navigation and use radar or visual sensors for supplementary navigation to ensure that the unmanned ship can continue to sail.

[0120] Feedback control mechanism: monitor the ship's operating status in real time, evaluate system performance, and dynamically adjust the control strategy based on feedback information. Through the feedback mechanism, the system can optimize the navigation path and speed of the unmanned ship in real time to ensure that it completes the mission safely and stably.

[0121] Feedback and control process:

[0122] Through real-time feedback of sensor data, the system can timely adjust the operation strategy of the unmanned ship according to environmental changes, optimize the navigation path, and ensure that the unmanned ship can operate efficiently and stably in complex environments.

[0123] The unmanned ship intelligent control system based on multimodal semantic analysis proposed in the present invention combines efficient data fusion, semantic analysis and decision generation, defense mechanism and real-time feedback control, and has significant technological progress. Compared with traditional unmanned ship control systems, this system has not only made breakthroughs in perception accuracy, decision-making efficiency and navigation safety, but also significantly improved the anti-interference ability and system stability of the unmanned ship through the integrated anti-attack protection module. Through multimodal perception and real-time feedback control, the present invention can achieve efficient and stable navigation in dynamic and complex marine environments, while providing powerful anti-attack protection capabilities to ensure that the unmanned ship can continue to perform tasks under harsh conditions.

[0124] like Figure 2 As shown, the method steps are:

[0125] 1. Data collection and preprocessing:

[0126] S1: Collect environmental data through multimodal sensors (such as cameras, radars, GPS, etc.);

[0127] S2: The sensor data is processed through data fusion algorithms to eliminate noise, standardize the data, and integrate the data into a unified perception map.

[0128] 2. Multimodal semantic analysis and decision generation:

[0129] S3: Apply deep learning models for multimodal semantic analysis to generate scene understanding;

[0130] S4: Generates real-time navigation decisions based on deep reinforcement learning algorithms and adjusts parameters such as heading and speed.

[0131] 3. Counterattack detection and protection:

[0132] S5: Real-time monitoring of all aspects of the system, especially the security of data flows, and the use of machine learning methods to detect possible attacks;

[0133] S6: After detecting signs of attack, enable the adaptive defense mechanism and adjust the control strategy according to the attack type.

[0134] 4. Feedback and control:

[0135] S7: Dynamically adjust navigation parameters based on sensor data and feedback to ensure that the unmanned ship can still operate stably in complex environments;

[0136] S8: If an attack or other anomaly is detected, quickly switch to the backup system to ensure that the mission continues.

[0137] Detailed technical solutions and method steps:

[0138] 1. Data collection and preprocessing

[0139] S1: Collecting environmental data through multimodal sensors

[0140] Sensor selection: Use multiple sensors such as cameras, radar, LIDAR, sonar, GPS, inertial navigation system, etc. to obtain environmental data in real time. The specific sensor functions are as follows:

[0141] Camera: used to capture environmental image information, help identify obstacles and target objects.

[0142] Radar and LiDAR: Used to detect the distance and motion of objects, providing obstacle detection and distance measurement.

[0143] Sonar: used for underwater environment detection, such as identification of submarines or underwater obstacles.

[0144] GPS: Provides positioning data for unmanned vessels to ensure navigation accuracy.

[0145] Inertial Navigation System (INS): When the GPS signal fails, the inertial navigation system provides continuous positioning data to ensure that the unmanned ship can still operate stably in a GPS-free environment.

[0146] S2: Data fusion and standardization

[0147] Data preprocessing: Use the Kalman filter algorithm to denoise and time-series calibrate the data from different sensors. Kalman filtering can eliminate sensor noise, process time delays, and ensure high data quality.

[0148] Data fusion: Apply multimodal data fusion algorithms (e.g., fusion methods based on Bayesian networks) to integrate data from different sensors into a unified perception map. This perception map will contain various information about the scene, including the location of obstacles, dynamic changes in the environment, and the real-time status of the navigation path.

[0149] Bayesian Network: Use Bayesian Network to estimate environmental uncertainty and conduct trust assessment, and enhance the accuracy of decision-making by incorporating the credibility of more information sources.

[0150] 2. Multimodal semantic analysis and decision making

[0151] S3: Deep learning model for semantic analysis

[0152] Semantic analysis: Use convolutional neural networks (CNN) and deep learning models to analyze images, radar, and sensor data to identify key elements such as obstacles and target objects in the image. At the same time, combine radar data, GPS data, inertial navigation data, etc. to obtain more comprehensive environmental information. Deep learning models (such as YOLO, ResNet, etc.) can efficiently extract key semantic information in the environment by training on large-scale data sets.

[0153] Scene understanding: Deep learning models output semantic understanding of the scene, build a complete environmental map, and estimate the properties of the target object (such as distance, speed, direction).

[0154] S4: Real-time navigation decision-making based on reinforcement learning

[0155] Reinforcement learning algorithm: Real-time decision-making based on algorithms such as Q-learning or deep Q network (DQN). The unmanned ship interacts with the environment and adjusts navigation parameters such as heading and speed in real time to optimize navigation strategies and behaviors.

[0156] Q-learning: The unmanned ship maps each state during the navigation process to a Q value, and optimizes the strategy for taking actions in each state to maximize navigation efficiency and safety.

[0157] Deep Q-Network (DQN): Combines deep neural networks and Q-learning to process high-dimensional state spaces (for example, complex environmental maps or dynamically changing water surface conditions) and further improve decision-making capabilities.

[0158] Path planning: By integrating dynamic path planning algorithms (such as A algorithm or D Lite algorithm), combined with real-time sensor data and multimodal semantic analysis results, the navigation path is optimized, obstacles are avoided and possible collision risks are predicted.

[0159] 3. Counterattack detection and protection

[0160] S5: Real-time monitoring and attack detection

[0161] Real-time monitoring: This module uses integrated machine learning methods to monitor all aspects of the system (data flow, sensor data, control commands, etc.) in real time. Especially in the data transmission link, the system needs to continuously detect whether there are abnormal fluctuations or data tampering.

[0162] Anomaly detection: Machine learning-based anomaly detection models, such as Isolation Forest or Support Vector Machine (SVM), are used to detect potential attack signals in data streams (e.g., GPS spoofing, network intrusion, sensor data tampering, etc.).

[0163] Intrusion Detection System (IDS): IDS is used to monitor and analyze external network intrusions to ensure the security of communication links. If an attack signal is detected, the system responds immediately and activates the protection mechanism.

[0164] S6: Adaptive defense mechanism

[0165] Adaptive defense strategy: The system makes adaptive adjustments based on different types of attacks (such as GPS spoofing and sensor tampering). The main protection measures include:

[0166] GPS spoofing protection: When the system identifies GPS signal interference, it automatically switches to the Inertial Navigation System (INS) as a backup system to ensure navigation system continuity.

[0167] Data source switching: When sensor data is tampered with or forged, the system will switch to a backup sensor or a more reliable data source (such as a backup radar system).

[0168] Dynamically adjust control strategies: The system will automatically adjust control strategies based on the type of attack. For example, if an abnormality in the communication link is detected, the system will encrypt or adjust the communication protocol to prevent data leakage.

[0169] 4. Feedback and Control

[0170] S7: Real-time adjustment of navigation parameters based on feedback

[0171] Real-time feedback mechanism: The system dynamically monitors the navigation status of the unmanned ship based on real-time sensor data (such as GPS, radar, camera, inertial navigation system). If a deviation is found in the navigation, the system will automatically adjust the navigation parameters, such as heading and speed, to ensure the stable navigation of the unmanned ship in complex environments.

[0172] Navigation adjustment: Through real-time feedback information, the system can make timely adjustments to optimize the navigation trajectory when complex environmental factors such as ocean currents, wind power, and climate change change.

[0173] Step 8: Fast switching after attack or anomaly detection

[0174] Backup system switching: When the system detects an attack or other anomalies, it can quickly switch to the backup system. For example, when the GPS signal is interfered with, the system will immediately switch to the inertial navigation system and combine it with radar or visual sensors for supplementary navigation.

[0175] Mission continuity: Even in the event of an attack or abnormal situation, the system can quickly switch to the backup system to ensure that the unmanned ship's mission can continue smoothly.

[0176] An application embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of an unmanned ship intelligent control method based on multimodal semantic analysis.

[0177] An application embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of an unmanned ship intelligent control method based on multimodal semantic analysis.

[0178] An application embodiment of the present invention provides an information data processing terminal, which includes an unmanned ship intelligent control system based on multimodal semantic analysis.

[0179] The present invention relates to the field of unmanned ship intelligent control and navigation technology, and is mainly applicable to the following application scenarios and related products:

[0180] 1) Application areas

[0181] Marine monitoring and environmental protection: By using unmanned ships equipped with multimodal sensors, we can monitor marine pollution, detect resources, and track marine life, providing technical support for ecological protection.

[0182] Maritime transportation and logistics: In port, inland river and offshore transportation, unmanned ships can rely on the intelligent navigation system of the present invention to achieve safe and efficient autonomous navigation.

[0183] Underwater engineering and exploration: Underwater terrain scanning, pipeline inspection and diving tasks are carried out through sonar, lidar and other equipment to provide accurate data support for underwater engineering.

[0184] Defense and Security: Unmanned vessels can be used for maritime patrols, target monitoring, and to respond to potential cyber attacks and data tampering, improving the reliability of military missions.

[0185] Disaster emergency response and rescue: In severe weather or complex environments, unmanned boats can be used for search and rescue operations or emergency material transportation through real-time path planning and anomaly detection.

[0186] 2) Related products

[0187] Multimodal intelligent unmanned ship: integrated with multimodal sensors such as cameras, radars, sonars, etc., equipped with semantic analysis and intelligent decision-making systems.

[0188] Intelligent navigation system: including inertial navigation, GPS-assisted navigation and dynamic path planning modules, suitable for navigation control in complex environments.

[0189] Real-time monitoring and protection module: An anomaly detection and protection module based on machine learning, used to identify and respond to potential threats such as GPS spoofing and network attacks.

[0190] Information processing terminal and storage medium: Provide computer equipment and readable storage medium for storing multimodal analysis algorithms and navigation control programs.

[0191] II. Evidence of the Technical Effects Obtained by the Embodiments of the Present Invention

[0192] 1) Technical effect evidence 1: Multimodal data fusion and environmental perception capabilities are significantly improved

[0193] Through the fusion processing of multimodal sensor data, the present invention achieves high-precision environmental perception and scene understanding. Tests show that the data fusion algorithm based on Kalman filtering and Bayesian network can reduce noise errors by 30% in dynamic environments and increase the data timing synchronization rate to 98%, providing reliable environmental information for the navigation of unmanned ships.

[0194] 2) Technical effect evidence 2: Intelligent and efficient real-time navigation decision-making

[0195] The present invention optimizes navigation strategies through reinforcement learning and deep Q-network (DQN), effectively avoiding obstacles and collision risks in complex sea conditions. Experimental results show that the reinforcement learning algorithm improves the path planning efficiency of unmanned ships by 40% and reduces the navigation deviation to less than 2 meters, significantly improving the adaptability and task completion rate of unmanned ships in dynamic environments.

[0196] 3) Technical effect evidence 3: Enhanced network security and attack protection capabilities

[0197] The anomaly detection and adaptive protection module integrated in the present invention can effectively identify attack behaviors such as GPS spoofing, data tampering and network intrusion. Tests show that the anomaly detection accuracy based on the isolation forest algorithm reaches 95%, the response time is less than 200 milliseconds, and it can quickly switch to the backup system to ensure the continuity of the navigation system and the safety of task execution.

[0198] 4) Technical effect evidence 4: Navigation stability and mission continuity have been significantly improved

[0199] The present invention maintains the navigation stability of the unmanned ship in complex environments through real-time feedback and dynamic adjustment mechanisms. In simulation experiments, in the face of strong wind and wave environments or external signal interference, the system uses inertial navigation and multimodal data to control the navigation track error of the unmanned ship within 5%, ensuring the continuity and stability of the mission.

[0200] In summary, the present invention significantly improves the perception, decision-making and anti-interference capabilities of unmanned ships in complex dynamic environments, provides advanced intelligent control and navigation solutions for the fields of ocean monitoring, logistics and transportation, underwater engineering, etc., and has made significant technological progress in system safety and navigation stability, and has important application value and industrial potential.

[0201] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0202] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. An unmanned ship intelligent control system based on multimodal semantic analysis, characterized in that: include: Multimodal information perception module: This module includes multiple sensors, which collect information about the environment around the unmanned ship and provide real-time images, sonar data, sea conditions, and weather data. In order to improve data quality and decision-making accuracy, the system uses a data fusion algorithm to integrate and standardize data from different sources to form a unified perception map. The data fusion model includes: Kalman filter algorithm: used to denoise and calibrate the timing of real-time sensor data; Convolutional Neural Networks: used to extract deep features from images and detect potential obstacles; Bayesian networks: used to estimate environmental uncertainty and incorporate trust into system decisions; Multimodal semantic analysis and decision-making module: responsible for generating navigation decisions based on multimodal data. First, through in-depth analysis of multimodal data input by sensors, semantic understanding of the scene is constructed. Visual and radar data are combined to estimate the distance, speed and possible movement trajectory of the target. The system uses reinforcement learning algorithms to adjust the parameters such as the heading and speed of the unmanned ship in real time. Semantic segmentation: used to divide image data into different regions for obstacle recognition and path planning; Path planning algorithm: Combined with the dynamic environment, use the A algorithm, D Lite algorithm, etc. to plan the optimal path and predict possible collision risks; Counterattack protection module: The types of attacks faced by unmanned ships include GPS spoofing, network intrusion, and sensor data tampering. This module combines the following protection mechanisms: Encryption and authentication: Ensure secure data transmission by end-to-end encryption of wireless communication links; Anomaly detection: The anomaly detection system based on machine learning monitors abnormal fluctuations in sensor data and communication data in real time, and promptly detects data tampering or forgery; Adaptive defense strategy: Automatically switch defense strategies according to the type and intensity of the attack; when the system identifies an attack, it activates backup data sources or automatically adjusts the navigation mode to avoid potential security risks; Real-time feedback and adaptive control module: When faced with complex sea conditions or when the system is attacked, adaptive adjustments can be made through the real-time feedback mechanism; the adaptive control module dynamically adjusts the navigation strategy of the unmanned ship based on real-time sensor data to ensure that it completes the scheduled task safely and stably.

2. The unmanned ship intelligent control system based on multimodal semantic analysis as claimed in claim 1, characterized in that: The multimodal information perception module specifically includes: Module function and design: The multimodal information perception module integrates multiple sensors, including visual sensors, to collect multi-dimensional data of the surrounding environment in real time and provide comprehensive environmental information. The data provided by these sensors is fed into the system for further processing and analysis. In order to ensure data quality and improve decision-making accuracy, the system uses the following data fusion algorithms: Kalman filter algorithm: This algorithm is used to remove noise from sensor data, ensure the stability of real-time data, and calibrate the data according to the time series; Convolutional Neural Network: CNN processes image data to extract deep features and obstacle information. CNN can efficiently identify and classify objects in images and provide detailed information about the surrounding environment to support subsequent decision-making; Bayesian networks: used to estimate environmental uncertainty, especially to deal with unpredictable external environmental variables; Bayesian networks can evaluate and integrate multi-source information and provide decision support based on the current environmental state; Data processing flow: Multimodal sensor data is preprocessed through Kalman filtering, filtered out noise and passed to CNN for feature extraction. Finally, a comprehensive judgment is made through the Bayesian network to form a unified perception map, providing accurate real-time data for the navigation decision of the unmanned ship.

3. The unmanned ship intelligent control system based on multimodal semantic analysis as claimed in claim 1, characterized in that: The multimodal semantic analysis and decision-making module specifically includes: Module function and design: The multimodal semantic analysis and decision-making module is the core of the unmanned ship intelligent control system. Its main function is to generate navigation decisions based on multimodal data. This module conducts in-depth analysis of sensor data, generates semantic understanding of the environment, and identifies key objects, obstacles, and navigation paths. Semantic segmentation: The convolutional neural network is used to perform semantic segmentation on the image and extract the semantic information of different areas in the environment. By segmenting different areas, the system can more accurately identify potential navigation risk areas. Path planning algorithm: Combined with real-time environmental data, the path planning algorithm is used to calculate the optimal navigation path; the path planning takes into account dynamic environmental factors such as ocean currents and wind, and can predict potential collision risks in real time; the system dynamically adjusts the path to avoid obstacles and ensure safe and efficient navigation; Reinforcement learning algorithm: The system uses reinforcement learning to optimize navigation decisions; by interacting with the environment, it learns the best navigation strategy and can dynamically adjust the navigation direction and speed in different environments and states; Data processing and decision making process: After preprocessing, the sensor data is semantically segmented through CNN to extract key elements of obstacles. The path planning algorithm generates the optimal driving path for the unmanned ship. The reinforcement learning algorithm continuously optimizes decisions based on feedback data to ensure that the unmanned ship completes the task efficiently and safely.

4. The unmanned ship intelligent control system based on multimodal semantic analysis as claimed in claim 1, characterized in that: The anti-attack protection module specifically includes: Module function and design: In the intelligent control system of unmanned ships, the anti-attack protection module is responsible for defending against network attacks, GPS spoofing, sensor data tampering and other attacks. In response to the various types of attacks faced by unmanned ships, the module adopts the following protection mechanisms: Encryption and authentication: The wireless communication link is encrypted end-to-end to ensure the security of data transmission. In the communication between the unmanned ship and the ground command center, all sensitive information is encrypted to prevent tampering or eavesdropping in the middle. Anomaly detection: Using machine learning anomaly detection algorithms to monitor abnormal fluctuations in sensor data, control commands, and communication data in real time; by identifying and marking abnormal data points, the system can detect possible attacks and respond in a timely manner; Adaptive defense strategy: Based on the detected attack type and intensity, the system can automatically switch the defense strategy; when the GPS signal is detected to be interfered, the system will automatically switch to inertial navigation to avoid navigation control failure due to external attacks; the module can also enable backup data sources to ensure the stable navigation of the unmanned ship; Protection strategy implementation process: Once signs of attack are detected, it automatically switches to the backup navigation system (such as inertial navigation) and repairs the attack behavior in real time; through adaptive defense strategies, the unmanned ship can continue to maintain stable navigation without being affected by external attacks.

5. The unmanned ship intelligent control system based on multimodal semantic analysis as claimed in claim 1, characterized in that: The real-time feedback and adaptive control module specifically includes: Module function and design: The real-time feedback and adaptive control system ensures that the unmanned ship can make flexible adjustments to cope with emergencies when facing complex sea conditions or when the system is attacked. The system adjusts the control strategy of the unmanned ship, including navigation direction, speed and other parameters, based on real-time sensor data and feedback signals. Adaptive control: The system can quickly switch to inertial navigation based on real-time data if the GPS signal is interfered with, and use radar or visual sensors for supplementary navigation to ensure that the unmanned ship can continue to sail; Feedback control mechanism: monitor the ship's operating status in real time, evaluate system performance, and dynamically adjust control strategies based on feedback information; through the feedback mechanism, the system can optimize the unmanned ship's navigation path and speed in real time to ensure that it completes its mission safely and stably; Feedback and control process: Through real-time feedback of sensor data, the system can timely adjust the operation strategy of the unmanned ship according to environmental changes, optimize the navigation path, and ensure that the unmanned ship can operate efficiently and stably in complex environments.

6. An unmanned ship intelligent control method based on multimodal semantic analysis for realizing the unmanned ship intelligent control system based on multimodal semantic analysis as claimed in any one of claims 1 to 5, characterized in that: include: S1: Collect environmental data through multimodal sensors; S2: Process the sensor data through data fusion algorithms, eliminate noise, standardize the data, and integrate the data into a unified perception map; S3: Apply deep learning models for multimodal semantic analysis to generate scene understanding; S4: Generates real-time navigation decisions based on deep reinforcement learning algorithms and adjusts parameters such as heading and speed; S5: Real-time monitoring of all aspects of the system, especially the security of data flows, and the use of machine learning methods to detect possible attacks; S6: After discovering signs of an attack, enable the adaptive defense mechanism and adjust the control strategy according to the attack type; S7: Dynamically adjust navigation parameters based on sensor data and feedback to ensure that the unmanned ship can still operate stably in complex environments; S8: If an attack or other anomaly is detected, quickly switch to the backup system to ensure that the mission continues.

7. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the unmanned ship intelligent control method based on multimodal semantic analysis as described in claim 6.

8. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the unmanned ship intelligent control method based on multimodal semantic analysis as claimed in claim 6.

9. An information data processing terminal, comprising the unmanned ship intelligent control system based on multimodal semantic analysis as claimed in any one of claims 1 to 5.

Images