Automatic ship driving method and system based on large model visual perception

Abstract

The invention relates to the technical field of ship navigation and control, and provides a ship automatic driving method and system based on large model visual perception. According to the invention, through integration of a multi-source sensing acquisition technology, omnibearing and high-precision sensing of sea surface and underwater environment is realized; according to the method, multi-source data such as visible light, infrared, sonar and AIS system data are collected, the robustness of environmental perception is enhanced by using a large visual model based on a Transform architecture and a multi-source feature enhancement algorithm, a plurality of candidate trajectories are further generated in cooperation with a diffusion model, a navigation rule base is called to carry out compliance screening on the optimal trajectory, and then a decision instruction is output. According to the method, automatic driving of the ship under low visibility or complex sea conditions is realized, the safety and decision-making efficiency of ship navigation are remarkably improved, various navigation scenes can be flexibly coped, and more intelligent and reliable navigation support is provided for the ship.

Description

Technical Field

[0001] This invention relates to the field of ship navigation and control technology, and more specifically, to a ship autopilot method and system based on large model visual perception. Background Technology

[0002] With increasingly frequent maritime activities, the demand for ship navigation safety and intelligence has significantly increased. Traditional maritime systems, which mainly rely on single sensors or basic data fusion technologies, are ill-equipped to handle the challenges of dynamic target perception and real-time decision-making in complex marine environments. Current technological trends focus on multimodal data collaborative processing, integrating multi-source data such as visible light, infrared, sonar, and Automatic Identification System (AIS) data to achieve comprehensive and accurate environmental perception.

[0003] Traditional maritime systems suffer from weak multi-source data collaboration capabilities and insufficient robustness in environmental perception. Individual sensors are susceptible to interference from lighting, visibility, or sea conditions, leading to data loss or error accumulation. For multi-source data analysis, current basic data fusion methods lack dynamic weight allocation mechanisms, making it difficult to adapt to the feature association requirements of complex scenarios. At the trajectory decision-making level, traditional methods rely on static rule bases, failing to generate multi-path candidate solutions in real time and perform compliance verification, resulting in low decision-making efficiency and limited security. Furthermore, traditional systems lack closed-loop feedback optimization mechanisms; navigation parameter adjustments rely on manual intervention, hindering continuous adaptive control. Remote monitoring functions are also relatively isolated, with high data interaction latency, failing to support real-time scheduling and overall performance improvement. Summary of the Invention

[0004] To overcome the problems of weak multi-source data collaboration capability and insufficient environmental perception robustness in traditional navigation systems described in the prior art, this invention provides a ship autopilot method and system based on large-model visual perception.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: The ship autopilot method based on large-model visual perception includes the following steps: Using visible light cameras, infrared thermal imagers, sonar equipment, and AIS systems, current visible light images, thermal imaging data, underwater sonar data, and AIS data are collected and preprocessed to form a standardized dataset. The standardized dataset is input into a pre-trained large visual model based on the Transformer architecture. The large visual model extracts single-source features from the standardized dataset, combines a multi-source feature enhancement algorithm to fuse the single-source features, and then performs boundary feature enhancement processing to obtain environmental perception features. The environmental perception features are input into a pre-trained diffusion model. The diffusion model generates multiple candidate trajectories based on a dynamic target relationship graph constructed from AIS data. After filtering the candidate trajectories by calling a preset navigation rule base, the optimal trajectory is output and a decision instruction is generated. The steering control unit and propulsion control unit deployed on the ship receive the decision commands and execute them to control the ship's course and speed.

[0006] In a preferred embodiment, the underwater sonar data includes two-dimensional acoustic images of underwater obstacles acquired by side-scan sonar, and distance and contour data of underwater targets acquired by forward-looking sonar; the AIS data includes the vessel's Maritime Mobile Service Identifier (MMSI), latitude and longitude, heading, speed, and dynamic information of surrounding vessels; then, the acquisition and preprocessing of the current visible light image, thermal imaging data, underwater sonar data, and AIS data includes the following steps: Acquire current visible light images, thermal imaging data, underwater sonar data, and AIS data; The visible light image, the thermal imaging data, the underwater sonar data, and the AIS data are time-stamped and synchronized. The visible light image, thermal imaging data, underwater sonar data, and AIS data that have been synchronized with time stamps are denoised, encoded, and beamformed to form a standardized data set.

[0007] In a preferred embodiment, the large visual model includes 24-32 encoder layers and 28-36 decoder layers; each encoder layer is configured with 8-32 self-attention heads, the computation range of which covers the entire input data; the input layer of the large visual model is configured with an interface adapted to multimodal data, the input layer is used to receive the RGB features of the visible light image, the temperature field features of the thermal imaging data, the acoustic reflection intensity features of the underwater sonar data, and the structured vector features of the AIS data, and converts the various feature data into feature vectors of the same dimension through feature mapping before inputting them into the encoder.

[0008] In a preferred embodiment, the large visual model is configured with a multi-source feature enhancement algorithm; its expression is:

[0009] in, This refers to features for global environment perception. For the first i Single-source characteristics of sensor-like devices The mean of multiple source features; Represents the self-attention weights. This is the attention enhancement coefficient; iIndicates the sensor type identifier, and , This refers to a visible light camera. I Indicates an infrared thermal imager. S Indicates sonar equipment. A Indicates the AIS system; For the first i Confidence weights for sensor-like devices The first The inherent reliability coefficient of this type of sensor, The first The sensor's environmental adaptability coefficient This is the adjustment coefficient.

[0010] In a preferred embodiment, the dynamic target relationship map is constructed by collecting AIS data over a certain period of time; the steps include: S010. Extract the ship's MMSI, latitude and longitude, heading, speed and navigation status information from the AIS data; S020. Convert the ship's latitude and longitude coordinates to the UTM coordinate system; S030. Calculate the relative distance, relative heading, and relative speed between the current vessel and any surrounding vessel to determine the spatial positional relationship and motion trend correlation between the targets. S040. Construct a target relationship graph with ships as target nodes, and add type labels and static attributes to each target node in combination with the environmental perception type in the environmental perception features. S050. Update the AIS data and the environmental perception features at preset time intervals, and repeat steps S010 to S030 to dynamically update the UTM coordinates, heading, speed and motion trend association of any target node in the target relationship graph to form a dynamic target relationship graph.

[0011] In a preferred embodiment, the diffusion model generates multiple candidate trajectories based on a dynamic target relationship graph constructed from AIS data, including the following steps: The AIS data currently collected by the AIS system is used as the initial state data of the ship; During the forward diffusion process, Gaussian noise is gradually added to the initial state data of the ship to generate multiple sets of noisy virtual state data; In the reverse diffusion process, a U-Net network is used, with the spatial position relationship and motion trend association between the corresponding target nodes in the dynamic target relationship graph as the condition constraint, to perform noise reduction iteration on the virtual state data, and each iteration outputs a set of potential trajectory parameters; When the number of iterations reaches a preset termination condition, the diffusion model outputs a certain number of candidate trajectories; wherein, the candidate trajectories include future...t The position coordinate sequence, heading change curve, and speed adjustment sequence within minutes; t This is the preset time length.

[0012] In a preferred embodiment, the method further includes the following steps: After executing the decision command, the actual driving status data of the ship is collected by sensors deployed on the ship; the actual driving status data includes the actual heading, actual speed, and the real-time latitude and longitude position of the ship.

[0013] Based on the actual driving status data, the decision command, and the optimal trajectory, a comprehensive deviation value is calculated using a ship driving comprehensive deviation evaluation algorithm. Its expression is: ; in, For the actual course, To determine the course; This is the actual speed. To determine the cruising speed; Initial speed; The decision instruction is adjusted based on the comprehensive deviation value; its expression is: ; in, For heading adjustment, This is the heading adjustment factor; This is a sign function; it outputs 1 when the input is positive, -1 when the input is negative, and 0 when the input is zero. For speed adjustment amount, This is the speed adjustment factor.

[0014] In a preferred embodiment, the steering gear control unit includes a servo driver and an angle sensor. The servo driver receives a heading adjustment angle signal from the decision command to drive the rudder blade to rotate by a corresponding angle. The angle sensor collects the actual rotation angle of the rudder blade to obtain the actual heading of the ship. The propeller control unit includes a power regulator and a speed sensor. The power regulator receives a speed adjustment signal from the decision command to adjust the output power of the ship's propeller. The speed sensor collects the actual speed of the propeller to obtain the actual speed of the ship.

[0015] In a preferred embodiment, the method further includes the following steps: uploading the actual driving status data, the decision command, the optimal trajectory, and the comprehensive deviation value to the shore-based monitoring system.

[0016] Furthermore, this invention also proposes a ship autopilot system based on large-model visual perception, applying the ship autopilot method based on large-model visual perception proposed in this invention. The system includes: The multi-source sensing and acquisition module is equipped with a visible light camera, an infrared thermal imager, a sonar device, and an AIS system. It is used to collect current visible light images, thermal imaging data, underwater sonar data, and AIS data about surrounding vessels and preprocess them to form a standardized data set. The large model fusion processing module is equipped with a pre-trained visual large model based on the Transformer architecture, which is used to extract single-source features from the standardized dataset, fuse the single-source features with a multi-source feature enhancement algorithm, and output environmental perception features after boundary feature enhancement processing. The multimodal trajectory decision module is configured with a pre-trained diffusion model and a dynamic target relationship graph constructed based on AIS data. The diffusion model generates multiple candidate trajectories based on the dynamic target relationship graph constructed based on AIS data, and outputs the optimal trajectory and generates decision instructions after filtering the candidate trajectories by calling a preset navigation rule base. The navigation control execution module, which is equipped with a steering gear control unit and a propeller control unit, is used to receive the decision commands and execute them to control the ship's course and speed.

[0017] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention achieves comprehensive and high-precision perception of the sea surface and underwater environment by integrating multi-source sensing and acquisition technologies. Specifically, it collects multi-source data such as visible light, infrared, sonar, and AIS system data, and uses a large visual model based on the Transformer architecture and multi-source feature enhancement algorithms to enhance the robustness of environmental perception. Furthermore, it uses a diffusion model to generate multiple candidate trajectories, and then calls the navigation rule base to select the optimal trajectory based on compliance before outputting decision instructions. This enables ships to autopilot in low visibility or complex sea conditions, significantly improving the safety and decision-making efficiency of ship navigation, and can flexibly respond to various navigation scenarios, providing ships with more intelligent and reliable navigation support. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a ship autopilot method according to an embodiment of the present invention.

[0019] Figure 2 This is an architectural diagram of a ship autopilot system according to an embodiment of the present invention. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Example

[0024] This embodiment proposes a ship autopilot method based on large-model visual perception, such as... Figure 1 The diagram shown is a flowchart of the ship autopilot method of this embodiment.

[0025] The ship autopilot method proposed in this embodiment includes the following steps: S100 uses a visible light camera, an infrared thermal imager, sonar equipment, and an AIS system to collect current visible light images, thermal imaging data, underwater sonar data, and AIS data, and preprocesses them to form a standardized dataset. S200. Input the standardized dataset into a pre-trained visual large model based on the Transformer architecture. The visual large model extracts single-source features from the standardized dataset, combines a multi-source feature enhancement algorithm to fuse the single-source features, and then obtains environmental perception features through boundary feature enhancement processing. S300. The environmental perception features are input into a pre-trained diffusion model. The diffusion model generates multiple candidate trajectories based on a dynamic target relationship graph constructed from AIS data. After filtering the candidate trajectories by calling a preset navigation rule base, the optimal trajectory is output and a decision instruction is generated. S400, the steering control unit and propulsion control unit deployed on the ship receive the decision commands and execute them to control the ship's course and speed.

[0026] This embodiment collects multi-source data such as visible light, infrared, sonar, and AIS data, and uses a large visual model based on the Transformer architecture and multi-source feature enhancement algorithms to enhance the robustness of environmental perception. It further combines the diffusion model to generate the optimal trajectory and output decision commands, thereby realizing the ship's autonomous driving in complex environments.

[0027] In this embodiment, the visible light camera may optionally be a high-definition industrial camera, used to acquire visible light images of the sea surface and other ship targets to provide basic visual information support; the infrared thermal imager may optionally be an uncooled focal plane array camera, used to acquire thermal imaging data of surrounding targets in nighttime or low-visibility environments, including the outlines and positions of surrounding ships, buoys and other targets; the sonar equipment may optionally be a side-scan sonar and a forward-looking sonar, used to acquire two-dimensional acoustic images of underwater obstacles and distance and outline data of underwater targets ahead, such as the distribution range and shape of potential obstacles such as underwater reefs and shipwrecks; the AIS system may optionally be a Class A device, used to acquire the ship's Maritime Mobile Service Identifier (MMSI), latitude and longitude, heading, speed and dynamic information of surrounding ships.

[0028] AIS data is a set of digital navigation information generated and transmitted by the AIS system, mainly used to achieve automatic identification and traffic monitoring between ships and between ships and shore. It includes static information, dynamic information, and voyage-related information. Static information includes ship name, call sign, MMSI, IMO number, length, beam, and ship type; dynamic information includes real-time updated latitude and longitude, speed over land (SOG), heading over land (COG), heading, rate of turn, and navigation status; voyage-related information includes port of destination, estimated time of arrival (ETA), ship draft, type of cargo, and number of crew members. The AIS data collected in this embodiment can be selected based on the specific implementation scenario, and is not limited here.

[0029] Furthermore, this embodiment may optionally perform standardized preprocessing on multi-source data such as visible light images, thermal imaging data, underwater sonar data, and AIS data to adapt to the input requirements of large visual models based on the Transformer architecture.

[0030] In an optional embodiment, step S100 utilizes a visible light camera, an infrared thermal imager, a sonar device, and an AIS system to acquire current visible light images, thermal imaging data, underwater sonar data, and AIS data, and performs preprocessing. Specific steps include: S110: Acquire current visible light images, thermal imaging data, underwater sonar data, and AIS data; S120. Synchronize the visible light image, the thermal imaging data, the underwater sonar data, and the AIS data using timestamps; S130. The visible light image, thermal imaging data, underwater sonar data and AIS data that have been time-stamped are respectively denoised, encoded and beamformed to form a standardized data set.

[0031] This embodiment ensures consistency across data sources in terms of time by synchronizing data from different sources using timestamps, thus avoiding data matching errors caused by time differences. For example, denoising the data from these different sources may optionally include eliminating image jitter, water mist interference, and background noise in sonar data to improve data purity; denoising the data from these different sources may optionally include converting various heterogeneous data into a unified format for efficient reading and processing by subsequent modules; beamforming processing of the data from these different sources can enhance the signal strength and directivity of sonar data, improving the accuracy of underwater target detection. Ultimately, a standardized dataset is formed, providing high-quality, uniformly formatted input data for the large model fusion processing module, ensuring the accuracy and efficiency of subsequent processing steps.

[0032] In one optional embodiment, the large visual model based on the Transformer architecture includes 24 to 32 encoder layers and 28 to 36 decoder layers; wherein each encoder layer is configured with 8 to 32 self-attention heads, and the calculation range of the self-attention heads covers the entire input data, which can fully mine the global correlation information in the data.

[0033] The input layer of the large-scale visual model is configured with an interface adapted to multimodal data. This input layer receives RGB features from the visible light image, temperature field features from the thermal imaging data, acoustic reflection intensity features from the underwater sonar data, and structured vector features from the AIS data. After converting these various feature data into feature vectors of the same dimension through feature mapping, the vectors are input into the encoder. The encoder and decoder then perform single-source feature extraction, ensuring that the unique information from each data source is fully extracted, laying the foundation for subsequent fusion. Furthermore, the large-scale visual model combines a multi-source feature enhancement algorithm to perform feature fusion on the extracted single-source features, and then performs boundary feature enhancement processing to obtain environmental perception features.

[0034] The large-scale visual model in this embodiment is pre-trained before use. Optionally, pre-training is performed on a dataset containing scene-annotated data. The annotated data includes targets such as ships, buoys, and reefs under different lighting conditions and sea states, enabling it to accurately identify and extract features from various maritime targets. This effectively adapts to the data processing needs of low visibility and complex environments at night. For example, the pre-training of the large-scale visual model is performed on a dataset containing 1 million frames of maritime scene annotation data. This dataset covers various targets (e.g., ships, buoys, and reefs) under different sea states, lighting conditions, and weather conditions.

[0035] It should be noted that the number of encoder and decoder layers in this large visual model is set according to actual usage needs. Preferably, it is configured with 24 encoder layers and 28 decoder layers. The number of self-attention heads configured on the encoder is set according to the actual hardware parameters and fine-grained requirements. Preferably, the encoder is configured with 32 self-attention heads.

[0036] Furthermore, in an optional embodiment, the large visual model is configured with a multi-source feature enhancement algorithm; its expression is:

[0037] in, This refers to features for global environment perception. For the first i Single-source characteristics of sensor-like devices The mean of multiple source features; Represents the self-attention weights. This is the attention enhancement coefficient; i Indicates the sensor type identifier, and , This refers to a visible light camera. I Indicates an infrared thermal imager. S Indicates sonar equipment. A Indicates the AIS system; For the first i Confidence weights for sensor-like devices The first The inherent reliability coefficient of this type of sensor, The first The sensor's environmental adaptability coefficient This is the adjustment coefficient.

[0038] In this embodiment, by dynamically calculating the confidence weights of the sensors, the weight allocation can be dynamically adjusted according to the actual working performance of each sensor in low-visibility or complex nighttime scenarios. This allows sensor data with high reliability and adaptability to play a greater role in the fusion process. A multi-source feature enhancement algorithm integrates visible light, infrared, sonar, and AIS data, avoiding the limitations of a single data source and improving the comprehensiveness and accuracy of feature representation. After feature fusion of single-source features, boundary feature enhancement processing further highlights the boundary differences between the target and the background, reducing recognition errors caused by target blurring in nighttime environments. Finally, high-accuracy environmental perception features are output, providing precise environmental perception basis for subsequent diffusion models to make optimal trajectory decisions.

[0039] It should be noted that the sensors referred to in this embodiment are visible light cameras, infrared thermal imagers, sonar equipment, and AIS systems deployed on ships. They can also be other sensors that may be configured or deployed on ships, such as angle sensors, speed sensors, GPS, heading sensors, speedometers, etc. The environmental perception features in this embodiment include the position, heading, and speed of surrounding ships, as well as the position and shape of underwater obstacles (e.g., reefs, buoys). They can also include other environmental factors that may affect the ship's navigation trajectory selection, such as wind direction, tides, and sea conditions, and can be set according to the application scenario.

[0040] In an optional embodiment, the dynamic target relationship map is constructed by collecting AIS data over a certain period of time; the steps include: S010. Extract the ship's MMSI, latitude and longitude, heading, speed and navigation status information from the AIS data; S020. Convert the ship's latitude and longitude coordinates to the UTM coordinate system; S030. Calculate the relative distance, relative heading, and relative speed between the current vessel and any surrounding vessel to determine the spatial positional relationship and motion trend correlation between the targets. S040. Construct a target relationship graph with ships as target nodes, and add type labels and static attributes to each target node in combination with the environmental perception type in the environmental perception features. S050. Update the AIS data and the environmental perception features at preset time intervals, and repeat steps S010 to S030 to dynamically update the UTM coordinates, heading, speed and motion trend association of any target node in the target relationship graph to form a dynamic target relationship graph.

[0041] Converting ship latitude and longitude coordinates to the UTM coordinate system can eliminate positional deviations caused by different coordinate systems and ensure the consistency of spatial positioning; determining the spatial positional relationships and motion trend correlations between targets can help predict the existence of collision risks in advance; and combining the environmental perception types in the environmental perception features with adding type labels (e.g., cargo ship, buoy, reef) and static attributes (e.g., ship length, ship width, ship tonnage, draft, buoy height, reef height) to each target node can provide richer reference information for trajectory decision-making.

[0042] For example, regarding the construction of the dynamic target relationship graph, the construction of the dynamic target relationship graph should be completed at least before the candidate trajectory generation using the diffusion model in step S300, preferably before step S100, that is, before the execution of the ship's autopilot task, so as to improve the efficiency and accuracy of decision command generation.

[0043] For example, the preset time interval in step S050 is set to 10 to 30 seconds, preferably 10 seconds, to update AIS data and environmental perception results at regular intervals, and to update the position, motion parameters and relationships of target nodes in the map synchronously, forming a real-time dynamic target relationship map, which can promptly grasp changes in the surrounding environment and avoid decision-making errors due to information lag.

[0044] Furthermore, in an optional embodiment, the diffusion model generates multiple candidate trajectories based on a dynamic target relationship graph constructed from AIS data, including the following steps: The AIS data currently collected by the AIS system is used as the initial state data of the ship; During the forward diffusion process, Gaussian noise is gradually added to the initial state data of the ship to generate multiple sets of noisy virtual state data; In the reverse diffusion process, a U-Net network is used, with the spatial position relationship and motion trend association between the corresponding target nodes in the dynamic target relationship graph as the condition constraint, to perform noise reduction iteration on the virtual state data, and each iteration outputs a set of potential trajectory parameters; When the number of iterations reaches a preset termination condition, the diffusion model outputs a certain number of candidate trajectories; wherein, the candidate trajectories include future... t The position coordinate sequence, heading change curve, and speed adjustment sequence within minutes; t This is the preset time length.

[0045] In this embodiment, the diffusion model includes a forward diffusion process and a reverse generation process. During the forward diffusion process, Gaussian noise is gradually added to the initial ship state data to simulate various changes in navigation state that may occur in low visibility or complex nighttime scenarios. Preferably, the latest observed state from the historical trajectory in AIS data is used as the initial ship state data.

[0046] For example, the number of iterations is determined by the number of candidate trajectories required. For instance, the model sets 3 to 5 sets of iteration termination conditions, which correspond to generating 3 to 5 candidate trajectories. Each candidate trajectory contains a sequence of position coordinates, a heading change curve, and a speed adjustment sequence for the next 3 to 10 minutes, providing sufficient trajectory options for subsequent screening.

[0047] After generating candidate trajectories, the navigation rules database is further invoked to screen all candidate trajectories for compliance, eliminating those that do not comply with collision avoidance rules and channel speed limits, ensuring that the final output decision instructions meet both safe navigation requirements and maritime traffic rules, providing legal and safe path guidance for ship navigation.

[0048] For example, the navigation rules library used in this embodiment includes a full digital model of the COLREGS International Collision Avoidance Code, divided into three sub-libraries according to the type of encounter situation: face-to-face, overtaking, and cross encounter. Each sub-library includes the applicable conditions, behavioral requirements, and quantitative thresholds: the face-to-face sub-library sets the threshold for the angle between the two vessels' headings to be 170°~190°, requiring a avoidance turning angle of not less than 30°; the overtaking sub-library sets the relative speed threshold between the overtaking vessel and the overtaken vessel to be 2 knots or more, requiring a lateral distance of not less than 2 nautical miles between the overtaking vessel and the overtaken vessel; the cross encounter sub-library sets the heading priority threshold for right-hand vessels, requiring a avoidance distance of not less than 1 nautical mile for vessels approaching from the right, and requiring vessels approaching from the left to actively decelerate to less than 50% of their original speed. The rule library also includes supplementary rules for special scenarios, setting a speed limit of 10 knots in bridge areas and a buffer distance of 0.5 nautical miles at the boundary of prohibited areas. It should be noted that the above navigation rules library can be constructed and adjusted by selecting other applicable navigation rules according to the application scenario, and is not limited in this embodiment.

[0049] This embodiment filters the candidate trajectories by calling a preset navigation rule base, outputs the optimal trajectory, and generates a decision instruction. The decision instruction may optionally include a heading adjustment angle signal, a speed adjustment signal, etc., and is specifically generated based on the position coordinate sequence, heading change curve, and speed adjustment sequence contained in the optimal trajectory.

[0050] Furthermore, in an optional embodiment, the method further includes the following steps: S500: Collect actual ship navigation status data through sensors deployed on the ship; the actual navigation status data includes actual heading, actual speed, and the ship's real-time latitude and longitude position; calculate a comprehensive deviation value based on the actual navigation status data, the decision command, and the optimal trajectory using a ship navigation comprehensive deviation evaluation algorithm. ; and, adjust the decision instruction based on the comprehensive deviation value.

[0051] Among them, the comprehensive deviation value The expression is: ; in, For the actual course, To determine the course; This is the actual speed. To determine the cruising speed; Initial speed; The decision instruction is adjusted based on the comprehensive deviation value; its expression is: ; in, For heading adjustment, This is the heading adjustment factor; This is a sign function; it outputs 1 when the input is positive, -1 when the input is negative, and 0 when the input is zero. For speed adjustment amount, This is the speed adjustment factor.

[0052] In this embodiment, a comprehensive deviation evaluation algorithm for ship navigation is adopted. The comprehensive deviation value is calculated by combining the actual course, the decision course, the actual speed, the decision speed, and the initial speed, thereby quantifying the degree of deviation between the ship's actual navigation state and the ideal state. Furthermore, based on the calculated comprehensive deviation value, a ship navigation parameter deviation feedback adjustment algorithm is used. This algorithm combines the course adjustment coefficient, the speed adjustment coefficient, the comprehensive deviation value, the sign of the difference between the actual and decision courses, the difference between the actual and decision speeds, and the initial speed to calculate the course adjustment amount and the speed adjustment amount, respectively. This enables dynamic correction decision-making for actual navigation deviations, thereby improving navigation accuracy and safety.

[0053] Furthermore, in an optional embodiment, the method further includes the following steps: S600: Upload the actual driving status data, the decision command, the optimal trajectory, and the comprehensive deviation value to the shore-based monitoring system.

[0054] In this embodiment, by uploading the ship's actual driving status data, comprehensive deviation value, adjusted parameters and other relevant data to the shore-based monitoring system in real time, remote data interaction is completed. This allows shore-based staff to monitor the ship's driving status in near-shore waterways at night when visibility is low or sea conditions are complex, promptly grasp whether there are any abnormalities in the ship (e.g., whether there is a continuous deviation or whether it is facing a sudden danger), and intervene remotely when necessary, further ensuring the safety of ship navigation.

[0055] Further, in an optional embodiment, the steering gear control unit includes a servo driver and an angle sensor. The servo driver receives a heading adjustment angle signal from the decision command to drive the rudder blade to rotate by a corresponding angle. The angle sensor acquires the actual rotation angle of the rudder blade to obtain the actual heading of the ship. The propeller control unit includes a power regulator and a speed sensor. The power regulator receives a speed adjustment signal from the decision command to adjust the output power of the ship's propeller. The speed sensor acquires the actual rotational speed of the propeller to obtain the actual speed of the ship.

[0056] In this embodiment, GPS receivers, heading sensors, and speedometers can be used to obtain the ship's real-time latitude and longitude position, actual navigation direction, and actual navigation speed to assist in dynamic correction decisions for actual navigation deviations. Example

[0057] This embodiment proposes a ship autopilot system based on large-model visual perception, applying the ship autopilot method proposed in Embodiment 1. For example... Figure 2 The diagram shown is an architecture diagram of the ship autopilot system in this embodiment.

[0058] The ship autopilot system based on large-model visual perception proposed in this embodiment includes: The multi-source sensing and acquisition module is equipped with a visible light camera, an infrared thermal imager, a sonar device, and an AIS system. It is used to collect current visible light images, thermal imaging data, underwater sonar data, and AIS data about surrounding vessels and preprocess them to form a standardized data set. The large model fusion processing module is equipped with a pre-trained visual large model based on the Transformer architecture, which is used to extract single-source features from the standardized dataset, fuse the single-source features with a multi-source feature enhancement algorithm, and output environmental perception features after boundary feature enhancement processing. The multimodal trajectory decision module is configured with a pre-trained diffusion model and a dynamic target relationship graph constructed based on AIS data. The diffusion model generates multiple candidate trajectories based on the dynamic target relationship graph constructed based on AIS data, and outputs the optimal trajectory and generates decision instructions after filtering the candidate trajectories by calling a preset navigation rule base. The navigation control execution module, which is equipped with a steering gear control unit and a propeller control unit, is used to receive the decision commands and execute them to control the ship's course and speed.

[0059] In one optional embodiment, the steering gear control unit includes a servo driver and an angle sensor. The servo driver receives a heading adjustment angle signal from the decision command to drive the rudder blade to rotate by a corresponding angle. The angle sensor collects the actual rotation angle of the rudder blade to obtain the ship's actual heading. The propeller control unit includes a power regulator and a speed sensor. The power regulator receives a speed adjustment signal from the decision command to adjust the output power of the ship's propeller. The speed sensor collects the actual rotational speed of the propeller to obtain the ship's actual speed. Furthermore, a GPS receiver, heading sensor, and speedometer can be used to acquire the ship's real-time latitude and longitude position, actual navigation direction, and actual navigation speed to assist in dynamic correction decisions regarding actual navigation deviations.

[0060] In an optional embodiment, the system further includes a closed-loop optimization interaction module, which is configured to calculate a comprehensive deviation value based on actual driving state data, decision instructions and optimal trajectory output by the multimodal trajectory decision module, using a ship driving comprehensive deviation evaluation algorithm. Then based on the comprehensive deviation value Adjusting decision-making instructions to achieve dynamic correction of actual driving deviations; and, The actual driving status data, decision instructions, optimal trajectory, and comprehensive deviation value are uploaded to the shore-based monitoring system, so that shore-based staff can monitor the driving status of ships in near-shore channels in low visibility or complex sea conditions at night and promptly detect any abnormalities.

[0061] It should be noted that the ship autopilot system in this embodiment uses the ship autopilot method proposed in Embodiment 1. Therefore, all optional implementation methods are also applicable to this system and will not be described in detail.

[0062] The terminology used in the accompanying drawings is for illustrative purposes only and should not be construed as limiting the invention. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A ship autopilot method based on large-model visual perception, characterized in that, Includes the following steps: Using visible light cameras, infrared thermal imagers, sonar equipment, and AIS systems, current visible light images, thermal imaging data, underwater sonar data, and AIS data are collected and preprocessed to form a standardized dataset. The standardized dataset is input into a pre-trained large visual model based on the Transformer architecture. The large visual model extracts single-source features from the standardized dataset, combines a multi-source feature enhancement algorithm to fuse the single-source features, and then performs boundary feature enhancement processing to obtain environmental perception features. The environmental perception features are input into a pre-trained diffusion model. The diffusion model generates multiple candidate trajectories based on a dynamic target relationship graph constructed from AIS data. After filtering the candidate trajectories by calling a preset navigation rule base, the optimal trajectory is output and a decision instruction is generated. The steering control unit and propulsion control unit deployed on the ship receive the decision commands and execute them to control the ship's course and speed.

2. The ship autopilot method according to claim 1, characterized in that, The underwater sonar data includes two-dimensional acoustic images of underwater obstacles acquired by side-scan sonar, and distance and contour data of underwater targets acquired by forward-looking sonar; the AIS data includes the ship's MMSI, latitude and longitude, heading, speed, and dynamic information of surrounding ships. The acquisition and preprocessing of current visible light images, thermal imaging data, underwater sonar data, and AIS data includes the following steps: Acquire current visible light images, thermal imaging data, underwater sonar data, and AIS data; The visible light image, the thermal imaging data, the underwater sonar data, and the AIS data are time-stamped and synchronized. The visible light image, thermal imaging data, underwater sonar data, and AIS data that have been synchronized with time stamps are denoised, encoded, and beamformed to form a standardized data set.

3. The ship autopilot method according to claim 1, characterized in that, The large visual model includes 24-32 encoder layers and 28-36 decoder layers; each encoder layer is configured with 8-32 self-attention heads, and the calculation range of the self-attention heads covers the entire input data. The input layer of the large visual model is configured with an interface that adapts to multimodal data. The input layer is used to receive the RGB features of the visible light image, the temperature field features of the thermal imaging data, the acoustic reflection intensity features of the underwater sonar data, and the structured vector features of the AIS data. After converting the various feature data into feature vectors of the same dimension through feature mapping, the data is input into the encoder.

4. The ship autopilot method according to claim 3, characterized in that, The large visual model is configured with a multi-source feature enhancement algorithm; Its expression is: ； in, This refers to features for global environment perception. For the first i Single-source characteristics of sensor-like devices The mean of multiple source features; Represents the self-attention weights. This is the attention enhancement coefficient; i Indicates the sensor type identifier, and , This refers to a visible light camera. I Indicates an infrared thermal imager. S Indicates sonar equipment. A Indicates the AIS system; For the first i Confidence weights for sensor-like devices The first The inherent reliability coefficient of this type of sensor, The first The sensor's environmental adaptability coefficient This is the adjustment coefficient.

5. The ship autopilot method according to claim 1, characterized in that, The dynamic target relationship map is constructed by collecting AIS data over a certain period of time; The steps include: S010. Extract the ship's MMSI, latitude and longitude, heading, speed and navigation status information from the AIS data; S020. Convert the ship's latitude and longitude coordinates to the UTM coordinate system; S030. Calculate the relative distance, relative heading, and relative speed between the current vessel and any surrounding vessel to determine the spatial positional relationship and motion trend correlation between the targets. S040. Construct a target relationship graph with ships as target nodes, and add type labels and static attributes to each target node in combination with the environmental perception type in the environmental perception features. S050. Update the AIS data and the environmental perception features at preset time intervals, and repeat steps S010 to S030 to dynamically update the UTM coordinates, heading, speed and motion trend association of any target node in the target relationship graph to form a dynamic target relationship graph.

6. The ship autopilot method according to claim 5, characterized in that, The diffusion model generates multiple candidate trajectories based on a dynamic target relationship graph constructed from AIS data, including the following steps: The AIS data currently collected by the AIS system is used as the initial state data of the ship; During the forward diffusion process, Gaussian noise is gradually added to the initial state data of the ship to generate multiple sets of noisy virtual state data; In the reverse diffusion process, a U-Net network is used, with the spatial position relationship and motion trend association between the corresponding target nodes in the dynamic target relationship map as the condition constraint, to perform noise reduction iteration on the virtual state data, and each iteration outputs a set of potential trajectory parameters; When the number of iterations reaches a preset termination condition, the diffusion model outputs a certain number of candidate trajectories; wherein, the candidate trajectories include future... t The position coordinate sequence, heading change curve, and speed adjustment sequence within minutes; t This is the preset time length.

7. The ship autopilot method according to claim 1, characterized in that, The method further includes the following steps: After executing the decision command, the ship's actual driving status data is collected by sensors deployed on the ship; the actual driving status data includes the actual heading, actual speed, and the ship's real-time latitude and longitude position; Based on the actual driving status data, the decision command, and the optimal trajectory, a comprehensive deviation value is calculated using a ship driving comprehensive deviation evaluation algorithm. Its expression is: ； in, For the actual course, To determine the course; This is the actual speed. To determine the cruising speed; Initial speed; The decision instruction is adjusted based on the comprehensive deviation value; its expression is: ； in, For heading adjustment, This is the heading adjustment factor; This is a sign function; it outputs 1 when the input is positive, -1 when the input is negative, and 0 when the input is zero. For speed adjustment amount, This is the speed adjustment factor.

8. The ship autopilot method according to claim 7, characterized in that, The steering control unit includes a servo driver and an angle sensor. The servo driver is used to receive the heading adjustment angle signal in the decision command to drive the rudder blade to rotate by the corresponding angle. The angle sensor is used to collect the actual rotation angle of the rudder blade to obtain the actual heading of the ship. The propulsion control unit includes a power regulator and a speed sensor. The power regulator is used to receive the speed adjustment signal in the decision command to adjust the output power of the ship's propulsion. The speed sensor is used to collect the actual speed of the propulsion to obtain the actual speed of the ship.

9. The ship autopilot method according to claim 7, characterized in that, The method further includes the following steps: The actual driving status data, the decision command, the optimal trajectory, and the comprehensive deviation value are uploaded to the shore-based monitoring system.

10. A ship autopilot system based on large-model visual perception, employing the ship autopilot method based on large-model visual perception as described in any one of claims 1 to 9, characterized in that, include: The multi-source sensing and acquisition module is equipped with a visible light camera, an infrared thermal imager, a sonar device, and an AIS system. It is used to collect current visible light images, thermal imaging data, underwater sonar data, and AIS data about surrounding vessels and preprocess them to form a standardized data set. The large model fusion processing module is equipped with a pre-trained visual large model based on the Transformer architecture, which is used to extract single-source features from the standardized dataset, fuse the single-source features with a multi-source feature enhancement algorithm, and output environmental perception features after boundary feature enhancement processing. The multimodal trajectory decision module is configured with a pre-trained diffusion model and a dynamic target relationship graph constructed based on AIS data. The diffusion model generates multiple candidate trajectories based on the dynamic target relationship graph constructed based on AIS data, and outputs the optimal trajectory and generates decision instructions after filtering the candidate trajectories by calling a preset navigation rule base. The navigation control execution module, which is equipped with a steering gear control unit and a propeller control unit, is used to receive the decision commands and execute them to control the ship's course and speed.

Images