Ship autonomous collision avoidance decision-making method and system based on cognitive entropy near-end policy optimization

Abstract

The application discloses a ship autonomous collision avoidance decision-making method and system based on cognitive entropy proximal policy optimization, and belongs to the field of intelligent ship control. The method comprises the following steps: acquiring ship and environment information, and constructing a state vector containing a collision risk index; inputting the state vector into a cognitive entropy proximal policy optimization decision-making model, introducing a cognitive entropy regularization term in the proximal policy optimization framework to regulate the exploration and balance of the strategy, realizing a smooth transition from high exploration to high utilization, and outputting a propelling force / rudder angle instruction; after executing the instruction, calculating a multi-component reward based on target proximity, track deviation, fused collision risk and COLREGs compliance; and iteratively optimizing network parameters and using the reward and state feedback. The application solves the problems of difficult balance between exploration and utilization, low convergence efficiency of reinforcement learning in ship collision avoidance, and improves the decision-making safety, compliance and training stability.

Description

Technical Field

[0001] This invention relates to the field of intelligent ship control, and more specifically, to a method and system for autonomous collision avoidance decision-making for ships based on cognitive entropy-based near-end strategy optimization. Background Technology

[0002] With the development of intelligent shipping technology, unmanned surface vessels (USVs) are increasingly being used in marine transportation, patrol, surveying, and other scenarios. Achieving safe, reliable, and compliant autonomous collision avoidance is one of the core challenges for USVs to become practical. Traditional collision avoidance algorithms, such as artificial potential field methods and dynamic window methods, have limited adaptability in scenarios involving dynamic encounters between multiple vessels and complex sea conditions.

[0003] In recent years, deep reinforcement learning has been widely applied to ship path planning and collision avoidance control due to its powerful ability to learn optimal policies from interactions. Proximal policy optimization (PPO), as a classic policy gradient algorithm, has become one of the mainstream choices in this field due to its good stability and ease of parameter tuning. However, directly applying PPO to complex ship collision avoidance decisions still faces many challenges: 1) The challenge of balancing exploration and exploitation: The state-action space of collision avoidance tasks is complex, and the sparse reward problem is prominent. Sufficient exploration is needed in the early stages of training to find feasible policies, but fixed or linearly decaying entropy coefficients make it difficult to achieve an adaptive and smooth transition from sufficient exploration to efficient exploitation, easily leading to early convergence to suboptimal policies or insufficient exploration in the later stages, making it unable to adapt to dynamic environments. Comparative paper CN120161842A attempts to balance global and local considerations by switching between SAC and PPO algorithms, but it does not solve the dynamic adjustment problem of exploration-exploitation within a single algorithm framework. 2) Complex reward function design: Ship collision avoidance requires simultaneous consideration of multiple objectives such as route tracking, collision risk, COLREGs rules, and motion smoothness. Designing a reasonable multi-component reward function to guide the agent in learning safe, compliant, and efficient strategies is crucial. While comparative documents CN112947431B and CN120335492A design multi-objective rewards, they do not deeply integrate DCPA / TCPA into a continuous risk indicator for precise quantification of collision threats. 3) Robustness of the decision-making model in complex multi-ship scenarios: Existing methods are effective in single or simple scenarios, but in dense, dynamic multi-ship encounter scenarios, decision-making is prone to instability or violations of navigation rules.

[0004] In summary, there is an urgent need for a ship autonomous collision avoidance decision-making method and system that can adaptively adjust the exploration intensity during training, effectively quantify dynamic collision risks, and deeply integrate navigation rule constraints. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing technologies and provide a ship autonomous collision avoidance decision-making method and system based on cognitive entropy proximal strategy optimization. By designing a cognitive entropy dynamic annealing mechanism, the PPO algorithm can adaptively balance exploration and utilization in stages during training. Combined with refined state construction and multi-objective reward functions, it improves the convergence speed, safety, rule compliance and overall robustness of ship autonomous collision avoidance decision-making in complex scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a ship autonomous collision avoidance decision-making method based on cognitive entropy proximal strategy optimization, comprising the following steps:

[0008] S1. Obtain the current ship status information and environmental perception information containing at least one target ship or obstacle;

[0009] S2. Based on its own ship state information and environmental perception information, a short-term navigation state vector is constructed, which includes its own motion state, target point information, channel constraints and fused collision risk indicators.

[0010] S3. Input the short-term navigation state vector into the cognitive entropy proximal policy optimization decision model. The decision model, based on the current training stage, uses a dynamically changing cognitive entropy weight function. Adjust the weight of the strategy entropy term in the total loss function and output continuous motion control commands, including thrust commands and / or rudder angle commands;

[0011] S4. Execute motion control commands, update the ship's motion status, and calculate multi-component real-time rewards based on the updated status. The multi-component real-time rewards include at least target rewards, track tracking rewards, collision risk penalties, and rule compliance rewards.

[0012] S5. Feed the multi-component instant rewards and updated states back to the decision model to iteratively update the parameters of the policy network and value network in the decision model, and update the cognitive entropy weight function according to the training progress. .

[0013] Furthermore, the loss function of the cognitive entropy proximal policy optimization decision model. for:

[0014]

[0015] in, The policy truncation loss term for near-end policy optimization; This is the error term for estimating the value function; The policy entropy characterizes the randomness of the action distribution; The weighting coefficient for the value error term; The cognitive entropy weight function is a function that varies with time or the number of training iterations. The dynamic coefficient of change;

[0016] Cognitive entropy weighting function Using the Sigmoid function to generate an S-shaped annealing curve, its expression is:

[0017]

[0018] in, The maximum entropy weight coefficient is set during training to limit the agent's overexploration in the early stages; This is the minimum entropy weight coefficient, used to prevent the strategy from completely losing its exploration ability in the later stages; This is the rate control coefficient, used for adjustment. The speed of development; These are transition point control parameters used to determine... The main period during which the value decreases from high to low; This refers to the growth time of the situational awareness cluster.

[0019] Furthermore, through configuration , , , Parameters, make The training process exhibits phased characteristics:

[0020] In the early stages of training When, satisfy ≈ Strategy entropy term Dominant in total loss, encouraging extensive exploration of the state-action space;

[0021] During the training period hour, Rapid decline, exploration and utilization tend to reach equilibrium;

[0022] During the later stages of training When, satisfy ≈ Strategy loss item It dominates the overall losses, and the strategy tends to be stable and exploitative.

[0023] Furthermore, in step S2, the fused collision risk index (CR) is constructed based on the nearest encounter distance (DCPA) and the nearest encounter time (TCPA), and satisfies:

[0024]

[0025] in, This indicates that you will encounter a distance in the near future. Indicates the time that will be encountered soon; and These are distance-sensitive factors and time-sensitive factors, used to adjust the model's sensitivity to... and The degree of sensitivity.

[0026] Furthermore, the collision risk penalty in step S4 is positively correlated with the merged collision risk index, and the rule compliance reward is used to incentivize or punish behaviors that comply with or violate the International Maritime Collision Avoidance Regulations (COLREGs); multi-component instant rewards The following formulas are used for comprehensive calculation:

[0027]

[0028] in, The target reward is used to characterize the ship's propulsion effect towards a target point or a predetermined route; For track tracking rewards, used to measure heading / course deviation; As a penalty for collision risk, by and Common constraints generate risks; Rewards for rule compliance; This is a penalty for smoother maneuvers, used to suppress excessive changes in propulsion and rudder angle; These are weighting coefficients that are dynamically adjusted according to the environment.

[0029] Furthermore, in step S2, the short-term navigation state vector includes at least the following dimensions: the speed and angular velocity of the vessel itself in the local coordinate system, the relative bearing and distance to the next target point, the lateral deviation from the reference channel, and the relative position, relative speed, and corresponding fused collision risk index CR for each target vessel.

[0030] Furthermore, the decision model in step S3 uses fixed time intervals. Perform forward computation and output action control commands; cognitive entropy weighting function The update cycle is greater than Update once after every N strategy iterations. The value of .

[0031] Secondly, the present invention provides a ship autonomous collision avoidance decision-making system based on cognitive entropy proximal strategy optimization, comprising:

[0032] The environmental perception module is configured to acquire the ship's current status information and environmental perception information including at least one target ship or obstacle.

[0033] The state construction module is configured to construct a short-term navigation state vector based on its own ship state information and environmental perception information, which includes its own motion state, target point information, channel constraints and fused collision risk indicators.

[0034] The cognitive entropy decision module is configured to input the short-term navigation state vector into the cognitive entropy proximal policy optimization decision model. The decision model then dynamically adjusts the decision based on the current training phase using a dynamically changing cognitive entropy weight function. Adjust the weight of the strategy entropy term in the total loss function and output continuous motion control commands, including thrust commands and / or rudder angle commands;

[0035] The execution and reward generation module is configured to execute motion control commands, update the ship's motion status, and calculate multi-component real-time rewards based on the updated status. The multi-component real-time rewards include at least target rewards, track tracking rewards, collision risk penalties, and rule compliance rewards.

[0036] The model optimization and entropy weight adjustment module is configured to feed back multi-component real-time rewards and updated states to the decision model for iterative updates of the policy network and value network parameters, while simultaneously updating the cognitive entropy weight function according to the training progress. .

[0037] In another aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the above-described ship autonomous collision avoidance decision-making methods based on cognitive entropy proximal strategy optimization.

[0038] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements a ship autonomous collision avoidance decision-making method for cognitive entropy proximal strategy optimization as described above.

[0039] The above technical solution has the following technical effects:

[0040] This invention introduces a cognitive entropy dynamic adjustment mechanism based on the S-shaped annealing curve, enabling the PPO algorithm to autonomously and smoothly transition from full exploration to efficient utilization during training. This effectively solves the problem of dynamic balance between exploration and utilization in existing reinforcement learning methods for ship collision avoidance tasks. Furthermore, it constructs a fusion of continuous collision risk indicators... The state space and reward function that integrates multi-dimensional objectives (route tracking, risk avoidance, and rule compliance) provide the agent with accurate and structured learning signals, thereby directly improving the safety, route compliance, and overall adaptability of the final decision strategy in complex multi-ship encounter scenarios. The ship autonomous collision avoidance decision-making method based on cognitive entropy proximal strategy optimization proposed in this application has good portability and engineering implementation potential. Attached Figure Description

[0041] Figure 1 A flowchart of a ship autonomous collision avoidance decision-making method based on cognitive entropy proximal strategy optimization provided in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of a ship autonomous collision avoidance decision-making system based on cognitive entropy proximal strategy optimization, provided in another embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the structure of an electronic device according to another embodiment of the present invention. Detailed Implementation

[0044] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0045] With the continuous expansion of maritime transport and the rapid development of intelligent shipping technology, unmanned surface vehicles (USVs) are widely used in port operations, waterway inspections, and marine observation. USVs possess advantages such as all-weather operation, remote control, and autonomous decision-making, and are expected to significantly reduce labor costs while improving shipping safety and operational efficiency.

[0046] However, achieving safe and reliable autonomous collision avoidance in the complex and ever-changing maritime environment remains a significant challenge. On the one hand, traditional path planning and collision avoidance algorithms (such as A) still face considerable difficulties. While algorithms such as artificial potential fields and genetic algorithms are widely used, they typically rely on pre-built environmental models and have limited adaptability to multi-ship encounters, dynamic obstacles, and random disturbances. Furthermore, they struggle to simultaneously achieve smoothness of the trajectory and computational efficiency. On the other hand, intelligent decision-making methods, represented by Deep Reinforcement Learning (DRL), can automatically learn collision avoidance strategies through interaction with the environment in a high-dimensional state space and have been widely used in path planning and autonomous ship collision avoidance research. Existing work based on algorithms such as PPO, DDPG, and DQN has achieved some success in typical scenarios, but significant challenges remain when dealing with high-dimensional, highly uncertain maritime collision avoidance tasks, including difficulties in balancing exploration and utilization, low convergence efficiency in complex environments, and a lack of interpretability and rule constraints in collision avoidance decisions.

[0047] In summary, there is an urgent need to propose a deep reinforcement learning collision avoidance algorithm that can adaptively adjust the exploration intensity according to the learning progress of the agent and combine ship motion models with collision risk assessment, so as to improve training convergence efficiency and decision robustness while ensuring compliance with international maritime collision avoidance regulations (COLREGs).

[0048] The following is combined Figure 1 The flowchart below provides a detailed explanation of the method of the present invention.

[0049] A first aspect of the present invention provides a ship autonomous collision avoidance decision-making method based on cognitive entropy proximal strategy optimization, comprising the following steps:

[0050] S1. Obtain the current ship status information and environmental perception information containing at least one target ship or obstacle;

[0051] S2. Based on its own ship state information and environmental perception information, a short-term navigation state vector is constructed, which includes its own motion state, target point information, channel constraints and fused collision risk indicators.

[0052] S3. Input the short-term navigation state vector into the cognitive entropy proximal policy optimization decision model. The decision model, based on the current training stage, uses a dynamically changing cognitive entropy weight function. Adjust the weight of the strategy entropy term in the total loss function and output continuous motion control commands, including thrust commands and / or rudder angle commands;

[0053] S4. Execute motion control commands, update the ship's motion status, and calculate multi-component real-time rewards based on the updated status. The multi-component real-time rewards include at least target rewards, track tracking rewards, collision risk penalties, and rule compliance rewards.

[0054] S5. Feed the multi-component instant rewards and updated states back to the decision model to iteratively update the parameters of the policy network and value network in the decision model, and update the cognitive entropy weight function according to the training progress. .

[0055] This embodiment covers the complete closed-loop process of autonomous collision avoidance decision-making for ships. In one implementation, the autonomous ship acquires its own attitude, speed, and target ship dynamic information through GPS, IMU, AIS, and radar, calculates a collision risk index including DCPA and TCPA, and constructs a multi-dimensional short-term navigation state vector. This state vector is input into a decision model based on the PPO framework, which internally uses a cognitive entropy weighting function that evolves over time. The intensity of strategy exploration is adjusted. The model outputs thrust and rudder angle commands to control the ship's movements. After the actions are executed, the system comprehensively calculates multi-component rewards based on target approach, track tracking deviation, collision risk changes, and compliance with COLREGs rules. These rewards, along with new state data, are then used to synchronously update the parameters and entropy weights of the decision network. The value is then used to enter the next decision-making cycle.

[0056] In the above embodiments, the vessel's own state information may include, but is not limited to: the vessel's heading angle, rate of change of heading angle, rudder angle, rate of change of rudder angle, speed, and global coordinates; while environmental perception information may include, but is not limited to: the relative bearing and coordinates of the target point, the target vessel's heading, speed, relative position, and the fused collision risk index calculated based on DCPA / TCPA. Short-term navigation state data includes the vessel's own navigation state, the target point state, the target vessel state, and the reference channel state, used to characterize the comprehensive situation of the unmanned vessel in complex multi-vehicle encounter scenarios. The short-term navigation state data is input into a predetermined cognitive entropy proximal strategy optimization decision model, also known as the cognitive entropy proximal strategy optimization (CEPPO) decision layer, or a cognitive entropy proximal strategy optimization agent. Under the action of the cognitive entropy adjustment mechanism, this model performs strategy evaluation and action sampling on the short-term navigation state data, and outputs continuous rudder angle control commands and / or equivalent control actions. By executing the continuous rudder angle control commands to update the ship's motion state, the ship's position, heading, and speed at the next moment are calculated using the ship's kinematics / dynamics model or a high-fidelity simulation environment. The distance, bearing, and collision risk indicators related to the target point, reference channel, and target ship are recalculated, and an instant reward signal is generated accordingly. The instant reward signal comprehensively considers factors such as target approach efficiency, track tracking error, collision risk level, COLREGs rule compliance, and motion smoothness.

[0057] The method described above links environmental perception, state construction, decision output, reward evaluation, and parameter updating into a self-consistent closed loop. Its key difference from simply executing a fixed policy or using statically explored parameters lies in the introduction of a dynamic entropy weight adjustment mechanism strongly coupled with the training process. This mechanism ensures that the policy learning process no longer relies on a manually preset decay plan, but rather... The functional form transforms the abstract concept of "exploration" into a precisely controllable, smoothly changing numerical variable over time. As a result, the system's behavior can autonomously and predictably transition from extensive random trials in the early stages of training to stable, optimal execution in later stages. This transformation is built into the algorithmic framework, ensuring that the learning process leads to a stable and effective strategy under different task initializations and random seeds.

[0058] In some embodiments, the loss function of the cognitive entropy proximal policy optimization decision model for:

[0059]

[0060] in, The policy truncation loss term for near-end policy optimization; This is the error term for estimating the value function; The policy entropy characterizes the randomness of the action distribution; The weighting coefficient for the value error term; The cognitive entropy weight function is a function that varies with time or the number of training iterations. The dynamic coefficient of change;

[0061] Cognitive entropy weighting function Using the Sigmoid function to generate an S-shaped annealing curve, its expression is:

[0062]

[0063] in, The maximum entropy weight coefficient is set during training to limit the agent's overexploration in the early stages; This is the minimum entropy weight coefficient, used to prevent the strategy from completely losing its exploration ability in the later stages; This is the rate control coefficient, used for adjustment. The speed of development; These are transition point control parameters used to determine... The main period during which the value decreases from high to low; This refers to the growth time of the situational awareness cluster.

[0064] This embodiment specifically defines the mathematical forms of the loss function and entropy weight function of the decision model. In one implementation, the optimization objective of the policy network is... It consists of three parts: standard PPO shearing items Error term of value function and by Weighted policy entropy .in, Set as the Sigmoid function:

[0065]

[0066] For example, setting =0.05, =0.001, =0.003, =250000, and over a total of one million steps of training, the entropy weights will exhibit a smooth "S"-shaped decay.

[0067] This embodiment incorporates the entropy term into the loss function as dynamic weights, essentially introducing a time-varying regularization term into the policy optimization process. In the early stages of training ( much smaller ), near Higher entropy rewards encourage greater randomness in action distribution, thus preventing premature entry into local optimum action patterns (such as always going straight or always turning) in the early stages of collision avoidance tasks with sparse rewards. As training progresses... nearby, Entering the rapid change zone, the contribution of entropy reward decreases rapidly, and the focus of the optimization process gradually shifts from encouraging randomness to directly optimizing policy performance based on reward signals. Towards the later stages of training (when t is much greater than...),... ), Stable to A very small entropy term is preserved to prevent the policy from degenerating completely into a deterministic function and losing the fine-tuning ability needed to cope with rare states. This Sigmoid curve-based adjustment method, compared to linear decay, provides a more explicit inflection point in the transition phase, making the shift from discovery to exploitation clearer and more controllable.

[0068] In some embodiments, by configuration , , , Parameters, make The training process exhibits phased characteristics:

[0069] In the early stages of training When, satisfy ≈ Strategy entropy term Dominant in total loss, encouraging extensive exploration of the state-action space;

[0070] During the training period hour, Rapid decline, exploration and utilization tend to reach equilibrium;

[0071] During the later stages of training When, satisfy ≈ Strategy loss item It dominates the overall losses, and the strategy tends to be stable and exploitative.

[0072] In the above embodiments, the cognitive entropy weighting function In the early stages of training, the following conditions must be met:

[0073]

[0074] Early training :at this time Strategy entropy term When total loss dominates, agents are encouraged to explore the state-action space extensively to avoid getting trapped in local optima.

[0075] Entropy weight function during training satisfy:

[0076]

[0077] Mid-training Entropy coefficient is the cognitive entropy weighting function. The decline is rapid, and exploration and exploitation gradually reach equilibrium. At this point, the policy optimization term in the total loss function ( ) and value function estimation term ( The increased weight of ) guides the agent to favor high-yield actions, accelerating convergence.

[0078] Entropy weight function in the later stage of training satisfy:

[0079]

[0080] Late training :at this time As the entropy coefficient stabilizes, the weight of the policy entropy term is reduced to a minimum, and the loss function is dominated by policy optimization, ensuring policy stability and utilization.

[0081] This embodiment further elaborates on the entropy weighting function. The phased training characteristics achieved through parameter configuration. For example, by setting a larger... (e.g., 0.1) and smaller (e.g., 0.001) can prolong the high exploration phase; conversely, setting a smaller value will prolong it. and larger This allows the exploration phase to end quickly, accelerating convergence. In the simulation experiment, a set of parameters is configured as follows ( =0.08, =0.002, =0.005, =200 The other group is ( ), =0.03, =0.0005, =0.01, =100 Both are adapted to maritime traffic scenarios of varying complexity.

[0082] In the above embodiments, , , and These four parameters together constitute a four-dimensional control space describing the "exploration-exploitation" balance strategy. and The upper and lower boundaries of the exploration intensity are defined to ensure that the scope of exploration behavior is always constrained within a reasonable physical and task range, avoiding ineffective extreme random actions. The parameter k controls the "rapidity" of the exploration intensity decay; a larger k value... A higher k value implies a faster transition from exploration-driven to exploit-driven strategies, suitable for scenarios with relatively simple state spaces and where optimal strategies are easier to discover; a smaller k value corresponds to a slower transition, allowing agents to conduct more thorough searches in complex, multi-peak reward landscapes. This anchors the central moment of this transition, and its relationship with the total training time determines the proportion of each stage. This parameterized design allows the method to systematically adapt to collision avoidance learning tasks of different difficulties and risk preferences by adjusting a few intuitive parameters, rather than relying on targeted adjustments to the algorithm structure.

[0083] In some embodiments, the above-mentioned motion space adopts a continuous thrust-torque control form, and the control quantity is expressed as:

[0084]

[0085] in, , These correspond to the longitudinal thrust generated by the main propeller and the torque controlling the ship's rotation about its vertical axis (yaw axis), respectively. This motion space design can directly drive the physical engine to simulate the ship's real response, which not only simplifies the motion execution model but also improves the agent's control accuracy and environmental adaptability in complex collision avoidance tasks.

[0086] In some embodiments, in step S2, the fused collision risk index (CR) is constructed based on the nearest encounter distance (DCPA) and the nearest encounter time (TCPA), and satisfies:

[0087]

[0088] in, This indicates that you will encounter a distance in the near future. Indicates that the time will be met soon; and These are distance-sensitive factors and time-sensitive factors, used to adjust the model's sensitivity to... and The degree of sensitivity.

[0089] This embodiment details the calculation method for the fusion collision risk index CR. (The above...) In the calculation formula, the smaller The value makes the model more sensitive to changes in proximity distance, while a smaller value... The value indicates that the strategy responds more aggressively to time urgency. For example, The value is taken as 1.5 times the safe meeting distance (e.g., 0.5 nautical miles). The value is set to twice the time required for emergency avoidance maneuvers (e.g., 300 seconds). When the target ship... For 0.1 nautical miles, When the time is 100 seconds, the calculated CR value will be significantly higher than For 0.5 nautical miles, This was done over a 500-second period, thus quantifying different threat levels.

[0090] In the above embodiments, The formula uses the difference between two negative exponent terms to... and These two scalars, with different physical meanings, are merged into a single, continuous risk measure. (Exponential function) and All inputs are mapped to the interval (0,1], and the values ​​monotonically decrease as the input increases. Difference form This makes it possible when the target ship simultaneously has a very small (The first term is close to 1) and very small When the second term is close to 1, the CR value approaches 0, indicating extremely high risk; when Very large (the first term is close to 0) and When it is also very large (the second term is close to 0), The value also approaches 0, indicating extremely low risk. This non-linear fusion method is more sensitive to the essence of collision risk—"close distance and tight time"—than linear weighting or threshold judgment, providing a continuous and nuanced gradient signal for subsequent rewards and penalties, and avoiding the sudden decision-making changes caused by binarized risk judgment.

[0091] In some embodiments, the collision risk penalty in step S4 is positively correlated with the fused collision risk index, and the rule compliance reward is used to incentivize or penalize behaviors that comply with or violate the International Maritime Collision Avoidance Regulations (COLREGs); multi-component instant rewards The following formulas are used for comprehensive calculation:

[0092]

[0093] in, The target reward is used to characterize the ship's propulsion effect towards a target point or a predetermined route; For track tracking rewards, used to measure heading / course deviation; As a penalty for collision risk, by and Common constraints generate risks; Rewards for rule compliance; This is a penalty for smoother maneuvers, used to suppress excessive changes in propulsion and rudder angle; These are weighting coefficients that are dynamically adjusted according to the environment.

[0094] This embodiment defines the specific structure and calculation of a multi-component instantaneous reward function. For example, in a simulation training session, the weights are set as follows: =1.2 (target reward) =0.8 (tracking reward) =2.5 (risk penalty) =1.5 (rule reward) =0.1 (Action Penalty). When the ship successfully approaches the target in a single decision ( ), maintain track ( However, it slightly violates the rules of intersection and encounter ( When this happens, the total reward will reflect this trade-off.

[0095] In the above embodiments, the structure of the reward function incorporates multiple objectives of collision avoidance decision-making—navigation efficiency (…). ), route maintenance ( ), security ), compliance ( and handling stability — Decoupled into independent addable terms. Each term is linked by its corresponding weight. These weights, when scaled, essentially define the agent's priorities when faced with conflicting objectives. High Drive the intelligent agent to The indicated risks elicit a highly sensitive response; This directly encodes COLREGs terms as learnable value signals. This explicit, structured reward design, unlike using a single reward signal synthesized through complex rules, allows the policy network to more clearly distinguish the impact of different behaviors on different objectives during the learning process, thus learning a balanced, multi-objective integrated behavioral pattern. The independence of each component also facilitates fine-tuning in the later stages of training; for example, after basic safety is guaranteed, by increasing... To further improve navigation efficiency.

[0096] In the above embodiments, the multi-component instantaneous reward includes at least: target reward, navigation reward, collision avoidance reward, and / or rule constraint reward, wherein the target reward is used to encourage the unmanned surface vessel to effectively advance towards the target point; the navigation reward is used to constrain the unmanned surface vessel to navigate smoothly along the reference course; and the collision avoidance reward and collision risk penalty are used to reflect... / and The changing trend; rule-based constraints and rewards are used to constrain unmanned vessels to comply with COLREGs-related clauses in typical encounter scenarios such as direct encounters, intersections, and overtaking.

[0097] In some embodiments, in step S2, the short-term navigation state vector includes at least the following dimensions: the vessel's own velocity and angular velocity in the local coordinate system, its relative bearing and distance to the next target point, its lateral deviation from the reference channel, and its relative position, relative speed, and corresponding fused collision risk index for each target vessel. .

[0098] This embodiment details the specific dimensional composition of the short-term navigation state vector. In an implementation of a two-ship encounter scenario, the state vector includes: longitudinal velocity, lateral velocity, heading angle, rate of change of heading angle, rudder angle, and rate of change of rudder angle in the ship's own coordinate system; sine and cosine values ​​of the target point's relative azimuth angle α and its distance; lateral deviation from the planned route; and for a single target ship, its relative position, relative velocity, and corresponding fusion collision risk index. value.

[0099] In the above embodiments, the setting of the short-term navigation state vector follows the principle of "completeness and necessity," while considering both the efficiency of representation and the directness of decision-making. Using velocity in the ship's coordinate system instead of the geodetic coordinate system allows the control commands (thrust, torque) output by the policy network to be physically directly correlated with the state input, simplifying the learning mapping. Introducing the sine and cosine values ​​of the target point's azimuth avoids the problem of abrupt changes in angle values ​​at the boundary between 0 degrees and 360 degrees, providing continuous direction encoding. The dynamic information (position, velocity) of each target ship and its derived advanced risk indicators are integrated into the system. This integration provides the policy network with features across multiple levels, from raw data to semantic information. This combination allows the network to perform reactive collision avoidance directly based on geometric relationships, or to conduct more forward-looking planning based on risk indicators. The state vectors do not contain high-dimensional redundant data such as raw images or point clouds, ensuring the computational efficiency of the decision model and making it suitable for the real-time requirements of ship-embedded systems.

[0100] In some embodiments, the decision model in step S3 is set at fixed time intervals. Perform forward computation and output action control commands; cognitive entropy weighting function The update cycle is greater than Update once after every N strategy iterations. The value of .

[0101] This embodiment specifies the time scale for decision-making and entropy weight updates. For example, one setting is: decision control cycle. The interval is 1 second, meaning that a control command is calculated and output once per second based on the latest state. The cognitive entropy weight... The update cycle is once every 2048 time steps (approximately 34 minutes of training data). This means that in 2048 consecutive decisions, The strategy remains unchanged, allowing for learning and experience gathering under a relatively stable exploration intensity.

[0102] In the above embodiments, this dual-timescale design decouples the "policy execution" and "meta-parameter adjustment" processes, resulting in a shorter decision-making cycle. (Second-level) ensures that the control system can respond promptly to external dynamic environments (such as the maneuvering of other vessels), meeting the requirements for real-time collision avoidance. The longer entropy weight update cycle (thousand-step level) guarantees... The stability of change. The value of the entropy weight is adjusted based on the overall learning progress reflected by empirical data from a past batch (e.g., 2048 steps), rather than an immediate reaction to the result of a single step. This avoids oscillations in the entropy weight due to random fluctuations in the reward for a single step, ensuring the stability of the training process. Placing the entropy weight update and the policy network parameter update within the same batch loop also simplifies the implementation of the training process. This design allows the system to maintain a slow, robust, and long-term convergent learning pace while responding quickly.

[0103] refer to Figure 2 To facilitate understanding of the second aspect of the invention, a ship autonomous collision avoidance decision-making system based on cognitive entropy proximal strategy optimization is provided, comprising:

[0104] The environmental perception module is configured to acquire the ship's current status information and environmental perception information including at least one target ship or obstacle.

[0105] The state construction module is configured to construct a short-term navigation state vector based on its own ship state information and environmental perception information, which includes its own motion state, target point information, channel constraints and fused collision risk indicators.

[0106] The cognitive entropy decision module is configured to input the short-term navigation state vector into the cognitive entropy proximal policy optimization decision model. The decision model then dynamically adjusts the decision based on the current training phase using a dynamically changing cognitive entropy weight function. Adjust the weight of the strategy entropy term in the total loss function and output continuous motion control commands, including thrust commands and / or rudder angle commands;

[0107] The execution and reward generation module is configured to execute motion control commands, update the ship's motion status, and calculate multi-component real-time rewards based on the updated status. The multi-component real-time rewards include at least target rewards, track tracking rewards, collision risk penalties, and rule compliance rewards.

[0108] The model optimization and entropy weight adjustment module is configured to feed back multi-component real-time rewards and updated states to the decision model for iterative updates of the policy network and value network parameters, while simultaneously updating the cognitive entropy weight function according to the training progress. .

[0109] This embodiment provides a hardware system for implementing a ship autonomous collision avoidance decision-making method based on cognitive entropy proximal strategy optimization. The system includes: an environment perception module composed of multi-source sensors (including but not limited to GPS, AIS, and radar) and a data fusion unit; a state construction module with an operational state construction algorithm; a computational unit deploying the CEPPO decision model as a cognitive entropy decision-making module; an execution and reward generation module connected to the ship's actuators (steering gear, propellers) and integrating a reward calculator; and a module responsible for backpropagation calculation and... The model optimization and entropy weight adjustment modules are based on value iteration. These modules communicate via the vehicle bus or internal data bus.

[0110] In the above embodiments, each module carries a specific functional stage in the method flow, and the data flow between modules strictly corresponds to the information transfer between method steps. The separation of the perception module and the state construction module conforms to the hierarchical processing conventions of modern ship perception systems. Treating the cognitive entropy decision module as an independent computing unit is beneficial for accelerating computation using GPUs or dedicated AI chips. The integration of the execution and reward generation modules realizes a closed loop from digital instructions to physical actions and then to digital evaluation. The model optimization module can run in the background or in the cloud for non-real-time deep training. This modular design not only clearly divides the system functions and reduces coupling, but also facilitates system upgrades and maintenance—for example, the decision model algorithm or perception fusion strategy can be upgraded separately without affecting other modules. It provides a stable, reliable, and scalable physical carrier for the corresponding algorithms.

[0111] In some embodiments, such as Figure 3 This application also provides an electronic device, including a processor 301, a memory 302 and a bus 303, and a computer program stored in the memory. When the processor executes the program, it implements any of the above-described ship autonomous collision avoidance decision-making methods based on cognitive entropy proximal strategy optimization.

[0112] Furthermore, as an executable solution, the ship autonomous collision avoidance decision-making system based on cognitive entropy proximal strategy optimization can be a computer unit, which can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described computer unit structure is merely an example and does not constitute a limitation on the computer unit; it may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0113] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0114] In some embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the ship autonomous collision avoidance decision-making method based on cognitive entropy proximal strategy optimization described above in the embodiments of the present invention.

[0115] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0116] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A ship autonomous collision avoidance decision-making method based on cognitive entropy proximal strategy optimization, characterized in that, Includes the following steps: S1. Obtain the current ship status information and environmental perception information containing at least one target ship or obstacle; S2. Based on the ship's own state information and the environmental perception information, construct a short-term navigation state vector that includes its own motion state, target point information, channel constraints and fused collision risk indicators. S3. Input the short-term navigation state vector into the cognitive entropy proximal policy optimization decision model. The decision model, based on the current training stage, uses a dynamically changing cognitive entropy weight function. Adjust the weight of the strategy entropy term in the total loss function and output continuous motion control commands, including thrust commands and / or rudder angle commands; S4. Execute the motion control command, update the ship's motion state, and calculate a multi-component real-time reward based on the updated state. The multi-component real-time reward includes at least the target reward, track tracking reward, collision risk penalty, and rule compliance reward. S5. Feed the multi-component instant reward and the updated state back to the decision model for iteratively updating the parameters of the policy network and value network in the decision model, and simultaneously update the cognitive entropy weight function according to the training progress. ; in, The cognitive entropy weight function is defined as a function that varies with time or the number of training iterations. The dynamic coefficient of change; The cognitive entropy weighting function Using the Sigmoid function form, its expression is: In the formula, The maximum entropy weight coefficient is set during training to limit the agent's overexploration in the early stages; This is the minimum entropy weight coefficient, used to prevent the strategy from completely losing its exploration ability in the later stages; This is the rate control coefficient, used for adjustment. The speed of development; These are transition point control parameters used to determine... The main period during which the value decreases from high to low; The growth time of the situational awareness cluster; In step S2, the fused collision risk index CR is constructed based on the nearest encounter distance (DCPA) and the nearest encounter time (TCPA), and satisfies: In the formula, This indicates that you will encounter a distance in the near future. Indicates the time that will be encountered soon; and These are distance-sensitive factors and time-sensitive factors, used to adjust the model's sensitivity to... and The degree of sensitivity.

2. The method according to claim 1, characterized in that, The loss function of the cognitive entropy proximal strategy optimization decision model for: in, Policy truncation loss term optimized for near-end policies; This is the error term for estimating the value function; The policy entropy characterizes the randomness of the action distribution; The weighting coefficient for the value error term; Let be the cognitive entropy weighting function.

3. The method according to claim 1 or 2, characterized in that, By configuration , , , Parameters, make The training process exhibits phased characteristics: In the early stages of training When, satisfy ≈ Strategy entropy term Dominant in total loss, encouraging extensive exploration of the state-action space; During the training period hour, Rapid decline, exploration and utilization tend to reach equilibrium; During the later stages of training When, satisfy ≈ Strategy loss item It dominates the overall losses, and the strategy tends to be stable and exploitative.

4. The method according to claim 1, characterized in that, The collision risk penalty in step S4 is positively correlated with the merged collision risk index, and the rule compliance reward is used to incentivize or penalize behaviors that comply with or violate the International Maritime Collision Avoidance Code. The multi-component instant reward The following formulas are used for comprehensive calculation: in, The target reward is used to characterize the ship's propulsion effect towards a target point or a predetermined route; For track tracking rewards, used to measure heading / course deviation; As a penalty for collision risk, by and Common constraints generate risks; Rewards for rule compliance; This is a penalty for smoother maneuvers, used to suppress excessive changes in propulsion and rudder angle; These are weighting coefficients that are dynamically adjusted according to the environment.

5. The method according to claim 1, characterized in that, In step S2, the short-term navigation state vector includes at least the following dimensions: the speed and angular velocity of the vessel in the local coordinate system, the relative bearing and distance to the next target point, the lateral deviation from the reference channel, and the relative position, relative speed, and corresponding fused collision risk index CR for each target vessel.

6. The method according to claim 1, characterized in that, The decision model in step S3 is at a fixed time interval. Perform forward computation and output action control commands; the cognitive entropy weighting function The update cycle is greater than Update once after every N strategy iterations. The value of .

7. A ship autonomous collision avoidance decision-making system based on cognitive entropy proximal strategy optimization, characterized in that, include: The environmental perception module is configured to acquire the ship's current status information and environmental perception information including at least one target ship or obstacle. The state construction module is configured to construct a short-term navigation state vector based on the ship's own state information and the environmental perception information. This vector includes the ship's own motion state, target point information, channel constraints, and a fused collision risk index. The fused collision risk index CR is constructed based on the nearest encounter distance (DCPA) and the nearest encounter time (TCPA), and satisfies the following: In the formula, This indicates that you will encounter a distance in the near future. Indicates the time that will be encountered soon; and These are distance-sensitive factors and time-sensitive factors, used to adjust the model's sensitivity to... and The degree of sensitivity; The cognitive entropy decision module is configured to input the short-term navigation state vector into the cognitive entropy proximal policy optimization decision model, which, based on the current training phase, uses a dynamically changing cognitive entropy weight function. Adjust the weight of the strategy entropy term in the total loss function and output continuous motion control commands, including thrust commands and / or rudder angle commands; The execution and reward generation module is configured to execute the action control command, update the ship's motion state, and calculate a multi-component real-time reward based on the updated state. The multi-component real-time reward includes at least a target reward, a track tracking reward, a collision risk penalty, and a rule compliance reward. The model optimization and entropy weight adjustment module is configured to feed back the multi-component instantaneous reward and the updated state to the decision model, for iteratively updating the parameters of the policy network and value network in the decision model, and simultaneously updating the cognitive entropy weight function according to the training progress. ; in, The cognitive entropy weight function is defined as a function that varies with time or the number of training iterations. The dynamic coefficient of change; The cognitive entropy weighting function Using the Sigmoid function form, its expression is: In the formula, The maximum entropy weight coefficient is set during training to limit the agent's overexploration in the early stages; This is the minimum entropy weight coefficient, used to prevent the strategy from completely losing its exploration ability in the later stages; This is the rate control coefficient, used for adjustment. The speed of development; These are transition point control parameters used to determine... The main period during which the value decreases from high to low; This refers to the growth time of the situational awareness cluster.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.

Images