Unmanned ship dynamic collision avoidance decision-making method and system based on diffusion model and adversarial reinforcement learning

Abstract

The invention provides an unmanned ship dynamic collision avoidance decision-making method and system based on a diffusion model and adversarial reinforcement learning. Comprising the steps of 1, collecting expert collision avoidance data; 2, training a diffusion model; step 3, performing data enhancement; 4, building a training environment; step 5, constructing a generator and a discriminator; step 6, updating the discriminator; step 7, updating the generator; and step 8, completing training. The diffusion model is trained through limited expert sample data, the trained diffusion model is used to generate additional collision avoidance data, and the additional data and the original expert data jointly participate in the training process of adversarial inverse reinforcement learning, so that the problem that the expert collision avoidance data is limited can be effectively solved. And an engineering application plan is provided for the proposed algorithm, the algorithm proposed by the invention is verified, and a systematic scheme is formed by combining the theory and the reality.

Description

Technical Field

[0001] The invention relates to the technical field of unmanned boat collision avoidance, in particular to an unmanned boat dynamic collision avoidance decision method and system based on a diffusion model and adversarial reinforcement learning. Background Art

[0002] Unmanned boats play an increasingly important role in the civil and military fields and can be used for tasks such as water quality monitoring, combating piracy, and maritime search and rescue. In order to ensure the success of these tasks, unmanned boats must have the ability to autonomously avoid dynamic obstacles. During navigation, dynamically sailing ships are the most common type of obstacles encountered by unmanned boats. However, with the advancement of globalization, the number of ships at sea has increased significantly, and the navigation conditions faced by unmanned boats have become increasingly complex, which poses a severe challenge to their collision avoidance decision-making capabilities.

[0003] In the research on dynamic collision avoidance of unmanned boats, the main algorithms can be divided into three categories: 1) Rule-based collision avoidance algorithms: They rely on predefined navigation rules such as the International Regulations for Preventing Collisions at Sea (COLREG). Such algorithms provide collision avoidance decisions based on applicable rules, but it may be difficult to find appropriate response strategies in scenarios not covered. 2) Optimization-based collision avoidance algorithms: By establishing a decision model, the collision avoidance problem is transformed into an optimization problem, and the optimal path is solved using relevant algorithms (such as model predictive control, artificial potential field, and genetic algorithm). Although these methods can handle multi-objective and constraint problems, they have high computational complexity, poor real-time performance, and are difficult to apply in dynamic environments. 3) Learning-based collision avoidance algorithms: Neural network strategies are trained through a large amount of historical data to make collision avoidance decisions. These data can come from real scenes or simulated environments. This type of method includes reinforcement learning and deep reinforcement learning, which can show strong adaptability in complex environments and handle nonlinear, highly constrained, and dynamic collision avoidance scenarios.

[0004] At present, learning-based collision avoidance algorithms are considered to be one of the most promising solutions because they can not only make optimal decisions quickly in dynamic environments, but also fully consider constraints such as ship maneuvering characteristics. This type of algorithm trains the agent to learn the optimal strategy by maximizing the cumulative reward signal generated by the interaction between the unmanned boat and the environment. However, the performance of the algorithm depends to a large extent on the design of the reward function. For dynamic collision avoidance tasks, the design of a reasonable reward function is extremely challenging. In contrast, expert collision avoidance data is simpler to obtain, but the amount is often limited and cannot cover all possible collision avoidance scenarios.

[0005] Existing technologies do not fully consider the scale constraints of expert collision avoidance data. When expert data is insufficient, it is difficult to effectively learn high-quality collision avoidance decisions. Inverse reinforcement learning requires the derivation of a reward function before reinforcement learning. This process not only requires a lot of computational cost, but also increases the computational time. At the same time, the derived reward function is easily affected by insufficient data, thereby reducing the effectiveness of the strategy. Adversarial imitation learning directly learns strategies without paying attention to the reward function, and cannot explicitly derive reward signals, resulting in insufficient behavior explanation ability and unstable training process. Adversarial inverse reinforcement learning combines the advantages of inverse reinforcement learning and adversarial imitation learning. It can not only effectively learn strategies, but also derive reward functions that explain behaviors, thereby improving the generalization ability of strategies. However, adversarial inverse reinforcement learning relies on sufficient expert data. In addition, there are few existing engineering applications of technologies. At present, most reinforcement learning algorithms in the field of ships are mostly in the simulation experiment stage. Due to factors such as the computing power of unmanned boat edge computing equipment, no major breakthroughs have been made. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a dynamic collision avoidance decision method and system for an unmanned boat based on a diffusion model and adversarial reinforcement learning. By introducing a diffusion model to enhance limited expert data, the existing algorithm is effectively alleviated from being highly dependent on reward functions and expert data.

[0007] To achieve the above object, the present invention adopts the following technical solution: a dynamic collision avoidance decision method for unmanned boats based on diffusion model and adversarial reinforcement learning, comprising the following steps:

[0008] Step 1, collect expert collision avoidance data of the SoftActor-Critic based collision avoidance model in the simulation environment;

[0009] Step 2, training the diffusion model by adding noise to the input data and updating the neural network weights using the back-propagation algorithm to minimize the difference between the noise predicted by the neural network and the actual added noise;

[0010] Step 3: Perform data enhancement. During the data enhancement process, the diffusion model starts with a set of random noises and gradually removes the noises by using the neural network trained in step 2 to generate samples similar to the distribution of the real data. Through the denoising operation of multiple time steps, the diffusion model can generate additional samples similar to the original data distribution from completely random noises.

[0011] Step 4: Use the ship motion model to build a training environment;

[0012] Step 5: Build the generator and discriminator;

[0013] Step 6, update the discriminator;

[0014] Step 7, update the generator;

[0015] Step 8: Complete the training.

[0016] In a preferred embodiment, the expert collision avoidance data p in step 1 is E It consists of several lines (s, a, s′, d), where s represents the current state of the environment, a is the collision avoidance action taken by the expert collision avoidance model in the current state, s′ is the state of the environment at the next moment after executing action a, and d indicates whether the encounter scene is over.

[0017] In a preferred embodiment, the diffusion model in step 2 is composed of a three-layer fully connected neural network, and the loss function for measuring the difference is as follows:

[0018]

[0019] Where N is the neural network parameter, is the noise predicted by the neural network, ε is the actual noise added, x t For actual data, is the expectation of the noise distribution.

[0020] In a preferred embodiment, the training of the diffusion model in step 2 specifically includes the following steps:

[0021] Step 2.1: For the expert collision avoidance data p E Sampling is performed to obtain sampling data x0;

[0022] Step 2.2, select a time step t, t is randomly sampled from a uniform distribution; add noise to the sampled data x0 to generate noisy data x t ; The noise size is determined by the noise schedule β t Control; β t is linearly uniformly distributed, that is, β t The calculation of is shown in the following formula (2):

[0023]

[0024] where β start is the initial noise intensity, β end represents the final noise intensity, T represents the total number of time steps;

[0025] Step 2.3, use the neural network with parameter N to predict the noise ε N (x t ,t);

[0026] Step 2.4, calculate the loss function L N ;

[0027] Step 2.5, through the back propagation algorithm, according to L N Update the model's parameters N to minimize the difference between the predicted noise and the true noise.

[0028] In a preferred embodiment, the specific process of data enhancement in step 3 is as follows:

[0029] Step 3.1, randomly sample the initial noise sample x from the standard Gaussian distribution T ; This sample represents the initial completely noisy data;

[0030] Step 3.2, select a time step t = T, and gradually perform the denoising process starting from T;

[0031] Step 3.3, for each time step t, use the diffusion model trained in step 2 to predict the current noise

[0032] Step 3.4, according to the noise ε N (x t ,t), calculate the denoised data x t-1 ;

[0033] Step 3.5: Repeat steps 3.3 and 3.4 to obtain the generated new sample x0.

[0034] In a preferred embodiment, the ship motion model used in step 4 is a simplified dynamic model, and the model structure is as follows:

[0035]

[0036] Among them, (x, y, ψ) represents the position and heading of the unmanned boat in the earth-fixed coordinate system; (u, v, r) represents the longitudinal velocity, transverse velocity and yaw angular velocity of the unmanned boat in the hull-fixed coordinate system; (m1, m2) is the additional mass coefficient, (x u ,y v ) is the linear damping coefficient, J is the moment of inertia, and (T, τ) represent the changes of force and torque acting on the unmanned boat, respectively.

[0037] In a preferred embodiment, in step 5, the discriminator is composed of a value estimation network and a reward estimation network, each network contains three hidden layers, and each layer has 256 neurons; the generator is composed of an action network, a Q1 network, a Q2 network, a target Q1 network and a target Q2 network, each of which is composed of three hidden layers, and each layer has 256 neurons.

[0038] In a preferred embodiment, the objective function of the discriminator in step 6 is designed to minimize the expert collision avoidance data p E And the generator generates data pG The cross entropy loss between is shown in formula (4):

[0039]

[0040] Where (s, a) represents the state and action respectively. Represents the distribution of expert data p E The expected calculation is performed on the (s, a) sampled in the above example. D(s, a) is the output of the discriminator with (s, a) as input. The reward signal R(s, a) output by the discriminator is calculated based on D(s, a). R(s, a) is used to update the generator. The calculation model of this reward signal is defined as:

[0041] R(s,a)=log(1-D(s,a))-log(D(s,a)) (5)

[0042] The specific process of updating the discriminator is divided into the following steps:

[0043] Step 6.1: For the expert collision avoidance data p E And the generator generates data p G Conduct random sampling;

[0044] Step 6.2, calculate the output D(s,a) of the discriminator;

[0045] Step 6.3, calculate the loss function L(D);

[0046] Step 6.4, update the parameters of the discriminator according to the loss function through the back-propagation algorithm.

[0047] In a preferred embodiment, the step 7 specifically includes the following steps:

[0048] Step 7.1, the generator generates data p using the current strategy π generator G ;

[0049] Step 7.2, generate data p G Take samples;

[0050] Step 7.3, input the sampled data in step 7.2 into R(s,a) calculated in the discriminator, and use it as the reward signal in reinforcement learning to update the generator;

[0051] Step 7.4, update the generator strategy π according to the SAC algorithm;

[0052] This is achieved by maximizing the cumulative reward J(π) given by the discriminator. The generator is updated using SAC. The calculation of J(π) is as follows:

[0053]

[0054] in is the strategy entropy, α is the temperature coefficient, R n (s t ,a t ) is the reward signal output by the discriminator.

[0055] In a preferred embodiment, the control process steps of the unmanned boat collision avoidance decision system are as follows:

[0056] Step 1), establish communication between the PC and the unmanned boat main control system;

[0057] Step 2), the unmanned boat main control system receives the sensor data and pre-processes it;

[0058] Step 3), pre-processing data is sent to the PC;

[0059] Step 4), the PC processes the data and makes decisions based on the pre-trained model;

[0060] Step 5), the decision result is sent from the PC to the main control system;

[0061] Step 6), the main control system distributes the thrust and sends it to the bottom control unit;

[0062] Step 7), implement collision avoidance decision.

[0063] Compared with the prior art, the present invention has the following beneficial effects: the collision avoidance algorithm provided by the present invention is based on the adversarial inverse reinforcement learning algorithm to form an unmanned boat collision avoidance strategy combined with a diffusion model. The algorithm uses the reward signal given by the discriminator to solve the problem of difficult design of the reward function. At the same time, the diffusion model is used for data increase to ensure that an effective collision avoidance strategy is learned under the condition of limited expert collision avoidance samples, thereby improving the algorithm training effect. The present invention forms a complete system solution from algorithm to factory application, which not only includes algorithm research, but also fully explains the actual verification of the algorithm. This solution establishes communication between the PC and the boat main control system through the TCP / IP protocol, and the decision is executed by the PC, which solves the problem of insufficient computing power of the boat main control system. The collision avoidance decision is calculated by receiving the boat-borne sensor data, and the thrust distribution is realized by the main control system, achieving a breakthrough from simulation to reality, making the verification of the collision avoidance algorithm more reliable and promoting ship collision avoidance research. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a structural diagram of the unmanned boat collision avoidance decision model of a preferred embodiment of the present invention.

[0065] Figure 2 This is a flow chart of diffusion model training according to a preferred embodiment of the present invention.

[0066] Figure 3 The following is a flowchart of the unmanned boat collision avoidance decision-making according to a preferred embodiment of the present invention.

[0067] Figure 4 Schematic diagram of the relative position and relative heading between two ships in a preferred embodiment of the present invention

[0068] Figure 5 This is a flowchart of unmanned ship collision avoidance in the case of multiple ships encountering each other in a preferred embodiment of the present invention DETAILED DESCRIPTION

[0069] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0070] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0071] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0072] A dynamic collision avoidance decision method for unmanned boats based on diffusion model and adversarial reinforcement learning, reference Figure 1-5 , the structure of the unmanned boat dynamic collision avoidance decision algorithm is referenced Figure 1 , the specific steps are as follows:

[0073] Step 1, collect expert collision avoidance data;

[0074] Step 2, training the diffusion model;

[0075] Step 3: perform data enhancement;

[0076] Step 4: Build a training environment;

[0077] Step 5: Build the generator and discriminator;

[0078] Step 6, update the discriminator;

[0079] Step 7, update the generator;

[0080] Step 8: Complete the training.

[0081] For step 1, considering that there are few actual applications of unmanned boats, it is very difficult to collect real marine collision avoidance data for model training. Therefore, the expert collision avoidance data mainly used in the algorithm training process is the data from the collision avoidance model based on SoftActor-Critic (SAC) in the simulation environment. E It consists of several lines (s,a,s′,d).

[0082] For step 2, the diffusion model consists of three layers of fully connected neural networks, with 256 neurons in each layer. During the training process, the model adds noise to the input data and updates the neural network weights using the back-propagation algorithm to minimize the difference between the noise predicted by the neural network and the actual added noise. The loss function that measures this difference is as follows:

[0083]

[0084] Where N is the neural network parameter, is the noise predicted by the neural network, ε is the actual noise added, x t For actual data, is the expectation of the noise distribution.

[0085] The diffusion model training process is as follows Figure 2 As shown, the specific steps are as follows:

[0086] Step 2.1: For the expert collision avoidance data p E Sampling is performed to obtain sampling data x0.

[0087] Step 2.2, select a time step t, t is randomly sampled from a uniform distribution. Add noise to x0 to generate noisy data x t The noise level is determined by the noise schedule β t Control. t is linearly uniformly distributed, that is, β t The calculation of is shown in the following formula (2):

[0088]

[0089] where β start is the initial noise intensity, β end represents the final noise intensity, and T represents the total number of time steps.

[0090] Step 2.3, use the neural network with parameter N to predict the noise ∈ N (x t ,t).

[0091] Step 2.4, calculate the loss function L N .

[0092] Step 2.5, through the back propagation algorithm, according to L N Update the model's parameters N to minimize the difference between the predicted noise and the true noise.

[0093] For step 3, during the data augmentation process, the model starts with a set of random noise and generates samples similar to the real data distribution by gradually removing the noise using the neural network trained in step 2. Through the denoising operation over multiple time steps, the model is able to generate additional samples similar to the original data distribution from completely random noise.

[0094] The specific process of data enhancement is as follows:

[0095] Step 3.1, randomly sample the initial noise sample x from the standard Gaussian distribution T . This sample represents the initial completely noisy data.

[0096] Step 3.2, select a time step t=T, and gradually perform the denoising process starting from T.

[0097] Step 3.3: For each time step t, use the diffusion model trained in step 2 to predict the current noise ε N (x t ,t).

[0098] Step 3.4, according to the noise ε N (x t ,t), calculate the denoised data x t-1 .

[0099] Step 3.5: Repeat steps 3.3 and 3.4 to obtain the generated new sample x0.

[0100] For step 4, you need to use the ship motion model to build a training environment. The ship motion model used is a simplified dynamic model, and the model structure is as follows:

[0101]

[0102] Among them, (x, y, ψ) represents the position and heading of the unmanned boat in the earth-fixed coordinate system; (u, v, r) represents the longitudinal velocity, transverse velocity and yaw angular velocity of the unmanned boat in the hull-fixed coordinate system; (m1, m2) is the additional mass coefficient, (x u ,y v ) is the linear damping coefficient, J is the moment of inertia, and (T, τ) represent the changes of force and torque acting on the unmanned boat, respectively.

[0103] For step 5, the discriminator consists of a value estimation network and a reward estimation network, each of which contains three hidden layers and 256 neurons in each layer. The generator consists of an action network, a Q1 network, a Q2 network, a target Q1 network, and a target Q2 network, each of which consists of three hidden layers and 256 neurons in each layer.

[0104] For step 6, the main purpose of the discriminator is to distinguish the expert collision avoidance data p E And the generator generates data p G , whose objective function is usually designed to minimize the cross entropy loss between the two data, as shown in equation (4).

[0105]

[0106] Where (s, a) represents the state and action respectively. Represents the distribution of expert data pE The expected calculation is performed on the (s, a) sampled in the above example. D(s, a) is the output of the discriminator with (s, a) as input. The reward signal R(s, a) output by the discriminator can be calculated based on D(s, a). R(s, a) is mainly used to update the generator. The calculation model of this reward signal is usually defined as:

[0107] R(s,a)=log(1-D(s,a))-log(D(s,a)) (5)

[0108] The specific process of updating the discriminator can be divided into the following steps:

[0109] Step 6.1, for p E and p G Perform random sampling.

[0110] Step 6.2, calculate the output D(s,a) of the discriminator.

[0111] Step 6.3, calculate the loss function L(D).

[0112] Step 6.4, update the parameters of the discriminator according to the loss function through the back-propagation algorithm.

[0113] For step 7, the main purpose of the generator is to train an action network π to maximize the indistinguishability between the data it generates and the expert collision avoidance data. This goal can be achieved by maximizing the cumulative reward J(π) given by the discriminator. Since the generator in the present invention is updated using SAC, the calculation of J(π) is as follows:

[0114]

[0115] in is the strategy entropy, α is the temperature coefficient, R n (s t ,a t ) is the reward signal output by the discriminator.

[0116] The specific process of generator update can be divided into the following steps:

[0117] Step 7.1, the generator generates data p using the current strategy π G .

[0118] Step 7.2, generate data p G Take samples.

[0119] Step 7.3: Input the sampled data in step 7.2 into R(s,a) calculated in the discriminator, and use it as the reward signal in reinforcement learning to update the generator.

[0120] Step 7.4, update the generator’s strategy π according to the SAC algorithm.

[0121] For step 8, steps 6 and 7 need to be performed continuously until the total training step length is completed and the training is ended.

[0122] Reference for control process of unmanned boat collision avoidance decision system Figure 3 Here are the steps:

[0123] Step 1), establish communication between the PC and the unmanned boat main control system;

[0124] Step 2), the unmanned boat main control system receives the sensor data and pre-processes it;

[0125] Step 3), pre-processing data is sent to the PC;

[0126] Step 4), the PC processes the data and makes decisions based on the pre-trained model;

[0127] Step 5), the decision result is sent from the PC to the main control system;

[0128] Step 6), the main control system distributes the thrust and sends it to the bottom control unit;

[0129] Step 7), implement collision avoidance decision.

[0130] For step 1), the present invention establishes communication between the PC and the unmanned boat main control system through the TCP / IP protocol. First, the PC creates a Socket connection and establishes a connection with the unmanned boat main control system through a specified IP address and port number. After the main control system monitors the connection request and accepts the connection request from the PC, the two sides start data transmission.

[0131] For step 2), when the algorithm of the present invention is applied in engineering, it mainly obtains millimeter wave radar and GPS data to make collision avoidance decisions. After receiving the raw data, the main control performs data processing to obtain the position and posture of the own ship and the target ship information.

[0132] For step 3), the PC receives the ship's position and obstacle information data.

[0133] For step 4), the PC calculates the collision avoidance model input based on the data information and the collision avoidance model makes a decision.

[0134] For step 5), the decision result is sent from the PC to the unmanned boat main control system.

[0135] For step 6): the main control system generates a thrust distribution instruction according to the received collision avoidance decision.

[0136] For step 7): the thrust distribution command is sent by the main control system to the underlying control unit to implement collision avoidance decision.

[0137] In order to ensure navigation safety, effective decision-making strategies need to be developed. Adversarial inverse reinforcement learning performs well in this process and can train a decision model that can generate decisions that meet the dynamic constraints of the unmanned boat without relying on artificial reward functions. However, there is a disadvantage in the research process that it relies too much on the scale of expert collision avoidance data. The diffusion model is introduced for data enhancement to solve the decision-making problem under limited samples.

[0138] Considering that the International Regulations for Preventing Collisions at Sea stipulate that ships need to change their course rather than their speed as much as possible during the avoidance process, the decision variable of the collision avoidance model can be set to the torque required for the unmanned ship to turn. That is, the action space of the model is:

[0139] a∈[τ min ,τ max ](7)

[0140] where τ min and τ max They are respectively the minimum and maximum steering torque that the unmanned boat control unit can provide.

[0141] The model state space consists of three parts:

[0142] s={s OS ,s TS ,s Goal} (8)

[0143] where s OS Used to describe information related to the own ship (OS). TS Used to describe the information related to the target ship (TS).Goal Used to describe information related to the target point.

[0144] s OS 、s TS and Goal The specific composition is as follows:

[0145]

[0146] in Represents the distance from OS to TS and the target point. Indicates the relative position of OS to TS and target. is the relative heading between OS and TS. V TS is the speed of TS. OL Indicates the encounter between OS and TS, while C RI represents the collision risk between OS and TS. r = 4m is the safety domain radius of TS.

[0147] The relative heading and relative orientation between the two ships are shown in the figure below: Figure 4 As shown, the calculation formula is as follows:

[0148]

[0149] C RI The value range is from 0 to 1. RI = 0, it means no collision will occur; when C RI =1, it means that a collision has occurred. RI This is calculated using a five-factor risk assessment model as follows:

[0150] C RI =J·M T

[0151] J = [α DCPA ,α TCPA ,α B ,α D ,α K ]

[0152] M=[γ DCPA ,γ TCPA ,γ B ,γ D ,γ K ] (11)

[0153] Where J represents the effect of each factor on C RIThe variable M represents the membership function of each factor described in formula (11). If the OS encounters multiple TSs with collision risk at the same time during navigation, the most dangerous TS is calculated by formula (11) and the navigation information of the TS is used to calculate the model input for decision making. Figure 5 shown.

[0154] The present invention integrates adversarial inverse reinforcement learning and diffusion model, alleviates the model's dependence on data scale without using a carefully designed reward function, and enables unmanned ship collision avoidance decision-making to be realized.

[0155] Based on the proposed collision avoidance decision algorithm, the present invention fully explains the algorithm engineering verification, proposes a reasonable technical solution, and combines theory with practice to form a complete collision avoidance system.

Claims

1. A dynamic collision avoidance decision method for unmanned boats based on diffusion model and adversarial reinforcement learning, characterized in that: The following steps are involved: Step 1, collect expert collision avoidance data of the expert collision avoidance model based on SoftActor-Critic in the simulation environment; Step 2, training the diffusion model by adding noise to the input data and updating the neural network weights using the back-propagation algorithm to minimize the difference between the noise predicted by the neural network and the actual added noise; Step 3: perform data enhancement; In the data augmentation process, the diffusion model starts with a set of random noises and gradually removes the noise by using the neural network trained in step 2 to generate samples similar to the distribution of the real data. Through the denoising operation of multiple time steps, the diffusion model is able to generate additional samples similar to the original data distribution from completely random noises. Step 4: Use the ship motion model to build a training environment; Step 5: Build the generator and discriminator; Step 6, update the discriminator; Step 7, update the generator; Step 8: Complete the training.

2. According to claim 1, a dynamic collision avoidance decision method for unmanned boats based on diffusion model and adversarial reinforcement learning is characterized in that: The expert collision avoidance data p in step 1 E It consists of several lines (s, a, s′, d); s represents the current state of the environment, a is the collision avoidance action taken by the expert collision avoidance model in the current state, s′ is the state of the environment at the next moment after executing action a, and d indicates whether the encounter scene is over.

3. The unmanned boat dynamic collision avoidance decision method based on diffusion model and adversarial reinforcement learning according to claim 1 is characterized in that: The diffusion model in step 2 is composed of a three-layer fully connected neural network, and the loss function for measuring the difference is as follows: Where N is the neural network parameter, is the noise predicted by the neural network, ε is the actual noise added, x t For actual data, is the expectation of the noise distribution.

4. The unmanned boat dynamic collision avoidance decision method based on diffusion model and adversarial reinforcement learning according to claim 3 is characterized in that: The training of the diffusion model in step 2 specifically includes the following steps: Step 2.1: For the expert collision avoidance data p E Sampling is performed to obtain sampling data x0; Step 2.2, select a time step t, t is randomly sampled from a uniform distribution; add noise to the sampled data x0 to generate noisy data x t ; The noise size is determined by the noise schedule β t Control; β t is linearly uniformly distributed, that is, β t The calculation of is shown in the following formula (2): where β start is the initial noise intensity, β end represents the final noise intensity, T represents the total number of time steps; Step 2.3, use the neural network with parameter N to predict the noise Step 2.4, calculate the loss function Step 2.5, through the back propagation algorithm, according to Update the model's parameters N to minimize the difference between the predicted noise and the true noise.

5. The unmanned boat dynamic collision avoidance decision method based on diffusion model and adversarial reinforcement learning according to claim 1 is characterized in that: The specific process of data enhancement in step 3 is as follows: Step 3.1, randomly sample the initial noise sample x from the standard Gaussian distribution T ; This sample represents the initial completely noisy data; Step 3.2, select a time step t = T, and gradually perform the denoising process starting from T; Step 3.3, for each time step t, use the diffusion model trained in step 2 to predict the current noise Step 3.4, according to the noise Calculate the denoised data x t-1 ; Step 3.5: Repeat steps 3.3 and 3.4 to obtain the generated new sample x0.

6. The unmanned boat dynamic collision avoidance decision method based on diffusion model and adversarial reinforcement learning according to claim 1 is characterized in that: The ship motion model used in step 4 is a simplified dynamic model, and the model structure is as follows: Among them, (x, y, ψ) represents the position and heading of the unmanned boat in the earth-fixed coordinate system; (u, v, r) represents the longitudinal velocity, transverse velocity and yaw angular velocity of the unmanned boat in the hull-fixed coordinate system; (m1, m2) is the additional mass coefficient, (x u ,y v ) is the linear damping coefficient, J is the moment of inertia, and (T, τ) represent the changes of force and torque acting on the unmanned boat, respectively.

7. The unmanned boat dynamic collision avoidance decision method based on diffusion model and adversarial reinforcement learning according to claim 1 is characterized in that: The objective function of the discriminator in step 6 is designed to minimize the expert collision avoidance data p E And the generator generates data p G The cross entropy loss between is shown in formula (4): Where (s, a) represents the state and action respectively. Represents the distribution of expert data p E The expected calculation is performed on the (s, a) sampled in the above example. D(s, a) is the output of the discriminator with (s, a) as input. The reward signal R(s, a) output by the discriminator is calculated based on D(s, a). R(s, a) is used to update the generator. The calculation model of this reward signal is defined as: R(s,a)=log(1-D(s,a))-log(D(s,a))(5) The specific process of updating the discriminator is divided into the following steps: Step 6.1: For the expert collision avoidance data p E And the generator generates data p G Conduct random sampling; Step 6.2, calculate the output D(s,a) of the discriminator; Step 6.3, calculate the loss function Step 6.4, update the parameters of the discriminator according to the loss function through the back-propagation algorithm.

8. The unmanned boat dynamic collision avoidance decision-making method based on diffusion model and adversarial reinforcement learning according to claim 1 is characterized in that: The step 7 specifically includes the following steps: Step 7.1, the generator generates data p using the current strategy π generator G ; Step 7.2, generate data p G Take samples; Step 7.3, input the sampled data in step 7.2 into R(s,a) calculated in the discriminator, and use it as the reward signal in reinforcement learning to update the generator; Step 7.4, update the generator strategy π according to the SAC algorithm; This is achieved by maximizing the cumulative reward J(π) given by the discriminator. The generator is updated using SAC. The calculation of J(π) is as follows: in is the strategy entropy, α is the temperature coefficient, R n (s t ,a t ) is the reward signal output by the discriminator.

9. The unmanned boat dynamic collision avoidance decision method based on diffusion model and adversarial reinforcement learning according to claim 1, characterized in that: The control process steps of the unmanned boat collision avoidance decision system are as follows: Step 1), establish communication between the PC and the unmanned boat main control system; Step 2), the unmanned boat main control system receives the sensor data and pre-processes it; Step 3), pre-processing data is sent to the PC; Step 4), the PC processes the data and makes decisions based on the pre-trained model; Step 5), the decision result is sent from the PC to the main control system; Step 6), the main control system distributes thrust and sends it to the bottom control unit; Step 7), implement collision avoidance decision.

10. A dynamic collision avoidance decision system for unmanned boats based on diffusion model and adversarial reinforcement learning, characterized in that: A dynamic collision avoidance decision method for an unmanned boat based on a diffusion model and adversarial reinforcement learning as described in any one of claims 1 to 9 is run.

Images