An unmanned ship dynamic collision avoidance decision method and system based on a diffusion model and adversarial inverse reinforcement learning

Abstract

The application provides an unmanned ship dynamic collision avoidance decision method and system based on a diffusion model and an adversarial inverse reinforcement learning. The method comprises the following steps: 1, collecting expert collision avoidance data; 2, training a diffusion model; 3, performing data enhancement; 4, building a training environment; 5, constructing a generator and a discriminator; 6, updating the discriminator; 7, updating the generator; and 8, completing the training. The diffusion model is trained by using limited expert sample data, additional collision avoidance data is generated by using the trained diffusion model, and the additional data and the original expert data jointly participate in the training process of the adversarial inverse reinforcement learning, so that the problem of limited expert collision avoidance data can be effectively solved. Furthermore, an engineering application plan is provided for the proposed algorithm, the algorithm is verified, and a systematic solution is formed by combining theory with practice.

Description

Technical Field

[0001] This invention relates to the field of unmanned surface vessel (USV) collision avoidance technology, and in particular to a dynamic collision avoidance decision-making method and system for USVs based on a diffusion model and adversarial inverse reinforcement learning. Background Technology

[0002] Unmanned surface vessels (USVs) are playing an increasingly important role in both civilian and military fields, serving tasks such as water quality monitoring, anti-piracy operations, and maritime search and rescue. To ensure the success of these missions, USVs must possess the ability to autonomously avoid dynamic obstacles. During navigation, dynamically moving vessels are the most common type of obstacle encountered by USVs. However, with the advancement of globalization and the significant increase in the number of ships at sea, the navigational situations faced by USVs are becoming increasingly complex, posing a serious challenge to their collision avoidance decision-making capabilities.

[0003] In the study of dynamic collision avoidance by unmanned surface vessels (USVs), the main algorithms can be divided into three categories: 1) Rule-based collision avoidance algorithms: These rely on predefined navigation rules such as the Collision Avoidance Regulations at Sea (COLREG). These algorithms provide collision avoidance decisions based on applicable rules, but may struggle to find suitable coping strategies in scenarios not covered by these rules. 2) Optimization-based collision avoidance algorithms: These algorithms transform the collision avoidance problem into an optimization problem by establishing a decision model and use relevant algorithms (such as model predictive control, artificial potential fields, and genetic algorithms) to solve for the optimal path. Although these methods can handle multi-objective and constrained problems, they have high computational complexity, poor real-time performance, and are difficult to apply in dynamic environments. 3) Learning-based collision avoidance algorithms: These algorithms train neural network strategies using a large amount of historical data to make collision avoidance decisions. This data can come from real-world scenarios or simulated environments. This category includes reinforcement learning and deep reinforcement learning, which can exhibit strong adaptability in complex environments and handle nonlinear, highly constrained, and dynamic collision avoidance scenarios.

[0004] Currently, learning-based collision avoidance algorithms are considered one of the most promising solutions because they can make optimal decisions quickly in dynamic environments while fully considering constraints such as ship handling characteristics. These algorithms train the agent to learn the optimal strategy by maximizing the cumulative reward signal generated by the interaction between the unmanned surface vessel (USV) and the environment. However, the performance of these algorithms largely depends on the design of the reward function. Designing a reasonable reward function is extremely challenging for dynamic collision avoidance tasks. In contrast, acquiring expert collision avoidance data is simpler, but it is often limited in quantity and cannot cover all possible collision avoidance scenarios.

[0005] Current technologies do not fully consider the scale constraints of expert collision avoidance data. When expert data is insufficient, it is difficult to effectively learn high-quality collision avoidance decisions. Inverse reinforcement learning requires deriving the reward function before reinforcement learning, which not only incurs significant computational costs but also increases computation time. Furthermore, the derived reward function is susceptible to data insufficiency, reducing policy effectiveness. Adversarial imitation learning directly learns the policy without regard to the reward function, failing to explicitly derive the reward signal, resulting in insufficient behavioral explanation and unstable training. Adversarial inverse reinforcement learning combines the advantages of inverse reinforcement learning and adversarial imitation learning, effectively learning the policy and deriving the reward function to explain behavior, thereby improving the policy's generalization ability. However, adversarial inverse reinforcement learning relies heavily on sufficient expert data. Moreover, current technologies have limited engineering applications. Most reinforcement learning algorithms in the marine field are currently limited to simulation experiments, and significant breakthroughs have not been achieved due to factors such as the computing power of edge computing devices for unmanned surface vessels. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a dynamic collision avoidance decision-making method and system for unmanned surface vessels based on diffusion model and adversarial inverse reinforcement learning. By introducing diffusion model to enhance limited expert data, the invention effectively alleviates the high dependence of existing algorithms on reward functions and expert data.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a dynamic collision avoidance decision-making method for unmanned surface vessels based on a diffusion model and adversarial inverse reinforcement learning, comprising the following steps:

[0008] Step 1: Collect expert collision avoidance data of the SoftActor-Critic-based collision avoidance model in the simulation environment;

[0009] Step 2: Train the diffusion model by adding noise to the input data and updating the neural network weights using the backpropagation algorithm to minimize the difference between the neural network's predicted noise and the actual added noise.

[0010] Step 3, perform data augmentation; during the data augmentation process, the diffusion model starts with a set of random noise and gradually removes the noise by using the neural network trained in Step 2 to generate samples similar to the real data distribution; through denoising operations at multiple time steps, the diffusion model can generate additional samples similar to the original data distribution from completely random noise.

[0011] Step 4: Use a ship motion model to build a training environment;

[0012] Step 5: Construct the generator and discriminator;

[0013] Step 6: Update the discriminator;

[0014] Step 7, update the generator;

[0015] Step 8: Complete the training.

[0016] In a preferred embodiment, the expert collision avoidance data p in step 1 E It consists of several (s,a,s′,d) statements, where s represents the current environmental state, a is the collision avoidance action taken by the expert collision avoidance model in the current state, s′ is the environmental state at the next moment after the action a is executed, and d indicates whether the encounter scenario has ended.

[0017] In a preferred embodiment, the diffusion model in step 2 consists of a three-layer fully connected neural network, and the loss function for measuring the difference is as follows:

[0018]

[0019] Where N is the neural network parameter. The neural network predicts noise, ε represents the actual noise added, and x represents the noise level. t For actual data, This represents the expected noise distribution.

[0020] In a preferred embodiment, step 2, training the diffusion model, specifically includes the following steps:

[0021] Step 2.1, analyze the expert collision avoidance data p E Sampling is performed to obtain sampled data x0;

[0022] Step 2.2: Select a time step t, which is randomly sampled from a uniform distribution; add noise to the sampled data x0 to generate noisy data x. t The noise level is determined by the set noise scheduling β. t Control; β t It is linearly uniformly distributed, i.e., β t The calculation is shown in the following formula (2):

[0023]

[0024] Where β start It is the initial noise intensity, β end The final noise intensity is represented by T, and the total number of time steps is represented by T.

[0025] Step 2.3: Use a neural network with parameter N to predict the noise ε. N (x t ,t);

[0026] Step 2.4, calculate the loss function L N ;

[0027] Step 2.5, using the backpropagation algorithm, based on L N Update the model's parameters N to minimize the difference between the predicted noise and the actual noise.

[0028] In a preferred embodiment, the data augmentation process in step 3 is as follows:

[0029] Step 3.1: Randomly sample initial noise samples x from a standard Gaussian distribution. T This sample represents the initial, completely noisy data.

[0030] Step 3.2: Select a time step t = T, and start the denoising process step by step from T.

[0031] Step 3.3: For each time step t, use the diffusion model trained in Step 2 to predict the current noise.

[0032] Step 3.4, based on the noise ε N (x t ,t), calculate the denoised data x t-1 ;

[0033] Step 3.5: Repeat steps 3.3 and 3.4 to obtain the new sample x0.

[0034] In a preferred embodiment, the ship motion model used in step 4 is a simplified dynamic model, and the model structure is shown below:

[0035]

[0036] Where (x,y,ψ) represents the position and heading of the unmanned surface vessel (USV) in the Earth's fixed coordinate system; (u,v,r) represents the pitch velocity, yaw velocity, and yaw rate of the USV in the hull's fixed coordinate system; (m1,m2) are the added mass coefficients, (x u ,y v ) is the linear damping coefficient, J is the moment of inertia, and (T,τ) represent the changes in force and torque acting on the unmanned surface vessel, respectively.

[0037] In a preferred embodiment, in step 5, the discriminator consists of a value estimation network and a reward estimation network, each network containing three hidden layers with 256 neurons per layer; the generator consists of an action network, a Q1 network, a Q2 network, a target Q1 network, and a target Q2 network, each network consisting of three hidden layers with 256 neurons per layer.

[0038] In a preferred embodiment, the objective function of the discriminator in step 6 is designed to minimize the expert collision avoidance data p. E And the generator generates data pG The cross-entropy loss between them is shown in equation (4):

[0039]

[0040] Where (s, a) represent the state and action, respectively. Indicates the distribution p of expert data E The expected value of (s,a) obtained from the mid-sample is calculated, and D(s,a) is the output of the discriminator with (s,a) as input. The reward signal R(s,a) output by the discriminator is calculated based on D(s,a), and R(s,a) is used to update the generator. The calculation model for this reward signal is defined as follows:

[0041] R(s,a)=log(1-D(s,a))-log(D(s,a)) (5)

[0042] The discriminator update process consists of the following steps:

[0043] Step 6.1, analyze the expert collision avoidance data p E And the generator generates data p G Perform random sampling;

[0044] Step 6.2, calculate the output D(s,a) of the discriminator;

[0045] Step 6.3, calculate the loss function L(D);

[0046] Step 6.4: Update the discriminator parameters according to the loss function using the backpropagation algorithm.

[0047] In a preferred embodiment, step 7 specifically includes the following steps:

[0048] Step 7.1, the generator uses the current policy π generator to generate data p. G ;

[0049] Step 7.2, generate data p G Perform sampling;

[0050] Step 7.3: Input the sampled data from Step 7.2 into the discriminator to calculate R(s,a), and use it as the reward signal in reinforcement learning to update the generator;

[0051] Step 7.4: Update the generator's policy π according to the SAC algorithm;

[0052] This is achieved by maximizing the cumulative reward J(π) given by the discriminator. The generator is updated using SAC, and J(π) is calculated as follows (6):

[0053]

[0054] in Let R be the strategy entropy, α be the temperature coefficient, and R be the... n (s t ,a t ) is the reward signal output by the discriminator.

[0055] In a preferred embodiment, the control process steps of the unmanned surface vessel collision avoidance decision system are as follows:

[0056] Step 1) Establish communication between the PC and the unmanned surface vessel's main control system;

[0057] Step 2), the unmanned surface vessel's main control system receives and preprocesses sensor data;

[0058] Step 3), send the preprocessed data to the PC;

[0059] Step 4): The PC processes the data and combines it with the pre-trained model to make a decision;

[0060] Step 5): The decision result is sent from the PC to the main control system;

[0061] Step 6): The main control system distributes thrust and sends the data to the lower-level control unit.

[0062] Step 7) Implement collision avoidance decision.

[0063] Compared with existing technologies, this invention has the following advantages: The collision avoidance algorithm provided by this invention is based on adversarial reinforcement learning algorithms, forming an unmanned surface vessel (USV) collision avoidance strategy combined with a diffusion model. This algorithm uses the reward signal provided by the discriminator, solving the problem of difficult reward function design. Simultaneously, it uses a diffusion model to increase data, ensuring that an effective collision avoidance strategy is learned even with limited expert collision avoidance samples, thus improving the algorithm's training effect. This invention forms a complete system solution from algorithm to factory application, encompassing not only algorithm research but also providing a thorough explanation of the algorithm's practical verification. This solution establishes communication between the PC and the vessel's main control system via the TCP / IP protocol. Decision-making is executed by the PC, solving the problem of insufficient computing power in the vessel's main control system. By receiving data from onboard sensors to calculate collision avoidance decisions, the main control system implements thrust allocation, achieving a breakthrough from simulation to reality, making collision avoidance algorithm verification more reliable, and promoting research on ship collision avoidance. Attached Figure Description

[0064] Figure 1 This is a structural diagram of the collision avoidance decision model for an unmanned surface vessel according to a preferred embodiment of the present invention.

[0065] Figure 2 This is a flowchart illustrating the diffusion model training process of a preferred embodiment of the present invention.

[0066] Figure 3 This is a flowchart illustrating the collision avoidance decision-making process of an unmanned surface vessel according to a preferred embodiment of the present invention.

[0067] Figure 4 This is a schematic diagram illustrating the relative bearing and relative heading between two ships according to a preferred embodiment of the present invention.

[0068] Figure 5 This is a flowchart illustrating the collision avoidance process for unmanned vessels in multi-vehicle encounters, as per a preferred embodiment of the present invention. Detailed Implementation

[0069] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0070] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0071] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0072] A dynamic collision avoidance decision-making method for unmanned surface vessels based on diffusion models and adversarial inverse reinforcement learning, referencing Figure 1-5 The structure of the dynamic collision avoidance decision algorithm for unmanned surface vessels is referenced. Figure 1 The specific steps are as follows:

[0073] Step 1: Collect expert collision avoidance data;

[0074] Step 2, train the diffusion model;

[0075] Step 3: Perform data augmentation;

[0076] Step 4: Set up the training environment;

[0077] Step 5: Construct the generator and discriminator;

[0078] Step 6: Update the discriminator;

[0079] Step 7, update the generator;

[0080] Step 8: Complete the training.

[0081] For step 1, considering the limited practical applications of unmanned surface vessels (USVs) and the difficulty in collecting real-world maritime collision avoidance data for model training, the expert collision avoidance data primarily used in the algorithm training process comes from the SoftActor-Critic (SAC)-based collision avoidance model in a simulation environment. The expert collision avoidance data p used... E It consists of several (s,a,s′,d).

[0082] For step 2, the diffusion model consists of a three-layer fully connected neural network, with 256 neurons in each layer. During training, the model minimizes the difference between the neural network's predicted noise and the actual added noise by adding noise to the input data and updating the neural network weights using the backpropagation algorithm. The loss function that measures this difference is as follows:

[0083]

[0084] Where N is the neural network parameter. The neural network predicts noise, ε represents the actual noise added, and x represents the noise level. t For actual data, This represents the expected noise distribution.

[0085] The training process of the diffusion model is as follows Figure 2 As shown, the specific steps are as follows:

[0086] Step 2.1, analyze the expert collision avoidance data p E Sampling is performed to obtain sampled data x0.

[0087] Step 2.2: Select a time step t, where t is randomly sampled from a uniform distribution. Add noise to x0 to generate noisy data x. t The noise level is determined by the set noise scheduling β. t Control. β t It is linearly uniformly distributed, i.e., β t The calculation is shown in the following formula (2):

[0088]

[0089] Where β start It is the initial noise intensity, β end The final noise intensity is represented by T, and the total number of time steps is represented by T.

[0090] Step 2.3, use a neural network with parameter N to predict noise ∈ N (x t ,t).

[0091] Step 2.4, calculate the loss function L N .

[0092] Step 2.5, using the backpropagation algorithm, based on L N Update the model's parameters N to minimize the difference between the predicted noise and the actual noise.

[0093] For step 3, during data augmentation, the model starts with a set of random noise and gradually removes the noise using the neural network trained in step 2, generating samples similar to the real data distribution. Through denoising operations at multiple time steps, the model is able to generate additional samples that resemble the original data distribution from completely random noise.

[0094] The specific steps of data augmentation are as follows:

[0095] Step 3.1: Randomly sample initial noise samples x from a standard Gaussian distribution. T This sample represents the initial, completely noisy data.

[0096] Step 3.2: Select a time step t = T, and start the denoising process step by step from T.

[0097] Step 3.3: For each time step t, use the diffusion model trained in Step 2 to predict the current noise ε. N (x t ,t).

[0098] Step 3.4, based on the noise ε N (x t ,t), calculate the denoised data x t-1 .

[0099] Step 3.5: Repeat steps 3.3 and 3.4 to obtain the new sample x0.

[0100] For step 4, a training environment needs to be built using a ship motion model. The ship motion model used is a simplified dynamic model, and its structure is shown below:

[0101]

[0102] Where (x,y,ψ) represents the position and heading of the unmanned surface vessel (USV) in the Earth's fixed coordinate system; (u,v,r) represents the pitch velocity, yaw velocity, and yaw rate of the USV in the hull's fixed coordinate system; (m1,m2) are the added mass coefficients, (x u ,y v ) is the linear damping coefficient, J is the moment of inertia, and (T,τ) represent the changes in force and torque acting on the unmanned surface vessel, respectively.

[0103] For step 5, the discriminator consists of a value estimation network and a reward estimation network, each containing three hidden layers with 256 neurons per layer. The generator consists of an action network, a Q1 network, a Q2 network, a target Q1 network, and a target Q2 network, all of which have three hidden layers with 256 neurons per layer.

[0104] For step 6, the main purpose of the discriminator is to distinguish expert collision avoidance data p. E And the generator generates data p G The objective function is usually designed to minimize the cross-entropy loss between the two types of data, as shown in equation (4).

[0105]

[0106] Where (s, a) represent the state and action, respectively. Indicates the distribution of expert data pE The expected value of (s,a) obtained from the sampling is calculated, and D(s,a) is the output of the discriminator with (s,a) as input. Based on D(s,a), the reward signal R(s,a) output by the discriminator can be calculated. R(s,a) is mainly used to update the generator. The calculation model for this reward signal is usually defined as:

[0107] R(s,a)=log(1-D(s,a))-log(D(s,a)) (5)

[0108] The discriminator update process can be divided into the following steps:

[0109] Step 6.1, for p E and p G Perform random sampling.

[0110] Step 6.2: Calculate the output D(s,a) of the discriminator.

[0111] Step 6.3, calculate the loss function L(D).

[0112] Step 6.4: Update the discriminator parameters according to the loss function using the backpropagation algorithm.

[0113] For step 7, the main objective of the generator is to train an action network π that maximizes the indistinguishability between the generated data and the expert collision avoidance data. This objective can be achieved by maximizing the cumulative reward J(π) given by the discriminator. Since the generator in this invention uses SAC for updating, J(π) is calculated as follows (6):

[0114]

[0115] in Let R be the strategy entropy, α be the temperature coefficient, and R be the... n (s t ,a t ) is the reward signal output by the discriminator.

[0116] The generator update process can be divided into the following steps:

[0117] Step 7.1, the generator uses the current strategy π to generate data p. G .

[0118] Step 7.2, generate data p G Perform sampling.

[0119] Step 7.3: Input the sampled data from step 7.2 into the discriminator to calculate R(s,a), and use it as the reward signal in reinforcement learning to update the generator.

[0120] Step 7.4: Update the generator's policy π according to the SAC algorithm.

[0121] For step 8, steps 6 and 7 need to be performed continuously until the total training step length is completed and the training ends.

[0122] Unmanned surface vessel collision avoidance decision system control process reference Figure 3 The steps are as follows:

[0123] Step 1) Establish communication between the PC and the unmanned surface vessel's main control system;

[0124] Step 2), the unmanned surface vessel's main control system receives and preprocesses sensor data;

[0125] Step 3), send the preprocessed data to the PC;

[0126] Step 4): The PC processes the data and combines it with the pre-trained model to make a decision;

[0127] Step 5): The decision result is sent from the PC to the main control system;

[0128] Step 6): The main control system distributes thrust and sends the data to the lower-level control unit.

[0129] Step 7) Implement collision avoidance decision.

[0130] For step 1), this invention establishes communication between the PC and the unmanned surface vessel (USV) main control system via the TCP / IP protocol. First, the PC creates a Socket connection and establishes a connection with the USV main control system using a specified IP address and port number. After the main control system listens for connection requests and accepts the PC's connection request, data transmission begins between the two parties.

[0131] For step 2), when the algorithm of this invention is applied in engineering, it mainly acquires millimeter-wave radar and GPS data to make collision avoidance decisions. After receiving the raw data, the main controller processes the data to obtain the ship's attitude and the target ship's information.

[0132] For step 3), the PC receives the ship's position and obstacle information data.

[0133] For step 4), the PC calculates the collision avoidance model input based on the data information and the collision avoidance model makes the decision.

[0134] For step 5), the decision result is sent from the PC to the unmanned surface vessel's main control system.

[0135] For step 6): The main control system generates thrust allocation instructions based on the received collision avoidance decisions.

[0136] For step 7): the thrust distribution command is sent from the main control system to the lower control unit to realize the collision avoidance decision.

[0137] To ensure navigational safety, effective decision-making strategies are essential. Adversarial inverse reinforcement learning excels in this process, capable of training decision models that conform to the dynamic constraints of unmanned surface vessels (USVs) without relying on artificial reward functions. However, it suffers from an over-reliance on the scale of expert collision avoidance data. Therefore, a diffusion model is introduced for data augmentation to address decision-making problems with limited samples.

[0138] Considering that the International Maritime Collision Avoidance Regulations (IMC) require vessels to change course rather than speed during collision avoidance, the decision variable for the collision avoidance model can be set as the torque required for the unmanned vessel to turn. That is, the model's action space is:

[0139] a∈[τ min ,τ max (7)

[0140] Where τ min and τ max These are the minimum and maximum steering torques that the unmanned surface vessel control unit can provide, respectively.

[0141] The model's state space consists of three parts:

[0142] s={s OS ,s TS ,s Goal} (8)

[0143] Where s OS Used to describe information related to the owner ship (OS), s TS Used to describe information related to the other party's vessel (Targetship, TS), sGoal Used to describe information related to the target point.

[0144] s OS s TS and s Goal The specific components are as follows:

[0145]

[0146] in This represents the distance from OS to TS and the target point. This indicates the relative orientation of OS to TS and the target. It is the relative heading between OS and TS. V TS It's the speed of TS. C OL This indicates the meeting situation between OS and TS, while C RI This represents the collision risk between the OS and TS. r = 4m is the safety domain radius of the TS.

[0147] A diagram showing the relative headings and bearings between the two ships is shown below. Figure 4 As shown, the calculation formula is as follows:

[0148]

[0149] C RI The value of C ranges from 0 to 1. RI When C = 0, it means no collision will occur; when C = 0, it means no collision will occur. RI When C = 1, it means a collision has occurred. RI The calculation, using a five-factor risk assessment model, is as follows:

[0150] C RI =J·M T

[0151] J = [α] DCPA ,α TCPA ,α B ,α D ,α K ]

[0152] M = [γ DCPA ,γ TCPA ,γ B ,γ D ,γ K (11)

[0153] Where J represents the effect of each factor on C. RIThe weight matrix of the degree of influence. Variable M represents the membership function of each factor described in formula (11). If the OS encounters multiple TSs with collision risk during navigation, the most dangerous TS is calculated using formula (11), and its navigation information is used as input to the model for decision-making. The collision avoidance process is as follows: Figure 5 As shown.

[0154] This invention integrates adversarial inverse reinforcement learning and a diffusion model, which alleviates the model's dependence on data scale without using a carefully designed reward function, thus enabling collision avoidance decisions for unmanned vessels.

[0155] Based on the proposed collision avoidance decision algorithm, this invention provides a thorough explanation of the algorithm's engineering verification, proposes a reasonable technical solution, and combines theory with practice to form a complete collision avoidance system.

Claims

1. An unmanned surface vehicle dynamic collision avoidance decision-making method based on a diffusion model and adversarial inverse reinforcement learning, characterized in that, The method comprises the following steps: Step 1, collecting expert collision avoidance data of a Soft Actor-Critic-based expert collision avoidance model in a simulation environment; Step 2, training a diffusion model by adding noise to input data and updating neural network weights using a backpropagation algorithm to minimize the difference between the neural network predicted noise and the actual added noise; Step 3, data augmentation; During the data augmentation process, the diffusion model starts from a set of random noise and gradually removes the noise by using the neural network trained in step 2 to generate samples similar to the real data distribution; through multiple time step denoising operations, the diffusion model can generate additional samples similar to the original data distribution from completely random noise; Step 4, building a training environment using a ship motion model; Step 5, constructing a generator and a discriminator; Step 6, updating the discriminator; Step 7, updating the generator; Step 8, completing the training; The diffusion model in step 2 is composed of three fully connected neural networks, and the loss function for measuring the difference is as follows: （1） wherein is a neural network parameter, is a neural network predicted noise, is an actual added noise, is actual data, is an expectation of a noise distribution; The diffusion model in step 2 is trained specifically as follows: Step 2.1, sampling expert collision avoidance data sampling to obtain sample data ; Step 2.2, select a time step , is randomly sampled from a uniform distribution; the sampled data is added with noise to generate noisy data ; the noise size is controlled by a set noise schedule ; is a linear uniform distribution, i.e. The calculation of is shown in the following formula (2): （2） wherein is the initial noise intensity, denotes the final noise intensity, denotes the total number of time steps; Step 2.3, using a neural network with parameters predicted noise ; Step 2.

4. Compute loss function ; Step 2.5, updating the parameters of the model by a backpropagation algorithm according to updating the parameters of the model to minimize the difference between the predicted noise and the real noise.

2. The unmanned surface vehicle dynamic collision avoidance decision-making method based on diffusion model and adversarial inverse reinforcement learning according to claim 1, characterized in that, Expert collision avoidance data in step 1 Composed of several articles Composition; of which Indicates the current state of the environment. It is the collision avoidance action taken by the expert collision avoidance model under the current conditions. It is to perform an action The environmental state in the next moment. Indicates whether the scene has ended.

3. The unmanned surface vehicle dynamic collision avoidance decision-making method based on diffusion model and adversarial inverse reinforcement learning according to claim 1, characterized in that, The specific process of data augmentation in step 3 is as follows: Step 3.

1. Randomly sample initial noise samples from a standard Gaussian distribution ; this sample represents the initially fully noisy data; Step 3.2, select a time step and stepwise from start the denoising process; Step 3.3, for each time step , predict the current noise using the diffusion model trained in step 2 Step 3.4, calculating the denoised data from the noise ;​ Step 3.5: Steps 3.3 and 3.4 are repeated to obtain a new sample generated .

4. The unmanned surface vehicle dynamic collision avoidance decision-making method based on diffusion model and adversarial inverse reinforcement learning according to claim 1, characterized in that, The ship motion model used in step 4 is a simplified dynamic model, and the model structure is as follows: （3） wherein, denotes the position and heading of the USV in the earth-fixed coordinate system; denotes the surge, sway and yaw angular velocities of the USV in the body-fixed coordinate system; is the added mass coefficient, is the linear damping coefficient, is the moment of inertia, denote the changes in forces and moments acting on the USV, respectively.

5. The unmanned surface vehicle dynamic collision avoidance decision-making method based on diffusion model and adversarial inverse reinforcement learning according to claim 1, characterized in that, The objective function of the discriminator in step 6 is designed to minimize the cross-entropy loss between the expert collision avoidance data and the generated data , as shown in equation (4): J(θD) = -E[log(D(x)) + log(l - D(G(z))) ] (4) （4） wherein respectively denote state and action, denotes the expectation computation on the samples from the expert data distribution ; the output of the discriminator with as input; the reward signal is computed according to the discriminator output , is used to implement the generator update; the computation model of this reward signal is defined as: （5） The specific process of updating the discriminator is divided into the following steps: Step 6.1, expert collision avoidance data and generator generated data random sampling is performed; Step 6.2, calculating the output of the discriminator ; Step 6.3, compute loss function ; Step 6.4, update the parameters of the discriminator according to the loss function through the backpropagation algorithm.

6. The unmanned surface vehicle dynamic collision avoidance decision-making method based on diffusion model and adversarial inverse reinforcement learning according to claim 1, characterized in that, The step 7 specifically comprises the following steps: Step 7.1, Generator uses current policy Generator generates data ; Step 7.2, generating data sampling; Step 7.3, the sampled data in step 7.2 is input into the discriminator to calculate and it is used as a reward signal in reinforcement learning to update the generator; Step 7.

4. Update the generator's policy according to the SAC algorithm ; By maximizing the cumulative reward given by the discriminator The generator is updated using SAC, The calculation is as follows (6): （6） wherein is the policy entropy, is the temperature coefficient, is the reward signal of the discriminator output.

7. The unmanned surface vehicle dynamic collision avoidance decision-making method based on diffusion model and adversarial inverse reinforcement learning according to claim 1, characterized in that, The control flow steps of the unmanned ship collision avoidance decision system are as follows: Step 1), establish communication between the PC end and the unmanned ship master control system; Step 2), the unmanned ship master control system receives sensor data and pre-processes it; Step 3), send the pre-processed data to the PC end; Step 4), the PC end processes the data and makes decisions in combination with the pre-trained model; Step 5), the decision result is sent to the master control system by the PC end; Step 6), the master control system performs thrust allocation and sends it to the underlying control unit; Step 7), implement collision avoidance decision.

8. An unmanned surface vehicle dynamic collision avoidance decision system based on diffusion model and adversarial inverse reinforcement learning, characterized in that, A dynamic collision avoidance decision method for unmanned ships based on diffusion model and adversarial inverse reinforcement learning is run.

Images