Training methods and devices for obstacle avoidance modules of unmanned surface vessels, and methods for exploring unknown areas.

Abstract

This invention discloses a training method and device for an unmanned surface vessel (USV) obstacle avoidance module, and a method for exploring unknown areas. The USV obstacle avoidance module has a TD3 agent, and the training method includes: acquiring the agent's current observation state s. i Execute the current action a i The observed state s at the next moment after environmental feedback i 'and reward r i , with (s i ,a i ,r i ,s i The current sample is stored in the experience cache, where the reward r is... i This includes a distance reward reflecting the UAV's current position relative to the target position and a safety reward reflecting the degree of safety between the UAV and obstacles. A sample weight is calculated for each sample, which is proportional to the sum of the squared residuals of the critic network and the absolute values ​​of the losses of the actor network. Based on these sample weights, a number of samples are extracted to train the agent and update the network parameters. The larger the sample weight, the higher the probability of it being extracted. By calculating the weights and sampling according to them, a UAV obstacle avoidance module with high accuracy can be obtained.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and more specifically, relates to a training method and device for an unmanned surface vessel obstacle avoidance module, and a method for exploring unknown areas. Background Technology

[0002] With the rapid development of the current economy, more and more unknown areas at sea need to be explored, or emergency rescue is needed in the event of a disaster at sea. However, because these unknown areas are unexplored, they are full of various potential dangers, and some unknown areas or rescue areas are not even suitable for human travel.

[0003] Currently, many deep reinforcement learning-based algorithms are used in research on unknown region exploration. Within the field of deep reinforcement learning, algorithms such as PPO and DDPG are widely applied in unknown region detection. For unmanned surface vessels (USVs) exploring unknown regions, the most crucial requirement is intelligent obstacle avoidance; however, current algorithms generally suffer from low obstacle avoidance accuracy. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an unmanned surface vessel obstacle avoidance module training method and training device, and an unknown area exploration method, the purpose of which is to improve the obstacle avoidance accuracy of unmanned surface vessels when exploring unknown areas.

[0005] To achieve the above objectives, according to one aspect of the present invention, a training method for an unmanned surface vessel (USV) obstacle avoidance module is provided, wherein the USV obstacle avoidance module has a TD3 agent, and the training method includes:

[0006] Step S1: Initialize the observation states of each network and environment of the agent and the initial actions of the agent;

[0007] Step S2: Obtain the agent's current observation state s i Execute the current action a i The observed state s at the next moment after environmental feedback i 'and reward r i , with (s i ,a i ,r i ,s i The current sample is stored in the experience cache, where the reward r is... i It includes a distance bonus based on the current position of the unmanned surface vessel (USV) relative to the target position, and a safety bonus based on the degree of safety between the USV and obstacles.

[0008] Step S3: After a preset number of samples are stored in the experience cache, the sample weight of each sample is calculated. The sample weight is proportional to the sum of the squared residuals of the critic network and the absolute value of the loss of the actor network.

[0009] Step S4: Select a number of samples based on the sample weights to train the agent and update the network parameters of the agent. The larger the sample weight, the greater the probability of it being selected.

[0010] In one embodiment, it further includes:

[0011] Step S5: Determine whether the number of inner loop iterations meets the requirements. If not, re-initialize the initial actions of the agent and jump to step S2 to execute the inner loop from step S2 to step S5 until the number of inner loop iterations reaches the first preset value and then exit the inner loop.

[0012] Step S6: Determine whether the number of outer loop iterations meets the requirements. If not, reinitialize the observation state of the environment and the initial actions of the agent, then jump to step S2 to execute the outer loop from step S2 to step S6 until the number of outer loop iterations reaches the second preset value and then exit the outer loop.

[0013] In one embodiment, the reward r i It also includes one or more of the following: angular velocity bonus, linear velocity bonus, and time bonus; among which,

[0014] The angular velocity reward reflects the current angular velocity of the unmanned surface vessel.

[0015] The linear velocity reward reflects the current linear velocity of the unmanned surface vessel;

[0016] The time reward reflects the length of time it takes for the unmanned surface vessel to reach the target location.

[0017] In one embodiment, the formula for calculating the sample weight of each sample is:

[0018]

[0019] Where, p i δ represents the probability before normalization. i Represents the residual of any critic network, max 1,2 This indicates that the larger of the squared residuals from the two critic networks is taken, ▽ a Q(s i ,a i |θ Q ) represents the loss of the actor network, where Q(s) i ,a i |θ Q) represents the current observation state s i The current action a is executed in the actor network. i The Q-value, θ, output by any subsequent critic network Q Here are the current network parameters for the critic network, ▽ a () denotes the gradient operator, max 1,2 |▽ a Q(s i ,a i |θ Q | indicates taking the larger of the two absolute values ​​of the loss in the agent network, where η is an empirical weight parameter used to balance the residual and loss by orders of magnitude, and τ p α and τ are the replenishment probability and balance factor determined experimentally, respectively. p α is a positive number less than 0.3, k represents the number of samples in the current experience cache, and P(i) represents the sample weights after normalization.

[0020] In one embodiment, the residual δ of any critic network i The calculation formula is:

[0021] δ i =r i +γQ(s i ',a i '|θ Q ')-Q(s i ,a i |θ Q )

[0022] Where γ represents the attenuation factor and its value ranges from 0.96 to 0.99, Q(s) i ',a i '|θ Q ') represents the state s under observation. i 'The target network of the next actor executes action a' i 'The value θ corresponds to the Q-value output of the critic target network. Q 'This refers to the current network parameters of the target network in the critic section.'

[0023] In one embodiment, the agent is augmented with noise N. t Noise N t The calculation formula is:

[0024]

[0025] Where N(0,σ) 2 ) represents Gaussian noise with a mean of 0 and a variance of σ, and ε is the noise limit value determined based on experimental results.

[0026] According to a second aspect of the present invention, a training device for an unmanned surface vessel (USV) obstacle avoidance module is provided, the USV obstacle avoidance module having a TD3 intelligent agent, the training device comprising:

[0027] The initialization module is used to initialize the observation states of the various networks and environment of the agent and the initial actions of the agent.

[0028] The sample generation module is used to obtain the observed state s of the agent at each time step. i Next, execute action a i The observed state s at the next moment after environmental feedback i 'and reward r i , with (s i ,a i ,r i ,s i The sample at each time step is stored in the experience cache, where the reward r is... i It includes a distance bonus based on the current position of the unmanned surface vessel (USV) relative to the target position, and a safety bonus based on the degree of safety between the USV and obstacles.

[0029] The sample weight calculation module is used to calculate the sample weight of each sample after a preset number of samples are stored in the experience cache. The sample weight is proportional to the sum of the squared residuals of the critic network and the absolute value of the loss of the actor network.

[0030] The training module is used to select a number of samples based on sample weights to train the agent and update the network parameters of the agent. The larger the sample weight, the greater the probability of it being selected.

[0031] In one embodiment, the formula for calculating the sample weight of each sample is:

[0032]

[0033] Where, p i δ represents the probability before normalization. i Represents the residual of any critic network, max 1,2 This indicates that the larger of the squared residuals from the two critic networks is taken, ▽ a Q(s i ,a i |θ Q ) represents the loss of the actor network, where Q(s) i ,a i |θ Q ) represents the current observation state s i The current action a is executed in the actor network.i The Q-value, θ, output by any subsequent critic network Q Here are the current network parameters for the critic network, ▽ a () denotes the gradient operator, max 1,2 |▽ a Q(s i ,a i |θ Q | indicates taking the larger of the two absolute values ​​of the loss in the agent network, where η is an empirical weight parameter used to balance the residual and loss by orders of magnitude, and τ p α and τ are the replenishment probability and balance factor determined experimentally, respectively. p α is a positive number less than 0.3, k represents the number of samples in the current experience cache, and P(i) represents the sample weights after normalization.

[0034] According to a third aspect of the present invention, a method for exploring unknown areas by an unmanned surface vessel is provided, comprising:

[0035] Determine the next target location to be reached, collect the current environmental observation status and input it into the unmanned surface vessel obstacle avoidance module, obtain the corresponding actions and drive the unmanned surface vessel to move;

[0036] The unmanned surface vessel obstacle avoidance module is obtained according to the unmanned surface vessel obstacle avoidance module training method described above.

[0037] In one embodiment, multiple unmanned surface vessels (USVs) form a swarm, and each USV in the swarm communicates with each other. The exploration method further includes:

[0038] The Voronoi partitioning method dynamically divides the corresponding exploration area for each unmanned surface vessel.

[0039] Each unmanned surface vessel (USV) determines whether its current location has been detected. If so, it does not perform detection and moves directly to the next target location. If not, it performs detection before moving to the next target location and marks its current location as detected and broadcasts it to all USVs.

[0040] Each unmanned surface vessel (USV) determines its next target location based on a calculation formula for a coordinated exploration strategy. The calculation formula for this coordinated exploration strategy is as follows:

[0041]

[0042] Make the distance potential Ω in The boundary node corresponding to the minimum value is taken as the next target position of the i-th unmanned surface vessel;

[0043] Where λ is a pre-defined scalar coefficient, and d inThis represents the distance between the current position of the i-th unmanned surface vessel and the n-th boundary node. This represents the distance between the first target position and the nth boundary point of the i-th unmanned surface vessel (USV). Each boundary node is at the same distance from the current position of the i-th USV.

[0044] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0045] This invention optimizes the training method for the obstacle avoidance module of an unmanned surface vessel (USV). The USV obstacle avoidance module uses an agent employing the TD3 algorithm for obstacle avoidance. Before deployment, the agent is trained by repeatedly performing an action-feedback state and reward-action-feedback state and reward iterations to acquire a large number of samples. Random samples are then randomly selected for training and updating network parameters. Before random sample selection, this invention calculates the sample weight of each sample in the experience cache. The sample weight reflects the importance of the corresponding sample and is correlated with the sampling probability. A higher sample weight indicates higher training value, a higher probability of being selected for training, and better training results. Furthermore, this invention analyzes the calculation parameters for sample weights, selecting the residual of the critic network and the loss of the actor network as key parameters. The sample weight is proportional to the sum of the squared residual of the critic network and the absolute value of the loss of the actor network. Analysis shows that the larger the sum of these two values, the greater the action deviation generated by the actor network, and the more focused the training needs to be on that sample. By using the training methods described above, it is possible to obtain an obstacle avoidance module for unmanned surface vessels with high obstacle avoidance accuracy.

[0046] Furthermore, by optimizing reward calculations, the driving parameters of the unmanned surface vessel can be kept within a suitable range, thereby increasing exploration speed.

[0047] Furthermore, by calculating the sample weights using the given formula, an obstacle avoidance module for unmanned surface vessels with higher obstacle avoidance accuracy can be trained more effectively.

[0048] Furthermore, by limiting noise, it can be controlled within a reasonable range, ensuring that a large amount of noise will not affect the results, thereby accelerating the training speed of the network. Attached Figure Description

[0049] Figure 1 This is a block diagram of a TD3 intelligent agent;

[0050] Figure 2 This is a flowchart of the steps of the unmanned surface vessel obstacle avoidance module training method in one embodiment;

[0051] Figure 3This is a flowchart of the steps of the unmanned surface vessel obstacle avoidance module training method in another embodiment;

[0052] Figure 4 This is a structural block diagram of an unmanned surface vessel obstacle avoidance module training device in one embodiment;

[0053] Figure 5 This is a flowchart illustrating the steps involved in an unmanned surface vessel swarm exploring an unknown area, as described in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0055] The training method for the obstacle avoidance module of the unmanned surface vessel in this invention involves the TD3 algorithm. To better understand this invention, a brief introduction to TD3 will be given first.

[0056] TD3 is a well-researched deterministic deep reinforcement learning algorithm, such as... Figure 1 As shown, it contains six networks: actor network, critic network 1, critic network 2, actor target network, critic target network 1, and critic target network 2. This architecture is a conventional architecture, and its specific working method is as follows: the agent obtains the current state s of the environment. t , will state s t Feed into the actor network to obtain action a t And the state s of the next moment t+1 ; to convert state-action pairs (s t ,a t The data is fed into critic network 1 and critic network 2 to obtain the Q-value Q1 used to evaluate the above state-action pair. t and Q2 t ; The state s of the next moment t+1 The action a is fed into the actor target network to obtain the action the agent should take in the next moment. t+1 , to state action pair (s t+1 ,a t+1 The data is fed into critic target network 1 and critic target network 2 to obtain the Q-value Q1' at the next time step. t+1 and Q2' t+1 Select Q1' t+1 and Q2' t+1 The smaller value min{Q1' t+1 and Q2't+1 According to Q1 t and min{Q1' t+1 and Q2' t+1 Calculate gradients to update the parameters of critic network 1, based on Q2. t and min{Q1' t+1 and Q2' t+1 Update the parameters of critic network 2, and select Q1. t and Q2 t The smaller value min{Q1 t Q2 t} Calculate about action a t The gradient is used to update the parameters of the actor network; after the above network parameters are updated several times, the parameters of the target network are updated using a soft update method.

[0057] Based on this, the present invention proposes a training method for an unmanned surface vessel obstacle avoidance module, such as... Figure 2 The diagram shown is a flowchart of the training method for the obstacle avoidance module of an unmanned surface vessel in one embodiment. The steps involved are described in detail below.

[0058] Step S100: Initialize the observation states of each network and environment of the agent and the initial actions of the agent.

[0059] This includes initializing the networks of the TD3 agent, specifically the network parameters of the six networks: actor network, critic network 1, critic network 2, actor target network, critic target network 1, and critic target network 2.

[0060] Define the network parameter θ as any critic network parameter. Q Define the network parameters of the actor network as θ μ Define the network parameters of any critic target network as θ′ Q Define the network parameters of the actor target network as θ′ μ Specifically, the network parameters of the actor network, critic network 1, and critic network 2 can be randomly initialized first. Then, the weight parameters θ′ of the target critic network can be initialized using the following formula. Q The weight parameters θ′ of the target actor network μ ,

[0061]

[0062] In this context, the network parameters of critic network 1 are assigned to critic target network 1, and the network parameters of critic network 2 are assigned to critic target network 2.

[0063] In this scenario, the environmental observation status includes the unmanned surface vessel's distance measurement data of the surrounding environment, its position data relative to the target, and its own speed data; while the actions are the unmanned surface vessel's driving data.

[0064] Understandably, the initial initialization also includes clearing the experience cache data.

[0065] In one embodiment, noise in the agent was also optimized, and the noise was calculated according to the following formula:

[0066]

[0067] Where N(0,σ) 2 ) represents Gaussian noise with a mean of 0 and a variance of σ, and ε is a noise limit value determined based on experimental results. In this embodiment, by optimizing the noise as described above, the training speed of the network is accelerated while ensuring that a large range of noise does not affect the results.

[0068] Step S200: Obtain the agent's current observation state s i Execute the current action a i The observed state s at the next moment after environmental feedback i 'and reward r i , with (s i ,a i ,r i ,s i The current sample is stored in the experience cache, where the reward r is... i It includes a distance bonus based on the current position of the unmanned surface vessel (USV) relative to the target position, and a safety bonus based on the degree of safety between the USV and obstacles.

[0069] After initializing the observed state of the environment and the initial action of the agent, the agent is run, continuously executing a repetitive process of action-feedback state and reward-action-feedback state and reward, where the next observed state s is the feedback. i 'and also the current state s i Continue executing the loop. Each time an action is executed, a corresponding set of (s) can be obtained. i ,a i ,r i ,s i The data is stored as samples in the experience cache. By repeatedly executing the process, a large number of samples can be stored in the experience cache.

[0070] Among them, the reward r i Includes distance reward r d and security rewards r cl .

[0071] For distance reward r d For example, if the unmanned surface vessel reaches its designated target position at the next specified moment, a larger reward is given; otherwise, a smaller, continuous reward is given. In other words, the closer it gets to the target position, the greater the reward r. d The larger the size, the more it drives the unmanned surface vessel to move toward the target.

[0072] For security reward r cl For example, if the current distance between the unmanned surface vessel and the obstacle is not less than twice the prescribed maximum safe distance, a reward can be set. cl The value is 0 if the current distance between the unmanned surface vessel and the obstacle is greater than the specified maximum safe distance d. omax And less than twice the specified maximum safety distance 2d omax Given a small negative penalty reward r cp If the current distance between the unmanned surface vessel and the obstacle is less than the specified maximum safe distance d. omax Given a large negative penalty reward r cpo The first penalty is equivalent to reminding the unmanned surface vessel that its current situation may be very dangerous, and the second penalty is to order the unmanned surface vessel to stop.

[0073] In one embodiment, the reward r i It also includes angular velocity bonus r av Linear velocity reward r lv Time reward r t Any one or more of the following.

[0074] Specifically, regarding the angular velocity reward r av Generally speaking, if the absolute value of the angular velocity is greater than the specified maximum angular velocity v... amax Given a negative penalty reward r ap In other cases, the reward is 0.

[0075] Specifically, regarding the linear velocity reward r lv In other words, if the linear velocity is lower than the specified minimum linear velocity v lmin Given a negative penalty reward r lp In other cases, the reward is 0.

[0076] Specifically, regarding the time reward r t In this scenario, if the time taken to reach the destination is relatively short, a reward of κΔt is given, which is the difference between the specified time and the time taken. κ represents a weighting factor, used to maintain this reward at the same order of magnitude as all the other rewards mentioned above, and Δt represents the difference between the specified time and the time taken. If the time taken to reach the destination exceeds the specified time, the reward is zero. Increasing the time reward can improve the speed of exploration.

[0077] Step S300: After a preset number of samples are stored in the experience cache, calculate the sample weight of each sample. The sample weight is proportional to the sum of the squared residuals of the critic network and the absolute value of the loss of the actor network.

[0078] In traditional techniques, samples are drawn with equal probability, meaning each sample has the same chance of being selected. However, different samples have different training values. To improve training effectiveness, this invention analyzes the correlation between the magnitude of the relevant state variables of each network in TD3 after each action and the training process. It finds that the larger the sum of the squared residuals of the critic network and the absolute values ​​of the losses of the actor networks, the more effective the corresponding sample is for training. Based on this, before updating network parameters by drawing samples, this invention first calculates the sample weight of each sample in the experience cache. This sample weight is proportional to the sum of the squared residuals of the critic network and the absolute values ​​of the losses of the actor networks. The larger the sum of the squared residuals of the critic network and the absolute values ​​of the losses of the actor networks, the higher the training value of the sample, the larger the calculated sample weight, and the greater the probability of it being drawn subsequently.

[0079] In a specific embodiment, through optimized design, the sample weights can be calculated using the following specific formula:

[0080]

[0081] Where, p i δ represents the probability before normalization. i Represents the residual of any critic network, max 1,2 This means selecting the larger of the squared residuals from the agent's two critic networks, specifically the larger of the squared residuals from critic network 1 and critic network 2, to increase the probability of data with smaller weights being sampled. ▽ a Q(s i ,a i |θ Q ) represents the loss of the actor network, where Q(s) i ,a i |θ Q ) represents the state s under observation. i The current action a is executed in the actor network. i The Q-value, θ, output by any subsequent critic network Q Here are the current network parameters for the critic network, ▽ a () denotes the gradient operator, max 1,2 |▽ a Q(s i ,ai |θ Q The expression `|` indicates that the larger of the two absolute values ​​of the loss in the actor network is selected. Specifically, the larger of the loss calculated based on the Q-value of critic network 1 (e.g., Q1) and the loss calculated based on the Q-value of critic network 2 (e.g., Q2) is chosen to increase the probability of sampling data with smaller weights. `η` is an empirical parameter used to balance the weights between the residuals and the loss, typically taking the order of magnitude difference between the squared residuals and the loss. This maintains consistency in the order of magnitude between the residuals and the loss, preventing one term from being too large and causing the result to be determined by only one term. `τ` p α and τ are the replenishment probability and balance factor determined experimentally, respectively. p A positive number less than 0.3 is used. For some data, the probability of being sampled is extremely small, almost zero, but this does not mean that such data is not worth training. Therefore, to ensure that such data can also be sampled and used for training, a small positive number τ is added. p This ensures that such data can be sampled. Specifically, a value of 0.1 can be used; α is a positive number less than 1. A value of 0 indicates uniform sampling, meaning each sample has an equal probability of being sampled. A value of 1 indicates sampling based on weighted probability. A value between 0 and 1 represents a balance between the two sampling methods, specifically a value of 0.8. k represents the number of samples in the current experience cache. P(i) represents the normalized sample weights.

[0082] In one embodiment, the formula for calculating the residuals for each critic network is as follows:

[0083] δ i =r i +γQ(s i ′,a i ′|θ Q ′)-Q(s i ,a i |θ Q )Formula (4)

[0084] Where γ represents the attenuation factor and its value ranges from 0.96 to 0.99, Q(s) i ,a i |θ Q ) represents the state s under observation. i The current action a is executed in the actor network. i The Q-value, θ, output by any subsequent critic network Q These are the current network parameters of the critic network. Q(s) i ',a i '|θ Q ') represents the state s under observation. i'The target network of the next actor executes action a' i 'The value θ corresponds to the Q-value output of the critic target network. Q 'Supposes the current network parameters of the target network and the observation state s'. i 'Execute the current action a for the actor network' i The observation status after environmental feedback.

[0085] Step S400: Select a number of samples according to the sample weights to train the agent and update the network parameters of the agent. The larger the sample weight, the greater the probability of it being selected.

[0086] The network parameters are updated and trained by randomly selecting samples from the experience cache based on the sample weights. The larger the sample weight, the greater the probability of it being selected. Compared with equal probability sampling, sampling based on sample weights can sample data that is more valuable, difficult to train, and requires multiple training sessions, so that the neural network can learn the features of the data better.

[0087] When updating network parameters, you can update the parameters of the critic network and then update the parameters of the actor network after updating the critic network t times.

[0088] Specifically, each time the critic network is updated, N sets of data are retrieved from the empirical cache based on the weights of each sample to calculate the loss function of the critic network, and the parameters of the critic network are updated using the gradient of the loss function. The process is shown in the following formula:

[0089]

[0090] Where, ω i The importance sampling weights are represented by β, N represents the number of samples drawn, and β represents the hyperparameter. i ',a i '|θ Q ') represents the Q-value of the target network, Q(s,a|θ) Q ) represents the Q-value of the critic network, min 1,2 This means selecting the smaller Q value from the two networks to address the problem of Q overestimation.

[0091] After updating the parameters of the critic network t times, the actor network is updated once using the following formula to achieve a delayed update strategy:

[0092]

[0093] Simultaneously update the parameters of the target network using the following formula:

[0094]

[0095] Here, τ is a weight parameter that makes the parameter update of the target network a soft update.

[0096] By following the steps described above, a round of updates to the overall network parameters of the agent can be completed.

[0097] Updates to agent networks typically involve multiple rounds; therefore, in one embodiment, such as Figure 3 As shown, it also includes the following steps:

[0098] Step S500: Determine whether the number of inner loop iterations meets the requirements. If not, re-initialize the initial actions of the agent and jump to step S200 to execute the inner loop from step S200 to step S500 until the number of inner loop iterations reaches the first preset value and then exit the inner loop.

[0099] Step S600: Determine whether the number of outer loops meets the requirements. If not, reinitialize the observation state of the environment and the initial action of the agent, and then jump to step S200 to execute the outer loop from step S200 to step S600 until the number of outer loops reaches the second preset value and then exit the outer loop.

[0100] In other words, the inner loop repeatedly executes the initial actions of the agent up to step S400 each time the observation state of the environment is initialized. This means that after each environment initialization, the inner loop updates the agent's parameters multiple times. The outer loop also repeatedly executes the initialization of the environment's observation state and completes the inner loop after each initialization. Each outer loop contains multiple inner loops to achieve more rounds of parameter updates. Ultimately, the trained agent, i.e., the unmanned surface vessel obstacle avoidance module, is obtained.

[0101] Accordingly, the present invention also relates to an unmanned surface vessel (USV) obstacle avoidance module training device, which corresponds to the USV obstacle avoidance module training method described above and can be used to execute the USV obstacle avoidance module training method described above. Specifically, as follows... Figure 4 As shown, the unmanned surface vessel obstacle avoidance module training device can be divided into at least the following four functional modules, namely:

[0102] The initialization module is used to initialize the observation states of the various networks and environment of the agent and the initial actions of the agent.

[0103] The sample generation module is used to obtain the observed state s of the agent at each time step. i Next, execute action a i The observed state s at the next moment after environmental feedback i 'and reward r i , with (s i ,a i ,ri ,s i The sample at each time step is stored in the experience cache, where the reward r is... i It includes a distance bonus based on the current position of the unmanned surface vessel (USV) relative to the target position, and a safety bonus based on the degree of safety between the USV and obstacles.

[0104] The sample weight calculation module is used to calculate the sample weight of each sample after a preset number of samples are stored in the experience cache. The sample weight is proportional to the sum of the squared residuals of the critic network and the absolute value of the loss of the actor network.

[0105] The training module is used to select a number of samples based on the sample weights to train the agent and update the network parameters of the agent. The larger the sample weight, the greater the probability of it being selected.

[0106] Parameters are typically updated in multiple rounds; therefore, in one embodiment, it further includes:

[0107] The loop control module is used to control the number of times the initial action of the agent is repeatedly executed to take out a number of samples to train the agent and update the network parameters of the agent to meet the preset requirements.

[0108] The external loop control module is used to control the repeated execution of the observation state of the initialization environment to ensure that the number of times the external loop completes the internal loop meets the preset requirements.

[0109] Since this unmanned surface vessel obstacle avoidance module training device corresponds to the unmanned surface vessel obstacle avoidance module training method described above, the specific functions of each module are adapted to the corresponding method steps described above, and will not be repeated here.

[0110] Accordingly, the present invention also relates to a method for exploring unknown areas by unmanned surface vessels, which mainly includes:

[0111] The next target location is determined, the current environmental observation status is collected and input into the unmanned surface vessel obstacle avoidance module, the corresponding actions are obtained and the unmanned surface vessel is driven to move. The unmanned surface vessel obstacle avoidance module is obtained according to the unmanned surface vessel obstacle avoidance module training method described above.

[0112] Specifically, once the next target location is determined and the observation status of the environment in which the unmanned surface vessel is located is collected, the data is input into the obstacle avoidance module of the unmanned surface vessel, namely the TD3 intelligent agent, which can then decide on the next action.

[0113] In one embodiment, when exploring unknown areas at sea, a swarm of unmanned surface vessels (USVs) is typically formed, and a communication network is established between the USVs to ensure smooth communication between them. Figure 5As shown, the specific exploration process includes: first, dividing each unmanned surface vessel into Voronoi partitions; if there is no information node at the current position, then placing an information node; searching for the boundary point of the next time step in the Voronoi partition and updating the local and global information node networks; then, each unmanned surface vessel takes action to reach the boundary point of the next time step; repeating the above process, if there is no boundary point for each information node, then the exploration is complete.

[0114] Specifically, the process can be divided into the following steps:

[0115] The Voronoi partitioning method dynamically divides the corresponding exploration area for each unmanned surface vessel.

[0116] Each unmanned surface vessel (USV) determines whether its current location is already an information node. If so, it means it has been detected and will not be detected. Instead, it searches for the next boundary point (i.e., the next target location) in the corresponding Voronoi partition. If not, it sets its current location as an information node to indicate that it has been detected and broadcasts this information to all USVs. It then performs its detection work and searches for the next boundary point (i.e., the next target location) in the corresponding Voronoi partition.

[0117] Each unmanned surface vessel moves to the next boundary point based on the output action decision until the search is completed.

[0118] Each unmanned surface vessel (USV) determines its next target location based on a calculation formula for a coordinated exploration strategy. The formula for this coordinated exploration strategy is as follows:

[0119]

[0120] Make the distance potential Ω in The boundary node corresponding to the minimum value is taken as the next target position of the i-th unmanned surface vessel;

[0121] Where λ is a pre-defined scalar coefficient; if it is 0, it indicates that the unmanned surface vessel (USV) adopts a breadth-first strategy; if it is 1, it indicates that the USV adopts a depth-first strategy; if it is between 0 and 1, it indicates a balance between the two strategies. in This represents the distance between the current position of the i-th unmanned surface vessel and the n-th boundary node. This represents the distance between the first target position and the nth boundary point of the i-th unmanned surface vessel (USV). Each boundary node is at the same distance from the current position of the i-th USV.

[0122] In summary, this invention improves upon the existing TD3 algorithm. Before randomly selecting samples, it calculates the sample weight of each sample in the experience cache. The sample weight reflects the importance of the corresponding sample, and is correlated with the sampling probability. The higher the sample weight, the higher the training value of the corresponding sample, the greater the probability of it being selected for training, and the better the training effect, resulting in a high-accuracy obstacle avoidance module for unmanned surface vessels. Furthermore, by optimizing the reward function, the exploration speed can also be improved.

[0123] Those skilled in the art will readily understand that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method for an unmanned surface vessel obstacle avoidance module, characterized in that, The unmanned surface vessel obstacle avoidance module has an agent based on the TD3 algorithm, and the training method includes: Step S1: Initialize the observation states of each network and environment of the agent and the initial actions of the agent; Step S2: Obtain the agent's current observation state Execute the current action The observation state at the next moment after environmental feedback and rewards ,by The current sample is stored in the experience cache, where the reward... It includes a distance bonus based on the current position of the unmanned surface vessel (USV) relative to the target position, and a safety bonus based on the degree of safety between the USV and obstacles. Step S3: After a preset number of samples are stored in the experience cache, the sample weight of each sample is calculated. The sample weight is proportional to the sum of the squared residuals of the critic network and the absolute value of the loss of the actor network. Step S4: Select a number of samples according to the sample weights to train the agent and update the network parameters of the agent. The larger the sample weight, the greater the probability of it being selected. The formula for calculating the sample weight of each sample is: in, This represents the probability before normalization. Represents the residual of any critic network. This means taking the larger of the squared residuals from the agent's two critic networks. This represents the loss of the actor network, where, This indicates the current observation state. The next actor network executes the current action. The Q-value output by any subsequent critic network. These are the current network parameters of the critic network. Represents the gradient operator, This means taking the larger of the two absolute values ​​of the loss in the actor network. These are empirical parameters for weighting, which are orders of magnitude between residuals and losses. and These are the replenishment probability and balance factor, determined experimentally. For positive numbers less than 0.3 A positive number less than 1, where k represents the number of samples in the current experience cache. This represents the sample weights after normalization.

2. The training method for the obstacle avoidance module of an unmanned surface vessel as described in claim 1, characterized in that, Also includes: Step S5: Determine whether the number of inner loop iterations meets the requirements. If not, re-initialize the initial actions of the agent and jump to step S2 to execute the inner loop from step S2 to step S5 until the number of inner loop iterations reaches the first preset value and then exit the inner loop. Step S6: Determine whether the number of outer loop iterations meets the requirements. If not, reinitialize the observation state of the environment and the initial actions of the agent, then jump to step S2 to execute the outer loop from step S2 to step S6 until the number of outer loop iterations reaches the second preset value and then exit the outer loop.

3. The training method for the obstacle avoidance module of an unmanned surface vessel as described in claim 1, characterized in that, award It also includes one or more of the following: angular velocity bonus, linear velocity bonus, and time bonus; among which, The angular velocity reward reflects the current angular velocity of the unmanned surface vessel. The linear velocity reward reflects the current linear velocity of the unmanned surface vessel; The time reward reflects the length of time it takes for the unmanned surface vessel to reach the target location.

4. The training method for the obstacle avoidance module of an unmanned surface vessel as described in claim 1, characterized in that, Residuals of any critic network The calculation formula is: in, This represents the attenuation factor, and its value ranges from 0.96 to 0.

99. In the observation state The target network of the next actor performs the action. The Q-value corresponding to the critic target network output is then... The current network parameters of the target network in the critic section; In the observation state The next actor network executes the current action. The Q-value corresponding to the output of the critic network is then... These are the current network parameters for the critic network.

5. The training method for the obstacle avoidance module of an unmanned surface vessel as described in claim 1, characterized in that, The agent is accompanied by noise. ,noise The calculation formula is: in, This indicates that the mean is 0 and the variance is 0. Gaussian noise, This is the noise limit value determined based on experimental results.

6. A training device for an unmanned surface vessel obstacle avoidance module, characterized in that, The unmanned surface vessel obstacle avoidance module has a TD3 intelligent agent, and the training device includes: The initialization module is used to initialize the observation states of the various networks and environment of the agent and the initial actions of the agent. The sample generation module is used to obtain the observed state of the agent at each time step. Next action The observation state at the next moment after environmental feedback and rewards ,by The reward is stored as a sample at each moment in the experience cache. It includes a distance bonus based on the current position of the unmanned surface vessel (USV) relative to the target position, and a safety bonus based on the degree of safety between the USV and obstacles. The sample weight calculation module is used to calculate the sample weight of each sample after a preset number of samples are stored in the experience cache. The sample weight is proportional to the sum of the squared residuals of the critic network and the absolute value of the loss of the actor network. The training module is used to select a number of samples based on the sample weights to train the agent and update the network parameters of the agent. The larger the sample weight, the greater the probability of it being selected. The formula for calculating the sample weight of each sample is: in, This represents the probability before normalization. Represents the residual of any critic network. This means taking the larger of the squared residuals from the agent's two critic networks. This represents the loss of the actor network, where, This indicates the current observation state. The next actor network executes the current action. The Q-value output by any subsequent critic network. These are the current network parameters of the critic network. Represents the gradient operator, This means taking the larger of the two absolute values ​​of the loss in the actor network. These are empirical parameters for weighting, which are orders of magnitude between residuals and losses. and These are the replenishment probability and balance factor determined experimentally. For positive numbers less than 0.3 A positive number less than 1, where k represents the number of samples in the current experience cache. This represents the sample weights after normalization.

7. A method for exploring unknown areas using unmanned surface vessels, characterized in that, include: Determine the next target location to be reached, collect the current environmental observation status and input it into the unmanned surface vessel obstacle avoidance module, obtain the corresponding actions and drive the unmanned surface vessel to move; The unmanned surface vessel obstacle avoidance module is obtained by the unmanned surface vessel obstacle avoidance module training method according to any one of claims 1 to 5.

8. The method for exploring unknown areas by unmanned surface vessels as described in claim 7, characterized in that, Multiple unmanned surface vessels (USVs) form a swarm, with each USV communicating with the others. The exploration method also includes: The Voronoi partitioning method dynamically divides the corresponding exploration area for each unmanned surface vessel. Each unmanned surface vessel (USV) determines whether its current location has been detected. If so, it does not perform detection and moves directly to the next target location. If not, it performs detection before moving to the next target location, and marks its current location as detected and broadcasts it to all USVs. Each unmanned surface vessel (USV) determines its next target location based on a calculation formula for a coordinated exploration strategy. The calculation formula for this coordinated exploration strategy is as follows: Make distance potential The boundary node corresponding to the minimum value is taken as the next target position of the i-th unmanned surface vessel; in, These are pre-set scalar coefficients. This represents the distance between the current position of the i-th unmanned surface vessel and the n-th boundary node. This represents the distance between the first target position and the nth boundary point of the i-th unmanned surface vessel (USV). Each boundary node is at the same distance from the current position of the i-th USV.

Images