A near-end strategy optimization method for path tracking of unmanned surface vessels based on priority experience replay, computer equipment and storage media

Abstract

This invention relates to the field of unmanned surface vessel (USV) path tracking control technology, specifically disclosing a method, computer equipment, and storage medium for path tracking of USVs based on prioritizing experience replay optimization of near-end policy. The method includes setting up a prioritizing experience replay memory pool, an action space, a state space, and a reward function; constructing a network model based on a near-end policy optimization algorithm, and training the network model using samples from the prioritizing experience replay memory pool; generating current state information based on images acquired by the USV, the current position information of the USV, and the position information of matching points on the reference trajectory; and prioritizing the training of samples with poor action performance based on priority sampling data. This invention employs a prioritizing experience replay memory pool to ensure rapid model convergence, and the network model is constructed based on near-end policy optimization. The use of a state space and reward function tailored to environmental conditions improves the algorithm's adaptability and stability to the environment.

Description

Technical Field

[0001] This invention belongs to the field of unmanned surface vessel path tracking and control technology, specifically involving a near-end strategy optimization method for unmanned surface vessels based on priority experience playback, computer equipment, and storage medium. Background Technology

[0002] With the development of technology, more and more intelligent unmanned systems are appearing in people's field of vision. Unmanned vehicles and drones can complete many land and air tasks, and on the water, unmanned surface vessels (USVs) have always been a research hotspot. Depending on different needs, USVs can complete different types of tasks such as cruising, data collection, and detection. In particular, some high-risk operations require the participation of USVs. Compared with manual operations, USVs have the characteristics of low cost, strong adaptability, and low risk, so more and more water surface operations are choosing to be completed using USVs.

[0003] Path tracking of unmanned surface vessels (USVs) has always been a challenging research area. Compared to land and air environments, the water environment is more complex and contains more unknowns, posing greater challenges to the stability and environmental adaptability of algorithms. Existing algorithms, due to the sparse rewards of USVs, typically require large datasets to converge and exhibit low stability. Therefore, improving the convergence speed and stability of the algorithms is an urgent problem to be solved. Summary of the Invention

[0004] In view of the problems and shortcomings of the existing technology, the purpose of this invention is to provide a path tracking method for unmanned surface vessels based on priority experience playback and a computer device and storage medium.

[0005] To achieve the objectives of this invention, the technical solution adopted is as follows:

[0006] The first aspect of this invention provides a method for optimizing the path tracking of unmanned surface vessels (USVs) based on priority experience replay, comprising the following steps:

[0007] S1: Set up a priority experience replay memory pool, action space, state space and reward function, collect the state, action and reward of the unmanned surface vessel during path tracking, and obtain samples of the unmanned surface vessel in the simulation environment;

[0008] S2: Store the samples in the priority experience replay memory pool according to priority;

[0009] S3: Build a network model based on the near-end policy optimization algorithm, and train the network model using samples in the priority experience replay memory pool;

[0010] S4: Input the trained network model into the computer equipment of the unmanned surface vessel;

[0011] S5: Transmit the mission command to the unmanned surface vessel equipped with the network model;

[0012] S6: Based on the mission command, obtain the image captured by the camera of the unmanned surface vessel at the current moment, the position information of the unmanned surface vessel at the current moment, and the reference trajectory of the unmanned surface vessel;

[0013] S7: Using the image, the unmanned surface vessel's position information, and the matching point position information on the reference trajectory corresponding to the current position, obtain the current state of the unmanned surface vessel;

[0014] S8: Input the current state of the unmanned surface vessel into the network model, and the network model outputs the action to be executed to the unmanned surface vessel;

[0015] S9: The unmanned surface vessel executes the output action of the network model;

[0016] S10: Repeat steps S6 to S9 until the tracking control of the reference trajectory is completed.

[0017] According to the above-described path tracking method for unmanned surface vessels based on priority experience replay, preferably, the state space in step S1 is represented as follows:

[0018] s t ={ima.ge t L t b t}

[0019] In the formula s t Representing the state space, ima.ge t L represents the image acquired at the current moment. t =(x t y t This indicates the current location information of the unmanned surface vessel. This indicates the position information of the matching point on the reference trajectory corresponding to the current position of the unmanned surface vessel.

[0020] Based on the above-described method for optimizing path tracking of unmanned surface vessels using a near-end strategy based on priority experience replay, preferably, the reward function in step S1 is expressed as:

[0021]

[0022] In the formula, e represents a constant, and x t+1 , The x-coordinates of the unmanned surface vessel and the corresponding matching point on the reference trajectory are represented in sequence, respectively. t+1 , The r represents the ordinate position of the unmanned surface vessel at the next moment, and the ordinate position of the corresponding matching point on the reference trajectory, respectively. b It is a constant.

[0023] According to the above-described path tracking method for unmanned surface vessels (USVs) based on priority experience replay, preferably, the action space in step S1 is:

[0024] a t ={F t δ t}

[0025] In the formula F t Indicates driving force, δ t Indicates the rudder angle.

[0026] According to the above-described method for optimizing the path tracking of unmanned surface vessels (USVs) based on priority experience replay, preferably, the priority of the priority experience replay memory pool in step S1 is represented as follows:

[0027]

[0028] In the formula, β and σ are both constants, and (x0, y0) is the initial position of the unmanned surface vessel.

[0029] Based on the above-described path tracking method for unmanned surface vessels (USVs) optimized by a near-end strategy using priority experience replay, preferably, the objective function of the network model in step S3 is:

[0030]

[0031] in, R(τ)=r1+γr2+γ 2 r2+γ 3 r3+…+γ n-1 r n

[0032] In the formula, θ represents the parameters of the network model, and p θ (a t |s t pθ represents the probability of the current model's action output. k (a t |s t ) represents the action output probability of the model before the update, τ represents a batch of data sampled according to priority, ε = 0.2, b represents the expected total reward, γ = 0.1 represents the discount factor, and k = {1, 2, 3, 4} represents the number of times the model is updated with a batch of data.

[0033] According to the above-described method for optimizing the path tracking of unmanned surface vessels (USVs) based on priority experience replay, preferably, the update process of the network model parameter θ is as follows:

[0034]

[0035] In the formula θ i+1 θ represents the parameters of the updated network model. i λ represents the parameters of the network model before the update, and λ represents the step size factor.

[0036] Based on the above-mentioned near-end strategy optimization method for unmanned surface vessels (USVs) based on priority experience replay, preferably, the specific steps for training the network model in step S3 are as follows:

[0037] S301: Randomly initialize the parameters θ0 of the network model and the position (x0, y0) of the unmanned surface vessel;

[0038] S302: Acquire the image captured by the camera at the current moment, the current position information of the unmanned surface vessel, and the reference trajectory of the unmanned surface vessel;

[0039] S303: Image captured by the unmanned surface vessel's camera at the current moment. t Current location information L t =(x t y t ), and the location information of the matching points on the reference trajectory corresponding to the current location. Get the current state value S t ={image t L t b t};

[0040] S304: Input the current state value into the network model and output action a. t Command the unmanned surface vessel to perform the action value, and return the reward value r after performing the action. t ;

[0041] S305: Image captured by the camera after the action has been performed. t+1 Location information of unmanned surface vessel (USV) L t+1 =(x t+1 y t+1 And the location information of the corresponding matching points on the reference trajectory. Obtain the state value s at the next moment. t+1 ={image t+1 L t+1 b t+1};

[0042] S306: Using the unmanned surface vessel's position information at the next moment L t+1 =(x t+1 y t+1 The position information of the matching point on the reference trajectory corresponding to the next moment's position. Calculate priority pt;

[0043] S307: via priority p t (s) t a t r t s t+1 Stored in the priority experience replay memory pool;

[0044] S308: Update the priority experience replay memory pool;

[0045] S309: Repeat S302 to S308 to obtain data samples;

[0046] S310: According to priority p t N data points are sampled from the priority experience replay memory pool and packaged into the network model to update the network model's parameters.

[0047] A second aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step in the near-end strategy optimization path tracking method for unmanned surface vessels based on priority experience replay as described in the first aspect.

[0048] A third aspect of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a computer processor, implements any step in the near-end strategy optimization method for unmanned surface vessels based on priority experience replay as described in the first aspect.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] This invention calculates the unmanned surface vessel's position information L at the next moment when storing samples. t+1 =(x t+1 y t+1 ), the position information of the matching point on the reference trajectory corresponding to the position at the next moment. Calculate priority p t Each sample has a priority label. The greater the difference between the actual location information and the location information of the matching point, the worse the action effect and the higher the priority. The priority is related to the sampling probability when sampling data. The higher the priority of the sample, the greater the probability that it will be sampled when training the model. By prioritizing the training of samples with poor performance, we can not only improve the convergence speed of the model, but also improve the robustness of the model and enhance the stability and accuracy of the algorithm control. The network model is built based on the proximal policy optimization algorithm and can control continuous actions. By adopting a state space and reward function that are set in combination with environmental conditions, the algorithm's adaptability and stability to the environment are improved. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the method for optimizing the path tracking of unmanned surface vessels based on priority experience replay according to the present invention.

[0052] Figure 2 This is a schematic diagram of the planar motion of an unmanned surface vessel.

[0053] Figure 3 The process of storing samples in the memory pool for priority experience playback. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example 1

[0056] A near-end strategy optimization method for unmanned surface vessels based on priority experience replay, such as... Figure 1 As shown, the specific steps are as follows:

[0057] S1: Set up a priority experience replay memory pool, action space, state space, and reward function to collect the state, actions, and rewards during unmanned surface vessel path tracking;

[0058] S2: Store the samples obtained by the unmanned surface vessel in the simulation environment in the priority experience replay memory pool according to priority;

[0059] S3: Samples in the sample-first experience replay memory pool are used to train the network model based on the near-end policy optimization algorithm;

[0060] S4: Input the trained network model into the computer equipment of the unmanned surface vessel;

[0061] S5: Transmit the mission command to the unmanned surface vessel equipped with the network model;

[0062] S6: Based on the mission command, obtain the image captured by the camera of the unmanned surface vessel at the current moment, the position information of the unmanned surface vessel at the current moment, and the reference trajectory of the unmanned surface vessel;

[0063] S7: Using the image, the unmanned surface vessel's position information, and the matching point position information on the reference trajectory corresponding to the current position, obtain the current state of the unmanned surface vessel;

[0064] S8: Input the current state of the unmanned surface vessel into the network model based on the near-end policy optimization algorithm, and the network model outputs the action to be executed to the unmanned surface vessel;

[0065] S9: The unmanned surface vessel executes the output action of the network model;

[0066] S10: Repeat steps S6 to S9 until the tracking control of the reference trajectory is completed.

[0067] In this embodiment, the network model consists of one input layer, one normalization layer, six hidden layers, and two fully connected layers. The normalization layer is used to standardize parameters at different scales in the state. Each hidden layer has 256 units, the activation function is the sigmoid function, and the Adam gradient descent optimizer is used with a learning rate of 0.01. The model is trained after collecting 2000 samples, with 300 samples extracted for each training iteration.

[0068] Firstly, in the research of unmanned surface vessels (USVs), six-degree-of-freedom USVs are the most widely used. This invention only considers the USV's motion on the water surface, without considering the stability of the cargo it carries. The simplified model is as follows: Figure 2 As shown, the action space in step S1 above is therefore represented as:

[0069] a t ={F t δ t}

[0070] In the formula F t Indicates driving force, δ t Indicates the rudder angle.

[0071] This method is primarily aimed at small unmanned surface vessels (USVs) used for cruising and surveying. These USVs have relatively low propulsion, which can be expressed as power, typically between 0 and 100W, and rudder angles between -45° and +45°. Therefore, in the experiments of this invention, F... t ∈[0, 100], δ t ∈[-45, 45]. This invention employs a near-end policy optimization algorithm, which can handle continuous action values ​​without discretizing the action space. The distribution of actions is established through the probability output by the policy model, and the execution actions of the unmanned surface vessel can be obtained by sampling the action distribution according to the probability.

[0072] Secondly, this invention conducts simulation experiments on the Unity 3D platform. The Wave Package plugin in Unity 3D simulates ocean waves, and obstacles are set to represent reefs, ships, and small islands. The environment on water is much more complex than on land or in the air; therefore, the state space of the unmanned surface vessel needs to consider multiple factors. The state space in step S1 is represented as follows:

[0073] s t ={image t L t b t}

[0074] In the formula, image tThis represents the image acquired at the current moment. In the experiment, the camera was set to capture images at a frequency of 30 frames per second. L t =(x t y t This indicates the current location information of the unmanned surface vessel. This indicates the location information of the matching point on the reference trajectory corresponding to the current position of the unmanned surface vessel.

[0075] Images acquired by cameras on unmanned surface vessels (USVs) contain information such as water waves, surrounding obstacles, and target point locations. Designing these images in the state space allows these factors to be taken into account, improving the accuracy of the USV's path tracking control algorithm and its adaptability to the environment. The current position information of the USV and the position information of the matching point on the reference trajectory can make the state space better reflect whether the USV has deviated from the reference trajectory.

[0076] The reward function is involved in the network model's decision-making and convergence; therefore, the reward function r in step S1 of this invention... t Represented as:

[0077]

[0078] In the formula, e represents a constant, and x t+1 , The x-coordinates of the unmanned surface vessel and the corresponding matching point on the reference trajectory are represented in sequence, respectively. t+1 , The r represents the ordinate position of the unmanned surface vessel at the next moment, and the ordinate position of the corresponding matching point on the reference trajectory, respectively. b It is a constant.

[0079] In this embodiment r b We set 0.2 as the baseline for the reward function to ensure that the reward values ​​are not all positive.

[0080] The reference trajectory is the desired motion path for the unmanned surface vessel (USV). The reward function of this invention is designed by combining the USV's position information with the position information of matching points on the reference trajectory. The greater the difference between the USV's position information and the position information of the matching points on the reference trajectory, the smaller the reward is, and vice versa. This ensures that the network model's output tends to move along the direction of the reference trajectory. b Ensuring that the reward values ​​are not all positive can improve the model's learning ability.

[0081] The memory pool of this invention is designed based on the idea of ​​prioritizing experience playback, and stores samples using the unmanned surface vessel's next-moment position information L. t+1 =(x t+1 y t+1 ), the location information of the corresponding matching points on the reference trajectory Calculate priority p t Each sample has a priority label. The greater the difference between the actual location information and the location information of the matching point, the worse the effect of the action, and the higher the priority. The priority is related to the sampling probability when sampling data. The higher the priority of the sample, the greater the probability of the sample being sampled when training the model. By prioritizing the training of samples with poor performance, we can not only improve the convergence speed of the model, but also improve the robustness of the model and enhance the stability and accuracy of the algorithm control.

[0082] In step S1 of this embodiment, the priority calculation for the priority experience playback memory pool is set as follows:

[0083]

[0084] In the formula, β and σ are both constants, and (x0, y0) is the initial position of the unmanned surface vessel. In this embodiment, β is taken as 0.1 and σ is taken as 100.

[0085] In this embodiment, the reason for using the priority experience replay technique without setting a correction coefficient is that the objective function of the near-end policy optimization algorithm already considers the error caused by the change in distribution, so there is no need to set a separate correction coefficient. The next moment's position information L of the unmanned surface vessel. t+1 =(x t+1 y t+1 The position information of the matching point on the reference trajectory The larger the gap, the higher the calculated p. t The larger the value of β, the higher the priority of the sample, and the higher the probability that the sample will be sampled during data collection. β is used to ensure that p t Not zero.

[0086] The process of storing samples in the priority experience replay memory pool is as follows: Figure 3 As shown, during the training process, the next moment's position information L of the unmanned surface vessel is used... t+1 =(x t+1 y t+1 ), the location information of the corresponding matching points on the reference trajectory Calculate priority p t The current state, the action performed in the current state, the reward after performing the action in the current state, and the state at the next moment are stored in the priority experience replay memory pool according to priority.

[0087] The reference trajectory for unmanned surface vessel path tracking obtained in step S6 above based on the curve interpolation method can be expressed as:

[0088] y = c5x 5 +c4x 4 +c3x 3+c2x 2 +c1x+c0

[0089] In the formula, c0, c1, c2, c3, c4, and c5 are all constants.

[0090] In step S5 above, the objective function of the network model constructed based on the near-end policy optimization algorithm is expressed as:

[0091]

[0092] in, R(τ)=r1+γr2+γ 2 r2+γ 3 r3+…+γ n-1 r n

[0093] In the formula, θ represents the parameters of the network model, and p θ (a t |s t ) represents the probability of the current model's action output. Let τ represent the probability of the action output of the model before the update, ε = 0.2, b represent the expected total reward, γ = 0.1 represent the discount factor, and k = {1, 2, 3, 4} represent the number of times the model is updated with a batch of data.

[0094] The parameter update process of the network model is as follows:

[0095] In the formula θ i+1 θ represents the parameters of the updated network model. i This represents the parameters of the network model before the update, and λ represents the step size factor. In this embodiment, λ = 0.01.

[0096] In step S3, after constructing the network model, the specific steps for training it are as follows:

[0097] S301: Randomly initialize the parameters θ0 of the network model and the position (x0, y0) of the unmanned surface vessel;

[0098] S302: Acquire the image captured by the camera at the current moment, the current position information of the unmanned surface vessel, and the reference trajectory of the unmanned surface vessel;

[0099] S303: Image captured by the unmanned surface vessel's camera at the current moment. t Current location information L t =(x t y t ), and the location information of the matching points on the reference trajectory corresponding to the current location. Obtain the current state value s of the unmanned surface vessel.t ={image t L t b t};

[0100] S304: Input the current state value of the unmanned surface vessel into the network model and output action a. t Command the unmanned surface vessel to perform the action value, and return the reward value r after performing the action. t ;

[0101] S305: Image captured by the camera after the action has been performed. t+1 Location information of unmanned surface vessel (USV) L t+1 =(x t+1 y t+1 And the location information of the corresponding matching points on the reference trajectory. Obtain the state value s at the next moment. t+1 ={image t+1 L t+1 b t+1};

[0102] S306: Utilizing the unmanned surface vessel's position information at the next moment L t+1 =(x t+1 y t+1 The position information of the matching point on the reference trajectory corresponding to the next moment's position. Calculate priority pt;

[0103] S307: via priority p t (s) t a t r t s t+1 Stored in the priority experience replay memory pool;

[0104] S308: Update the priority experience replay memory pool;

[0105] S309: Repeat steps S302 to S308 to obtain 2000 data samples;

[0106] S310: Retrieve from the priority experience replay memory pool according to priority p t 300 data points were sampled and packaged into a batch, which was then input into the network model to update the model's parameters.

[0107] Example 2

[0108] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step in a proximal strategy optimization method for unmanned surface vessels based on priority experience replay as described in Embodiment 1.

[0109] Furthermore, the method for path tracking of unmanned surface vessels (USVs) based on priority experience replay, as described in Embodiment 1, can be implemented as a computer software program. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, it performs the functions defined in the method of this application.

[0110] Example 3

[0111] A computer-readable storage medium storing a computer program that, when executed by a processor, implements any step of a method for path tracking of unmanned surface vessels based on a proximity strategy optimized by prioritizing experience replay, as described in Embodiment 1.

[0112] The computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0113] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] In summary, this invention effectively overcomes the shortcomings of the prior art and has high industrial applicability. The above embodiments are intended to illustrate the substantive content of this invention, but are not intended to limit the scope of protection of this invention. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this invention without departing from the essence and scope of protection of this invention.

Claims

1. A path tracking method for unmanned surface vessels (USVs) based on priority experience replay optimization, characterized in that, Includes the following steps: S1: Set up a priority experience replay memory pool, action space, state space and reward function, collect the state, action and reward of the unmanned surface vessel during path tracking, and obtain samples of the unmanned surface vessel in the simulation environment; The state space is represented as follows: = { ， ， } In the formula Representing the state space, This represents the image acquired at the current moment. =( , This indicates the current location information of the unmanned surface vessel. =( , This indicates the position information of the matching point on the reference trajectory corresponding to the current position of the unmanned surface vessel; The reward function is expressed as follows: In the formula Represents a constant. , These represent the x-coordinate position of the unmanned surface vessel at the next moment, and the x-coordinate position of the corresponding matching point on the reference trajectory, respectively. These represent the ordinate position of the unmanned surface vessel at the next moment, and the ordinate position of the corresponding matching point on the reference trajectory, respectively. It is a constant; The action space is: ={ ， } In the formula Indicates driving force. Represents the rudder angle; S2: Store the samples in the priority experience replay memory pool according to priority; The priority of the priority experience replay memory pool is represented as follows: In the formula β and All are constants, , ( ) represents the initial position of the unmanned surface vessel. , These represent the x-coordinate position of the unmanned surface vessel at the next moment, and the x-coordinate position of the corresponding matching point on the reference trajectory, respectively. The coordinates of the unmanned surface vessel (USV) and the coordinates of the matching point on the reference trajectory are respectively represented at the next moment. S3: Establish a network model based on the near-end policy optimization algorithm, and train the network model using samples in the priority experience replay memory pool; S4: Input the trained network model into the computer equipment of the unmanned surface vessel; S5: Transmit the mission command to the unmanned surface vessel equipped with the network model; S6: Based on the mission command, obtain the image captured by the camera of the unmanned surface vessel at the current moment, the position information of the unmanned surface vessel at the current moment, and the reference trajectory of the unmanned surface vessel; S7: Using the image, the unmanned surface vessel's position information, and the matching point position information on the reference trajectory corresponding to the current position, obtain the current state of the unmanned surface vessel; S8: Input the current state of the unmanned surface vessel into the network model, and the network model outputs the action to be executed to the unmanned surface vessel; S9: The unmanned surface vessel executes the output action of the network model; S10: Repeat steps S6 to S9 until the tracking control of the reference trajectory is completed.

2. The method for path tracking of unmanned surface vessels based on priority experience replay according to any one of claims 1, characterized in that, The objective function of the network model described in step S3 is: in, , + In the formula The parameters representing the network model, This represents the probability of the current model's action output. This represents the probability of the model's action output before the update. This indicates a batch of data sampled according to priority. b represents the expected total reward. Let k represent the discount factor, and k = {1, 2, 3, 4} represent the number of times the model is updated with a batch of data.

3. The method for path tracking of unmanned surface vessels based on priority experience replay according to claim 2, characterized in that, The parameters of the network model The update process is as follows: In the formula This represents the parameters of the updated network model. This represents the parameters of the network model before the update. This represents the step size factor.

4. The method for path tracking of unmanned surface vessels based on priority experience replay according to claim 2, characterized in that, The specific steps for training the network model described in step S3 are as follows: S301: Randomly initialize the parameters of the network model and the location of the unmanned surface vessel ( , ); S302: Acquire the image captured by the camera at the current moment, the current position information of the unmanned surface vessel, and the reference trajectory of the unmanned surface vessel; S303: Image captured by the unmanned surface vessel's camera at the current moment. Current location information =( , ), and the location information of the matching points on the reference trajectory corresponding to the current location. =( , Get the current state value ={ , , }; S304: Input the current state value into the network model and output the action. Command the unmanned surface vessel to perform the action value, and return the reward value after performing the action. ; S305: An image captured by the camera after the action has been performed. Location information of unmanned surface vessels =( , And the location information of the corresponding matching points on the reference trajectory. =( , Obtain the state value at the next moment. ={ , , }; S306: Using the unmanned surface vessel's position information at the next moment =( , The position information of the matching point on the reference trajectory corresponding to the next moment's position. =( , ) Calculation priority ; S307: Priority Will( , , Stored in the priority experience replay memory pool; S308: Update the priority experience replay memory pool; S309: Repeat S302 to S308 to obtain data samples; S310: Based on priority N data points are sampled from the priority experience replay memory pool and packaged into the network model to update the network model's parameters.

5. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements any step in the near-end strategy optimization method for unmanned surface vessels based on priority experience replay as described in any of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer processor, implements any step in the near-end strategy optimization path tracking method for unmanned surface vessels based on priority experience replay as described in any of claims 1-4.

Images