Method and device for optimizing tracking trajectory of unmanned ship, medium and product

Abstract

The invention discloses an optimization method and device for an unmanned ship tracking trajectory, a medium and a product, and the method comprises the steps: inputting a state quantity at a current moment into an unmanned ship trajectory tracking network model, and obtaining an action at a next moment; according to the state quantity of the next moment and a preset condition, determining a tracking trajectory of the unmanned ship in the whole tracking continuous process; according to the duration of the whole tracking duration process, the number of actions executed by the unmanned ship and the empirical data corresponding to each action, local rewards in each group of empirical data are updated, and target empirical data are obtained; based on a priority experience playback algorithm, sampling the target experience data to obtain a plurality of training samples; training the unmanned ship trajectory tracking network model according to the plurality of training samples to obtain a trained unmanned ship trajectory tracking network model; the tracking trajectory is optimized according to the trained unmanned ship trajectory tracking network model, the optimized tracking trajectory is determined, and the tracking precision and efficiency of the unmanned ship are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning technology, and in particular to an optimization method, device, medium and product for tracking trajectories of unmanned boats. Background Art

[0002] With the advancement of science and technology and the increase in military needs, unmanned boats are increasingly used in maritime patrols and military operations. Due to their advantage of being free from human intervention, unmanned boats can perform a variety of tasks in complex environments, including tracking and patrolling ships at sea and recovering submarines. In order to achieve efficient and precise control of unmanned boats in these tasks, trajectory tracking technology has become one of the key technologies.

[0003] Traditional target tracking methods usually rely on fixed control algorithms and preset target models, which are often incapable of handling dynamic environments and complex targets, which in turn affects the tracking trajectory of the unmanned boat and leads to low tracking accuracy and efficiency. Summary of the invention

[0004] The purpose of this application is to provide a method, device, medium and product for optimizing the tracking trajectory of an unmanned boat, so as to solve the problem that the tracking trajectory of the unmanned boat is low in accuracy, resulting in low tracking accuracy and efficiency of the unmanned boat.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for optimizing the tracking trajectory of an unmanned boat, comprising:

[0007] Obtain the state quantity of the unmanned boat at the current moment; the state quantity includes the current position of the unmanned boat, the current speed of the unmanned boat, the current position of the dynamic target, the current speed of the dynamic target, the current Euclidean distance between the unmanned boat and the dynamic target, the current heading angle of the unmanned boat and the current angle between the line connecting the unmanned boat and the dynamic target, and the current control quantity of the two propeller motors of the unmanned boat; the current control quantity of the two propeller motors of the unmanned boat is used as an action to drive the unmanned boat to move;

[0008] Input the current state into the unmanned boat trajectory tracking network model to obtain the action at the next moment;

[0009] Execute the action at the next moment, determine the state quantity at the next moment and the local reward for executing the action;

[0010] Determine the tracking trajectory of the unmanned boat during the entire tracking process according to the state quantity at the next moment and the preset conditions; the entire tracking process is from starting to track the dynamic target to stopping tracking the dynamic target;

[0011] According to the duration of the entire tracking process, the number of times the unmanned boat performs actions and the experience data corresponding to each action, the local reward in each set of experience data is updated to obtain target experience data; the experience data includes the action at the current moment, the state quantity at the next moment, the state quantity at the current moment and the local reward;

[0012] Based on the priority experience replay algorithm, the target experience data is sampled to obtain multiple training samples;

[0013] Training the unmanned boat trajectory tracking network model according to the multiple training samples to obtain a trained unmanned boat trajectory tracking network model;

[0014] The tracking trajectory is optimized according to the trained unmanned boat trajectory tracking network model to determine the optimized tracking trajectory.

[0015] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for optimizing the tracking trajectory of an unmanned boat.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for optimizing the tracking trajectory of an unmanned boat.

[0017] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for optimizing the tracking trajectory of an unmanned boat.

[0018] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0019] The present invention first obtains the state quantity of the unmanned boat at the current moment, inputs the unmanned boat trajectory tracking network model to obtain the action at the next moment, executes the action at the next moment to determine the state quantity at the next moment and the local reward of the action, and then obtains multiple sets of experience data in the whole process of the unmanned boat tracking the dynamic target, and for the local reward in each set of experience data, uses the duration of the unmanned boat tracking the dynamic target, the number of times the unmanned boat performs the action and the experience data corresponding to each action to update, so as to improve the accuracy of the local reward in each set of experience data. Then, the unmanned boat trajectory tracking network model is trained by sampling the target experience data with multiple training samples obtained by the priority experience replay algorithm, so that the unmanned boat trajectory tracking network model can learn useful information more efficiently during the training process, thereby improving the overall learning effect. The unmanned boat trajectory tracking network model after training is rapidly converged and has strong stability because of the priority experience replay algorithm. Therefore, the action at the next moment output by the trained unmanned boat trajectory tracking network model is more accurate, and then the accuracy of the optimized tracking trajectory determined based on the trained unmanned boat trajectory tracking network model is greatly improved, thereby effectively improving the tracking accuracy and efficiency of the unmanned boat. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic flow chart of a method for optimizing the tracking trajectory of an unmanned boat provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0024] like Figure 1 As shown, the present application provides a method for optimizing the tracking trajectory of an unmanned boat, comprising:

[0025] Step 101: Obtain the state quantity of the unmanned boat at the current moment; the state quantity includes the current position of the unmanned boat, the current speed of the unmanned boat, the current position of the dynamic target, the current speed of the dynamic target, the current Euclidean distance between the unmanned boat and the dynamic target, the current heading angle of the unmanned boat and the current angle between the line connecting the unmanned boat and the dynamic target, and the current control quantity of the two propeller motors of the unmanned boat; the current control quantity of the two propeller motors of the unmanned boat is used as an action to drive the unmanned boat to move.

[0026] Among them, the ground radar is used to obtain the real-time position and speed of the unmanned boat and the dynamic target, and the position and speed information is sent to the unmanned boat. The initial position of the unmanned boat and the dynamic target, as well as the movement mode of the dynamic target are set.

[0027] In practical applications, the virtual environment for unmanned boat tracking can be set up to include the model of the unmanned boat, the ocean environment, the model of the dynamic target (submarine / surface boat), etc.

[0028] Among them, the state quantity is s t Indicates that t Includes 12 parameters, specific The position of the unmanned boat in the XYZ direction of the world coordinate system is η u =(x u ,y u ,z u ), dynamic target position η d =(x d ,y d ,z d ), the speed of the unmanned boat is v u , the dynamic target speed is v d , the Euclidean distance between the unmanned boat and the dynamic target is Δl, and the angle between the current head direction of the unmanned boat and the line connecting the unmanned boat and the dynamic target is Control quantity of two propeller motors of unmanned boat Current actions of autonomous boat sports Used to control the unmanned boat to track dynamic targets. Calculate the unmanned boat action in training t When noise is added, the unmanned boat moves a t The calculation formula is as follows:

[0029] a t =μ(s t |θ μ )+∈

[0030] ∈=clip(N(0,σ),-b,b)

[0031] In the formula, s t Represents the state quantity at the current moment; θ μRepresents the policy network parameters; μ() represents the policy network; ∈ represents the random noise parameter; the clip function indicates that when N(0,σ)<-b, ∈=-b, when N(0,σ)>b, ∈=b, otherwise, ∈=N(0,σ); N(0,σ) means that it satisfies the normal distribution, -b and b represent fixed parameters, b>0.

[0032] Step 102: Input the state quantity at the current moment into the unmanned boat trajectory tracking network model to obtain the action at the next moment.

[0033] Among them, the unmanned boat trajectory tracking network model is the improved TwinDelayedDeep Deterministicpolicygradient, that is, the improved TD3, which is a reinforcement learning algorithm neural network.

[0034] In some embodiments, the network structure of the unmanned boat trajectory tracking network model specifically includes: a strategy network, a target strategy network, a value network and a target value network; the strategy network is used to input the state quantity at the current moment and output the action at the current moment; the value network is used to obtain the state quantity at the current moment and the action at the current moment in the strategy network, evaluate the state-action pair at the current moment and output a state value evaluation value; the state-action pair represents the state quantity and action at the same moment; the target strategy network is used to input the state quantity at the next moment and output the action at the next moment; the target value network is used to obtain the state quantity at the next moment and the action at the next moment in the target strategy network, and output the state value target value of the state-action pair at the next moment.

[0035] Among them, the policy network is the Actor network, the target policy network is the TargetActor network, the value network is the Critic1 network and the Critic2 network, and the target value network is the Target Critic1 network and the Target Critic2 network. Although the network architecture of the Actor network and the TargetActor network is the same, there are differences in their inputs and outputs; similarly, the Critic class network and the TargetCritic class network also share the same network architecture, but there are subtle differences between them.

[0036] The Actor network mainly includes 1 input layer, 4 hidden layers (f c 1. f c 2. f c 3. f c 4) and 1 output layer. The input layer contains 12 nodes, corresponding to the state quantity s tThe hidden layer is composed of linear layers, and the four hidden layers contain 1024, 1024, 512 and 256 nodes respectively; the output layer contains 2 nodes, corresponding to action a t The Critic network architecture (i.e., the value network and the target value network) consists of two branches. The first branch is similar to the Actor network, consisting of 1 input layer and 4 hidden layers. The input layer contains 12 nodes, corresponding to the state quantity s t The hidden layer is composed of linear layers, and the four hidden layers contain 1024, 1024, 512 and 256 nodes respectively. The other branch consists of 1 input layer and 2 hidden layers. The input layer is the output layer of the Actor network, that is, action a t , contains 2 nodes, and the two hidden layers contain 512 and 256 nodes respectively. The two branches have the same size of 256-node hidden layer, so the outputs of the two 256-node hidden layers are added and normalized and then output to the output layer. The output layer contains 1 node, which is the evaluation value of the state-action pair or the target value of the state-action pair.

[0037] Step 103: Execute the action at the next moment, determine the state quantity at the next moment and the local reward for executing the action.

[0038] Among them, the local reward function r is calculated based on the state quantity obtained at the next moment st (s t ).

[0039] In some embodiments, executing the action at the next moment, determining the state quantity at the next moment and the local reward for executing the action, specifically includes: executing the action at the next moment, obtaining the state quantity at the next moment; inputting the state quantity at the next moment into the local reward function to obtain the local reward for executing the action; the calculation formula of the local reward function is:

[0040]

[0041] Among them, r st (s t ) is the local reward function; A is the amplitude height; σ is the standard deviation; b is the location parameter; r 2t The reward function for the unmanned boat to track the dynamic target within a preset distance. R is the preset distance; |||| is the binary norm; ω1, ω2, ω3, ω4 and ω5 are hyperparameters; v u is the speed of the unmanned boat; v d is the speed of the dynamic target; Δl is the Euclidean distance between the unmanned boat and the dynamic target; is the angle between the head of the unmanned boat and the line connecting the unmanned boat and the dynamic target; τ ltis the control quantity of the propeller motor of the unmanned boat l at time t; τ rt is the control quantity of the propeller motor of the unmanned boat r at time t; τ lt-1 Control value of propeller motor of unmanned boat l at time t-1; τ rt-1 It is the control quantity of the propeller motor of the unmanned boat r at time t-1.

[0042] Step 104: Determine the tracking trajectory of the unmanned boat during the entire tracking process according to the state quantity at the next moment and the preset conditions; the entire tracking process is from starting to track the dynamic target to stopping tracking the dynamic target.

[0043] In some embodiments, the tracking trajectory of the unmanned boat during the entire tracking process is determined based on the state quantity at the next moment and the preset conditions, specifically including: obtaining the Euclidean distance between the unmanned boat and the dynamic target in the state quantity at the next moment; determining the preset conditions based on the Euclidean distance; the preset conditions include a first preset condition and a second preset condition; the first preset condition is that the Euclidean distance is not greater than the preset distance and the duration of the unmanned boat tracking the dynamic target exceeds the first preset time; the second preset condition is that the Euclidean distance is greater than the preset distance and the duration of the unmanned boat tracking the dynamic target does not exceed the second preset time; the first preset time is less than the second preset time; when the Euclidean distance and the duration of the unmanned boat tracking the dynamic target meet the first preset condition, it is determined that the unmanned boat has tracked the dynamic target, stops tracking, and determines the tracking trajectory; when the Euclidean distance and the duration of the unmanned boat tracking the dynamic target meet the second preset condition, it is determined that the unmanned boat cannot track the dynamic target, stops tracking, and determines the tracking trajectory.

[0044] Among them, K rounds of unmanned boat tracking dynamic targets can be set, each round represents a process from the start of tracking to the stop of tracking by the unmanned boat, and K represents the number.

[0045] For example, the first preset time is 30s and the second preset time is 60s. If the Euclidean distance between the unmanned boat and the dynamic target is less than the preset distance R and lasts for more than 30s, or the Euclidean distance between the two parties is not less than the preset distance R within 60s, the unmanned boat stops tracking, and the current training of the unmanned boat tracking the dynamic target ends.

[0046] Among them, the preset distance is preferably 1 meter.

[0047] Step 105: According to the duration of the entire tracking process, the number of times the unmanned boat performs an action and the experience data corresponding to each action, the local reward in each set of experience data is updated to obtain target experience data; the experience data includes the action at the current moment, the state quantity at the next moment, the state quantity at the current moment and the local reward.

[0048] Among them, multiple sets of experience data collected in the current game are obtained and stored in the experience pool. Assuming that K games need to be trained, multiple sets of experience data collected in K games are obtained and stored in the experience pool.

[0049] In some embodiments, based on the duration of the entire tracking process, the number of times the unmanned boat performs actions and the experience data corresponding to each action, the local reward in each set of experience data is updated to obtain the target experience data, specifically including: when the Euclidean distance and the duration of the unmanned boat tracking the dynamic target meet the second preset condition, it is determined that the unmanned boat obtains a timeout reward, and the local reward in each set of experience data is updated according to the timeout reward; the updating process of the local reward is: obtaining the local reward in any set of experience data; determining the local timeout reward according to the timeout reward and the number of times the unmanned boat performs actions; determining the updated local reward according to the local reward in the experience data and the local timeout reward; replacing the local reward in the experience data with the updated local reward.

[0050] In practical applications, the global reward function r is obtained based on multiple sets of experience data for each game. t (s1,a1,s2,a2,…,s T ).

[0051] Among them, the global reward function r t (s1,a1,s2,a2,…,s T ) is calculated as follows:

[0052]

[0053] Among them, T is the time when the unmanned boat stops tracking, r6 is the timeout reward; the timeout reward function is:

[0054]

[0055] Among them, t is the duration of the entire tracking process of the unmanned boat.

[0056] Specifically, according to the global reward function, the number of actions performed by the unmanned boat and the duration of this round, the local reward function r at each moment (i.e., each action of the unmanned boat) is updated. st , the updated global-local reward function is r t (s i ), which is the updated local reward.

[0057] The global-local reward function is calculated as:

[0058]

[0059] Among them, s represents the total number of actions performed by the unmanned boat during the entire tracking process; is the local timeout reward; ω6 is the hyperparameter; s i is the state quantity at time i.

[0060] According to the global-local reward function r t (s i ), that is, the updated local reward, modifies each set of experience data (s t ,a t ,r t ,s t ').

[0061] Step 106: Based on the priority experience replay algorithm, sample the target experience data to obtain multiple training samples.

[0062] In some embodiments, the calculation formula of the priority experience replay algorithm is:

[0063]

[0064] Among them, P(c) is a training sample (s t ,a t ,r t ,s t '), a t is the action at the current moment, s t ' is the state quantity at the next moment, s t is the state quantity at the current moment, r t is the local reward; c represents the sequence number of the extracted experience data; p k is the priority of the extracted empirical data; α is a preset parameter used to adjust the degree of priority sampling of training samples.

[0065] It is worth noting that the priority experience replay algorithm can reduce the correlation between data and enhance the ability to handle complex events. The specific effects include:

[0066] 1. Improve learning efficiency: By prioritizing experience replaying those that are more challenging or have a greater impact on the learning goal, the model can learn from important experiences faster.

[0067] 2. Accelerate convergence: Prioritizing replaying experience can help the model find the optimal strategy faster, thereby reducing the time required for training.

[0068] 3. Improved policy learning: By focusing on and replaying those experiences with greater impact, the model is able to better adjust its strategy and reduce the impact of negative experiences.

[0069] 4. Handling rare events: For those events that occur less frequently but have a greater impact on the final outcome, prioritizing experience replay can give the model more opportunities to learn.

[0070] 5. Enhanced stability: By selectively replaying important experiences, the volatility during training is reduced, making the learning process more stable.

[0071] In general, the key goal of prioritized experience replay is to enable the model to learn useful information more efficiently during training, thereby improving the overall learning effect.

[0072] Step 107: training the unmanned boat trajectory tracking network model according to the multiple training samples to obtain a trained unmanned boat trajectory tracking network model.

[0073] Among them, to determine whether the unmanned boat trajectory tracking network model is well trained, the number of rounds of the unmanned boat tracking dynamic target can be set to reach a predetermined stable cycle number M, and then the number of rounds-reward function curve graph can be checked. If the reward function has stabilized at cycle M, the cycle is terminated, the unmanned boat target trajectory tracking network model is trained, and the trained unmanned boat trajectory tracking network model is obtained, that is, the optimal network model. If the reward function has not stabilized, continue to repeat steps 101-107 to check whether the reward function in the number of rounds-reward function curve graph is stable until the reward function area is stable.

[0074] In some embodiments, the unmanned boat trajectory tracking network model is trained according to the multiple training samples to obtain the trained unmanned boat trajectory tracking network model, specifically including: inputting the state quantity at the current moment in the training sample into the policy network of the unmanned boat trajectory tracking network model, outputting the action at the current moment, and inputting the action at the current moment and the state at the current moment as the action-state pair at the current moment into the value network, and outputting the state value evaluation value of the state-action pair at the current moment; inputting the state quantity at the next moment in the training sample into the target policy network, and outputting the action at the next moment; and inputting the action at the next moment and the state quantity at the next moment in the training sample as the action-state pair at the next moment into the target value network, and obtaining the state value target value of the state-action pair at the next moment based on the local reward in the training sample; and minimizing the loss between the state value evaluation value of the state-action pair at the current moment and the state value target value of the state-action pair at the next moment by the gradient descent algorithm to obtain the trained unmanned boat target tracking network model.

[0075] In some embodiments, the parameter updating process of the policy network is: according to the action at the current moment and the state value evaluation value of the state-action pair at the current moment, the policy network is iteratively trained to update the parameters of the policy network; the parameter updating process of the value network is: according to the action at the next moment and the state value target value of the state-action pair at the next moment, the value network is iteratively trained to update the parameters of the value network; the parameter updating process of the target policy network is: updating the parameters of the target policy network through the updated parameters of the policy network and the soft update method; the parameter updating process of the target value network is: updating the parameters of the target value network through the updated parameters of the value network and the soft update method.

[0076] Specifically, the specific process of updating the network parameters of the unmanned boat trajectory tracking network model is as follows:

[0077] (1) Critic1 and Critic2 network parameter update process: Use the TargetActor network to calculate the state quantity s at the next moment t 'The next action a t ', action a t The calculation formula of ' is as follows:

[0078] a t '=μ'(s t '|θ μ′ )

[0079] Among them, θ μ' is the network parameter of the TargetActor network, μ'() represents the TargetActor network;

[0080] Then based on the target strategy smoothing regularization, in the target action a t 'Add noise to it:

[0081] a t '=μ'(s t '|θ μ′ )+∈

[0082] ∈=clip(N(0,σ),-b,b)

[0083] Among them, ∈ represents a random noise parameter; the clip function means that when N(0,σ)<-b, ∈=-b, when N(0,σ)>b, ∈=b, otherwise, ∈=N(0,σ); N(0,σ) means that it satisfies the normal distribution, -b and b represent fixed parameters, b>0.

[0084] Then, based on the idea of ​​dual networks, the Target Critic network is used to calculate the state-action pair (s t ,a t)’s state value target value y t , state value target value y t The calculation formula is as follows:

[0085]

[0086] Among them, r t is the reward function, γ is the discounted return rate, and θ i Q ' is the network parameter of Target Critic1 and Target Critic2 network, Q i '() represents the TargetCritic1 and TargetCritic2 networks.

[0087] Finally, the gradient descent algorithm is used to minimize the loss L between the evaluation value and the target value. ci , thereby updating the parameters in the Critic1 and Critic2 networks. The specific loss calculation formula is:

[0088]

[0089] Where Q i () represents the Target Critic1 and TargetCritic2 networks, s t Indicates the state quantity at the current moment; a t Represents the action at the current moment; θ i Q are the network parameters of Critic1 and Critic2 networks.

[0090] (2) Actor network parameter update process:

[0091] After the Ctitic1 and Critic2 networks are updated for d steps, the Actor network update is started, and the state quantity s at the current moment is calculated using the Actor network. t The current action a t :

[0092] a t =μ(s t |θ μ )

[0093] Among them, θ μ is the network parameter of the Actor network, μ() represents the Actor network;

[0094] It is worth noting that there is no need to add noise after calculating the action. Here we hope that the Actor network can be updated in the direction of the maximum value, so adding noise does not make sense.

[0095] Further, the state-action pair (s t ,a t )’s state value evaluation value q t , state value evaluation value q t The calculation formula is as follows, assuming that the Critic1 network is used:

[0096] q t =Q1(s t ,a t |θ Q1 )

[0097] Among them, θ Q1 are the network parameters of Critic1 network, Q1() represents Critic1 network;

[0098] Finally, the gradient ascent algorithm is used to maximize the state value evaluation value q t , thereby updating the parameters θ in the Actor network μ The reason why either Critic1 or Critic2 can be used to calculate the Q value is mainly because the purpose of the Actor network is to maximize the cumulative expected return, and there is no need to use the minimum value.

[0099] (3) Update the network parameters of TargetActor network and Target Critic.

[0100] The soft update method is adopted, in which the parameter update process of the TargetActor network is as follows:

[0101] θ μ′ =τθ μ +(1-τ)θ μ′

[0102] The Target Critic1 and Target Critic2 network update process is as follows:

[0103]

[0104] Among them, τ is the learning rate (momentum), τ∈(0,1), and is usually taken as 0.005.

[0105] Step 108: Optimize the tracking trajectory according to the trained unmanned boat trajectory tracking network model to determine the optimized tracking trajectory.

[0106] In some embodiments, the tracking trajectory is optimized according to the trained unmanned boat trajectory tracking network model to determine the optimized tracking trajectory, specifically including: inputting the target state quantity at the current moment into the trained unmanned boat trajectory tracking network model to obtain the target action of the unmanned boat at the next moment; executing the target action at the next moment to drive the unmanned boat to move, and obtaining the target state quantity at the next moment; using the target state quantity at the next moment as the target state quantity at the current moment, returning to the step of "inputting the target state quantity at the current moment into the trained unmanned boat trajectory tracking network model to obtain the target action of the unmanned boat at the next moment", until the Euclidean distance between the unmanned boat and the dynamic target and the duration of the unmanned boat tracking the dynamic target reach a first preset condition, stopping the tracking, and using the actions performed by the unmanned boat at each moment as the optimized tracking trajectory after the unmanned boat tracks the dynamic target.

[0107] The trained unmanned boat trajectory tracking network model is used as the unmanned boat target trajectory tracking network model, and S201-S205 are executed as follows:

[0108] S201: Control the radar to obtain the position, speed and other information of the unmanned boat and the dynamic target, and send it to the unmanned boat.

[0109] S202: The unmanned boat obtains and calculates the position and speed of the unmanned boat and the dynamic target, the Euclidean distance between the unmanned boat and the dynamic target, the angle between the current heading angle of the unmanned boat and the line connecting the unmanned boat and the dynamic target, and the control quantities of the two propeller motors of the unmanned boat.

[0110] S203: Input various parameters in S202 as state quantities into the unmanned boat target trajectory tracking network model to obtain the control quantities of two propellers of the unmanned boat. The unmanned boat target trajectory tracking network model is an optimal network model trained by the unmanned boat target trajectory tracking method based on reinforcement learning.

[0111] S204: Control the propeller movement according to the control amount to realize the movement of the unmanned boat.

[0112] S205: Repeat steps S201-S204 until the unmanned boat tracks the dynamic target.

[0113] Compared with the traditional method, the optimization method of unmanned boat tracking trajectory provided by this application has the following advantages:

[0114] 1. Strong adaptability: Reinforcement learning can adjust the action at the next moment by providing real-time feedback on the current state, adapt to the dynamic changes and complex environment of dynamic targets, and improve the tracking ability of unmanned boats in marine environments.

[0115] 2. High optimization performance: Through a lot of training and learning, reinforcement learning can find the optimal tracking trajectory and improve the tracking accuracy and efficiency of the unmanned boat.

[0116] 3. Intelligent decision-making: Reinforcement learning can take into account multiple factors and make more intelligent decisions, thereby improving the performance of unmanned boats in patrol and recovery tasks.

[0117] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0118] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0119] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRdM), magnetic memory (MRdM), ferroelectric memory (FRdM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RdM) or external cache memory, etc. By way of illustration and not limitation, RdM may be in various forms, such as static random access memory (SRdM) or dynamic random access memory (DRdM).

[0122] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0123] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for optimizing the tracking trajectory of an unmanned boat, characterized in that: The optimization method of the unmanned boat tracking trajectory includes: Obtain the state quantity of the unmanned boat at the current moment; the state quantity includes the current position of the unmanned boat, the current speed of the unmanned boat, the current position of the dynamic target, the current speed of the dynamic target, the current Euclidean distance between the unmanned boat and the dynamic target, the current heading angle of the unmanned boat and the current angle between the line connecting the unmanned boat and the dynamic target, and the current control quantity of the two propeller motors of the unmanned boat; the current control quantity of the two propeller motors of the unmanned boat is used as an action to drive the unmanned boat to move; Input the current state into the unmanned boat trajectory tracking network model to obtain the action at the next moment; Execute the action at the next moment, determine the state quantity at the next moment and the local reward for executing the action; Determine the tracking trajectory of the unmanned boat during the entire tracking process according to the state quantity at the next moment and the preset conditions; the entire tracking process is from starting to track the dynamic target to stopping tracking the dynamic target; According to the duration of the entire tracking process, the number of times the unmanned boat performs actions and the experience data corresponding to each action, the local reward in each set of experience data is updated to obtain target experience data; the experience data includes the action at the current moment, the state quantity at the next moment, the state quantity at the current moment and the local reward; Based on the priority experience replay algorithm, the target experience data is sampled to obtain multiple training samples; Training the unmanned boat trajectory tracking network model according to the multiple training samples to obtain a trained unmanned boat trajectory tracking network model; The tracking trajectory is optimized according to the trained unmanned boat trajectory tracking network model to determine the optimized tracking trajectory.

2. The method for optimizing the tracking trajectory of an unmanned boat according to claim 1, characterized in that: According to the state quantity and preset conditions at the next moment, the tracking trajectory of the unmanned boat during the entire tracking process is determined, including: Obtain the Euclidean distance between the unmanned boat and the dynamic target in the state quantity at the next moment; Determine a preset condition according to the Euclidean distance; the preset condition includes a first preset condition and a second preset condition; the first preset condition is that the Euclidean distance is not greater than the preset distance and the duration of the unmanned boat tracking the dynamic target exceeds the first preset time; the second preset condition is that the Euclidean distance is greater than the preset distance and the duration of the unmanned boat tracking the dynamic target does not exceed the second preset time; the first preset time is less than the second preset time; When the Euclidean distance and the duration of the unmanned boat tracking the dynamic target meet the first preset condition, determining that the unmanned boat has tracked the dynamic target, stopping the tracking, and determining the tracking trajectory; When the Euclidean distance and the duration of the unmanned boat tracking the dynamic target meet the second preset condition, it is determined that the unmanned boat cannot track the dynamic target, the tracking is stopped, and the tracking trajectory is determined.

3. The method for optimizing the tracking trajectory of an unmanned boat according to claim 1, characterized in that: According to the duration of the entire tracking process, the number of actions performed by the unmanned boat and the experience data corresponding to each action, the local reward in each set of experience data is updated to obtain the target experience data, specifically including: When the Euclidean distance and the duration of the unmanned boat tracking the dynamic target meet the second preset condition, it is determined that the unmanned boat obtains an overtime reward, and the local reward in each set of experience data is updated according to the overtime reward; the updating process of the local reward is: Get local rewards from any set of experience data; Determining a local timeout reward according to the timeout reward and the number of times the unmanned boat performs an action; Determine an updated local reward based on the local reward in the experience data and the local timeout reward; The updated local reward replaces the local reward in the experience data.

4. The method for optimizing the tracking trajectory of an unmanned boat according to claim 1, characterized in that: The network structure of the unmanned boat trajectory tracking network model specifically includes: strategy network, target strategy network, value network and target value network; The strategy network is used to input the state quantity at the current moment and output the action at the current moment; the value network is used to obtain the state quantity at the current moment and the action at the current moment in the strategy network, evaluate the state-action pair at the current moment and output the state value evaluation value; the state-action pair represents the state quantity and action at the same moment; The target strategy network is used to input the state quantity at the next moment and output the action at the next moment; the target value network is used to obtain the state quantity at the next moment and the action at the next moment in the target strategy network, and output the state value target value of the state-action pair at the next moment.

5. The method for optimizing the tracking trajectory of an unmanned boat according to claim 4, characterized in that: The method of training the unmanned boat trajectory tracking network model according to the plurality of training samples to obtain the trained unmanned boat trajectory tracking network model specifically includes: Input the state quantity at the current moment in the training sample into the strategy network of the unmanned boat trajectory tracking network model, output the action at the current moment, and input the action at the current moment and the state at the current moment as the action-state pair at the current moment into the value network, and output the state value evaluation value of the state-action pair at the current moment; Input the state quantity at the next moment in the training sample into the target strategy network, output the action at the next moment, and input the action at the next moment and the state quantity at the next moment in the training sample as the action-state pair at the next moment into the target value network, and obtain the state value target value of the state-action pair at the next moment based on the local reward in the training sample; The loss between the state value evaluation value of the state-action pair at the current moment and the state value target value of the state-action pair at the next moment is minimized by the gradient descent algorithm to obtain the trained unmanned boat target tracking network model.

6. The method for optimizing the tracking trajectory of an unmanned boat according to claim 4, characterized in that: The parameter updating process of the policy network is: according to the action at the current moment and the state value evaluation value of the state-action pair at the current moment, the policy network is iteratively trained to update the parameters of the policy network; The parameter updating process of the value network is: according to the action at the next moment and the state value target value of the state-action pair at the next moment, the value network is iteratively trained to update the parameters of the value network; The updating process of the parameters of the target policy network is as follows: updating the parameters of the target policy network by using the parameters of the updated policy network and a soft update method; The updating process of the parameters of the target value network is: updating the parameters of the target value network by using the parameters of the updated value network and a soft update method.

7. The method for optimizing the tracking trajectory of an unmanned boat according to claim 1, characterized in that: Execute the action at the next moment, determine the state quantity at the next moment and the local reward for executing the action, specifically including: Execute the action at the next moment and obtain the state quantity at the next moment; Inputting the state quantity at the next moment into the local reward function to obtain a local reward for executing the action; The calculation formula of the local reward function is: Among them, r st (s t ) is the local reward function; A is the amplitude height; σ is the standard deviation; b is the location parameter; r 2t The reward function for the unmanned boat to track the dynamic target within a preset distance. R is the preset distance; || || is the two-norm; ω1, ω2, ω3, ω4 and ω5 are hyperparameters; v u is the speed of the unmanned boat; v d is the speed of the dynamic target; Δl is the Euclidean distance between the unmanned boat and the dynamic target; It is the angle between the head of the unmanned boat and the line connecting the unmanned boat and the dynamic target; is the control quantity of the propeller motor of the unmanned boat l at time t; is the control quantity of the propeller motor of the unmanned boat r at time t; Control quantity of propeller motor of unmanned boat l at time t-1; It is the control quantity of the propeller motor of the unmanned boat r at time t-1.

8. The method for optimizing the tracking trajectory of an unmanned boat according to claim 1, characterized in that: The tracking trajectory is optimized according to the trained unmanned boat trajectory tracking network model to determine the optimized tracking trajectory, specifically including: The target state quantity at the current moment is input into the trained unmanned boat trajectory tracking network model to obtain the target action of the unmanned boat at the next moment; Execute the target action at the next moment to drive the unmanned boat to move, and obtain the target state quantity at the next moment; The target state quantity at the next moment is used as the target state quantity at the current moment, and the step of "inputting the target state quantity at the current moment into the trained unmanned boat trajectory tracking network model to obtain the target action of the unmanned boat at the next moment" is returned until the Euclidean distance between the unmanned boat and the dynamic target and the duration of the unmanned boat tracking the dynamic target reach the first preset condition, and the tracking is stopped, and the actions performed by the unmanned boat at each moment are used as the tracking trajectory after the unmanned boat tracks the dynamic target.

9. The method for optimizing the tracking trajectory of an unmanned boat according to claim 1, characterized in that: The calculation formula of the priority experience replay algorithm is: Among them, P(c) is a training sample (s t ,a t ,r t ,s t '), a t is the action at the current moment, s t ' is the state quantity at the next moment, s t is the state quantity at the current moment, r t is the local reward; c represents the sequence number of the extracted experience data; p k is the priority of the extracted empirical data; α is a preset parameter used to adjust the degree of priority sampling of training samples.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for optimizing the tracking trajectory of an unmanned boat according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for optimizing the tracking trajectory of an unmanned boat described in any one of claims 1 to 9 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for optimizing the tracking trajectory of an unmanned boat described in any one of claims 1 to 9 is implemented.

Images