Underwater multi-unmanned platform cooperative tracking method and system based on deep reinforcement learning

Abstract

The application belongs to the technical field of underwater multi-sensor target tracking, and discloses a kind of underwater multi-unmanned platform cooperative tracking method and system based on deep reinforcement learning;Among them, the underwater multi-unmanned platform cooperative tracking method includes: establishing end-to-end underwater multi-unmanned platform scheduling algorithm;Establish end-to-end underwater multi-unmanned platform data fusion algorithm;Combined with the Markov decision process model of modeling, introduce the MAPPO algorithm in deep reinforcement learning while learning multi-platform scheduling strategy, multi-platform data fusion strategy, evaluate the effectiveness of the current strategy through the feedback of the environment, and finally determine the optimal tracking strategy under the current environment.The application can effectively reduce the system energy consumption while significantly improving the system task execution efficiency and cooperative tracking performance, and can improve the application performance and system efficiency of underwater multi-unmanned platform in complex underwater target tracking environment.

Description

Technical Field

[0001] This invention belongs to the field of underwater multi-sensor target tracking technology, and specifically relates to an underwater multi-unmanned platform cooperative tracking method and system based on deep reinforcement learning. Background Technology

[0002] With the rapid development of underwater unmanned platforms, represented by autonomous underwater vehicles (AUVs), underwater multi-unmanned platform collaborative tracking systems, which are formed by networking multiple underwater unmanned platforms, have gradually matured and attracted much attention.

[0003] The underwater multi-unmanned platform cooperative tracking system can comprehensively utilize the reconnaissance, surveillance, detection, and communication capabilities of various sub-platforms to perform early detection, tracking, identification, and estimation of underwater targets, providing real-time information support for organized countermeasures. Compared with traditional underwater target tracking systems, the underwater multi-unmanned platform cooperative tracking system has the advantages of self-organization, strong fault tolerance, and strong stealth. However, due to the inherent structural characteristics and complex underwater environment, it also faces many challenges. On the one hand, underwater unmanned platforms need to use limited resources such as battery energy, computing performance, and communication performance to balance tracking performance and system resource consumption. On the other hand, affected by the complex underwater acoustic environment, the multi-source information obtained by the system is dynamically changing and subject to significant noise interference, with a severe lack of prior environmental knowledge.

[0004] The aforementioned situations have made the scheduling and data fusion of multiple underwater unmanned platforms (UAVs) crucial in underwater target tracking technology based on UAVs. Traditional scheduling and data fusion methods rely on prior environmental knowledge, making them ill-suited to dynamically changing underwater target tracking environments, leading to performance degradation. Furthermore, traditional underwater target tracking frameworks typically follow a step-by-step architecture, executing tracking tasks according to a "multi-platform scheduling - local estimation - multi-platform data fusion" process. This results in the system failing to address some tasks while simultaneously handling complex processes like multi-platform scheduling and data fusion, severely impacting system performance. Therefore, effectively scheduling multiple UAVs in a dynamic environment without prior knowledge to achieve high-efficiency tracking, fully utilizing dynamically changing information from multiple sources to find the optimal node fusion strategy to ensure data fusion accuracy meets tracking task requirements, and overcoming the limitations of traditional tracking frameworks to optimize the efficiency of collaborative tracking tasks under multiple complex scenarios have become significant challenges for underwater UAV systems. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for cooperative tracking of multiple underwater unmanned platforms based on deep reinforcement learning, in order to solve the problems of dynamic scheduling and data fusion of multiple underwater unmanned platforms under complex underwater environments and energy-constrained conditions. Specifically, this invention provides a cooperative tracking scheme for multiple underwater unmanned platforms based on deep reinforcement learning, constructing a novel end-to-end cooperative tracking framework. This framework enables high-efficiency target tracking of multiple underwater unmanned platforms under multi-source dynamic information, significantly improving system task execution efficiency and cooperative tracking performance while effectively reducing system energy consumption. It can enhance the application performance and system efficiency of multiple underwater unmanned platforms in complex underwater target tracking environments.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, this invention provides a method for cooperative tracking of multiple unmanned underwater platforms based on deep reinforcement learning, comprising the following steps:

[0008] This paper utilizes reinforcement learning to model the dynamic scheduling problem of multiple unmanned underwater platforms in complex underwater environments as a Markov decision process model. Furthermore, the design of the reward function incorporates an energy efficiency optimization objective function and a tracking performance evaluation function based on virtual data, thus establishing an end-to-end scheduling algorithm for multiple unmanned underwater platforms. This end-to-end scheduling algorithm includes a scheduling task state space, a scheduling task action space, a scheduling task reward function, and a multi-platform scheduling strategy.

[0009] This paper utilizes reinforcement learning to model the underwater multi-unmanned platform data fusion problem under multi-source dynamic information as a Markov decision process model. Furthermore, it incorporates a data fusion performance evaluation function based on virtual data into the design of the reward function, thereby establishing an end-to-end underwater multi-unmanned platform data fusion algorithm. The end-to-end underwater multi-unmanned platform data fusion algorithm includes a data fusion task state space, a data fusion task action space, a data fusion task reward function, and a multi-platform data fusion strategy.

[0010] By combining the end-to-end underwater multi-unmanned platform scheduling algorithm and the end-to-end underwater multi-unmanned platform data fusion algorithm, the Markov decision process model is established. The MAPPO algorithm in deep reinforcement learning is introduced to learn the multi-platform scheduling strategy and the multi-platform data fusion strategy. The effectiveness of the current strategy is evaluated through environmental feedback, and finally the optimal tracking strategy in the current environment is determined.

[0011] A further improvement of the present invention is that, in the end-to-end underwater multi-unmanned platform scheduling algorithm,

[0012] Scheduling task state space middle, kThe state space representation of the scheduled task at time t is as follows:

[0013] ;

[0014] In the formula, for k The state space of the scheduling task at any given moment; for k System tracking error at any given time; for k The system's remaining energy at a given time; To select a subset composed of ordinary platforms;

[0015] Scheduling task action space middle, k The task action space at any given time is represented as follows:

[0016] ;

[0017] In the formula, for k The action space of scheduling tasks at any given moment; A set of candidate general platforms; for The number of platforms in China;

[0018] Scheduled task reward function Represented as,

[0019] ;

[0020] In the formula, To coordinate system time tracking; For instant rewards; For settlement of rewards;

[0021] In the process of generating multi-platform scheduling strategies, a Plane Decision Process (PPO) is introduced, based on the established Markov decision process model, to generate scheduling strategies. The PPO consists of an execution network and an evaluation network; the execution network is used for strategy generation, and the evaluation network is used for strategy evaluation. End-to-end scheduling strategies are generated based on the execution network within the PPO.

[0022] k The loss function of the network at time t is shown below.

[0023] ;

[0024] In the formula, These are the execution network parameters in the scheduling task; This is a truncation function used to evaluate the scheduling policy function. Controlled Inside, For hyperparameters;

[0025] As shown below,

[0026] ;

[0027] In the formula, For parameter-based Strategies; For parameter-based Strategies; They are respectively k The scheduling task actions and status in real time;

[0028] for k The advantage estimation function at time 1 is shown below.

[0029] ;

[0030] In the formula, The total time required for the collaborative tracking task; Value function; All are discount factors; This is the cumulative time difference error;

[0031] The input to the execution network is the scheduling status of multiple unmanned platforms, and the output is the generated scheduling strategy for multiple unmanned platforms.

[0032] A further improvement of the present invention is that the instant reward is represented as follows:

[0033] , ;

[0034] In the formula, A joint factor for adjusting the proportions of the observation performance function and the energy loss function; A set of candidate general platforms; For observation performance function; This is the energy loss function;

[0035] , ;

[0036] In the formula, Fisher's information matrix;

[0037] ;

[0038] In the formula, This indicates the predicted location information of the target. This indicates the relative positional information between the candidate ordinary platform and the target predicted location;

[0039] ;

[0040] , ;

[0041] In the formula, N This refers to the number of ordinary platforms; For candidate platforms Remaining energy at any given moment; and These respectively represent the candidate platforms in Energy consumption required for constant communication with the central platform and ordinary platforms.

[0042] A further improvement of the present invention is that the settlement reward is represented as follows:

[0043] , ;

[0044] In the formula, A joint factor to balance the proportions of target tracking accuracy and system energy efficiency; For each round of training, the tracking accuracy is determined; For target accuracy; Target energy consumption; Energy consumption of the system per round of training; Total number of training rounds;

[0045] ;

[0046] In the formula, T For the duration of system tracking;

[0047] ;

[0048] ;

[0049] In the formula, for A typical platform Virtual measurement of time and actual measurement Virtual deviation between; for time and The Euclidean distance between them;

[0050] ;

[0051] In the formula, For the generated virtual measurement; The measurement equation is known; To pass state estimation The generated measurement prediction values; To measure the noise covariance.

[0052] A further improvement of the present invention lies in that, in the end-to-end underwater multi-unmanned platform data fusion algorithm,

[0053] Data fusion task state space middle, k Time-based data fusion task state space Represented as,

[0054] ;

[0055]

[0056] In the formula, Representative fusion estimate of covariance; This represents data fusion estimation; The total number of platforms participating in data fusion; and Representative Platform Local estimation of covariance and local state estimation, For the platform Corresponding data fusion weights;

[0057] Data fusion task action space middle, k The action space of the data fusion task at any given time is represented as follows:

[0058] ;

[0059] Data fusion task reward function Represented as,

[0060] ;

[0061] In the formula, For the duration of system collaborative tracking; For instant rewards; For settlement of rewards;

[0062] In the process of generating data fusion strategies, an end-to-end multi-platform data fusion strategy is generated based on the established Markov decision process model and the execution network in the PPO; among which,

[0063] k At what time, the loss function of the execution network for the data fusion task Represented as,

[0064] ;

[0065] In the formula, For execution network parameters in data fusion tasks; This is the evaluation function for the data fusion strategy;

[0066] ;

[0067] In the formula, For parameter-based Data strategy; For parameter-based Strategies; They are respectively k Real-time data fusion task actions and data fusion task status;

[0068] for k The advantage estimation function in the time-matter fusion task is shown below.

[0069] ;

[0070] The input to the execution network is the data fusion state, and the output is the generated data fusion strategy.

[0071] A further improvement of the present invention is the provision of instant rewards. Represented as,

[0072] , ;

[0073] ;

[0074] ;

[0075] In the formula, For those participating in data fusion The platform obtained Virtual measurement of time and actual measurement Virtual deviation between; for time and The Euclidean distance between them.

[0076] A further improvement of the present invention lies in the settlement of rewards. Represented as,

[0077] , ;

[0078] ;

[0079] In the formula, To estimate the target accuracy for data fusion; Estimating accuracy for each round of training data fusion; This refers to the total number of training rounds.

[0080] A further improvement of this invention lies in the following steps: The Markov decision process model established during the combination of the end-to-end underwater multi-unmanned platform scheduling algorithm and the end-to-end underwater multi-unmanned platform data fusion algorithm, along with the introduction of the MAPPO algorithm from deep reinforcement learning to simultaneously learn the multi-platform scheduling strategy and the multi-platform data fusion strategy, evaluate the effectiveness of the current strategy through environmental feedback, and finally determine the optimal tracking strategy under the current environment, specifically includes:

[0081] Based on the fundamental principles of MAPPO, the multi-unmanned platform scheduling task and the multi-unmanned platform data fusion task are treated as two intelligent agents performing independent tasks. These two agents interact with the environment through their respective execution networks to update their policies. A global evaluation network acquires the global state of both tasks and evaluates the policies generated by the two agents, achieving coordinated policy updates for both tasks and establishing an end-to-end underwater multi-unmanned platform cooperative tracking framework.

[0082] Global task state space modeling as ;

[0083] The global evaluation network combines the global task status to evaluate the strategies of both scheduling and data fusion tasks, enabling the strategies of both to be updated collaboratively in a global context.

[0084] The loss function of the global evaluation network is shown below.

[0085] ;

[0086] In the formula, To evaluate network parameters globally; Reporting for accumulated discounts; The number of agents; The value function of the new strategy; The value function of the old strategy;

[0087] The input to the global evaluation network is the global task state, and the output is the policy evaluation value.

[0088] A further improvement of the present invention is that, after establishing the end-to-end underwater multi-unmanned platform cooperative tracking framework, the end-to-end underwater multi-unmanned platform cooperative tracking process specifically includes:

[0089] (1) Initialization: Initialize the network parameters for scheduling task execution. Network parameters for data fusion task execution Global evaluation network parameters Initialize discount factor , and number of iterations Initialize the scheduling task state space Data fusion task state space and global task state space Track task-related parameters;

[0090] (2) Action update: Input the current scheduling status into the execution network of the scheduling task and the data fusion task respectively. The data fusion status and the old actions selected in the previous moment. Generate new actions at the current moment. ;

[0091] (3) Reward update: Update the rewards for scheduling tasks and data fusion tasks based on the new actions at the current moment;

[0092] (4) Global evaluation update: by inputting the global state at the current moment. Scheduling task actions Data fusion strategy Update the evaluation values ​​of the scheduling task strategy and the data fusion task strategy respectively;

[0093] (5) Execution network update: The gradient descent method is used to solve the loss function of the scheduling task execution network and the data fusion task execution network respectively, so as to update the execution network parameters and thus realize policy update; among which,

[0094] ;

[0095] (6) Global Evaluation Network Update: The global evaluation function loss function is solved globally using gradient descent to minimize the loss, thereby updating the global evaluation network parameters; where,

[0096] ;

[0097] (7) Repeat steps (2) to (6). This time, we completed end-to-end collaborative tracking of multiple unmanned underwater platforms.

[0098] In a second aspect, the present invention provides an underwater multi-unmanned platform cooperative tracking system based on deep reinforcement learning, comprising:

[0099] The first algorithm construction module is used to model the dynamic scheduling problem of underwater multi-unmanned platforms in complex underwater environments as a Markov decision process model using reinforcement learning. In the design of the reward function, the energy efficiency optimization objective function and the tracking performance evaluation function based on the virtual data method are combined to establish an end-to-end underwater multi-unmanned platform scheduling algorithm. The end-to-end underwater multi-unmanned platform scheduling algorithm includes a scheduling task state space, a scheduling task action space, a scheduling task reward function, and a multi-platform scheduling strategy.

[0100] The second algorithm construction module is used to model the underwater multi-unmanned platform data fusion problem under multi-source dynamic information as a Markov decision process model using reinforcement learning, and to combine a data fusion performance evaluation function based on virtual data method in the design of the reward function to establish an end-to-end underwater multi-unmanned platform data fusion algorithm; wherein, the end-to-end underwater multi-unmanned platform data fusion algorithm includes a data fusion task state space, a data fusion task action space, a data fusion task reward function, and a multi-platform data fusion strategy;

[0101] The tracking module is used to build a Markov decision process model by combining the end-to-end underwater multi-unmanned platform scheduling algorithm and the end-to-end underwater multi-unmanned platform data fusion algorithm. It introduces the MAPPO algorithm in deep reinforcement learning to learn the multi-platform scheduling strategy and the multi-platform data fusion strategy. It evaluates the effectiveness of the current strategy through environmental feedback and finally determines the optimal tracking strategy in the current environment.

[0102] Compared with the prior art, the present invention has the following beneficial effects:

[0103] This invention specifically provides a cooperative tracking scheme for underwater multi-unmanned platforms based on deep reinforcement learning. It constructs a novel end-to-end cooperative tracking framework for underwater multi-unmanned platforms, which can achieve high-efficiency target tracking of underwater multi-unmanned platforms under multi-source dynamic information. It can significantly improve the system's task execution efficiency and cooperative tracking performance while effectively reducing system energy consumption, and can improve the application performance and system efficiency of underwater multi-unmanned platforms in complex underwater target tracking environments. Specifically, in the technical solution of this invention, an end-to-end underwater multi-unmanned platform scheduling algorithm is established using reinforcement learning methods. This effectively solves the dynamic scheduling problem that traditional scheduling algorithms struggle with. Even without prior knowledge, the system dynamically adapts to the underwater environment, optimizing system resource allocation and task execution efficiency, and improving energy efficiency. Similarly, an end-to-end underwater multi-unmanned platform data fusion algorithm is established using reinforcement learning methods. This effectively solves the underwater multi-source dynamic information data fusion problem that traditional data fusion methods struggle with, optimizing the system's data fusion performance and execution efficiency in complex underwater environments, and improving the target tracking accuracy of the underwater multi-unmanned platform system. Furthermore, an underwater multi-unmanned platform cooperative tracking algorithm constructed using deep reinforcement learning effectively solves the complex multi-task problem that traditional step-by-step underwater target tracking algorithms struggle with. The end-to-end framework optimizes the entire cooperative tracking process, integrating multiple important aspects, including multi-platform scheduling and multi-platform data fusion, into a single framework. This enables the system to simultaneously handle multiple complex tasks such as scheduling and fusion, improving the cooperative tracking efficiency of underwater multi-unmanned platforms. Attached Figure Description

[0104] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0105] Figure 1 This is a flowchart illustrating a collaborative tracking method for underwater multi-unmanned platforms based on deep reinforcement learning, as described in an embodiment of the present invention.

[0106] Figure 2 This is a schematic diagram of the end-to-end underwater multi-unmanned platform scheduling algorithm constructed in an embodiment of the present invention.

[0107] Figure 3 This is a schematic diagram of the end-to-end underwater multi-unmanned platform data fusion algorithm constructed in an embodiment of the present invention.

[0108] Figure 4 This is a schematic diagram of the structure of the global evaluation network constructed in an embodiment of the present invention.

[0109] Figure 5 This is a schematic diagram of the end-to-end underwater multi-unmanned platform collaborative tracking framework used in an embodiment of the present invention.

[0110] Figure 6 This is a schematic diagram of an underwater multi-unmanned platform collaborative tracking system based on deep reinforcement learning, as described in an embodiment of the present invention. Detailed Implementation

[0111] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0112] Please see Figure 1 In this embodiment of the invention, a cooperative tracking algorithm for multiple underwater unmanned platforms based on deep reinforcement learning is provided, comprising the following steps:

[0113] Step S101 involves constructing an end-to-end underwater multi-unmanned platform scheduling framework using reinforcement learning. This framework models the dynamic scheduling problem of multiple unmanned platforms in complex underwater environments as a model-free, scalable Markov decision process. The reward function design incorporates an energy efficiency optimization objective function and a tracking performance evaluation function based on virtual data, thus establishing an end-to-end underwater multi-unmanned platform scheduling algorithm. Further, it consists of four main parts: the scheduling task state space, the scheduling task action space, the scheduling task reward function, and the multi-platform scheduling strategy.

[0114] Step S102 involves constructing an end-to-end underwater multi-unmanned platform data fusion framework using reinforcement learning. Specifically, reinforcement learning is used to model the underwater multi-unmanned platform data fusion problem under multi-source dynamic information as a Markov decision process model. The reward function design incorporates a data fusion performance evaluation function based on virtual data, thereby establishing an end-to-end underwater multi-unmanned platform scheduling algorithm. Further, it mainly consists of four parts: the data fusion task state space, the data fusion task action space, the data fusion task reward function, and the multi-platform data fusion strategy.

[0115] Step S103: MAPPO (Multi-Agent Proximal Policy Optimization) is used to learn scheduling strategies and construct an end-to-end underwater multi-unmanned platform cooperative tracking algorithm. Specifically, the MAPPO algorithm from deep reinforcement learning is introduced, combining the Markov decision process constructed in steps S101 and S102, to simultaneously learn multi-platform scheduling strategies and multi-platform data fusion strategies, optimizing training efficiency. Furthermore, the effectiveness of the current strategy is evaluated through environmental feedback to determine the optimal tracking strategy for the current environment.

[0116] This invention discloses a cooperative tracking method for multiple underwater unmanned platforms based on deep reinforcement learning. This method enables high-efficiency target tracking in multi-source dynamic information environments. It not only significantly reduces system energy consumption but also greatly improves task execution efficiency and cooperative tracking performance, enhancing the adaptability and system efficiency of multiple underwater unmanned platforms in complex underwater target tracking environments. The technical solution of this invention involves underwater sensor scheduling technology, underwater sensor data fusion technology, underwater target tracking technology, and deep reinforcement learning technology. The comprehensive application of these technologies enables the system to efficiently and accurately complete the cooperative tracking task of multiple underwater unmanned platforms. Specifically, regarding sensor scheduling, the technical solution of this invention adopts an end-to-end scheduling algorithm based on reinforcement learning, solving the problem that traditional scheduling algorithms are difficult to adapt to dynamic environmental changes. This algorithm can dynamically adapt to the underwater environment in the absence of prior knowledge, optimize system resource allocation, improve task execution efficiency, and significantly improve energy efficiency. Regarding data fusion, the technical solution of this invention utilizes reinforcement learning to construct a novel end-to-end data fusion algorithm, effectively solving the problem of underwater multi-source dynamic information data fusion. This innovative algorithm optimizes the data fusion performance and execution efficiency of the system in complex underwater environments, thereby improving the target tracking accuracy of underwater multi-unmanned platform systems. In terms of target tracking, the technical solution of this invention introduces deep reinforcement learning technology to construct a novel underwater multi-unmanned platform cooperative tracking algorithm. This algorithm overcomes the shortcomings of traditional step-by-step underwater target tracking algorithms in handling complex multi-task problems. It optimizes the entire cooperative tracking process through an end-to-end framework. This framework integrates important aspects such as multi-platform scheduling and multi-platform data fusion, enabling the system to simultaneously handle multiple complex tasks such as scheduling and fusion, thus significantly improving the cooperative tracking efficiency of underwater multi-unmanned platforms. In summary, this invention provides an efficient, accurate, and adaptive underwater multi-unmanned platform cooperative tracking solution, bringing new technological breakthroughs and development opportunities to the field of underwater target tracking.

[0117] In one embodiment of the present invention, reinforcement learning is used to model the scheduling problem between the central platform and ordinary platforms as a model-free, scalable Markov decision process, thereby constructing an end-to-end multi-unmanned platform data fusion framework, which mainly consists of four parts: a scheduling task state space. Scheduling task action space ,award and scheduling strategies Therefore, the specific establishment method is explained in the order of scheduling task state space modeling, scheduling task action space modeling, scheduling task reward function design, and end-to-end scheduling task strategy generation.

[0118] (1) State space modeling of scheduling tasks:

[0119] As a specific embodiment, the definition is as follows: k In real-time underwater multi-unmanned platforms, a dynamic cluster consisting of a central platform and multiple ordinary platforms collaborates on a tracking task. In this scenario, the scheduling problem between the central platform and the ordinary platforms becomes the central platform's responsibility. From the candidate general platform set Selected from the best A subset consisting of ordinary platforms To optimize the energy efficiency of underwater multi-unmanned platform systems by participating in underwater target tracking; among which...

[0120] The following conditions must be met: ;

[0121] In the formula, For the first i A regular platform, a regular platform The received sound intensity should meet the following requirements. ; Let be the sound intensity function. The sound intensity threshold required to complete the task is determined by its value and The number of members included is positively correlated; Let be the energy function. The energy threshold required to become a candidate for a regular platform.

[0122] Based on this, and combined with the basic theories of reinforcement learning, k State space at any given moment The results of the current scheduling task (including tracking performance evaluation, system energy consumption, etc.) and the current selection of ordinary platforms (i.e., the preferred subset of ordinary platforms) should be considered. It consists of two aspects, thus having ;

[0123] In the formula, fork The system tracking error at any given time is given by a tracking performance evaluation method based on virtual data. for k The system's remaining energy at a given time; To select a subset composed of ordinary platforms;

[0124] (2) Modeling the action space of scheduling tasks:

[0125] As a specific embodiment, the action space of the central platform for executing scheduling tasks Set as: From the set of candidate general platforms Optimized A subset of participants in underwater target tracking It is a discrete action space, and thus we have:

[0126] ;

[0127] (3) Design of the reward function for scheduling tasks:

[0128] As a specific example, the reward function for the scheduling task consists of two parts: an immediate reward based on the energy efficiency optimization objective and a settlement reward based on the virtual data method.

[0129] Explained, immediate reward is the reward obtained immediately after an action is performed, used to evaluate the merits of the strategy at the current moment; in scheduling tasks, immediate reward based on energy efficiency optimization objectives needs to consider both the observation performance and energy consumption of a typical platform; therefore, immediate reward consists of two parts: an observation performance function and an energy loss function.

[0130] In this embodiment of the invention, the Fisher information matrix can be obtained using the location of the candidate ordinary platform and the predicted location information of the target. , is represented as follows,

[0131] ;

[0132] In the formula, This indicates the predicted location information of the target. This indicates the relative positional information between the candidate ordinary platform and the target predicted position.

[0133] Based on this information matrix, it can be used to evaluate the richness of target information obtained by ordinary platforms, thereby measuring the observation performance of ordinary platforms. Therefore, the observation performance function is established as follows:

[0134] ;

[0135] The loss function can be implemented using a common platform. The total energy required for construction at any given time is considered. Based on the underwater energy consumption model, the candidate ordinary platform... exist The total energy required at any given moment is as follows:

[0136] ;

[0137] In the formula, and These respectively represent the candidate platforms in Energy consumption required for constant communication with the central platform and ordinary platforms; N This refers to the number of ordinary platforms; For candidate platforms The remaining energy at any given moment.

[0138] Based on this, the following energy loss function can be established:

[0139] ;

[0140] Based on the observation performance function and the energy loss function, the instantaneous reward can be constructed as follows:

[0141] ;

[0142] In the formula, To adjust the joint factor of the proportions of the observation performance function and the energy loss function; interpretively, when When the value is large, scheduling tends to improve system tracking performance. When the value is relatively small, the scheduling tends to reduce system energy consumption, and its value is determined according to task requirements.

[0143] Furthermore, the settlement reward is the total reward after each training round, used to evaluate the training effect. In the scheduling task, the effect of each training round should be reflected in the scheduling results, i.e., through system tracking performance and system energy consumption. Explaining this further, regarding system tracking performance, since the targets tracked by underwater multi-unmanned platforms are often non-cooperative targets, their true state values ​​are difficult to obtain. This renders the commonly used tracking performance evaluation method based on the difference between the true target state value and the state estimate inapplicable. Therefore, a virtual data method is introduced, which measures the virtual deviation between the virtual measurements generated from the state estimate and the actual measurements. To evaluate tracking performance.

[0144] The virtual measurement is defined to follow a Gaussian distribution, therefore we have ;

[0145] In the formula, For the generated virtual measurement, Given the measurement equations, To pass state estimation The generated measurement prediction values, To measure the noise covariance.

[0146] Therefore, we can conclude that, k The virtual deviation at time is represented as, ;

[0147] In the formula, for A typical platform Virtual measurement of time and actual measurement The virtual deviation between them for time and The Euclidean distance between them.

[0148] Based on the above formula, the tracking accuracy for each round of training can be constructed. As shown in the following formula,

[0149] ;

[0150] In the formula, T This refers to the duration of system tracking.

[0151] Regarding system energy consumption, it is determined by the total energy consumption of the system in each round. An evaluation is conducted; therefore, the settlement reward is constructed as follows:

[0152] ;

[0153] In the formula, For the total number of training rounds, For target accuracy, Both are determined by the target energy consumption and the mission requirements. The system energy consumption for each round of training. To improve the tracking accuracy in each training round, The joint factor balancing target tracking accuracy and system energy efficiency is determined by mission requirements. When the target tracking accuracy is high, the size of the settlement reward depends on the target tracking accuracy. When the value is large, the size of the settlement reward depends on the system's energy consumption.

[0154] In summary, considering the system collaborative tracking system time is Combining immediate rewards and settlement rewards, the scheduling task reward function can be constructed as follows:

[0155] .

[0156] (4) End-to-end scheduling strategy generation:

[0157] As a specific implementation, based on the established Markov decision process model, Proximal Policy Optimization (PPO) is introduced for scheduling policy generation. PPO consists of an execution network and an evaluation network; the execution network is used for policy generation, and the evaluation network is used for policy evaluation. Therefore, end-to-end scheduling policy generation is performed based on the execution network in PPO. k The loss function of the network at time t is as follows:

[0158] ;

[0159] In the formula, For the execution network parameters in the scheduling task, This is a manually designed truncation function; its role here is to modify the scheduling policy evaluation function. Controlled Inside, These are hyperparameters used to ensure the stability of the generation strategy.

[0160] here, As shown in the following formula:

[0161] ;

[0162] In the formula, For parameter-based strategy, For parameter-based strategy, They are respectively k The scheduling task actions and status at any given time.

[0163] for k The advantage estimation function at time t is shown in the following equation:

[0164] ;

[0165] In the formula, The total time required for collaborative tracking tasks. For value function, All are discount factors. This represents the cumulative time difference error.

[0166] Based on the aforementioned execution network, an end-to-end scheduling algorithm for multiple unmanned platforms can be established, such as... Figure 2 As shown; the execution network consists of an input layer, two 128-layer fully connected layers, and an output layer; the input is the multi-unmanned platform scheduling state, and the output is the generated multi-unmanned platform scheduling strategy. Due to the action space For a discrete action space, actions are generated through a multinomial distribution.

[0167] In a specific embodiment of the present invention, based on the fundamental theory of reinforcement learning, the data fusion problem among multiple unmanned platforms under multi-source dynamic information is modeled as a Markov decision process model using reinforcement learning methods, thereby constructing an end-to-end multi-unmanned platform data fusion framework, which mainly consists of four parts: the data fusion task state space. Data fusion task action space ,award and data fusion strategy Therefore, the specific establishment method is explained in the order of data fusion task state space modeling, data fusion task action space modeling, data fusion task reward function design, and data fusion task strategy generation.

[0168] (1) State space modeling for data fusion tasks:

[0169] As a specific embodiment, the definition is as follows: k The data fusion estimation method for time across multiple platforms is as follows.

[0170]

[0171] In the formula, Representative fusion estimate of covariance; This represents data fusion estimation; The total number of platforms participating in data fusion; and Representative Platform Local estimation of covariance and local state estimation, For the platform The corresponding data fusion weights.

[0172] At this point, the problem of multi-platform data fusion is to optimize the data fusion weights across multiple platforms to achieve the highest accuracy in data fusion estimation; based on this, and combined with the basic theories of reinforcement learning, k Time-based data fusion task state space The data fusion result at the current moment, including the data fusion estimate and the fusion estimate covariance, should give us: .

[0173] (2) Action space modeling for data fusion tasks:

[0174] As a specific example, the action space of a data fusion task Set as: Optimize the weighting of multi-platform data fusion ;consider A platform participating in data fusion, thus having ;

[0175] Since the fusion weight is in the range Since the data is constantly changing, the action space of the data fusion task is a continuous action space.

[0176] (3) Design of reward function for data fusion task:

[0177] As a specific embodiment, the reward function of the data fusion task consists of two parts: an immediate reward and a settlement reward. Since the data fusion strategy is evaluated only through the data fusion estimation performance, the virtual data method, the same as in step S101, is used to evaluate the data fusion estimation performance in order to construct the immediate reward and the settlement reward.

[0178] For instant rewards, consider k The virtual deviation at time is as follows:

[0179] ;

[0180] In the formula, For those participating in data fusion The platform obtained Virtual measurement of time and actual measurement The virtual deviation between them for time and The Euclidean distance between them.

[0181] Instant rewards can be constructed as follows:

[0182] ;

[0183] Meanwhile, the settlement reward is constructed from the global average virtual deviation as follows:

[0184] ;

[0185] ;

[0186] In the formula, For the total number of training rounds, To estimate the target accuracy for data fusion, Estimate the accuracy of the fusion of training data for each round.

[0187] In summary, considering the duration of system cooperative tracking is... Combining immediate rewards and settlement rewards, the scheduling task reward function can be constructed as follows:

[0188] .

[0189] (4) Generation of end-to-end data fusion strategy:

[0190] As a specific implementation, based on the established Markov decision process model, an end-to-end scheduling strategy is generated using the execution network in the PPO. k At what time, the loss function of the execution network for the data fusion task as follows:

[0191] ;

[0192] In the formula, For the execution network parameters in the data fusion task, The data fusion strategy evaluation function is shown in the following equation:

[0193] ;

[0194] In the formula, For parameter-based Data strategy, For parameter-based strategy, They are respectively k The data fusion task actions and data fusion task status at any given moment.

[0195] for k The advantage estimation function in the time-fusion task is shown in the following equation:

[0196] ;

[0197] Based on the aforementioned execution network, an end-to-end multi-unmanned platform data fusion algorithm can be constructed, such as... Figure 3 As shown; the execution network consists of an input layer, two 128-layer fully connected layers, and an output layer. The input is the data fusion state, and the output is the generated data fusion strategy. Due to the action space It is a continuous action space, and actions are generated through a normal distribution.

[0198] In a specific embodiment of the present invention, MAPPO is introduced to schedule and fuse multi-task strategy collaborative learning, combining the established end-to-end multi-unmanned platform scheduling algorithm and end-to-end multi-unmanned platform data fusion algorithm, thereby establishing an end-to-end underwater cooperative tracking algorithm; wherein, based on the end-to-end multi-unmanned platform scheduling algorithm and end-to-end multi-unmanned platform data fusion algorithm, two additional parts are added: a global task state space. 1. Global Evaluation Network. Therefore, the discussion will proceed from the order of global task state space modeling and global evaluation network design.

[0199] (1) Global task state space modeling:

[0200] As a specific implementation, underwater multi-unmanned platform cooperative tracking mainly includes two core tasks: multi-unmanned platform scheduling and multi-unmanned platform data fusion. Based on the basic concept of MAPPO, the multi-unmanned platform scheduling task and the multi-unmanned platform data fusion task are regarded as two intelligent agents performing independent tasks. Among them, the two intelligent agents interact with the environment through their own execution networks to update their policies. A global evaluation network is used to obtain the global state of the two tasks and evaluate the policies generated by the two intelligent agents, thereby achieving coordinated policy updates for the two tasks and establishing end-to-end underwater multi-unmanned platform cooperative tracking.

[0201] Therefore, the global task state space The states of these two core tasks need to be considered simultaneously to enable the global evaluation network to collaboratively optimize the policy. The model is as follows: .

[0202] (2) Global evaluation network design:

[0203] As a specific implementation, the global evaluation network combines the global task status to evaluate the strategies of both the scheduling task and the data fusion task, so that the strategies of both can be updated collaboratively in a global context.

[0204] The loss function of the global evaluation network is shown in the following equation.

[0205] ;

[0206] In the formula, To evaluate network parameters globally, To accumulate discount reports, Given the number of agents, and considering scheduling and data fusion, .here, For the new strategy value function, The value function is the old strategy value function. The specific structure of the global evaluation network is as follows: Figure 4 As shown, it consists of an input layer, two 128-layer fully connected layers, and an output layer. The input is the global task state, and the output is the policy evaluation value.

[0207] Based on the above methods, an end-to-end underwater multi-unmanned platform cooperative tracking framework can be constructed, such as... Figure 5 As shown. Therefore, the flow of the end-to-end underwater multi-unmanned platform cooperative tracking algorithm based on deep reinforcement learning is as follows:

[0208] (1) Initialization: Initialize the network parameters for scheduling task execution. Network parameters for data fusion task execution Global evaluation network parameters Initialize discount factor and Number of iterations Hyperparameters; Initialize the scheduling task state space Data fusion task state space Global task state space Tracking task-related parameters;

[0209] (2) Action update: Input the current scheduling status into the execution network of the scheduling task and the data fusion task respectively. The data fusion status and the old actions selected in the previous moment. Generate new actions at the current moment. ;

[0210] (3) Reward update: Update the rewards for scheduling and data fusion tasks based on the new actions at the current moment. ;

[0211] (4) Global evaluation update: by inputting the global state at the current moment. Scheduling task actions Data fusion strategy Update the evaluation values ​​of the scheduling task strategy and the data fusion task strategy respectively;

[0212] (5) Execution network update: The gradient descent method is used to solve the loss function of the scheduling task execution network and the data fusion task execution network respectively, so as to update the execution network parameters and thus realize policy update.

[0213] ;

[0214] (6) Global evaluation network update: The global evaluation function loss function is solved globally using the gradient descent method to minimize the loss, thereby updating the parameters of the global evaluation network.

[0215] ;

[0216] Repeat steps (2) through (6). Next, coordinate and track the process.

[0217] In summary, this invention discloses a cooperative tracking method for underwater multi-unmanned platforms based on deep reinforcement learning. The method includes: transforming the dynamic scheduling problem and the data fusion problem of multi-unmanned platforms in complex underwater environments into Markov decision process models using reinforcement learning; designing a reward function using virtual data to evaluate system tracking performance; and implementing reinforcement learning training in the absence of ground truth data on non-cooperative target trajectories, thereby establishing end-to-end underwater multi-unmanned platform scheduling and data fusion algorithms. Based on the above methods, a novel end-to-end underwater multi-unmanned platform cooperative tracking framework is constructed using the MAPPO cooperative learning scheduling strategy and data fusion strategy. This invention can effectively improve the accuracy of underwater multi-unmanned platform cooperative tracking, increase system task execution efficiency, reduce system energy consumption, and further extend the lifespan of the underwater multi-unmanned platform system.

[0218] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0219] Please see Figure 6 In this embodiment of the invention, a cooperative tracking system for multiple underwater unmanned platforms based on deep reinforcement learning is provided, comprising:

[0220] The first algorithm construction module is used to model the dynamic scheduling problem of underwater multi-unmanned platforms in complex underwater environments as a Markov decision process model using reinforcement learning. In the design of the reward function, the energy efficiency optimization objective function and the tracking performance evaluation function based on the virtual data method are combined to establish an end-to-end underwater multi-unmanned platform scheduling algorithm. The end-to-end underwater multi-unmanned platform scheduling algorithm includes a scheduling task state space, a scheduling task action space, a scheduling task reward function, and a multi-platform scheduling strategy.

[0221] The second algorithm construction module is used to model the underwater multi-unmanned platform data fusion problem under multi-source dynamic information as a Markov decision process model using reinforcement learning, and to combine a data fusion performance evaluation function based on virtual data method in the design of the reward function to establish an end-to-end underwater multi-unmanned platform data fusion algorithm; wherein, the end-to-end underwater multi-unmanned platform data fusion algorithm includes a data fusion task state space, a data fusion task action space, a data fusion task reward function, and a multi-platform data fusion strategy;

[0222] The tracking module is used to build a Markov decision process model by combining the end-to-end underwater multi-unmanned platform scheduling algorithm and the end-to-end underwater multi-unmanned platform data fusion algorithm. It introduces the MAPPO algorithm in deep reinforcement learning to learn the multi-platform scheduling strategy and the multi-platform data fusion strategy. It evaluates the effectiveness of the current strategy through environmental feedback and finally determines the optimal tracking strategy in the current environment.

[0223] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used to execute the operation of an underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning.

[0224] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning in the above embodiments.

[0225] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0226] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0227] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0228] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A cooperative tracking method for multiple unmanned underwater platforms based on deep reinforcement learning, characterized in that, Includes the following steps: This paper utilizes reinforcement learning to model the dynamic scheduling problem of multiple unmanned underwater platforms in complex underwater environments as a Markov decision process model. Furthermore, the design of the reward function incorporates an energy efficiency optimization objective function and a tracking performance evaluation function based on virtual data, thus establishing an end-to-end scheduling algorithm for multiple unmanned underwater platforms. This end-to-end scheduling algorithm includes a scheduling task state space, a scheduling task action space, a scheduling task reward function, and a multi-platform scheduling strategy. This paper utilizes reinforcement learning to model the underwater multi-unmanned platform data fusion problem under multi-source dynamic information as a Markov decision process model. Furthermore, it incorporates a data fusion performance evaluation function based on virtual data into the design of the reward function, thereby establishing an end-to-end underwater multi-unmanned platform data fusion algorithm. The end-to-end underwater multi-unmanned platform data fusion algorithm includes a data fusion task state space, a data fusion task action space, a data fusion task reward function, and a multi-platform data fusion strategy. By combining the end-to-end underwater multi-unmanned platform scheduling algorithm and the end-to-end underwater multi-unmanned platform data fusion algorithm, the Markov decision process model is established. The MAPPO algorithm in deep reinforcement learning is introduced to learn the multi-platform scheduling strategy and the multi-platform data fusion strategy. The effectiveness of the current strategy is evaluated through environmental feedback, and finally the optimal tracking strategy in the current environment is determined.

2. The underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning according to claim 1, characterized in that, In the end-to-end underwater multi-unmanned platform scheduling algorithm described above Scheduling task state space middle, k The state space representation of the scheduled task at time t is as follows: ； In the formula, for k The state space of the scheduling task at any given moment; for k System tracking error at any given time; for k The system's remaining energy at a given time; To select a subset composed of ordinary platforms; Scheduling task action space middle, k The task action space at any given time is represented as follows: ； In the formula, for k The action space of scheduling tasks at any given moment; A set of candidate general platforms; for The number of platforms in China; Scheduled task reward function Represented as, ； In the formula, To coordinate system time tracking; For instant rewards; For settlement of rewards; In the process of generating multi-platform scheduling strategies, a Plane Decision Process (PPO) is introduced, based on the established Markov decision process model, to generate scheduling strategies. The PPO consists of an execution network and an evaluation network; the execution network is used for strategy generation, and the evaluation network is used for strategy evaluation. End-to-end scheduling strategies are generated based on the execution network within the PPO. k The loss function of the network at time t is shown below. ； In the formula, These are the execution network parameters in the scheduling task; This is a truncation function used to evaluate the scheduling policy function. Controlled Inside, For hyperparameters; As shown below, ； In the formula, For parameter-based Strategies; For parameter-based Strategies; They are respectively k The scheduling task actions and status in real time; for k The advantage estimation function at time 1 is shown below. ； In the formula, The total time required for the collaborative tracking task; Value function; All are discount factors; This is the cumulative time difference error; The input to the execution network is the scheduling status of multiple unmanned platforms, and the output is the generated scheduling strategy for multiple unmanned platforms.

3. The underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning according to claim 2, characterized in that, Instant rewards are represented as follows: ， ； In the formula, A joint factor for adjusting the ratio of the observation performance function and the energy loss function; A set of candidate general platforms; For observation performance function; This is the energy loss function; ， ； In the formula, Fisher's information matrix; ； In the formula, This indicates the predicted location information of the target. This indicates the relative positional information between the candidate ordinary platform and the target predicted location; ； ， ； In the formula, N This refers to the number of ordinary platforms; For candidate platforms Remaining energy at any given moment; and These respectively represent the candidate platforms in Energy consumption required for constant communication with the central platform and ordinary platforms.

4. The underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning according to claim 2, characterized in that, Settlement rewards are represented as follows: ， ； In the formula, A joint factor to balance the proportions of target tracking accuracy and system energy efficiency; For each round of training, the tracking accuracy is determined; For target accuracy; Target energy consumption; Energy consumption of the system per round of training; Total number of training rounds; ； In the formula, T For the duration of system tracking; ； ； In the formula, for A typical platform Virtual measurement of time and actual measurement Virtual deviation between; for time and The Euclidean distance between them; ； In the formula, For the generated virtual measurement; The measurement equation is known; To pass state estimation The generated measurement prediction values; To measure the noise covariance.

5. The underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning according to claim 2, characterized in that, In the end-to-end underwater multi-unmanned platform data fusion algorithm described above Data fusion task state space middle, k Time-based data fusion task state space Represented as, ； In the formula, Representative fusion estimate of covariance; This represents data fusion estimation; The total number of platforms participating in data fusion; and Representative Platform Local estimation of covariance and local state estimation, For the platform Corresponding data fusion weights; Data fusion task action space middle, k The action space of the data fusion task at any given time is represented as follows: ； Data fusion task reward function Represented as, ； In the formula, For the duration of system collaborative tracking; For instant rewards; For settlement of rewards; In the process of generating data fusion strategies, an end-to-end multi-platform data fusion strategy is generated based on the established Markov decision process model and the execution network in the PPO; among which, k At what time, the loss function of the execution network for the data fusion task Represented as, ； In the formula, For execution network parameters in data fusion tasks; This is the evaluation function for the data fusion strategy; ； In the formula, For parameter-based Data strategy; For parameter-based Strategies; They are respectively k Real-time data fusion task actions and data fusion task status; for k The advantage estimation function in the time-matter fusion task is shown below. ； The input to the execution network is the data fusion state, and the output is the generated data fusion strategy.

6. The underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning according to claim 5, characterized in that, Instant rewards Represented as, ， ； ； ； In the formula, For those participating in data fusion The platform obtained Virtual measurement of time and actual measurement Virtual deviation between; for time and The Euclidean distance between them.

7. The underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning according to claim 5, characterized in that, Settlement Rewards Represented as, ， ； ； In the formula, To estimate the target accuracy for data fusion; Estimate the accuracy of the fusion data for each round of training; This refers to the total number of training rounds.

8. The underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning according to claim 5, characterized in that, The steps involved in establishing the Markov decision process model by combining the end-to-end underwater multi-unmanned platform scheduling algorithm and the end-to-end underwater multi-unmanned platform data fusion algorithm, introducing the MAPPO algorithm from deep reinforcement learning to learn the multi-platform scheduling strategy and the multi-platform data fusion strategy simultaneously, evaluating the effectiveness of the current strategy through environmental feedback, and finally determining the optimal tracking strategy in the current environment, specifically include: Based on the fundamental principles of MAPPO, the multi-unmanned platform scheduling task and the multi-unmanned platform data fusion task are treated as two intelligent agents performing independent tasks. These two agents interact with the environment through their respective execution networks to update their policies. A global evaluation network acquires the global state of both tasks and evaluates the policies generated by the two agents, achieving coordinated policy updates for both tasks and establishing an end-to-end underwater multi-unmanned platform cooperative tracking framework. Global task state space modeling as ; The global evaluation network combines the global task status to evaluate the strategies of both scheduling and data fusion tasks, enabling the strategies of both to be updated collaboratively in a global context. The loss function of the global evaluation network is shown below. ； In the formula, To evaluate network parameters globally; Reporting for accumulated discounts; The number of agents; The value function of the new strategy; The old strategy's value function; The input to the global evaluation network is the global task state, and the output is the policy evaluation value.

9. The underwater multi-unmanned platform cooperative tracking method based on deep reinforcement learning according to claim 8, characterized in that, After establishing the end-to-end underwater multi-unmanned platform cooperative tracking framework, the specific process of end-to-end underwater multi-unmanned platform cooperative tracking includes: (1) Initialization: Initialize the network parameters for scheduling task execution. Network parameters for data fusion task execution Global evaluation network parameters Initialize discount factor , and number of iterations Initialize the scheduling task state space Data fusion task state space and global task state space Track task-related parameters; (2) Action update: Input the current scheduling status into the execution network of the scheduling task and the data fusion task respectively. The data fusion status and the old actions selected in the previous moment. Generate new actions at the current moment. ; (3) Reward update: Update the rewards for scheduling tasks and data fusion tasks based on the new actions at the current moment; (4) Global evaluation update: by inputting the global state at the current moment. Scheduling task actions Data fusion strategy Update the evaluation values ​​of the scheduling task strategy and the data fusion task strategy respectively; (5) Execution network update: The gradient descent method is used to solve the loss function of the scheduling task execution network and the loss function of the data fusion task execution network respectively, so as to update the execution network parameters and thus realize the policy update; (6) Global evaluation network update: The global evaluation function loss function is solved by using the gradient descent method to minimize the loss, thereby updating the parameters of the global evaluation network; (7) Repeat steps (2) to (6). This time, we completed end-to-end collaborative tracking of multiple unmanned underwater platforms.

10. A cooperative tracking system for multiple unmanned underwater platforms based on deep reinforcement learning, characterized in that, include: The first algorithm construction module is used to model the dynamic scheduling problem of underwater multi-unmanned platforms in complex underwater environments as a Markov decision process model using reinforcement learning. In the design of the reward function, the energy efficiency optimization objective function and the tracking performance evaluation function based on the virtual data method are combined to establish an end-to-end underwater multi-unmanned platform scheduling algorithm. The end-to-end underwater multi-unmanned platform scheduling algorithm includes a scheduling task state space, a scheduling task action space, a scheduling task reward function, and a multi-platform scheduling strategy. The second algorithm construction module is used to model the underwater multi-unmanned platform data fusion problem under multi-source dynamic information as a Markov decision process model using reinforcement learning, and to combine a data fusion performance evaluation function based on virtual data method in the design of the reward function to establish an end-to-end underwater multi-unmanned platform data fusion algorithm; wherein, the end-to-end underwater multi-unmanned platform data fusion algorithm includes a data fusion task state space, a data fusion task action space, a data fusion task reward function, and a multi-platform data fusion strategy; The tracking module is used to build a Markov decision process model by combining the end-to-end underwater multi-unmanned platform scheduling algorithm and the end-to-end underwater multi-unmanned platform data fusion algorithm. It introduces the MAPPO algorithm in deep reinforcement learning to learn the multi-platform scheduling strategy and the multi-platform data fusion strategy. It evaluates the effectiveness of the current strategy through environmental feedback and finally determines the optimal tracking strategy in the current environment.

Images