Meta-reinforcement learning based multi-task acoustic decoy policy transfer method and system

Abstract

The application provides a multi-task acoustic decoy strategy migration method and system based on meta-reinforcement learning, comprising the following steps: constructing a meta-strategy model based on an acoustic decoy agent; obtaining observation information of the agent based on a multi-agent acoustic decoy-oriented confrontation simulation environment; collecting a four-tuple generated at each time step; the observation information of the agent at time t; the action of the agent at time t; the reward at time t; the observation information of the agent at time t+1; obtaining a trajectory set through multi-batch parallel simulation, representing the number of agents; training the meta-strategy model by using the trajectory set; and obtaining the corresponding acoustic decoy strategy of the agent based on the observation information of the agent by using the trained meta-strategy model.

Description

Technical Field

[0001] This invention relates to the field of strategy transfer technology for acoustic decoys autonomously countering torpedoes that perform multiple tasks, and more specifically, to a multi-task acoustic decoy strategy transfer method and system based on meta-reinforcement learning. Background Technology

[0002] In naval warfare, torpedoes remain one of the most significant threats to surface ships and submarines. To reduce the probability of being hit by torpedoes, acoustic decoys are widely used in naval equipment as a non-lethal (soft-kill) countermeasure. Existing acoustic decoys can be categorized into several types based on their deployment and mobility, including those that can be suspended and stationary, those launched by aircraft, those towed by ships, and those with autonomous movement capabilities. Each type of decoy has its own emphasis in terms of deployment method, mobility, and operational purpose, playing different roles in different tactical scenarios. Decoy control strategies generally fall into two categories: one is based on artificial rules or parameterized trajectories, designing decoy behavior patterns through experience or heuristic algorithms; the other relies on machine learning methods, especially modeling decoy countermeasures as Markov decision processes and employing reinforcement learning automatic learning strategies. While rule-based methods are simple to implement, their robustness to complex and ever-changing enemy situations is limited. Learning-based methods can automatically generate control strategies under complex dynamics and diverse target behaviors, but existing work mostly focuses on single torpedo models or single operating modes, lacking generalization and transfer mechanisms for various target search / guidance behaviors. In actual combat, torpedo search and attack behaviors often exhibit diversity, typically categorized into two main search patterns: linear search and circular search. Linear search can manifest as a straight trajectory or a "Z" shaped trajectory, while circular search may involve stationary circling or a figure-eight trajectory. Faced with these different search patterns, if the decoy strategy is only trained in one mode, its performance often degrades when deployed to another mode, making it difficult to meet the requirements of cross-scenario combat.

[0003] To achieve the goal of "one-time training, multiple applicability," a mechanism capable of capturing task differences and enabling rapid transfer needs to be introduced into the strategy structure and training paradigm. To verify the applicability of the method, this invention conducted experimental verification on two types of decoy platforms: a simple suspended acoustic decoy and a platform with motion control capabilities (self-propelled or maneuverable decoy). The experimental design aimed to train the strategy under a single torpedo behavior mode and verify its generalization ability and performance retention in the other three modes, covering mixed action scenarios such as continuous motion control output and discrete operating mode switching. Based on this, this invention proposes a multi-task acoustic decoy strategy transfer method and system based on meta-reinforcement learning. By learning task-level latent variables, the strategy is parameterized into a conditional policy that simultaneously accepts environmental observation and task embedding, thus enabling rapid transfer to other target tasks after training on a single source task. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a multi-task acoustic decoy policy transfer method and system based on meta-reinforcement learning.

[0005] A multi-task acoustic decoy policy transfer method based on meta-reinforcement learning, provided by the present invention, includes: Step S1: Construct a meta-policy model based on the acoustic decoy agent; Step S2: Construct an adversarial simulation environment for multi-agent acoustic decoys, and obtain the observation information of the agents based on the constructed adversarial simulation environment for multi-agent acoustic decoys. ; Step S3: Collect the quadruplets generated at each time step ;in, This represents the agent's observation information at time t; Let t be the action of the agent. Reward at time t; The data represents the agent's observations at time t+1; T is the time step; the trajectory set is obtained through multiple batches of parallel simulations. ,in, Indicates the number of agents; ; Step S4: Train the meta-policy model using the trajectory set to obtain the trained meta-policy model; Step S5: Use the trained meta-policy model to obtain the corresponding acoustic decoy strategy for the agent based on the agent's observation information; The intelligent agent is constructed based on the Actor-Critic algorithm; the acoustic decoy intelligent agent includes a mobile acoustic decoy intelligent agent and a non-mobile acoustic decoy intelligent agent.

[0006] Preferably, the observation information of the agent in step S2 This includes: distance thresholds for different torpedo patterns, the torpedo's azimuth angle or relative angle difference with the decoy, the distance between the submarine and the decoy, the submarine's azimuth angle with the decoy, the one-hot representation of the torpedo's current state machine intent, and the normalized time of the current simulation step; wherein, the different torpedo patterns include: straight line, zigzag, stationary rotation, and figure-eight trajectory.

[0007] Preferably, step S3 includes: The action of the agent at time t It is based on the current policy distribution Generate; where, These are private latent variables; The reward at time t include:

[0008] in, These are the weighting coefficients for each reward item, used to balance the relative importance between submarine survival and decoy effectiveness; As a reward for submarine survivability; Rewards are given for the effectiveness of acoustic decoys in attracting customers; ; in, Indicates the submarine at a certain time The reward; ; express The reward ;

[0009] in, , These are the weights; This is an indicator function used to characterize the success or failure of the interaction between the decoy and the torpedo.

[0010] Preferably, step S4 includes: during training, introducing a Kullback-Leibler divergence loss term to reflect the latent distribution of the task. Towards standard normal distribution Alignment:

[0011] Preferably, step S5 includes: For each intelligent agent Its trajectory obtained through interaction with the environment is as follows:

[0012] via variational task encoder Mapping to potential task distribution parameters:

[0013] in, This represents the mean of the potential distribution of the task. The standard deviation represents the potential distribution of the task. By employing reparameterization techniques, continuous latent vectors are sampled from this distribution:

[0014] in, This represents a noise sample that follows a standard normal distribution. Indicates the first An agent's implicit understanding and compressed representation of the current task is used to guide its subsequent policy selection.

[0015] A multi-task acoustic decoy policy transfer system based on meta-reinforcement learning, provided by the present invention, includes: Module M1: Constructing a meta-policy model based on acoustic decoy agents; Module M2: Constructs an adversarial simulation environment for multi-agent acoustic decoys, and obtains the observation information of the agents based on the constructed adversarial simulation environment for multi-agent acoustic decoys. ; Module M3: Collects the quadruplets generated at each time step. ;in, This represents the agent's observation information at time t; Let t be the action of the agent. Reward at time t; The data represents the agent's observations at time t+1; T is the time step; the trajectory set is obtained through multiple batches of parallel simulations. ,in, Indicates the number of agents; ; Module M4: Uses the trajectory set to train the meta-policy model, resulting in the trained meta-policy model; Module M5: Utilizes the trained meta-policy model to obtain the corresponding acoustic decoy strategy for the agent based on the agent's observation information; The intelligent agent is constructed based on the Actor-Critic algorithm; the acoustic decoy intelligent agent includes a mobile acoustic decoy intelligent agent and a non-mobile acoustic decoy intelligent agent.

[0016] Preferably, the observation information of the agent in module M2 This includes: distance thresholds for different torpedo patterns, the torpedo's azimuth angle or relative angle difference with the decoy, the distance between the submarine and the decoy, the submarine's azimuth angle with the decoy, the one-hot representation of the torpedo's current state machine intent, and the normalized time of the current simulation step; wherein, the different torpedo patterns include: straight line, zigzag, stationary rotation, and figure-eight trajectory.

[0017] Preferably, the module M3 includes: The action of the agent at time t It is based on the current policy distribution Generate; where, These are private latent variables; The reward at time t include:

[0018] in, These are the weighting coefficients for each reward item, used to balance the relative importance between submarine survival and decoy effectiveness; As a reward for submarine survivability; Rewards are given for the effectiveness of acoustic decoys in attracting customers; ; in, Indicates the submarine at a certain time The reward; ; express The reward ;

[0019] in, , These are the weights; This is an indicator function used to characterize the success or failure of the interaction between the decoy and the torpedo.

[0020] Preferably, module M4 includes: during training, introducing a Kullback-Leibler divergence loss term to reflect the latent distribution of the task. Towards standard normal distribution Alignment:

[0021] Preferably, the module M5 includes: For each intelligent agent Its trajectory obtained through interaction with the environment is as follows:

[0022] via variational task encoder Mapping to potential task distribution parameters:

[0023] in, This represents the mean of the potential distribution of the task. The standard deviation represents the potential distribution of the task. By employing reparameterization techniques, continuous latent vectors are sampled from this distribution:

[0024] in, This represents a noise sample that follows a standard normal distribution. Indicates the first An agent's implicit understanding and compressed representation of the current task is used to guide its subsequent policy selection.

[0025] Compared with the prior art, the present invention has the following beneficial effects: 1. As the number of training iterations increases, the cumulative reward of the agent gradually rises and tends to stabilize after a certain number of rounds. At the same time, the task success rate also exhibits similar convergence characteristics. In the initial stage, the success rate is low because the strategy has not yet been formed. With the accumulation of experience and continuous optimization of strategy updates, the success rate significantly improves and eventually converges to a high level. This result shows that the designed training mechanism can effectively drive the agent to gradually transition from random exploration to stable execution of the strategy, and has strong task adaptability and convergence. 2. This invention realizes the transfer of multi-task acoustic decoy strategies based on meta-reinforcement learning under different torpedo behavior modes; 3. This invention significantly enhances the sensitivity of the strategy to task changes, enabling the agent to quickly infer potential vectors and adapt when faced with unseen tasks, thereby achieving efficient transfer and robust generalization to complex dynamic environments. Attached Figure Description

[0026] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a multi-task acoustic decoy policy transfer method based on meta-reinforcement learning.

[0027] Figures 2a to 2b This diagram illustrates the convergence of success rate and reward with the number of training steps.

[0028] Figure 3 This is a diagram illustrating the performance comparison of different tasks. Detailed Implementation

[0029] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0030] Example 1 The multi-task acoustic decoy policy transfer method based on meta-reinforcement learning provided by the present invention, such as Figure 1 As shown, it includes: Step S1: Construct a meta-policy model based on the acoustic decoy agent; Specifically, the intelligent agent is an intelligent agent constructed based on the Actor-Critic algorithm; The acoustic decoy agent includes a mobile acoustic decoy agent and a non-mobile acoustic decoy agent; Acoustic decoys are key countermeasures for submarines used to confuse torpedoes, and are divided into two categories: movable (DecoyMove) and immovable (DecoyNoMove). Acoustic decoys possess different operating modes, such as deception, silence, or jamming. The deception mode simulates the acoustic characteristics of a submarine, while the jamming mode disrupts torpedo detection through noise signals. Movable acoustic decoys move in a set or random direction after launch, creating dynamic false targets; while immovable decoys remain in place after launch, attracting torpedo attention by continuously radiating signals. The strategic design of acoustic decoys is closely related to the timing of submarine deployment, together forming the submarine's defense system.

[0031] Each agent is internally configured with a task encoder based on variational inference, which generates private latent variables using only the agent's own trajectory information. Latent variables are not shared among agents. The task encoder parameters of each agent remain independent and are optimized together with the meta-policy model, thereby avoiding interference and collapse of latent representations among multiple agents. This independent design enables different individuals to construct differentiated task latent vectors based on their own trajectories, forming diverse policy patterns at the group level.

[0032] Step S2: Construct an adversarial simulation environment for multi-agent acoustic decoys, and obtain the observation information of the agents based on the constructed adversarial simulation environment for multi-agent acoustic decoys. ; Specifically, step S2 includes: First, an adversarial simulation environment for multi-agent acoustic decoys was established to simulate the dynamic interaction between the decoys and incoming torpedoes. This environment is based on a two-dimensional continuous space, and the kinematic and sensory characteristics of the torpedoes and acoustic decoys are defined to ensure the scalability and diversity of the experimental environment.

[0033] More specifically, let the set of enemy torpedo targets be:

[0034] Each torpedo target Having position vector velocity vector and perception radius .

[0035] The sensing area of ​​a torpedo in the environment can be defined as:

[0036] Its search pattern varies depending on the task settings, including linear, zigzag, stationary rotation, and figure-eight trajectories; Taking a linear torpedo as an example, its motion can be modeled as follows:

[0037] in, For time step; The figure-eight trajectory can be represented by a parametric curve, for example:

[0038] in, These are the trajectory amplitude and frequency parameters.

[0039] By randomizing the initial conditions and parameter settings, a wide variety of task instances can be generated under the same search mode, thereby improving the coverage of training samples.

[0040] The acoustic decoy set is defined as:

[0041] Each decoy has a position vector. The control variable is speed. and direction of movement .

[0042] Different types of decoys differ in their dynamic constraints: suspended decoys remain essentially stationary and their acoustic characteristics can be altered by switching operating modes; self-propelled decoys, on the other hand, possess a continuously controllable dynamic model, and their position evolution is determined by: The decision was made.

[0043] Furthermore, to simulate a realistic underwater acoustic environment, this invention incorporates simplified representations of multipath propagation and noise interference during the modeling process, modeling the torpedo's observation information as follows: 1 in, For an ideal acoustic propagation model, 1 represents the noise level. This design ensures that the torpedo's search and identification process has a certain degree of uncertainty, making it closer to real combat conditions.

[0044] Based on the adversarial simulation environment for multi-agent acoustic decoys constructed above, the observation information of the agents is obtained. This includes: distance thresholds for different torpedo modes, the torpedo's azimuth angle or relative angle difference with the decoy, the distance between the submarine and the decoy, the submarine's azimuth angle with the decoy, the one-hot representation of the torpedo's current state machine intent, and the normalized time of the current simulation step; wherein, the different torpedo modes include: straight line, zigzag, stationary rotation, and figure-eight trajectory.

[0045] Step S3: Collect the quadruplets generated at each time step ;in, This represents the agent's observation information at time t; Let t be the action of the agent. Reward at time t; The data represents the agent's observations at time t+1; T is the time step; the trajectory set is obtained through multiple batches of parallel simulations. ,in, Indicates the number of agents; These datasets maintain consistency with the current policy, ensuring the unbiasedness of policy gradient estimation, thus providing reliable support for subsequent meta-policy construction and hierarchical training.

[0046] More specifically, step S3 includes: The action of the agent at time t It is based on the current policy distribution Generation, independent of offline samples or external exploration noise; among which, These are private latent variables; The reward at time t This includes: abstracting the problem into a Markov Decision Process (MDP), at each time step... The environment depends on the current state. Combined action with acoustic decoys Generate a new state and give a reward To ensure consistency between the learning objectives and actual combat needs, the reward function considers both submarine survivability and the effectiveness of acoustic decoys.

[0047]

[0048] in, These are the weighting coefficients for each reward item, used to balance the relative importance between submarine survival and decoy effectiveness; As a reward for submarine survivability; Rewards are given for the effectiveness of acoustic decoys in attracting customers; Submarine survivability is modeled using a binarized function. A positive reward is given when the submarine is not hit by a torpedo, while a strong negative penalty is imposed when the submarine is hit. Its mathematical form is: ; in, Indicates the submarine at a certain time The reward; ; express The reward This is to highlight the importance of submarine survival; The effectiveness of acoustic decoys is measured by whether the torpedo is successfully attracted or interfered with. If the torpedo locks onto the acoustic decoy within a certain time, an additional positive reward is given; if the acoustic decoy is quickly identified or becomes ineffective, a negative reward is given. The expression is as follows:

[0049] in, , These are the weights; This is an indicator function used to characterize the success or failure of the interaction between the decoy and the torpedo.

[0050] Step S4: Train the meta-policy model using the trajectory set to obtain the trained meta-policy model; More specifically, step S4 includes: during training, introducing a Kullback-Leibler divergence loss term to reflect the latent distribution of the task. Towards standard normal distribution Alignment: .

[0051] Step S5: Use the trained meta-policy model to obtain the corresponding acoustic decoy strategy for the agent based on the agent's observation information; Specifically, step S5 includes: For each intelligent agent Its trajectory obtained through interaction with the environment is as follows:

[0052] via variational task encoder Mapping to potential task distribution parameters:

[0053] in, This represents the mean of the potential distribution of the task. The standard deviation represents the potential distribution of the task. By employing reparameterization techniques, continuous latent vectors are sampled from this distribution:

[0054] in, This represents a noise sample that follows a standard normal distribution. Indicates the first An agent's implicit understanding and compressed representation of the current task is used to guide its subsequent policy selection.

[0055] In this embodiment, during the execution phase, the latent vector and environmental observations are jointly input into the meta-policy model as the core variables for conditional decision-making. This mechanism significantly enhances the policy's sensitivity to task changes, enabling the agent to quickly infer the latent vector and adapt when faced with unseen tasks, thereby achieving efficient transfer and robust generalization to complex dynamic environments.

[0056] The present invention also provides a multi-task acoustic decoy policy transfer system based on meta-reinforcement learning. The multi-task acoustic decoy policy transfer system based on meta-reinforcement learning can be implemented by executing the process steps of the multi-task acoustic decoy policy transfer method based on meta-reinforcement learning. That is, those skilled in the art can understand the multi-task acoustic decoy policy transfer method based on meta-reinforcement learning as a preferred embodiment of the multi-task acoustic decoy policy transfer system based on meta-reinforcement learning.

[0057] Example 2 Example 2 is a preferred example of Example 1. A multi-task acoustic decoy policy transfer method based on meta-reinforcement learning, provided by the present invention, includes: First, a batch of task instances with different difficulties and characteristics are dynamically sampled from a preset task distribution. The current meta-policy model is then deployed in the corresponding environment to interact with the task, generating complete trajectory data including state, action, reward, and environmental feedback. All trajectory data is stored uniformly in an experience buffer and filtered based on reward sparsity and sample novelty to ensure that the data used during training has sufficient information gain and coverage.

[0058] Subsequently, a small batch of training samples is constructed based on the prioritized sampling trajectory to iteratively optimize the parameters of the meta-policy model. During the optimization process, a dynamic difficulty adjustment mechanism is introduced, which adaptively adjusts the task complexity based on recent success rates and convergence trends, thereby avoiding overfitting to simple scenarios or prematurely hitting performance bottlenecks in high-difficulty tasks.

[0059] In this embodiment, during training, a fixed evaluation interval is set to perform performance tests on an independent set of validation tasks for the current policy. Evaluation metrics include average reward and task success rate. The former measures the overall stability and efficiency of the policy over long-term interactions, while the latter reflects its ability to achieve key task objectives. By combining these two metrics, the policy's performance in generalization, reliability, and robustness can be comprehensively characterized. When performance stagnation or excessive fluctuations are detected, the system will trigger adaptive adjustments to the training schedule, such as increasing the diversity of sampling tasks or reallocating the sampling weights of the experience buffer, thereby further improving training efficiency and final performance.

[0060] like Figure 2a, Figure 2b , Figure 3 To verify the generalization performance of this invention in cross-task scenarios, transfer tests were conducted in four different but related task environments. The test results show that the agent exhibits strong adaptability without retraining. Its average reward and success rate in the four tasks remain at a high level, and the trend of the line graph is relatively smooth, indicating that the strategy has a certain degree of robustness and universality. This shows that the parameter sharing mechanism and policy update design introduced in the iterative training process not only improve the learning efficiency in single tasks, but also provide strong support for rapid adaptation in multi-task scenarios.

[0061] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0062] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A multi-task acoustic decoy policy transfer method based on meta-reinforcement learning, characterized in that, include: Step S1: Construct a meta-policy model based on the acoustic decoy agent; Step S2: Construct an adversarial simulation environment for multi-agent acoustic decoys, and obtain the observation information of the agents based on the constructed adversarial simulation environment for multi-agent acoustic decoys. ; Step S3: Collect the quadruplets generated at each time step ;in, This represents the agent's observation information at time t; Let t be the action of the agent. Reward at time t; The data represents the agent's observations at time t+1; T is the time step; the trajectory set is obtained through multiple batches of parallel simulations. ,in, Indicates the number of agents; ; Step S4: Train the meta-policy model using the trajectory set to obtain the trained meta-policy model; Step S5: Use the trained meta-policy model to obtain the corresponding acoustic decoy strategy for the agent based on the agent's observation information; The intelligent agent is constructed based on the Actor-Critic algorithm; the acoustic decoy intelligent agent includes a mobile acoustic decoy intelligent agent and a non-mobile acoustic decoy intelligent agent. Step S4 includes: during training, introducing a Kullback-Leibler divergence loss term to reflect the latent distribution of the task. Towards standard normal distribution Alignment: ； Step S5 includes: For each intelligent agent Its trajectory obtained through interaction with the environment is as follows: via variational task encoder Mapping to potential task distribution parameters: in, This represents the mean of the potential distribution of the task. The standard deviation represents the potential distribution of the task. By employing reparameterization techniques, continuous latent vectors are sampled from this distribution: in, This represents a noise sample that follows a standard normal distribution. Indicates the first An agent's implicit understanding and compressed representation of the current task is used to guide its subsequent policy selection.

2. The multi-task acoustic decoy policy transfer method based on meta-reinforcement learning according to claim 1, characterized in that, The observation information of the agent in step S2 This includes: distance thresholds for different torpedo patterns, the torpedo's azimuth angle or relative angle difference with the decoy, the distance between the submarine and the decoy, the submarine's azimuth angle with the decoy, the one-hot representation of the torpedo's current state machine intent, and the normalized time of the current simulation step; wherein, the different torpedo patterns include: straight line, zigzag, stationary rotation, and figure-eight trajectory.

3. The multi-task acoustic decoy policy transfer method based on meta-reinforcement learning according to claim 1, characterized in that, Step S3 includes: The action of the agent at time t It is based on the current policy distribution Generate; where, These are private latent variables; The reward at time t include: in, These are the weighting coefficients for each reward item, used to balance the relative importance between submarine survival and decoy effectiveness; As a reward for submarine survivability; Rewards are given for the effectiveness of acoustic decoys in attracting customers; ； in, Indicates the submarine at a certain time The reward; ; express The reward ; in, , These are the weights; This is an indicator function used to characterize the success or failure of the interaction between the decoy and the torpedo.

4. A multi-task acoustic decoy policy transfer system based on meta-reinforcement learning, characterized in that, include: Module M1: Constructing a meta-policy model based on acoustic decoy agents; Module M2: Constructs an adversarial simulation environment for multi-agent acoustic decoys, and obtains the observation information of the agents based on the constructed adversarial simulation environment for multi-agent acoustic decoys. ; Module M3: Collects the quadruplets generated at each time step. ;in, This represents the agent's observation information at time t; Let t be the action of the agent. Reward at time t; The data represents the agent's observations at time t+1; T is the time step; the trajectory set is obtained through multiple batches of parallel simulations. ,in, Indicates the number of agents; ; Module M4: Uses the trajectory set to train the meta-policy model, resulting in the trained meta-policy model; Module M5: Utilizes the trained meta-policy model to obtain the corresponding acoustic decoy strategy for the agent based on the agent's observation information; The intelligent agent is constructed based on the Actor-Critic algorithm; the acoustic decoy intelligent agent includes a mobile acoustic decoy intelligent agent and a non-mobile acoustic decoy intelligent agent. The module M4 includes: during training, introducing a Kullback–Leibler divergence loss term to represent the latent distribution of the task. Towards standard normal distribution Alignment: ； The module M5 includes: For each intelligent agent Its trajectory obtained through interaction with the environment is as follows: via variational task encoder Mapping to potential task distribution parameters: in, This represents the mean of the potential distribution of the task. The standard deviation represents the potential distribution of the task. By employing reparameterization techniques, continuous latent vectors are sampled from this distribution: in, This represents a noise sample that follows a standard normal distribution. Indicates the first An agent's implicit understanding and compressed representation of the current task is used to guide its subsequent policy selection.

5. The multi-task acoustic decoy policy transfer system based on meta-reinforcement learning according to claim 4, characterized in that, The observation information of the agent in module M2 This includes: distance thresholds for different torpedo patterns, the torpedo's azimuth angle or relative angle difference with the decoy, the distance between the submarine and the decoy, the submarine's azimuth angle with the decoy, the one-hot representation of the torpedo's current state machine intent, and the normalized time of the current simulation step; wherein, the different torpedo patterns include: straight line, zigzag, stationary rotation, and figure-eight trajectory.

6. The multi-task acoustic decoy policy transfer system based on meta-reinforcement learning according to claim 4, characterized in that, The module M3 includes: The action of the agent at time t It is based on the current policy distribution Generate; where, These are private latent variables; The reward at time t include: in, These are the weighting coefficients for each reward item, used to balance the relative importance between submarine survival and decoy effectiveness; As a reward for submarine survivability; Rewards are given for the effectiveness of acoustic decoys in attracting customers; ； in, Indicates the submarine at a certain time The reward; ; express The reward ; in, , These are the weights; This is an indicator function used to characterize the success or failure of the interaction between the decoy and the torpedo.

Images