A method and apparatus for calculating the reward value of a reinforcement learning acoustic decoy.

Abstract

This application discloses a method and apparatus for calculating the reward value of acoustic decoys in reinforcement learning. The method establishes variables based on the target recognition processing time and target recognition capability of dipping sonar, providing the target deception success rate and the average recognition processing time after the acoustic decoy appears. This allows for rapid calculation of the reward value, avoiding the problem of long processing times in Monte Carlo methods that affect the training speed of reinforcement learning agents. In some examples, the target success rate is weighted by considering the processing time required for subsequent actions such as attack and recognition after target recognition by an aerial platform, resulting in a more accurate calculation of the acoustic decoy reward value, thus avoiding the problem of long processing times in Monte Carlo methods that affect the training speed of reinforcement learning agents.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a method and apparatus for calculating the reward value of a reinforcement learning acoustic decoy. Background Technology

[0002] The purpose of reinforcement learning is to study and solve the sequential decision-making problem of agents. Reinforcement learning includes the agent, environment, observation state, action, and reward value, among which the reward value is an important factor guiding the agent to approach the training objective. The reward value plays a positive role in cold start and accelerating algorithm convergence in reinforcement learning.

[0003] In related technologies, by determining the reward threshold corresponding to a user, and based on the reward threshold and the pre-set reward values ​​corresponding to each user's single click, the probability of each reward value appearing within a predetermined time period is determined. Based on the probability, the reward value obtained by the user for each click within the predetermined time period is determined, so that the expected value of the reward value obtained by the user within the predetermined time period is less than or equal to the reward threshold, thereby controlling the reward value obtained by the user for each click and reducing costs.

[0004] The reward value cannot be automatically provided by the environment and is almost entirely determined by the agent designer. The reward value design is mainly based on relevant business knowledge. The acoustic decoy is equivalent to a submarine target T_sd with a constant target strength, possessing the ability to deceive active sonar such as dipping sonar. Currently, domestic research on the countermeasure effectiveness of acoustic decoys mainly focuses on torpedo countermeasures, lacking methods for countering other underwater acoustic equipment such as dipping sonar and calculating their effectiveness. This hinders the ability to guide agents to explore countermeasure methods outside of expert rules. The commonly used Monte Carlo method can calculate the effectiveness of acoustic decoys against dipping sonar, but it suffers from long computation times, affecting the training speed of reinforcement learning agents. Summary of the Invention

[0005] To solve or partially solve the above problems, this application provides a method and apparatus for calculating the reward value of a reinforcement learning acoustic decoy.

[0006] This application proposes a method for calculating the reward value of a reinforcement learning acoustic decoy. The method includes: initializing the identification processing time of the dipping sonar and a target identification capability variable; calculating the target identification capability variable based on the target identification capability variable to determine the success rate of a submarine using a preset number of acoustic decoys to counter the dipping sonar; calculating the average identification processing time of the acoustic decoy based on the dipping sonar identification processing time and the target identification capability variable; and determining the reward value of the acoustic decoy based on the average identification processing time and the target success rate.

[0007] In some examples, before calculating the reward value of the acoustic decoy based on the average recognition processing time and the target success rate, the method further includes: defining a processing time for processing the acoustic decoy based on the acoustic decoy, the processing time being used to weight the target success rate.

[0008] In some examples, initializing the dipping sonar identification processing time includes defining a single detection cycle for the dipping sonar and the identification time for each of the acoustic decoys.

[0009] In some examples, the target recognition capability variables include: the recognition rate of successfully recognizing a real target, the recognition rate of failing to recognize a real target, the recognition rate of failing to recognize a false target, and the recognition rate of successfully recognizing a false target; initializing the target recognition capability variables includes: setting the recognition rate of successfully recognizing a real target to α, setting the recognition rate of failing to recognize a real target to 1-α; setting the recognition rate of failing to recognize a false target to 1-β, and setting the recognition rate of successfully recognizing a false target to β.

[0010] In some examples, calculating the success rate of a submarine using a preset number of acoustic decoys to counter dipping sonar targets based on the target recognition capability variable includes: calculating the target success rate using a first preset formula and the target recognition capability variable to calculate the submarine's use of a preset number of acoustic decoys, wherein the first preset formula is:

[0011]

[0012] Where N is the sum of the preset number and the number of real targets, and i is the identification sequence position when a real target is successfully identified.

[0013] In some examples, calculating the average identification processing time of the acoustic decoy based on the dipping sonar identification processing time and the target identification capability variable includes: calculating the average identification processing time of the acoustic decoy based on a second preset formula, the dipping sonar identification processing time, and the target identification capability variable, wherein the second preset formula is:

[0014]

[0015] This application also proposes a device for calculating the reward value of a reinforcement learning acoustic decoy. The device includes: an initialization module for initializing the identification processing time of the dipping sonar and a target identification capability variable; a calculation module for calculating the target success rate of a submarine using a preset number of acoustic decoys to counter the dipping sonar based on the target identification capability variable; and calculating the average identification processing time of the acoustic decoy based on the dipping sonar identification processing time and the target identification capability variable; and a determination module for determining the reward value of the acoustic decoy based on the average identification processing time and the target success rate.

[0016] Compared with the prior art, this application has the following beneficial effects:

[0017] The technical solution provided in this application includes a method for calculating the reward value of a reinforcement learning acoustic decoy, comprising: initializing the identification processing time of the dipping sonar and the target identification capability variable; calculating the target success rate of a submarine using a preset number of acoustic decoys to counter the dipping sonar based on the target identification capability variable; calculating the average identification processing time of the acoustic decoy based on the dipping sonar identification processing time and the target identification capability variable; and determining the reward value of the acoustic decoy based on the average identification processing time and the target success rate. The method for calculating the reward value of a reinforcement learning acoustic decoy provided in this embodiment establishes a system based on the dipping sonar target identification processing time and the target identification capability variable, providing the target decoy's success rate and the average identification processing time after the acoustic decoy appears. This allows for rapid calculation of the reward value, avoiding the problem of long time cycles in Monte Carlo methods that affect the training speed of reinforcement learning agents. In some examples, the target success rate is weighted by considering the processing time required for subsequent operations such as attack and identification after target identification by an airborne platform, resulting in a more accurate calculation of the acoustic decoy reward value, thus avoiding the problem of long time cycles in Monte Carlo methods that affect the training speed of reinforcement learning agents. Attached Figure Description

[0018] Figure 1 This is a basic flowchart illustrating a method for calculating the reward value of a reinforcement learning acoustic decoy, as shown in an embodiment of this application.

[0019] Figure 2 This is a basic schematic diagram illustrating an identification process in an embodiment of this application. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0021] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0023] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0024] Example 1

[0025] Please see Figure 1 , Figure 1 This is an exemplary embodiment illustrating a method for calculating the reward value of a reinforcement learning acoustic decoy, the method comprising:

[0026] S101. Initialize the identification processing time and target identification capability variables of the dipping sonar;

[0027] S102. Calculate the success rate of the submarine using a preset number of acoustic decoys to counter dipping sonar targets based on the target recognition capability variables.

[0028] S103. Calculate the average recognition processing time of the acoustic decoy based on the dipping sonar recognition processing time and the target recognition capability variable;

[0029] S104. Determine the reward value of the acoustic decoy based on the average recognition processing time and the target success rate.

[0030] In some examples, before calculating the reward value of the acoustic decoy based on the average identification processing time and the target success rate, the method further includes: defining a processing time for processing the acoustic decoy based on the acoustic decoy, the processing time being used to weight the target success rate. It should be understood that after the acoustic decoy is successfully deceiving the target, the aerial platform can use weapons to attack. Typically, the aerial platform requires time for pre-attack preparation, attack, and post-attack identification; all of these times are considered as the processing time for processing the acoustic decoy, and this processing time is used as... minute.

[0031] In some examples, initializing the dipping sonar identification processing time includes: defining a single detection cycle for the dipping sonar and the identification time for each of the acoustic decoys; wherein the identification time for each of the acoustic decoys is the identification time for the sonar operator for each target, which is μ detection cycles. A single detection cycle τ for the dipping sonar is defined. Specifically,

[0032]

[0033] Where d max c0 is the maximum detection range of the dipping sonar, and c0 is the speed of sound.

[0034] In some examples, the target recognition capability variables include: the recognition rate of successfully identifying a real target, the recognition rate of failing to identify a real target, the recognition rate of failing to identify a false target, and the recognition rate of successfully identifying a false target. Initializing the target recognition capability variables includes: setting the recognition rate of successfully identifying a real target to α, setting the recognition rate of failing to identify a false target to 1-α, setting the recognition rate of failing to identify a false target to 1-β, and setting the recognition rate of successfully identifying a false target to β. Specifically, the recognition rate of successfully identifying a real target is also the rate of identifying a real target as a real target; the recognition rate of failing to identify a real target is also the rate of identifying a real target as a false target; the recognition rate of failing to identify a false target is also the rate of identifying a false target as a real target; and the recognition rate of successfully identifying a false target is also the rate of identifying a false target as a false target. The sonar operator's recognition rate of identifying a real target as a real target is α, the recognition rate of identifying a real target as a false target is 1-α, the recognition rate of identifying a false target as a real target is 1-β, and the recognition rate of identifying a false target as a false target is β.

[0035] In some examples, calculating the success rate of a submarine using a preset number of acoustic decoys to counter dipping sonar targets based on the target recognition capability variable includes: calculating the target success rate using a first preset formula and the target recognition capability variable to calculate the submarine's use of a preset number of acoustic decoys, wherein the first preset formula is:

[0036]

[0037] Where N is the sum of the preset number and the number of real targets, and i is the identification sequence position when a real target is successfully identified. That is, when there is 1 real target, the total number of real targets and acoustic decoys is N = γ + 1, and i represents the identification sequence position when the sonar operator successfully identifies the real target, provided that all false targets before the real target are successfully identified as false, with a probability of βi. 1 / N is a normalization factor, and the probability of the real target's position is evenly distributed across all positions.

[0038] In some examples, calculating the average identification processing time of the acoustic decoy based on the dipping sonar identification processing time and the target identification capability variable includes: calculating the average identification processing time of the acoustic decoy based on a second preset formula, the dipping sonar identification processing time, and the target identification capability variable, wherein the second preset formula is:

[0039]

[0040] Where, when i = 0, that is, the first selected target is identified as true, the probability is: This indicates that the target has the potential to be identified as a real target or a false target, with corresponding probabilities of 1 / 2 and 2 / 3 respectively. and At this point, the corresponding recognition period is 1 recognition period μτ. When i = 1, that is, the second selected target is identified as true, the premise is that all previously selected targets are identified as false, and the joint probability is... At this point, the corresponding recognition period is 2 recognition periods, or 2μτ; other selected cases follow the same logic. There is a hidden fundamental assumption here: the sonar operator believes that among the multiple targets existing in a small spatial area, only one is a true target, and once one target is identified as true, no further identification is performed. Figure 2 As shown, Figure 2 This is a basic schematic diagram of an identification process. From the perspective of the requirements for safe underwater navigation of submarines, this basic assumption is reasonable.

[0041] In some examples, the reward value for the acoustic decoy is determined based on the average recognition processing time and the target success rate:

[0042] The method for calculating the reward value of a reinforcement learning acoustic decoy provided in this embodiment includes: initializing the identification processing time of the dipping sonar and the target identification capability variable; calculating the target success rate of a submarine using a preset number of acoustic decoys to counter the dipping sonar based on the target identification capability variable; calculating the average identification processing time of the acoustic decoy based on the dipping sonar identification processing time and the target identification capability variable; and determining the reward value of the acoustic decoy based on the average identification processing time and the target success rate. The method for calculating the reward value of a reinforcement learning acoustic decoy provided in this embodiment establishes a system based on the target identification processing time of the dipping sonar and the target identification capability variable, providing the target decoy's success rate and the average identification processing time after the acoustic decoy appears. This allows for rapid calculation of the reward value, avoiding the problem of long time cycles in Monte Carlo methods that affect the training speed of reinforcement learning agents. In some examples, the target success rate is weighted by considering the processing time required for subsequent operations such as attack and identification after target identification by an aerial platform, resulting in a more accurate calculation of the acoustic decoy reward value, thus avoiding the problem of long time cycles in Monte Carlo methods that affect the training speed of reinforcement learning agents.

[0043] For example, assuming that the sonar operator's recognition rate of identifying a real target as a real target is α = 0.8 and the recognition rate of identifying a false target as a false target is β = 0.6, and after being deceived, the time φ = 10 minutes for the aviation platform to perform actions such as attack and recognition confirmation after the attack, then Table 1 shows the success rate of deceiving targets, the average recognition processing time, and the final reward value R after deploying 1 to 4 acoustic decoys.

[0044] Table 1 shows the effectiveness of acoustic decoys under the assumptions α = 0.8, β = 0.6.

[0045] Countermeasures Success rate of luring targets Average recognition processing time (s) Reward value R(s) 1. Bait 36％ 104.4 320.4 2 baits 47.7％ 130.5 416.8 3 baits 56.5％ 150.8 489.7 4 baits 63.1％ 165.6 544.26

[0046] If a submarine deploys decoys between itself and the dipping sonar, and then inserts a noise jammer, it can escape from half the contact distance of the dipping sonar in 410 seconds at high speed (12kt) (assuming the dipping sonar has a detection radius of 12 kilometers). The bonus value of deploying two decoys, 416.8 seconds, can help the submarine successfully counter and escape.

[0047] This application also proposes a device for calculating the reward value of a reinforcement learning acoustic decoy. The device includes: an initialization module for initializing the identification processing time of the dipping sonar and a target identification capability variable; a calculation module for calculating the target success rate of a submarine using a preset number of acoustic decoys to counter the dipping sonar based on the target identification capability variable; and calculating the average identification processing time of the acoustic decoy based on the dipping sonar identification processing time and the target identification capability variable; and a determination module for determining the reward value of the acoustic decoy based on the average identification processing time and the target success rate.

[0048] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0049] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0050] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0051] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0052] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] The technical solutions provided by the embodiments of the present invention have been described in detail above. Specific examples have been used in this patent to illustrate the principles and implementation methods of the embodiments of the present invention. The descriptions of the embodiments above are only for helping to understand the principles of the embodiments of the present invention. The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for calculating a reinforcement learning acoustic decoy reward value, characterized by, The method for calculating the reward value of the acoustic decoy in the reinforcement learning comprises the following steps: initializing the identification processing time of the sonar and a target identification capability variable; calculating the target success rate of the submarine using a preset number of acoustic decoys against the sonar based on the target identification capability variable; calculating the average identification processing time of the acoustic decoy based on the identification processing time of the sonar and the target identification capability variable; determining the reward value of the acoustic decoy based on the average identification processing time and the target success rate; the target identification capability variable comprises an identification rate of successfully identifying a real target, an identification rate of failing to identify a real target, an identification rate of failing to identify a false target, and an identification rate of successfully identifying a false target; initializing the target identification capability variable comprises the following steps: setting the identification rate of successfully identifying a real target as α and the identification rate of failing to identify a real target as 1-α; setting the identification rate of failing to identify a false target as 1-β and the identification rate of successfully identifying a false target as β; calculating the target success rate of the submarine using a preset number of acoustic decoys against the sonar based on the target identification capability variable comprises the following steps: calculating the target success rate of the submarine using a preset number of acoustic decoys based on a first preset formula and the target identification capability variable, wherein the first preset formula is as follows: wherein N is the sum of the preset number and the number of real targets, and i is the sequential position of the identification when a real target is successfully identified; calculating the average identification processing time of the acoustic decoy based on the identification processing time of the sonar and the target identification capability variable comprises the following steps: calculating the average identification processing time of the acoustic decoy based on a second preset formula and the identification processing time of the sonar and the target identification capability variable, wherein the second preset formula is as follows: ； wherein, the number of detection periods occupied by the single target identification time, is a single detection period length of the drape sonar; determining a reward value of the acoustic decoy based on the average recognition processing time and the target success rate is: x ; wherein, is the processing weighted time. 2.The method of claim 1, wherein, before calculating the reward value of the acoustic decoy based on the average identification processing time and the target success rate, the method further comprises the following steps: defining a processing time for processing the acoustic decoy based on the acoustic decoy, and the processing time is used for weighted processing of the target success rate.

3. The method of claim 2, wherein, initializing the identification processing time of the sonar comprises the following steps: defining a single detection period of the sonar and the identification time of each acoustic decoy.

4. A reinforcement learning acoustic decoy reward value computing apparatus characterized by comprising: The device for calculating the reward value of the acoustic decoy in the reinforcement learning comprises the following steps: an initialization module, which is configured to initialize the identification processing time of the sonar and a target identification capability variable; a calculation module, which is configured to calculate the target success rate of the submarine using a preset number of acoustic decoys against the sonar based on the target identification capability variable, and calculate the average identification processing time of the acoustic decoy based on the identification processing time of the sonar and the target identification capability variable; a determination module, which is configured to determine the reward value of the acoustic decoy based on the average identification processing time and the target success rate; the target identification capability variable comprises an identification rate of successfully identifying a real target, an identification rate of failing to identify a real target, an identification rate of failing to identify a false target, and an identification rate of successfully identifying a false target; initializing the target identification capability variable comprises the following steps: setting the identification rate of successfully identifying a real target as α and the identification rate of failing to identify a real target as 1-α; setting the identification rate of failing to identify a false target as 1-β and the identification rate of successfully identifying a false target as β; The recognition rate of failing to recognize the false target is set as 1-β, and the recognition rate of successfully recognizing the false target is set as β; The target success rate of the submarine using a preset number of acoustic decoys to resist the hanging sonar is calculated based on the target recognition capability variable, including: the target success rate of the submarine using a preset number of acoustic decoys is calculated by a first preset formula and the target recognition capability variable, and the first preset formula is: Wherein, N is the sum of the preset number and the number of true targets, and i is the order position of the recognition when the true target is successfully recognized; The average recognition processing time of the acoustic decoy is calculated based on the hanging sonar recognition processing time and the target recognition capability variable, including: the average recognition processing time of the acoustic decoy is calculated based on a second preset formula and the hanging sonar recognition processing time and the target recognition capability variable, and the second preset formula is: ； wherein, the number of detection periods occupied by the single target identification time, is a single detection period length of the drape sonar; determining a reward value of the acoustic decoy based on the average recognition processing time and the target success rate is: x ; wherein, is the processing weighted time.

Images