A deep reinforcement learning strategy transfer method based on fuzzy inference tree

Abstract

In order to solve the problem that the neural network obtained by using the deep reinforcement learning algorithm in the current pursuit and escape game intelligent decision-making has poor interpretability and debuggability, and is difficult to meet the actual deployment requirements, the present invention proposes a deep reinforcement learning strategy migration method based on a fuzzy inference tree. The present invention is guided by a fully trained network to be migrated, and trains the fuzzy inference tree through an intelligent optimization algorithm so that it has the same output as the trained network to be migrated under the same input, that is, the fuzzy inference tree learns the mapping relationship from state to action, and has the same decision-making ability as the deep reinforcement learning network to be migrated, thereby achieving the purpose of extracting the rules contained in the neural network in the deep reinforcement learning algorithm and migrating them to the fuzzy inference tree. Without changing the decision-making efficiency, the present invention improves the generation speed of decision-making instructions, and solves the problems of incomplete knowledge of pursuit and escape game information and poor debuggability and interpretability of decision-making methods.

Description

Technical Field

[0001] The present invention belongs to the field of computers and artificial intelligence. Background Art

[0002] Deep reinforcement learning, as an intelligent decision-making method, boasts strong search capabilities and excellent decision-making performance, making it suitable for solving the challenge of optimal decision-making in complex problems. However, in practice, the neural networks used to represent policies in deep reinforcement learning lack interpretability, making it difficult to effectively analyze and judge the decisions made by the agent, thus hindering the ability to achieve good human-machine trust. Furthermore, the fully connected nature of neural networks makes it difficult to locate and modify abnormal parameters when decisions are made abnormally. This significantly limits the practical deployment of deep reinforcement learning methods in applications such as aircraft control, autonomous driving, and industrial robotics, which require extremely high human-machine trust and policy debuggability.

[0003] Intelligent decision-making in dual-pilot pursuit-and-escape games refers to the process of leveraging technologies such as artificial intelligence and decision-making systems to enable the fleeing party to make assisted or autonomous decisions and thereby evade the pursuing party. This involves interactions between the pursuing party, the fleeing party, decoys, perception systems, and other elements, and can be applied to various pursuit-and-escape game environments. Intelligent decision-making in pursuit-and-escape games is a complex system involving multiple aspects, including data perception, data analysis, decision generation, evaluation, and execution. By comprehensively utilizing artificial intelligence, decision support systems, and other related technologies, intelligent and autonomous decision-making can be achieved by the fleeing party, increasing the success rate of the fleeing party in evading the pursuing party in pursuit-and-escape games.

[0004] Currently, mainstream reinforcement learning-based intelligent decision-making in pursuit-and-escape games uses a policy network trained with a reinforcement learning algorithm to directly generate control instructions, which are then executed by the actuator. The optimal policy is represented by the neural network and lacks interpretability. In actual deployments, especially on manned aircraft, it is generally required that any control instructions generated by the decision-making and control mechanisms be interpretable, and the algorithm generating the decision instructions must have definite boundaries. However, the optimal policy obtained through reinforcement learning training and represented by the neural network does not meet these two requirements. Furthermore, when a decision output error occurs in a certain state, it is impossible to locate the erroneous parameter in the neural network. Due to the fully connected nature of neural networks, modifying any parameter in the network will change the output values ​​corresponding to all input values. Therefore, even if a decision error in a certain state is located and fixed, there is no guarantee that the correction will not introduce new errors, which makes the neural network extremely difficult to debug. Summary of the Invention

[0005] In order to solve the technical problem that the neural network obtained by using deep reinforcement learning algorithm in the current pursuit and escape game intelligent decision-making has poor interpretability and debuggability and is difficult to meet the actual deployment requirements, the present invention proposes a deep reinforcement learning strategy transfer method based on fuzzy inference tree.

[0006] The technical solution of the present invention is:

[0007] A deep reinforcement learning strategy transfer method based on fuzzy inference tree, which is special in that it includes the following steps:

[0008] Step 1: Use a deep reinforcement learning algorithm to train the network to be transferred, where the network to be transferred is the policy network of the deep reinforcement learning algorithm or the value network used to represent the policy;

[0009] Step 2: Divide the state space of the pursuit-and-escape game intelligent decision-making task that requires decision optimization into equal intervals and uniformly sample it to obtain samples of state data that conform to a uniform distribution;

[0010] Step 3: Construct an optimal decision dataset and divide it into a training set and a test set. The optimal decision dataset consists of data and labels. The data is a sample of the state data obtained in step 2, and the label is the optimal action corresponding to the sample. The optimal action is obtained based on the parameters generated by the network to be transferred trained in step 1 and used to describe the distribution of the optimal action.

[0011] Step 4: Construct a fuzzy inference tree based on the pursuit-escape game intelligent decision-making task. The fuzzy inference tree includes multiple fuzzy inference units. The connection relationship between the fuzzy inference units is constructed based on the relationship between the pursuit-escape game intelligent decision-making task, the pursuit-escape motion state, the escape party's internal state, the pursuit-escape relative motion state and the decision instruction. The input of the fuzzy inference tree is a state signal, and the output is a decision instruction. The state signal includes the pursuit-escape motion state, the escape party's motion state and the escape party's internal state. The decision instruction includes the escape party's maneuvering parameters, the false target decoy deployment decision and the false target decoy deployment parameters.

[0012] Step 5: Using an intelligent optimization algorithm and the training set to perform migration training on the fuzzy inference tree, and using the test set to verify the fitting performance of the fuzzy inference tree;

[0013] Step 6: Use the average discounted cumulative return after at least 50 complete decision rounds to evaluate the decision performance of the fuzzy inference tree trained in step 5. If the average discounted cumulative return of the fuzzy inference tree is greater than or equal to 90% of the average discounted cumulative return of the network to be migrated trained in step 1, the decision performance evaluation of the fuzzy inference tree is qualified, the strategy migration is completed, and the current fuzzy inference tree is used to replace the network to be migrated trained in step 1, and the process ends; otherwise, return to step 5; if the decision performance of the fuzzy inference tree is still unqualified after 10 or more migration trainings, proceed to step 7;

[0014] Step 7: Optimize the structure of the fuzzy inference tree constructed in step 4, increase its complexity to improve the upper limit of its decision-making performance, and then return to step 5.

[0015] Furthermore, the deep reinforcement learning algorithm described in step 1 is DQN, DDPG, SAC, TD3, PPO, MADDPG or MAPPO.

[0016] Furthermore, the method for constructing the optimal decision dataset described in step 3 is:

[0017] First, the samples obtained in step 2 are input into the network to be transferred that was trained in step 1. The network to be transferred outputs parameters used to describe the optimal action distribution. Using these parameters, actions that satisfy Gaussian distribution, uniform distribution, or Bernoulli distribution are constructed and sampled to obtain the optimal action, which is used as the label.

[0018] Then, the labels are matched one-to-one with the samples obtained in step 2 to construct the optimal decision dataset;

[0019] Finally, the KL divergence is used to check and adjust the uniformity of the label distribution.

[0020] Furthermore, the fuzzy inference tree constructed in step 4 includes eight fuzzy inference units, namely, the escaping party state processing unit FIS-1, the pursuing party motion state processing unit FIS-2, the relative motion state processing unit FIS-3, the first intermediate signal conversion unit FIS-4, the second intermediate signal conversion unit FIS-5, the maneuver planning decision unit FIS-6, the false target decoy delivery decision unit FIS-7, and the maneuver parameter solution unit FIS-8.

[0021] The input of the escape party state processing unit FIS-1 is the escape party's position, speed, attitude, number of remaining false target decoys and control surface status, and the output is the escape party's spatial orientation, the escape party's control margin, and whether the escape party has enough remaining false target decoys;

[0022] The input of the chasing party motion state processing unit FIS-2 is the position, velocity and posture of the chasing party, and the output is the spatial orientation of the chasing party;

[0023] The input of the relative motion state processing unit FIS-3 is the relative distance, line of sight deflection and line of sight inclination of the two parties, and the output is the relative height advantage, relative energy advantage and relative direction of the two parties;

[0024] The inputs of the first intermediate signal conversion unit FIS-4 are the control margin of the escaping party and whether the remaining decoys of the escaping party are sufficient. The outputs are the escaping party's intention to release the decoys and the intention to change the maneuver of the escaping party.

[0025] The input of the second intermediate signal conversion unit FIS-5 is the spatial orientation of the pursuer, the relative height advantage, relative energy advantage and relative direction of the pursuer and the fleeing party, and the output is the threat level of the pursuer to the fleeing party;

[0026] The input of the maneuver planning and decision unit FIS-6 is the escaping party's intention to change the maneuver, the escaping party's spatial orientation, the pursuing party's spatial orientation, the relative height advantage, relative energy advantage, relative direction of the pursuing and escaping parties, and the threat level of the pursuing party to the escaping party. The output is the escaping party's maneuver type.

[0027] The input of the decoy deployment decision unit FIS-7 is the threat level of the pursuing party to the escaping party, the escaping party's willingness to deploy the decoy, and the escaping party's maneuver type. The output is whether the escaping party deploys the decoy and the deployment parameters of the decoy.

[0028] The input of the maneuver parameter calculation unit FIS-8 is the maneuver type and spatial orientation of the escaping party, and the output is the maneuver parameters of the escaping party.

[0029] Furthermore, the intelligent optimization algorithm described in step 5 is a genetic algorithm, a particle swarm algorithm, an ant colony algorithm or a simulated annealing algorithm.

[0030] Furthermore, the transfer training of the fuzzy inference tree in step 5 refers to regularizing the parameters of the fuzzy inference tree into a parameter sequence according to an arbitrary parameter order, and optimizing the parameter sequence by using a genetic algorithm, a particle swarm algorithm, an ant colony algorithm or a simulated annealing algorithm after binary encoding the parameter sequence.

[0031] Furthermore, in step 7, the method for optimizing the structure of the fuzzy inference tree constructed in step 4 is: increasing the number of divided fuzzy sets, reconstructing the fuzzification unit in each fuzzy inference unit in the fuzzy inference tree according to the current fuzzy set, and correspondingly increasing the number of fuzzy rules required for the fuzzy rule inference unit in each fuzzy inference unit, and / or decomposing at least one fuzzy inference unit in the fuzzy inference tree into multiple different fuzzy inference units according to the input signal.

[0032] Beneficial effects of the present invention:

[0033] The fuzzy inference tree performs reasoning based on fuzzy rules. Each corresponding relationship between input and output corresponds to a complete fuzzy rule chain. The reasoning process can be fully explained and tracked, and its decision-making has good interpretability. It can generate a clear and easy-to-understand rule set that describes the relationship between input variables and outputs, so that people can understand the process and reasons of decision-making. Therefore, when an error occurs, the erroneous fuzzy rule can be quickly located and corrected. At the same time, since fuzzy rules are conditionally triggered, changing a single rule will not affect the global input-output relationship, so the fuzzy inference tree also has good debuggability. In addition, compared with some complex neural network models, the fuzzy inference tree has lower computational complexity and is easy to implement and train quickly. The present invention makes full use of the deep reinforcement learning algorithm's ability to solve and explore complex decision-making problems and the interpretability and debuggability of the fuzzy inference tree algorithm to construct a policy migration method that has both performance and practicality. Specifically, the present invention uses a fully trained network to be transferred as a guide and trains a fuzzy inference tree using a genetic algorithm, particle swarm optimization algorithm, or other optimization algorithm to adjust the tree's parameters. This ensures that the fuzzy inference tree and the trained network to be transferred have the same output given the same input. This means that the fuzzy inference tree learns the mapping relationship between states and actions and has the same decision-making ability as the deep reinforcement learning network to be transferred (the policy network or the value network used to represent the policy). This allows the rules contained in the neural network of the deep reinforcement learning algorithm to be extracted and transferred to the fuzzy inference tree. Once the transfer is complete, the fuzzy inference tree can be used to replace the trained deep reinforcement learning network to be transferred. Using fuzzy inference trees to make decisions in the intelligent decision-making problem of pursuit and escape games not only takes advantage of the search and solution capabilities of deep reinforcement learning, but also migrates the optimal strategy obtained by reinforcement learning to the fuzzy inference tree through rule extraction. Without changing the decision-making efficiency, it not only improves the generation speed of decision-making instructions, but also can well solve the problems of incomplete knowledge of pursuit and escape game information, poor debuggability and interpretability of decision-making methods, and ultimately achieves the optimal mapping from sensor-level data to maneuver action output, so that the network to be migrated obtained by deep reinforcement learning training can meet the requirements of actual deployment and can be more effectively deployed on platforms such as manned aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the method of the present invention.

[0035] Figure 2 It is a structural diagram of the fuzzy reasoning unit in the present invention.

[0036] Figure 3 It is a schematic diagram of the fuzzy reasoning unit decomposition method used in the present invention.

[0037] Figure 4It is a structural diagram of the fuzzy inference tree involved in the embodiment of the present invention.

[0038] Figure 4 Description of each signal:

[0039] ① is the position, speed, attitude, number of remaining decoys, and control surface status of the escaping party; ② is the position, speed, and attitude of the pursuing party; ③ is the relative distance, sight angle, and sight inclination between the pursuing and escaping parties; ④ is the relative height advantage, relative energy advantage, and relative orientation of the pursuing and escaping parties; ⑤ is the spatial orientation of the pursuing party; ⑥ is whether the escaping party has sufficient remaining decoys; ⑦ is the control margin of the escaping party (referring to the extent of maneuvering the escaping party can still make); ⑧ is the spatial orientation of the escaping party; ⑨ is the escaping party's willingness to change maneuvers; ⑩ is the escaping party's willingness to deploy decoys. The degree of threat posed by the pursuing party to the fleeing party; The type of maneuver used by the escaping party; is the maneuvering parameter of the escaping party; Whether a false target bait is deployed for the escaping party; These are the parameters for placing false target bait. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0041] Reference Figure 1 The deep reinforcement learning strategy migration method based on fuzzy inference tree provided by the present invention comprises the following steps:

[0042] Step 1: Use a deep reinforcement learning algorithm to train the network to be transferred. The network to be transferred is the policy network of the deep reinforcement learning algorithm or the value network used to represent the policy.

[0043] For intelligent decision-making tasks in pursuit-and-escape games that require decision optimization, a deep reinforcement learning algorithm is used to train the transfer network. The transfer network's decision-making performance is evaluated and iteratively optimized until the average discounted cumulative reward reaches the expected level, resulting in a trained transfer network. Possible deep learning algorithms include Deep-Q-learning (DQN), Deep Deterministic Policy Gradient (DDPG), Soft Actor-Critic (SAC), Twin Delayed Deep Deterministic Policy Gradient (TD3), Proximal Policy Optimization (PPO), Multi-Agent Deep Deterministic Policy Gradient (MADDPG), or Multi-Agent Proximal Policy Optimization (MAPPO).

[0044] Step 2: Split the state space and sample it.

[0045] The state space of the pursuit and escape game intelligent decision-making task that requires decision optimization is evenly divided into multiple intervals according to certain intervals. A certain number of state data samples are evenly collected in each interval. After sampling, the distribution of the sample data needs to be verified, and the data that seriously deviates from the uniform distribution is eliminated to ensure that the sampling results conform to the uniform distribution and obtain samples of uniformly distributed state data.

[0046] Step 3: Construct the optimal decision dataset and divide it into training set and test set.

[0047] The optimal decision data set consists of data and labels. The data in the optimal decision data set is a sample of the uniformly distributed state data obtained by sampling in step 2. The label in the optimal decision data set is the optimal action corresponding to the sample one by one. The optimal action is obtained based on the parameters generated by the network to be migrated trained in step 1 and used to describe the distribution of the optimal action.

[0048] The specific method for constructing the optimal decision dataset is:

[0049] First, the samples obtained in step 2 are input as input signals to the network to be transferred that was trained in step 1. The network to be transferred outputs parameters used to describe the optimal action distribution (usually Gaussian distribution, uniform distribution, or Bernoulli distribution). The parameters are used to construct actions that satisfy the Gaussian distribution, uniform distribution, or Bernoulli distribution and sample them to obtain the optimal action, which is used as the label.

[0050] Then, the obtained labels are matched one-to-one with the samples obtained in step 2 to construct the optimal decision dataset.

[0051] Finally, KL divergence is used to check and adjust the uniformity of the label distribution.

[0052] After the construction is completed, the optimal decision dataset is divided into training set and test set according to a certain ratio.

[0053] Step 4: Construct the fuzzy inference tree.

[0054] A fuzzy inference tree is constructed by using a series of fuzzy inference units (FIS) through networked links between each other. The output of each fuzzy inference unit is connected to the input layer of different next-level fuzzy inference units, and the fuzzy inference units can be connected arbitrarily.

[0055] Each fuzzy inference unit (FIS) in the fuzzy inference tree consists of three parts: fuzzification unit, fuzzy rule inference unit and defuzzification unit. For example, its operation process is as follows: Figure 2 shown.

[0056] The input signal characteristics are divided into n fuzzy sets in advance. For example, if the input signal characteristic is the aircraft speed, the aircraft speed can be divided into three fuzzy sets: "high speed", "medium speed" and "low speed". Fuzzification units are constructed based on these n fuzzy sets. The fuzzification unit converts the input signal into the membership of each fuzzy set through the membership function, for example Figure 2In the example, L1 represents the membership vector of input signal X1 to each fuzzy set, L2 represents the membership vector of input signal X2 to each fuzzy set, and L3 represents the membership vector of input signal X3 to each fuzzy set. The membership function can be triangular, trapezoidal, or Gaussian. The membership of each input signal to each fuzzy set is between 0 and 1, and the sum of the memberships of each input signal to all fuzzy sets is 1. The fuzzy rule inference unit performs inference using memberships and fuzzy rules. For example, a fuzzy rule can take the form: "If L1 is A and L2 is B, then Z1 is C," where A and B are the specific values ​​of the membership vectors L1 and L2, respectively, and Z1 is the inference result, whose value is C. The defuzzification unit defuzzifies the inference results Z1 and Z2 to produce output results Y1 and Y2. Defuzzification methods include the area center method, the area bisection method, or the maximum membership method.

[0057] According to the actual intelligent decision-making tasks of the pursuit-escape game, the connection structure between the fuzzy reasoning units can be constructed based on the relationship between the information on the pursuit-escape game intelligent decision-making, such as the pursuit party's motion state, the escape party's state (including the escape party's motion state and internal state), and the relative motion state of the pursuit and escape parties, and the escape party's maneuvering parameters, false target decoy deployment decision, and false target decoy deployment parameters.

[0058] In this embodiment, the constructed fuzzy inference tree includes eight fuzzy inference units, namely, the escaping party state processing unit FIS-1, the pursuing party motion state processing unit FIS-2, the relative motion state processing unit FIS-3, the first intermediate signal conversion unit FIS-4, the second intermediate signal conversion unit FIS-5, the maneuver planning decision unit FIS-6, the false target decoy delivery decision unit FIS-7 and the maneuver parameter solution unit FIS-8. The input of the escaping party state processing unit FIS-1 is the escaping party's position, speed, attitude, number of remaining false target decoys and control surface status (for example, for an aircraft, the control surface status refers to the position of its various control surfaces), and the output is the escaping party's spatial orientation, the escaping party's control margin, and whether the escaping party's remaining false target decoys are sufficient; the input of the chasing party's motion state processing unit FIS-2 is the chasing party's position, speed and attitude, and the output is the chasing party's spatial orientation; the input of the relative motion state processing unit FIS-3 is the relative distance, line of sight deflection angle and line of sight inclination angle between the chasing and fleeing parties, and the output is the relative height advantage, relative energy advantage and relative direction of the chasing and fleeing parties; the input of the first intermediate signal conversion unit FIS-4 is the escaping party's control margin and whether the escaping party's remaining false target decoys are sufficient, and the output is the escaping party's intention to release false target decoys and the escaping party's intention to change its maneuvering action; The input of the second intermediate signal conversion unit FIS-5 is the spatial orientation of the pursuing party, the relative height advantage, relative energy advantage and relative direction of the pursuing and escaping parties, and the output is the threat level of the pursuing party to the escaping party; the input of the maneuver planning decision unit FIS-6 is the willingness of the escaping party to change its maneuver action, the spatial orientation of the escaping party, the spatial orientation of the pursuing party, the relative height advantage, relative energy advantage and relative direction of the pursuing and escaping parties, and the threat level of the pursuing party to the escaping party, and the output is the maneuver type of the escaping party; the input of the false target decoy release decision unit FIS-7 is the threat level of the pursuing party to the escaping party, the escaping party's willingness to release the false target decoy and the escaping party's maneuver type, and the output is whether the escaping party releases the false target decoy and the release parameters of the false target decoy; the input of the maneuver parameter solution unit FIS-8 is the maneuver type and spatial orientation of the escaping party, and the output is the maneuver parameters of the escaping party.

[0059] Step 5: Use the training set obtained in step 3 to perform migration training on the fuzzy inference tree constructed in step 4, and use the test set obtained in step 3 to verify the fitting performance of the fuzzy inference tree.

[0060] The fuzzy inference tree constructed in step 4 is transferred and trained using an intelligent optimization algorithm to adjust and optimize its parameters. The parameters of the fuzzy inference tree include the parameters of each fuzzy inference unit that constitutes the fuzzy inference tree. The parameters of the fuzzy inference unit include fuzzy inference rule parameters, membership function parameters, and defuzzification parameters. The parameters of these fuzzy inference units are regularized into a parameter sequence according to an arbitrary parameter order. After the parameter sequence is binary-encoded, it is optimized using an intelligent optimization algorithm such as a genetic algorithm, a particle swarm algorithm, an ant colony algorithm, or a simulated annealing algorithm. The optimization goal is to use the samples in the training set constructed in step 3 as the input data of the fuzzy inference tree, and the deviation between the output data obtained by the fuzzy inference tree and the labels corresponding to the samples in the training set and the input fuzzy inference tree is minimized. The deviation is measured using the mean square error loss function.

[0061] After the parameters of the fuzzy inference tree are obtained through iterative optimization, the fitting performance of the fuzzy inference tree is verified using the test set obtained in step 3.

[0062] Step 6: Evaluate the effectiveness of policy migration.

[0063] Since simple fitting performance verification cannot determine whether the fuzzy inference tree has the same decision-making performance as the trained network to be transferred, and it cannot determine whether the fuzzy inference tree constructed and trained in step 4 has the same feature representation ability as the trained network to be transferred. Therefore, the present invention performs a performance evaluation on the strategy represented by the fuzzy inference tree in a simulation environment. If the average discounted cumulative return obtained by using the fuzzy inference tree for decision-making in multiple (no less than 50) complete pursuit and escape game intelligent decision-making experiments is not less than 90% of the average discounted cumulative return obtained by using the network to be migrated trained in step 1 for decision-making, it can be regarded that the decision-making performance of the fuzzy inference tree has reached a level similar to that of the trained network to be migrated. At this time, it can be regarded that the decision-making performance evaluation of the fuzzy inference tree is qualified, and the strategy migration has been completed. In the actual pursuit and escape game intelligent decision-making task, the current fuzzy inference tree is used as the strategy network for generating decision signals, and the process ends; otherwise, it is regarded that the decision-making performance of the fuzzy inference tree is significantly different from that of the trained network to be migrated, and the process ends by returning to step 5 to re-transfer the fuzzy inference tree. If there is still a significant gap after multiple migration trainings (depending on the specific task, generally more than 10 times is regarded as multiple times), it means that the current structure of the fuzzy inference tree cannot meet the decision-making performance requirements of the trained network to be migrated, and then step 7 is entered.

[0064] Step 7: Optimize the structure of the fuzzy inference tree constructed in step 4.

[0065] The number of fuzzy inference units in a fuzzy inference tree and their connection structure determine the upper limit of the tree's decision-making performance. Since the number and connection structure of fuzzy inference units do not change during the fuzzy inference tree parameter optimization process, the upper limit of the fuzzy inference tree's decision-making performance is determined once the tree's structure is determined. If, during simulation verification, the fuzzy inference tree's decision-making performance fails to reach the level of the trained network to be transferred after multiple transfer training cycles (more than 10), the fuzzy inference tree structure needs to be modified. The complexity of the fuzzy inference tree should be increased based on the characteristics of the pursuit-escape game intelligent decision-making task, and then the process should proceed to step 5.

[0066] The main methods for modifying the fuzzy inference tree structure are:

[0067] 1. Increase the number of divided fuzzy sets, then reconstruct the fuzzification unit in each fuzzy inference unit (FIS) according to the current fuzzy sets, and correspondingly increase the number of fuzzy rules required by the fuzzy rule reasoning unit in each fuzzy inference unit (FIS).

[0068] 2. Decompose the original fuzzy inference unit into multiple different fuzzy inference units according to the input signal, for example Figure 3 As shown, the specific decomposition method is: if the number of input signals received by a fuzzy inference unit FIS-M is N and the number of output signals is P, then the input signals are divided into two groups of N-1 and 1 respectively, and the output signals are divided into two groups of P-1 and 1 respectively. At the same time, FIS-M is deleted, and fuzzy inference units FIS-M1 and FIS-M2 are constructed. FIS-M1 receives the first N-1 input signals of the original FIS-M, and the output signal is the first P-1 output signals of the original FIS-M1 and an auxiliary output signal Y_0. The input signal received by FIS-M2 is the auxiliary output signal Y_0 and the last output signal of the original FIS-M. Decomposing the fuzzy inference unit in this way can slow down the growth rate of the total number of fuzzy rules when increasing the number of fuzzy rules for each fuzzy inference unit, and at the same time provide a higher decision-making performance upper limit for the fuzzy inference tree. It can be done according to Figure 3 The exemplary method continues to decompose the fuzzy inference unit FIS-M1 until the decision performance of the entire fuzzy inference tree finally obtained passes the evaluation.

[0069] Example:

[0070] This embodiment illustrates the present invention with respect to the aircraft control decision of the escaping party in the intelligent decision-making task of the pursuit-escape game.

[0071] A deep reinforcement learning policy transfer method based on fuzzy inference tree. The detailed steps are as follows:

[0072] Step 1: Use a deep reinforcement learning algorithm to train the network to be transferred. The network to be transferred is the policy network of the deep reinforcement learning algorithm or the value network used to represent the policy.

[0073] In the intelligent decision-making task of the pursuit-escape game, the decision-making performance of the network to be migrated is measured by three indicators: the relative distance between the pursuing aircraft and the escaping aircraft at the final moment (hereinafter referred to as "pursuit distance"), the energy consumption of the escaping aircraft, and the consumption of false target decoys by the escaping aircraft.

[0074] The escaping plane The position coordinates at the moment are , the pursuing aircraft The position coordinates at the moment are , the zero time (starting time) of the pursuit-escape game is , the end time is During this process, the command overload of the escaping aircraft in each direction is , the pursuit aircraft's command overload in each direction is , the number of false target decoys consumed by the escaping aircraft is .

[0075] At the end of the intelligent decision-making task of the pursuit and escape game, The positions of the escaping aircraft and the chasing aircraft at the termination time are and If the pursuit distance is less than or equal to the specified miss distance L, that is:

[0076]

[0077] If the avoidance fails, the reward function returns a reward signal value of -1.

[0078] If the pursuit distance is greater than the specified miss distance L, that is:

[0079]

[0080] The avoidance is considered successful, and the miss amount reward, energy reward, and bait reward are recorded as 、 and ,in

[0081] =

[0082]

[0083]

[0084] Final reward signal The value of

[0085]

[0086] in, 、 and The weight of each reward can be adjusted according to the task tendency (tending to reduce the consumption of false target bait or reduce the energy consumption during the escape process under the premise of successful escape). The three weights meet and In the pursuit-escape game intelligent decision-making task, the decision-making performance index of the network to be transferred is equivalent to the reward signal The higher the value of , the higher the average value of the reward signal R obtained in multiple pursuit-escape game intelligent decision-making tasks, the better the decision-making performance of the network to be migrated. 2 represents the 2-norm of the vector.

[0087] The position, attitude and speed of the chasing and escaping aircraft are used as status signals, the control instructions of the escaping aircraft's various control surfaces are used as action signals, and the miss distance of the chasing aircraft, the energy consumption of the escaping aircraft during the evasion process and the consumption of false target bait are used as reward signals. , build the to-be-transferred network of the deep learning algorithm, and simulate the to-be-transferred network in the simulation environment, and use the deep reinforcement learning algorithm to search for the optimal strategy. After sufficient training until the to-be-transferred network converges and the reward signal After the value of rises to the expected level, an evaluation is performed. If the network to be transferred can achieve the corresponding performance indicators in the pursuit-escape game intelligent decision-making task, the training is completed. Otherwise, the reward function, network structure and / or training parameters are modified and retrained until the network to be transferred achieves the expected results. At this point, the trained network to be transferred is obtained.

[0088] Step 2: Split the state space and sample it.

[0089] The state space of the pursuit-and-escape game's intelligent decision-making task is evenly divided into multiple intervals at regular intervals. Within each interval, a certain number of state data samples are uniformly collected. This step requires evenly dividing the state space and sampling the same number of samples within each interval to ensure a uniform sampling distribution. Otherwise, the training of the fuzzy inference tree during rule transfer will be affected.

[0090] Step 3: Construct the optimal decision dataset and divide it into training set and test set.

[0091] An optimal decision dataset is constructed based on the samples collected in step 2 and the trained network to be migrated obtained in step 1. The data in the optimal decision dataset are samples of the uniformly distributed state data sampled in step 2. The labels in the optimal decision dataset are the optimal actions that correspond one-to-one to the samples of the state data. The optimal action is obtained by calculating the parameters used to describe the optimal action distribution using the sample data input into the trained network to be migrated obtained in step 1.

[0092] The specific method of constructing the optimal decision dataset is as follows:

[0093] First, the samples obtained in step 2 are input as input signals to the network to be transferred that was trained in step 1. The output signal of the network to be transferred is the Gaussian parameter used to describe the optimal action distribution. The Gaussian parameter is used to construct an action that satisfies the Gaussian distribution and sample it to obtain the optimal action, which is used as the label.

[0094] Then, the obtained labels are matched one-to-one with the samples obtained in step 2 to construct the optimal decision dataset.

[0095] Finally, KL divergence is used to check and adjust the uniformity of the label distribution in the optimal decision dataset.

[0096] After the construction is completed, the optimal decision dataset is divided into training set and test set in a ratio of 8:2.

[0097] Step 4: Construct the fuzzy inference tree.

[0098] According to the attributes of the pursuit-escape game intelligent decision-making task, the structure of the fuzzy reasoning tree is planned, and the number of fuzzy reasoning units in the fuzzy reasoning tree is determined. The pursuit-escape game task is divided into the branches of escaping party state processing, pursuing party motion state processing, relative motion state processing, intermediate signal conversion, maneuver planning decision, maneuver parameter solution and false target decoy delivery decision. Each branch contains several fuzzy reasoning units. In the same branch, except for the fuzzy reasoning units at the input and output ends, the remaining fuzzy reasoning units are only connected to the fuzzy reasoning units in this branch. The input ends of the three branches of escaping party state processing, pursuing party motion state processing and relative motion state processing only receive the input of state signals. The output ends of the two branches of maneuver parameter solution and false target decoy delivery decision directly constitute the output of the entire fuzzy reasoning tree. The input / output ends of other branches can be connected to the branches of the upper and lower layers. For details of the specific fuzzy reasoning tree structure and signal transmission process, please see the attached. Figure 4 .

[0099] Step 5: Perform migration training on the fuzzy inference tree constructed in step 4.

[0100] Step 5.1 parameter initialization;

[0101] After the fuzzy inference tree is constructed, the parameters of each fuzzy inference unit (fuzzy inference rule parameters, membership function parameters, and defuzzification parameters) must be initialized. This embodiment uses Gaussian distribution initialization to initialize the parameters of each fuzzy inference unit to specific values ​​(random values ​​sampled from a standard Gaussian distribution). After initialization, the parameters of all fuzzy inference units are organized into a parameter sequence in an arbitrary order, and this parameter sequence is converted into a binary code.

[0102] Step 5.2: Build the genetic algorithm model;

[0103] Based on the length of the binary code obtained in step 5.1 and the complexity of the intelligent decision-making task in the pursuit-escape game, construct a genetic algorithm model and determine the genetic algorithm parameters (population size, selection method, crossover probability, and mutation probability). Initialize the population. Based on the optimization objective, set the fitness function for the genetic algorithm iteration.

[0104] Step 5.3 Genetic algorithm iterative optimization;

[0105] Set the maximum number of iterations and the termination threshold for the genetic algorithm. In each iteration, the fitness of each individual in the population is calculated and selection, crossover, and mutation operations are performed based on the fitness to generate new individuals. Repeat this process until the individual with the highest fitness in the population reaches the termination threshold or the number of iterations reaches the maximum number. The binary code of this individual is then converted into the parameters of the fuzzy inference tree.

[0106] Step 5.4: Fitting performance verification;

[0107] After iteratively optimizing the parameters of the fuzzy inference tree in step 5.3, verify the fitting performance of the fuzzy inference tree using the test set constructed in step 3 to see whether it can accurately fit the data in the test set. If underfitting or overfitting occurs, adjust the population size, selection method, crossover probability, and / or mutation probability in the genetic algorithm and repeat step 5.3. If underfitting or overfitting does not occur, the migration training is complete and proceed to step 6.

[0108] Step 6: Evaluate the effectiveness of policy migration.

[0109] The decision-making performance of the fuzzy inference tree is evaluated. Specifically, the tuned fuzzy inference tree is inserted into the decision-making process, replacing the original network trained in step 1 to be transferred and interacting with the simulation environment. The fuzzy inference tree is then used to calculate the optimal action for the state of the pursuit-escape game intelligent decision-making task that requires decision optimization. After at least 50 complete decision-making rounds, the decision-making performance of the fuzzy inference tree is evaluated based on the average discounted cumulative reward. If the average discounted cumulative reward of the fuzzy inference tree is at least 90% of the average discounted cumulative reward obtained by the decision-making of the network trained in step 1, the fuzzy inference tree is considered to have similar decision-making performance to the network trained in step 1, indicating that the pursuit-escape game intelligent decision-making strategy of the network trained in step 1 has been fully transferred to the fuzzy inference tree. The current fuzzy inference tree is then used as the policy network to generate decision signals in the actual pursuit-escape game task, and the process ends. Otherwise, the fuzzy inference tree is considered to have failed to effectively inherit the pursuit-escape game intelligent decision-making strategy of the network trained in step 1. The process returns to step 5, re-modifies the genetic algorithm parameters, and retrains the fuzzy inference tree for transfer training. After the transfer training is complete, the decision-making performance is re-evaluated. If the decision-making performance evaluation of this step still fails after 10 repetitions of the migration training in step 5, it is considered that the decision-making performance upper limit of the current fuzzy inference tree cannot meet the requirements of representing the intelligent decision-making strategy of the pursuit-escape game contained in the network to be migrated. The structure of the fuzzy inference tree needs to be optimized, and step 7 is executed at this time.

[0110] Step 7: Optimize the structure of the fuzzy inference tree.

[0111] If the decision-making performance of the fuzzy inference tree fails to meet the expected requirements, the fuzzy tree structure constructed in step 4 needs to be adjusted and optimized. Specific methods include increasing the number of fuzzy inference units and / or performing fuzzy inference unit decomposition in at least one of the original branches: the fleeing party state processing, the pursuing party motion state processing, the relative motion state processing, the intermediate signal conversion, the maneuver planning decision, the maneuver parameter solution, and the decoy deployment decision.

[0112] After optimizing the structure of the fuzzy inference tree, it is necessary to re-train the migration and test its decision-making performance, that is, proceed to step 5.

[0113] Finally, it should be noted that when using the present invention to transfer strategies for different reinforcement learning algorithms, only the data source and generation method differ; the reinforcement learning algorithm training and rule extraction process are consistent with the present invention. Furthermore, the described embodiments are intended to provide a detailed description of the present invention and are not intended to limit its practical application.

Claims

1. A deep reinforcement learning policy transfer method based on fuzzy inference trees, used to transfer the decision-making ability of the policy network used to generate control instructions in intelligent decision-making tasks in pursuit-escape games in the fields of aircraft control, autonomous driving, and industrial robots, to improve the interpretability and debuggability of the policy network; characterized by: Including steps: Step 1: Use a deep reinforcement learning algorithm to train the network to be transferred, where the network to be transferred is the policy network of the deep reinforcement learning algorithm or the value network used to represent the policy; Step 2: Divide the state space of the pursuit-and-escape game intelligent decision-making task that requires decision optimization into equal intervals and uniformly sample it to obtain samples of state data that conform to a uniform distribution; Step 3: Construct an optimal decision dataset and divide it into a training set and a test set. The optimal decision dataset consists of data and labels. The data is a sample of the state data obtained in step 2, and the label is the optimal action corresponding to the sample. The optimal action is obtained based on the parameters generated by the network to be transferred trained in step 1 and used to describe the distribution of the optimal action. Step 4: construct a fuzzy inference tree according to the pursuit-escape game intelligent decision-making task, the fuzzy inference tree includes multiple fuzzy inference units, and the connection relationship between each fuzzy inference unit is constructed according to the relationship between the pursuit party's motion state, the escape party's motion state, the escape party's internal state and the relative motion state of the pursuit and escape parties and the decision instructions in the pursuit-escape game intelligent decision-making task; the input of the fuzzy inference tree is a state signal, and the output is a decision instruction; the state signal includes the pursuit party's motion state, the escape party's motion state and the escape party's internal state; the decision instruction includes the escape party's maneuvering parameters, false target bait release decision and false target bait release parameters; the pursuit party's motion state includes the pursuit party's position, speed and posture; the escape party's motion state includes the escape party's position, speed and posture; the escape party's internal state includes the remaining false target bait number and control surface state; the relative motion state of the pursuit and escape parties includes the relative distance, line of sight deflection angle and line of sight inclination angle between the pursuit and escape parties; Step 5: Using an intelligent optimization algorithm and the training set to perform migration training on the fuzzy inference tree, and using the test set to verify the fitting performance of the fuzzy inference tree; Step 6: Use the average discounted cumulative return after at least 50 complete decision rounds to evaluate the decision performance of the fuzzy inference tree trained in step 5. If the average discounted cumulative return of the fuzzy inference tree is greater than or equal to 90% of the average discounted cumulative return of the network to be migrated trained in step 1, the decision performance evaluation of the fuzzy inference tree is qualified, the strategy migration is completed, and the current fuzzy inference tree is used to replace the network to be migrated trained in step 1, and the process ends; Otherwise, return to step 5; if the decision-making performance of the fuzzy inference tree is still unqualified after 10 or more migration trainings, proceed to step 7; Step 7: Optimize the structure of the fuzzy inference tree constructed in step 4, increase its complexity to improve the upper limit of its decision-making performance, and then return to step 5.

2. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 1, characterized in that: The deep reinforcement learning algorithm described in step 1 is DQN, DDPG, SAC, TD3, PPO, MADDPG, or MAPPO.

3. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 1, characterized in that: The method for constructing the optimal decision dataset described in step 3 is: First, the samples obtained in step 2 are input into the network to be transferred that was trained in step 1. The network to be transferred outputs parameters used to describe the optimal action distribution. Using these parameters, actions that satisfy Gaussian distribution, uniform distribution, or Bernoulli distribution are constructed and sampled to obtain the optimal action, which is used as the label. Then, the labels are matched one-to-one with the samples obtained in step 2 to construct the optimal decision dataset; Finally, the KL divergence is used to check and adjust the uniformity of the label distribution.

4. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 1, characterized in that: The fuzzy inference tree constructed in step 4 includes eight fuzzy inference units: the escaping party state processing unit FIS-1, the pursuing party motion state processing unit FIS-2, the relative motion state processing unit FIS-3, the first intermediate signal conversion unit FIS-4, the second intermediate signal conversion unit FIS-5, the maneuver planning decision unit FIS-6, the false target decoy delivery decision unit FIS-7, and the maneuver parameter solution unit FIS-8. The input of the escape party state processing unit FIS-1 is the escape party's position, speed, attitude, number of remaining false target decoys and control surface status, and the output is the escape party's spatial orientation, the escape party's control margin, and whether the escape party has enough remaining false target decoys; The input of the chasing party motion state processing unit FIS-2 is the position, velocity and posture of the chasing party, and the output is the spatial orientation of the chasing party; The input of the relative motion state processing unit FIS-3 is the relative distance, line of sight deflection and line of sight inclination of the two parties, and the output is the relative height advantage, relative energy advantage and relative direction of the two parties; The inputs of the first intermediate signal conversion unit FIS-4 are the control margin of the escaping party and whether the remaining decoys of the escaping party are sufficient. The outputs are the escaping party's intention to release the decoys and the intention to change the maneuver of the escaping party. The input of the second intermediate signal conversion unit FIS-5 is the spatial orientation of the pursuer, the relative height advantage, relative energy advantage and relative direction of the pursuer and the fleeing party, and the output is the threat level of the pursuer to the fleeing party; The input of the maneuver planning and decision unit FIS-6 is the escaping party's intention to change the maneuver, the escaping party's spatial orientation, the pursuing party's spatial orientation, the relative height advantage, relative energy advantage, relative direction of the pursuing and escaping parties, and the threat level of the pursuing party to the escaping party. The output is the escaping party's maneuver type. The input of the decoy deployment decision unit FIS-7 is the threat level of the pursuing party to the escaping party, the escaping party's willingness to deploy the decoy, and the escaping party's maneuver type. The output is whether the escaping party deploys the decoy and the deployment parameters of the decoy. The input of the maneuver parameter calculation unit FIS-8 is the maneuver type and spatial orientation of the escaping party, and the output is the maneuver parameters of the escaping party.

5. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 1, characterized in that: The intelligent optimization algorithm described in step 5 is a genetic algorithm, a particle swarm algorithm, an ant colony algorithm or a simulated annealing algorithm.

6. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 5, characterized in that: The transfer training of the fuzzy inference tree in step 5 refers to regularizing the parameters of the fuzzy inference tree into a parameter sequence according to an arbitrary parameter order, and optimizing the parameter sequence by using a genetic algorithm, a particle swarm algorithm, an ant colony algorithm or a simulated annealing algorithm after binary encoding the parameter sequence.

7. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to any one of claims 1 to 6, characterized in that: The method for optimizing the structure of the fuzzy inference tree constructed in step 4 in step 7 is: increasing the number of divided fuzzy sets, reconstructing the fuzzification unit in each fuzzy inference unit in the fuzzy inference tree according to the current fuzzy set, and correspondingly increasing the number of fuzzy rules required by the fuzzy rule inference unit in each fuzzy inference unit, and / or decomposing at least one fuzzy inference unit in the fuzzy inference tree into multiple different fuzzy inference units according to the input signal.

Images