Deep reinforcement learning strategy migration method based on fuzzy inference tree

Abstract

The invention provides a deep reinforcement learning strategy migration method based on a fuzzy inference tree in order to solve the problems that a neural network obtained by using a deep reinforcement learning algorithm in a current pursuit game intelligent decision is poor in interpretability and debuggeability and is difficult to meet actual deployment requirements. The fully trained to-be-migrated network is used as guidance, and the fuzzy inference tree is trained through an intelligent optimization algorithm to have the same output with the trained to-be-migrated network under the same input, so that the fuzzy inference tree learns the mapping relation from the state to the action; the method has the same decision-making capability as a to-be-migrated network of deep reinforcement learning, thereby achieving the purpose of extracting and migrating rules contained in a neural network in a deep reinforcement learning algorithm to a fuzzy inference tree. According to the method, on the premise of not changing the decision efficiency, the generation speed of the decision instruction is increased, and the problems that pursuit game information is not completely known, and the debuggeability and interpretability of a decision method are poor are solved.

Description

Technical Field

[0001] The invention belongs to the field of computers and artificial intelligence. Background Art

[0002] As an intelligent decision-making method, deep reinforcement learning has the characteristics of strong search ability and good decision-making performance, and can solve the optimal decision-making problem of complex problems. However, in actual use, it is difficult to effectively analyze and judge the decisions made by the intelligent agent because the neural network used for policy representation in deep reinforcement learning is not interpretable, so it is difficult to achieve good human-machine mutual trust; at the same time, due to the fully connected characteristics of the neural network, it is difficult to locate and modify abnormal parameters when abnormal decisions occur. This greatly limits the actual deployment of deep reinforcement learning methods in application fields such as aircraft control, autonomous driving, and industrial robots, which have extremely high requirements for human-machine mutual trust and policy debuggability.

[0003] Intelligent decision-making in dual-machine pursuit and escape game refers to the process of using technologies such as artificial intelligence and decision-making systems to enable the escapee to make decisions assisted or autonomously, thereby avoiding the pursuer. It involves the interaction between the pursuer, the escapee, false target bait, the perception system, and other elements, and can be applied to various pursuit and escape game environments. Intelligent decision-making in pursuit and escape game is a complex system involving data perception, data analysis, decision generation, evaluation and execution. By comprehensively utilizing artificial intelligence, decision support systems and other related technologies, the escapee can make intelligent and autonomous decisions, thereby increasing the success rate of the escapee in avoiding the pursuer in the pursuit and escape game.

[0004] At present, the mainstream intelligent decision-making of pursuit and escape games based on reinforcement learning directly generates control instructions through the policy network obtained by reinforcement learning algorithm training and is executed by the executive agency. Its optimal strategy is represented by the neural network and is not interpretable. In actual deployment, especially when deployed on manned aircraft, it is generally required that any control instructions generated by the decision-making and control agencies must be interpretable, and the algorithm for generating decision instructions must have a certain boundary. However, the optimal strategy obtained by reinforcement learning training and represented by the neural network does not have the above two points. In addition, when the decision output in a certain state is wrong, it is impossible to locate the wrong parameter in the neural network, and due to the fully connected characteristics of the neural network, after modifying any parameter in the network, the output values ​​corresponding to all input values ​​will change. Therefore, even if the decision error in a certain state is located and fixed, it cannot be guaranteed that the correction will not bring new errors, which makes the debuggability of the neural network extremely poor. Summary of the invention

[0005] In order to solve the technical problem that the neural network obtained by using the deep reinforcement learning algorithm in the current pursuit and escape game intelligent decision-making has poor interpretability and debuggability and is difficult to meet the actual deployment requirements, the present invention proposes a deep reinforcement learning strategy transfer method based on fuzzy inference tree.

[0006] The technical solution of the present invention is:

[0007] A deep reinforcement learning strategy transfer method based on fuzzy inference tree, which is special in that it includes the following steps:

[0008] Step 1: Use a deep reinforcement learning algorithm to train the network to be transferred, where the network to be transferred is a policy network of the deep reinforcement learning algorithm or a value network used to characterize the policy;

[0009] Step 2: The state space of the pursuit and escape game intelligent decision-making task that needs to be optimized is divided into equal intervals and uniformly sampled to obtain samples of state data that conform to the uniform distribution;

[0010] Step 3: construct an optimal decision data set and divide it into a training set and a test set; the optimal decision data set consists of data and labels, the data is a sample of the state data obtained in step 2, the label is the optimal action corresponding to the sample, and the optimal action is obtained based on the parameters generated by the network to be migrated trained in step 1 and used to describe the distribution of the optimal action;

[0011] Step 4: construct a fuzzy inference tree according to the pursuit-escape game intelligent decision-making task, the fuzzy inference tree includes multiple fuzzy inference units, and the connection relationship between the fuzzy inference units is constructed according to the relationship between the pursuit party's motion state, the escape party's motion state, the escape party's internal state and the relative motion state of the pursuit and escape parties and the decision-making instructions in the pursuit-escape game intelligent decision-making task; the input of the fuzzy inference tree is a state signal, and the output is a decision-making instruction; the state signal includes the pursuit party's motion state, the escape party's motion state and the escape party's internal state; the decision-making instruction includes the escape party's maneuvering parameters, the false target bait delivery decision and the false target bait delivery parameters;

[0012] Step 5: Using an intelligent optimization algorithm and the training set to perform migration training on the fuzzy inference tree, and using the test set to verify the fitting performance of the fuzzy inference tree;

[0013] Step 6: Use the average discounted cumulative return after at least 50 rounds of complete decision-making to evaluate the decision-making performance of the fuzzy inference tree trained in step 5. If the average discounted cumulative return of the fuzzy inference tree is greater than or equal to 90% of the average discounted cumulative return of the network to be migrated trained in step 1, the decision-making performance evaluation of the fuzzy inference tree is qualified, the strategy migration is completed, and the current fuzzy inference tree is used to replace the network to be migrated trained in step 1, and the process ends; otherwise, return to step 5; if the decision-making performance of the fuzzy inference tree is still unqualified after 10 or more migration trainings, proceed to step 7;

[0014] Step 7: Optimize the structure of the fuzzy inference tree constructed in step 4, increase its complexity to improve the upper limit of its decision-making performance, and then return to step 5.

[0015] Furthermore, the deep reinforcement learning algorithm described in step 1 is DQN, DDPG, SAC, TD3, PPO, MADDPG or MAPPO.

[0016] Furthermore, the method for constructing the optimal decision data set described in step 3 is:

[0017] First, the samples obtained in step 2 are input into the network to be transferred that has been trained in step 1. The network to be transferred outputs parameters used to describe the optimal action distribution. The parameters are used to construct actions that satisfy Gaussian distribution, uniform distribution, or Bernoulli distribution and sample them to obtain the optimal action, which is used as the label.

[0018] Then, the labels are matched one by one with the samples obtained in step 2 to construct an optimal decision data set;

[0019] Finally, the KL divergence is used to check and adjust the uniformity of the label distribution.

[0020] Furthermore, the fuzzy inference tree constructed in step 4 includes 8 fuzzy inference units, namely, the escape party state processing unit FIS-1, the chase party motion state processing unit FIS-2, the relative motion state processing unit FIS-3, the first intermediate signal conversion unit FIS-4, the second intermediate signal conversion unit FIS-5, the maneuver planning decision unit FIS-6, the false target bait delivery decision unit FIS-7 and the maneuver parameter solution unit FIS-8;

[0021] The input of the escape party state processing unit FIS-1 is the position, speed, attitude, number of remaining false target decoys and control surface state of the escape party, and the output is the space orientation of the escape party, the control margin of the escape party, and whether the remaining false target decoys of the escape party are sufficient;

[0022] The input of the chasing party motion state processing unit FIS-2 is the chasing party's position, speed and posture, and the output is the chasing party's spatial orientation;

[0023] The input of the relative motion state processing unit FIS-3 is the relative distance, line of sight deflection angle and line of sight inclination of the two parties, and the output is the relative height advantage, relative energy advantage and relative direction of the two parties;

[0024] The input of the first intermediate signal conversion unit FIS-4 is the control margin of the escaping party and whether the remaining false target bait of the escaping party is sufficient, and the output is the intention of the escaping party to release the false target bait and the intention of the escaping party to change the maneuver;

[0025] The input of the second intermediate signal conversion unit FIS-5 is the spatial orientation of the chasing party, the relative height advantage, relative energy advantage and relative direction of the chasing and escaping parties, and the output is the threat level of the chasing party to the escaping party;

[0026] The input of the maneuver planning and decision unit FIS-6 is the escaping party's intention to change the maneuver, the escaping party's spatial orientation, the chasing party's spatial orientation, the relative height advantage, relative energy advantage, relative orientation of the chasing and escaping parties, and the threat level of the chasing party to the escaping party. The output is the escaping party's maneuver type.

[0027] The input of the false target decoy release decision unit FIS-7 is the threat level of the chasing party to the escaping party, the escaping party's willingness to release the false target decoy and the type of maneuver of the escaping party. The output is whether the escaping party releases the false target decoy and the release parameters of the false target decoy.

[0028] The input of the maneuver parameter solving unit FIS-8 is the maneuver type and spatial orientation of the escaping party, and the output is the maneuver parameter of the escaping party.

[0029] Furthermore, the intelligent optimization algorithm described in step 5 is a genetic algorithm, a particle swarm algorithm, an ant colony algorithm or a simulated annealing algorithm.

[0030] Furthermore, in step 5, the transfer training of the fuzzy inference tree refers to regularizing the parameters of the fuzzy inference tree into a parameter sequence according to an arbitrary parameter order, and optimizing the parameter sequence by using a genetic algorithm, a particle swarm algorithm, an ant colony algorithm or a simulated annealing algorithm after binary encoding the parameter sequence.

[0031] Furthermore, in step 7, the method for optimizing the structure of the fuzzy inference tree constructed in step 4 is: increasing the number of divided fuzzy sets, reconstructing the fuzzification unit in each fuzzy inference unit in the fuzzy inference tree according to the current fuzzy set, and correspondingly increasing the number of fuzzy rules required for the fuzzy rule inference unit in each fuzzy inference unit, and / or decomposing at least one fuzzy inference unit in the fuzzy inference tree into multiple different fuzzy inference units according to the input signal.

[0032] Beneficial effects of the present invention:

[0033] The fuzzy inference tree is based on fuzzy rules for reasoning. Each corresponding relationship between input and output corresponds to a complete fuzzy rule chain. The reasoning process can be fully explained and tracked. Its decision has good interpretability and can generate a clear and easy-to-understand rule set. It describes the relationship between input variables and outputs, allowing people to understand the process and reasons of decision-making. Therefore, when an error occurs, the erroneous fuzzy rules can be quickly located and corrected. At the same time, since fuzzy rules are conditionally triggered, changing a single rule will not affect the global input-output relationship, so the fuzzy inference tree also has good debuggability. In addition, compared with some complex neural network models, the fuzzy inference tree has lower computational complexity, is easy to implement and quickly train. The present invention makes full use of the deep reinforcement learning algorithm's ability to solve and explore complex decision-making problems and the interpretability and debuggability of the fuzzy inference tree algorithm to construct a strategy migration method with both performance and practicality. Specifically, the present invention uses a fully trained network to be transferred as a guide, and trains the fuzzy inference tree through a genetic algorithm, a particle swarm algorithm or other optimization algorithms to adjust the parameters of the fuzzy inference tree, so that the fuzzy inference tree and the trained network to be transferred have the same output under the same input, that is, the fuzzy inference tree learns the mapping relationship from state to action, and has the same decision-making ability as the network to be transferred (policy network or value network used to characterize the strategy) of deep reinforcement learning, thereby achieving the purpose of extracting the rules contained in the neural network in the deep reinforcement learning algorithm and migrating them to the fuzzy inference tree. After the migration is completed, the fuzzy inference tree can be used to replace the trained deep reinforcement learning network to be transferred. Using fuzzy inference trees to make decisions in the intelligent decision-making problem of pursuit and escape games not only utilizes the search and solution capabilities of deep reinforcement learning, but also migrates the optimal strategy obtained by reinforcement learning to the fuzzy inference tree through rule extraction. Without changing the decision-making efficiency, it not only improves the speed of generating decision instructions, but also can well solve the problems of incompletely knowable information in pursuit and escape games, poor debuggability and interpretability of decision-making methods. Ultimately, it can achieve the optimal mapping from sensor-level data to mobile action output, so that the network to be migrated obtained by deep reinforcement learning training can meet the requirements of actual deployment and can be more effectively deployed on manned aircraft and other platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the method of the present invention.

[0035] Figure 2 It is a structural diagram of the fuzzy reasoning unit in the present invention.

[0036] Figure 3 It is a schematic diagram of the fuzzy reasoning unit decomposition method used in the present invention.

[0037] Figure 4It is a structural diagram of the fuzzy inference tree involved in the embodiment of the present invention.

[0038] Figure 4 Description of each signal:

[0039] ① is the position, speed, attitude, number of remaining false target decoys and control surface status of the escaping party; ② is the position, speed and attitude of the chasing party; ③ is the relative distance, sight angle and sight inclination between the chasing and escaping parties; ④ is the relative height advantage, relative energy advantage and relative direction of the chasing and escaping parties; ⑤ is the spatial orientation of the chasing party; ⑥ is whether the remaining false target decoys of the escaping party are sufficient; ⑦ is the control margin of the escaping party (referring to how much maneuvering action the escaping party can still make); ⑧ is the spatial orientation of the escaping party; ⑨ is the willingness of the escaping party to change its maneuvering action; ⑩ is the willingness of the escaping party to release false target decoys; The degree of threat posed by the pursuing party to the fleeing party; The type of maneuver used by the escaping party; is the maneuvering parameter of the escaping party; Whether to place false target bait for the escaping party; These are the parameters for placing false target bait. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0041] Reference Figure 1 The deep reinforcement learning strategy migration method based on fuzzy inference tree provided by the present invention comprises the steps of:

[0042] Step 1: Use a deep reinforcement learning algorithm to train the network to be transferred, which is the policy network of the deep reinforcement learning algorithm or the value network used to characterize the policy.

[0043] For the intelligent decision-making tasks of the pursuit and escape game that require decision optimization, use the deep reinforcement learning algorithm to train the network to be transferred, evaluate the decision-making performance of the network to be transferred and make iterative optimization until the average discounted cumulative return value rises to the expected level, and obtain the trained network to be transferred. The deep learning algorithms that can be used here include DQN (Deep-Q-learning), DDPG (Deep Deterministic Policy Gradient), SAC (Soft Actor-Critic, Soft-AC algorithm), TD3 (Twin Delayed Deep Deterministic policy gradient, double delayed deep deterministic policy gradient algorithm), PPO (Proximal Policy Optimization), MADDPG (Multi Agent Deep Deterministic policy gradient) or MAPPO (Multi Agent Proximal Policy Optimization), etc.

[0044] Step 2: Split and sample the state space.

[0045] The state space of the pursuit and escape game intelligent decision-making task that requires decision optimization is evenly divided into multiple intervals according to certain intervals, and a certain number of state data samples are evenly collected in each interval. After sampling, the distribution of the sample data needs to be verified, and the data that seriously deviates from the uniform distribution is eliminated to ensure that the sampling results conform to the uniform distribution and obtain samples of uniformly distributed state data.

[0046] Step 3: Construct the optimal decision dataset and divide it into training set and test set.

[0047] The optimal decision data set consists of data and labels. The data in the optimal decision data set is a sample of the uniformly distributed state data sampled in step 2. The label in the optimal decision data set is the optimal action corresponding to the sample one by one. The optimal action is obtained based on the parameters generated by the network to be migrated trained in step 1 and used to describe the distribution of the optimal action.

[0048] The specific method for constructing the optimal decision data set is as follows:

[0049] First, the samples sampled in step 2 are input as input signals to the network to be transferred that has been trained in step 1. The network to be transferred outputs parameters used to describe the optimal action distribution (generally Gaussian distribution, uniform distribution or Bernoulli distribution). The parameters are used to construct actions that satisfy Gaussian distribution, uniform distribution or Bernoulli distribution and sample them to obtain the optimal action, which is used as the label.

[0050] Then, the obtained labels are matched one by one with the samples obtained in step 2 to construct the optimal decision data set.

[0051] Finally, KL divergence is used to check and adjust the uniformity of the label distribution.

[0052] After the construction is completed, the optimal decision data set is divided into training set and test set according to a certain ratio.

[0053] Step 4: Construct the fuzzy inference tree.

[0054] A fuzzy inference tree is constructed by using a series of fuzzy inference units (FIS) through networked links between each other. The output of each fuzzy inference unit is connected to the input layer of different next-level fuzzy inference units, and the fuzzy inference units can be connected arbitrarily.

[0055] Each fuzzy inference unit (FIS) in the fuzzy inference tree consists of three parts, namely, the fuzzification unit, the fuzzy rule inference unit and the defuzzification unit. For example, its operation process is as follows: Figure 2 shown.

[0056] The input signal characteristics are divided into n fuzzy sets in advance. For example, when the input signal characteristic is the aircraft speed, the aircraft speed can be divided into three fuzzy sets: "high speed", "medium speed" and "low speed". Fuzzification units are constructed based on these n fuzzy sets. The fuzzification unit converts the input signal into the membership of these input signals to each fuzzy set through the membership function, for example Figure 2L1 in the formula represents the membership vector of input signal X1 to each fuzzy set, L2 represents the membership vector of input signal X2 to each fuzzy set, and L3 represents the membership vector of input signal X3 to each fuzzy set; the membership function can be selected from triangular, trapezoidal or Gaussian membership functions; the membership of each input signal to each fuzzy set is a value between 0 and 1, and the sum of the membership of each input signal to all fuzzy sets is 1. The fuzzy rule reasoning unit makes inferences through membership and fuzzy rules. For example, the form of fuzzy rules can be: "If L1 is A and L2 is B, then Z1 is C", where A and B are the specific values ​​of membership vectors L1 and L2 respectively, and Z1 is the reasoning result, whose value is C. The defuzzification unit defuzzifies the reasoning results Z1 and Z2 to obtain output results Y1 and Y2; the defuzzification methods include area center method, area bisection method or maximum membership method.

[0057] According to the actual intelligent decision-making tasks of the pursuit and escape game, the connection structure between the fuzzy reasoning units can be constructed based on the relationship between the information on the pursuit and escape game intelligent decision-making, such as the motion state of the pursuing party, the state of the escaping party (including the motion state and internal state of the escaping party) and the relative motion state of the pursuing and escaping parties, and the maneuvering parameters of the escaping party, the false target bait release decision and the false target bait release parameters.

[0058] In this embodiment, the constructed fuzzy inference tree includes 8 fuzzy inference units, including the escaping party state processing unit FIS-1, the chasing party motion state processing unit FIS-2, the relative motion state processing unit FIS-3, the first intermediate signal conversion unit FIS-4, the second intermediate signal conversion unit FIS-5, the maneuver planning decision unit FIS-6, the false target bait delivery decision unit FIS-7 and the maneuver parameter solution unit FIS-8. The input of the escaping party state processing unit FIS-1 is the position, speed, attitude, number of remaining false target decoys and control surface state of the escaping party (for example, for an aircraft, the control surface state refers to the position of its various rudder surfaces), and the output is the spatial orientation of the escaping party, the control margin of the escaping party, and whether the remaining false target decoys of the escaping party are sufficient; the input of the chasing party motion state processing unit FIS-2 is the position, speed and attitude of the chasing party, and the output is the spatial orientation of the chasing party; the input of the relative motion state processing unit FIS-3 is the relative distance, line of sight deflection angle and line of sight inclination angle of the chasing and fleeing parties, and the output is the relative height advantage, relative energy advantage and relative direction of the chasing and fleeing parties; the input of the first intermediate signal conversion unit FIS-4 is the control margin of the escaping party, whether the remaining false target decoys of the escaping party are sufficient, and the output is the escaping party's intention to release the false target decoy and the escaping party's intention to change its maneuvering action; The input of the second intermediate signal conversion unit FIS-5 is the spatial orientation of the pursuing party, the relative height advantage, relative energy advantage and relative direction of the pursuing and escaping parties, and the output is the threat level of the pursuing party to the escaping party; the input of the maneuver planning decision unit FIS-6 is the willingness of the escaping party to change its maneuver, the spatial orientation of the escaping party, the spatial orientation of the pursuing party, the relative height advantage, relative energy advantage, relative direction of the pursuing and escaping parties, and the threat level of the pursuing party to the escaping party, and the output is the maneuver type of the escaping party; the input of the false target decoy release decision unit FIS-7 is the threat level of the pursuing party to the escaping party, the escaping party's willingness to release the false target decoy and the escaping party's maneuver type, and the output is whether the escaping party releases the false target decoy and the release parameters of the false target decoy; the input of the maneuver parameter solution unit FIS-8 is the maneuver type and spatial orientation of the escaping party, and the output is the maneuver parameters of the escaping party.

[0059] Step 5: Use the training set obtained in step 3 to perform migration training on the fuzzy inference tree constructed in step 4, and use the test set obtained in step 3 to verify the fitting performance of the fuzzy inference tree.

[0060] Use the intelligent optimization algorithm to perform migration training on the fuzzy inference tree constructed in step 4 to adjust and optimize its parameters. The parameters of the fuzzy inference tree include the parameters of each fuzzy inference unit that constitutes the fuzzy inference tree, and the parameters of the fuzzy inference unit include fuzzy inference rule parameters, membership function parameters and defuzzification parameters. The parameters of these fuzzy inference units are regularized into a parameter sequence according to an arbitrary parameter order, and the parameter sequence is binary encoded and optimized using an intelligent optimization algorithm such as a genetic algorithm, a particle swarm algorithm, an ant colony algorithm or a simulated annealing algorithm. The optimization goal is to use the samples in the training set constructed in step 3 as the input data of the fuzzy inference tree, and the deviation between the output data obtained by the fuzzy inference tree and the labels corresponding to the samples in the training set and the input fuzzy inference tree is minimized, and the deviation is measured using the mean square error loss function.

[0061] After the parameters of the fuzzy inference tree are obtained through iterative optimization, the fitting performance of the fuzzy inference tree is verified using the test set obtained in step 3.

[0062] Step 6: Evaluate the effectiveness of policy migration.

[0063] Since simple fitting performance verification cannot determine whether the fuzzy inference tree has the same decision-making performance as the trained network to be transferred, and it is impossible to determine whether the fuzzy inference tree constructed and trained in step 4 has the same feature representation ability as the trained network to be transferred. Therefore, the present invention performs a performance evaluation on the strategy represented by the fuzzy inference tree in a simulation environment. If the average discounted cumulative return obtained by using the fuzzy inference tree to make decisions in multiple (no less than 50) complete pursuit and escape game intelligent decision-making experiments is not less than 90% of the average discounted cumulative return obtained by using the network to be migrated trained in step 1 to make decisions, it can be regarded that the decision-making performance of the fuzzy inference tree has reached a level similar to that of the trained network to be migrated. At this time, it can be considered that the decision-making performance evaluation of the fuzzy inference tree is qualified, and the strategy migration has been completed. In the actual pursuit and escape game intelligent decision-making task, the current fuzzy inference tree is used as the strategy network for generating decision signals, and the process ends; otherwise, it is considered that the decision-making performance of the fuzzy inference tree is significantly different from that of the trained network to be migrated, then the process returns to step 5 to re-transfer the fuzzy inference tree. If there is still a significant gap after multiple migration trainings (depending on the specific task, generally more than 10 times is considered as multiple times), it means that the current structure of the fuzzy inference tree cannot meet the decision-making performance requirements of the trained network to be migrated, then step 7 is entered.

[0064] Step 7: Optimize the structure of the fuzzy inference tree constructed in step 4.

[0065] The number of fuzzy inference units in the fuzzy inference tree and their connection mode determine the upper limit of the decision-making performance of the fuzzy inference tree. The number and connection mode of fuzzy inference units will not change during the parameter optimization process of the fuzzy inference tree. Therefore, the upper limit of the decision-making performance of the fuzzy inference tree is determined after the structure of the fuzzy inference tree is determined. If it is found in the simulation verification that the decision-making performance of the fuzzy inference tree cannot reach the decision-making level of the trained network to be migrated after multiple migration trainings (more than 10 times), it is necessary to modify the structure of the fuzzy inference tree, increase the complexity of the fuzzy inference tree according to the characteristics of the intelligent decision-making task of the pursuit and escape game, and then proceed to step 5.

[0066] The main methods for modifying the fuzzy inference tree structure are:

[0067] 1. Increase the number of divided fuzzy sets, then reconstruct the fuzzification unit in each fuzzy inference unit (FIS) according to the current fuzzy sets, and correspondingly increase the number of fuzzy rules required by the fuzzy rule reasoning unit in each fuzzy inference unit (FIS).

[0068] 2. Decompose the original fuzzy inference unit into multiple different fuzzy inference units according to the input signal, for example Figure 3 As shown, the specific decomposition method is: if a fuzzy inference unit FIS-M receives N input signals and P output signals, the input signals are divided into two groups of N-1 and 1 respectively, and the output signals are divided into two groups of P-1 and 1 respectively. At the same time, FIS-M is deleted, and fuzzy inference units FIS-M1 and FIS-M2 are constructed. FIS-M1 receives the first N-1 input signals of the original FIS-M, and the output signal is the first P-1 output signals of the original FIS-M1 and an auxiliary output signal Y_0. The input signal received by FIS-M2 is the auxiliary output signal Y_0 and the last output signal of the original FIS-M. Decomposing the fuzzy inference unit in this way can slow down the growth rate of the total number of fuzzy rules when increasing the number of fuzzy rules for each fuzzy inference unit, and at the same time provide a higher decision-making performance upper limit for the fuzzy inference tree. It can be based on Figure 3 The exemplary method continues to decompose the fuzzy inference unit FIS-M1 until the decision performance of the entire fuzzy inference tree finally obtained passes the evaluation.

[0069] Example:

[0070] This embodiment is an explanation of the present invention with respect to the control decision of the escaping party's aircraft in the pursuit-escape game intelligent decision-making task.

[0071] A deep reinforcement learning strategy transfer method based on fuzzy inference tree. The detailed steps are as follows:

[0072] Step 1: Use a deep reinforcement learning algorithm to train the network to be transferred, which is the policy network of the deep reinforcement learning algorithm or the value network used to characterize the policy.

[0073] In the intelligent decision-making task of the pursuit-escape game, the decision-making performance of the network to be migrated is measured by three indicators: the relative distance between the pursuing aircraft and the escaping aircraft at the final moment (hereinafter referred to as "pursuit distance"), the energy consumption of the escaping aircraft, and the consumption of false target decoys of the escaping aircraft.

[0074] The escaping plane The position coordinates at time , the pursuing aircraft The position coordinates at time , the zero time (starting time) of the pursuit-escape game is , the end time is During this process, the command overload of the escaping aircraft in each direction is , the command overload of the chasing aircraft in each direction is , the number of false target decoys consumed by the escaping aircraft is .

[0075] At the end of the intelligent decision-making task of the pursuit and escape game, The positions of the escaping aircraft and the chasing aircraft at the termination time are and , if the pursuit distance is less than or equal to the specified miss distance L, that is:

[0076]

[0077] The avoidance is considered to have failed, and the reward function returns a reward signal value of -1.

[0078] If the pursuit distance is greater than the specified miss distance L, that is:

[0079]

[0080] If the avoidance is successful, the miss amount reward, energy reward, and bait reward are recorded as , and ,in

[0081] =

[0082]

[0083]

[0084] Final Reward Signal The value of

[0085]

[0086] in, , and The weight of each reward can be adjusted according to the task tendency (inclined to reduce the consumption of false target bait or reduce the energy consumption during the escape process under the premise of successful escape). The three weights meet and In the intelligent decision-making task of the pursuit-escape game, the decision-making performance index of the network to be migrated is equivalent to the reward signal The higher the value of , the higher the average value of the reward signal R obtained in multiple pursuit-escape game intelligent decision-making tasks, the better the decision-making performance of the network to be migrated. 2 represents the 2-norm of the vector.

[0087] The position, attitude and speed of the chasing and escaping aircraft are used as status signals, the control instructions of the escaping aircraft's various control surfaces are used as action signals, and the miss distance of the chasing aircraft, the energy consumption of the escaping aircraft during the avoidance process and the consumption of false target bait are used as reward signals. , build the network to be transferred by the deep learning algorithm, and simulate the network to be transferred in the simulation environment, and use the deep reinforcement learning algorithm to search for the optimal strategy. After sufficient training until the network to be transferred converges and the reward signal After the value of rises to the expected level, an evaluation is performed. If the network to be transferred can achieve the corresponding performance indicators in the pursuit and escape game intelligent decision-making task, the training is completed. Otherwise, the reward function, network structure and / or training parameters are modified and retrained until the network to be transferred achieves the expected effect. At this time, a well-trained network to be transferred is obtained.

[0088] Step 2: Split and sample the state space.

[0089] The state space of the pursuit and escape game intelligent decision-making task is evenly divided into multiple intervals at certain intervals, and a certain number of state data samples are evenly collected in each interval. In this step, the state space needs to be evenly divided into equal intervals and the same number of samples are sampled in each interval to ensure that the sampling results conform to the uniform distribution, otherwise it will affect the training of the fuzzy inference tree in rule migration.

[0090] Step 3: Construct the optimal decision dataset and divide it into training set and test set.

[0091] An optimal decision data set is constructed based on the samples collected in step 2 and the trained network to be migrated obtained in step 1. The data in the optimal decision data set are samples of the uniformly distributed state data sampled in step 2. The labels in the optimal decision data set are optimal actions that correspond one-to-one to the samples of the state data. The optimal action is obtained by calculating the parameters used to describe the distribution of the optimal action using the sample data input into the trained network to be migrated obtained in step 1.

[0092] The specific method of constructing the optimal decision data set is as follows:

[0093] First, the samples sampled in step 2 are input as input signals to the network to be transferred that has been trained in step 1. The output signal of the network to be transferred is a Gaussian parameter used to describe the optimal action distribution. The Gaussian parameter is used to construct an action that satisfies the Gaussian distribution and sample it to obtain the optimal action, which is used as the label.

[0094] Then, the obtained labels are matched one by one with the samples obtained in step 2 to construct the optimal decision data set.

[0095] Finally, KL divergence is used to check and adjust the uniformity of the label distribution in the optimal decision dataset.

[0096] After the construction is completed, the optimal decision dataset is divided into training set and test set in a ratio of 8:2.

[0097] Step 4: Construct the fuzzy inference tree.

[0098] According to the attributes of the pursuit-escape game intelligent decision-making task, the structure of the fuzzy reasoning tree is planned, and the number of fuzzy reasoning units in the fuzzy reasoning tree is determined. The pursuit-escape game task is divided into the branches of escaping party state processing, pursuing party motion state processing, relative motion state processing, intermediate signal conversion, maneuver planning decision, maneuver parameter solution and false target bait placement decision. Each branch contains several fuzzy reasoning units. In the same branch, except for the fuzzy reasoning units at the input and output ends, the remaining fuzzy reasoning units are only connected to the fuzzy reasoning units in this branch. The input ends of the three branches of escaping party state processing, pursuing party motion state processing and relative motion state processing only receive the input of state signals. The output ends of the two branches of maneuver parameter solution and false target bait placement decision directly constitute the output of the entire fuzzy reasoning tree. The input / output ends of other branches can be connected to the branches of the upper and lower layers. For details of the specific fuzzy reasoning tree structure and signal transmission process, please see the attached. Figure 4 .

[0099] Step 5: Perform migration training on the fuzzy inference tree constructed in step 4.

[0100] Step 5.1 parameter initialization;

[0101] After the fuzzy inference tree is constructed, the parameters of each fuzzy inference unit of the fuzzy inference tree (fuzzy inference rule parameters, membership function parameters and defuzzification parameters) need to be initialized. In this embodiment, the parameters of each fuzzy inference unit are initialized to a certain value (a random value obtained by sampling a standard Gaussian distribution) by using Gaussian distribution initialization. After the initialization is completed, the parameters of all fuzzy inference units are regularized into a parameter sequence in an arbitrary order, and the parameter sequence is converted into a binary code.

[0102] Step 5.2: construct the genetic algorithm model;

[0103] According to the length of the binary code obtained in step 5.1 and the complexity of the intelligent decision-making task of the pursuit and escape game, the genetic algorithm model is constructed, and the genetic algorithm parameters such as population size, selection method, crossover probability and mutation probability are determined, and the population is initialized. According to the optimization goal, the fitness function is set for the iteration of the genetic algorithm.

[0104] Step 5.3 genetic algorithm iterative optimization;

[0105] Set the maximum number of iterations and the iteration termination threshold of the genetic algorithm. In each iteration, the fitness of each individual in the population is calculated and selection, crossover, and mutation operations are performed based on the fitness to generate new individuals. Repeat the above iteration process until the individual with the highest fitness in the population reaches the iteration termination threshold or the number of iterations reaches the maximum number of iterations, and the binary code of the individual is solved as the parameters of the fuzzy inference tree.

[0106] Step 5.4: Fitting performance verification;

[0107] After iteratively optimizing the parameters of the fuzzy inference tree in step 5.3, verify the fitting performance of the fuzzy inference tree using the test set constructed in step 3 to see whether it can accurately fit the data in the test set. If underfitting or overfitting occurs, adjust the population size, selection method, crossover probability and / or mutation probability in the genetic algorithm and repeat step 5.3. If underfitting or overfitting does not occur, the migration training is completed and proceed to step 6.

[0108] Step 6: Evaluate the effectiveness of policy migration.

[0109] The decision-making performance of the fuzzy inference tree is evaluated. Specifically, the tuned fuzzy inference tree is substituted into the decision-making process, replacing the original network to be migrated trained in step 1 to interact with the simulation environment, and using the fuzzy inference tree to calculate the optimal action according to the state of the pursuit and escape game intelligent decision-making task that needs to be optimized. After at least 50 rounds of complete decision-making, the decision-making performance of the fuzzy inference tree is evaluated according to the average discounted cumulative return. If the average discounted cumulative return of the fuzzy inference tree is not less than 90% of the average discounted cumulative return obtained by the decision-making of the network to be migrated trained in step 1, it is considered that the fuzzy inference tree has a decision-making performance similar to that of the network to be migrated trained in step 1, indicating that the pursuit and escape game intelligent decision-making strategy in the network to be migrated trained in step 1 has been completely migrated to the fuzzy inference tree. In the actual pursuit and escape game task, the current fuzzy inference tree is used as the strategy network for generating decision signals, and the process ends. Otherwise, it is considered that the fuzzy inference tree fails to inherit the pursuit and escape game intelligent decision-making strategy of the network to be migrated trained in step 1 well, and it is necessary to return to step 5 to modify the genetic algorithm parameters and perform migration training of the fuzzy inference tree again. After the migration training is completed, the decision-making performance is re-evaluated. If the decision performance evaluation of this step still cannot be passed after repeating the migration training of step 5 for 10 times, it is considered that the decision performance upper limit of the current fuzzy inference tree cannot meet the requirements of representing the intelligent decision-making strategy of the pursuit and escape game contained in the network to be migrated, and the structure of the fuzzy inference tree needs to be optimized. At this time, execute step 7.

[0110] Step 7: Optimize the structure of the fuzzy inference tree.

[0111] If the decision performance of the fuzzy inference tree cannot meet the expected requirements, it is necessary to adjust and optimize the fuzzy tree structure constructed in step 4. The specific method includes adding the number of fuzzy inference units and / or decomposing the fuzzy inference units in at least one branch of the original escape party state processing, pursuit party motion state processing, relative motion state processing, intermediate signal conversion, maneuver planning decision, maneuver parameter solution and false target decoy delivery decision.

[0112] After optimizing the structure of the fuzzy inference tree, it is necessary to re-perform migration training and test its decision-making performance, that is, proceed to step 5.

[0113] Finally, it should be noted that when the present invention is used to transfer strategies for different reinforcement learning algorithms, only the source and generation method of the data are different, and the algorithm training and rule extraction process of reinforcement learning are consistent with the present invention. In addition, the described embodiments are intended to provide a detailed description of the present invention and do not limit the actual application.

Claims

1. A deep reinforcement learning strategy transfer method based on fuzzy inference tree, characterized in that: Includes steps: Step 1: Use a deep reinforcement learning algorithm to train the network to be transferred, where the network to be transferred is a policy network of the deep reinforcement learning algorithm or a value network used to characterize the policy; Step 2: The state space of the pursuit and escape game intelligent decision-making task that needs to be optimized is divided into equal intervals and uniformly sampled to obtain samples of state data that conform to the uniform distribution; Step 3: construct an optimal decision data set and divide it into a training set and a test set; the optimal decision data set consists of data and labels, the data is a sample of the state data obtained in step 2, the label is the optimal action corresponding to the sample, and the optimal action is obtained based on the parameters generated by the network to be migrated trained in step 1 and used to describe the distribution of the optimal action; Step 4: construct a fuzzy inference tree according to the pursuit-escape game intelligent decision-making task, the fuzzy inference tree includes multiple fuzzy inference units, and the connection relationship between the fuzzy inference units is constructed according to the relationship between the pursuit party's motion state, the escape party's motion state, the escape party's internal state and the relative motion state of the pursuit and escape parties and the decision-making instructions in the pursuit-escape game intelligent decision-making task; the input of the fuzzy inference tree is a state signal, and the output is a decision-making instruction; the state signal includes the pursuit party's motion state, the escape party's motion state and the escape party's internal state; the decision-making instruction includes the escape party's maneuvering parameters, the false target bait delivery decision and the false target bait delivery parameters; Step 5: Using an intelligent optimization algorithm and the training set to perform migration training on the fuzzy inference tree, and using the test set to verify the fitting performance of the fuzzy inference tree; Step 6: Use the average discounted cumulative return after at least 50 rounds of complete decision-making to evaluate the decision-making performance of the fuzzy inference tree trained in step 5. If the average discounted cumulative return of the fuzzy inference tree is greater than or equal to 90% of the average discounted cumulative return of the network to be migrated trained in step 1, the decision-making performance evaluation of the fuzzy inference tree is qualified, the strategy migration is completed, and the current fuzzy inference tree is used to replace the network to be migrated trained in step 1, and the process ends; Otherwise, return to step 5; if the decision-making performance of the fuzzy inference tree is still unqualified after 10 or more migration trainings, proceed to step 7; Step 7: Optimize the structure of the fuzzy inference tree constructed in step 4, increase its complexity to improve the upper limit of its decision-making performance, and then return to step 5.

2. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 1 is characterized in that: The deep reinforcement learning algorithm described in step 1 is DQN, DDPG, SAC, TD3, PPO, MADDPG, or MAPPO.

3. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 1, characterized in that: The method for constructing the optimal decision dataset described in step 3 is: First, the samples obtained in step 2 are input into the network to be transferred that has been trained in step 1. The network to be transferred outputs parameters used to describe the optimal action distribution. The parameters are used to construct actions that satisfy Gaussian distribution, uniform distribution, or Bernoulli distribution and sample them to obtain the optimal action, which is used as the label. Then, the labels are matched one by one with the samples obtained in step 2 to construct an optimal decision data set; Finally, the KL divergence is used to check and adjust the uniformity of the label distribution.

4. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 1, characterized in that: The fuzzy inference tree constructed in step 4 includes 8 fuzzy inference units, namely, the escape party state processing unit FIS-1, the chase party motion state processing unit FIS-2, the relative motion state processing unit FIS-3, the first intermediate signal conversion unit FIS-4, the second intermediate signal conversion unit FIS-5, the maneuver planning decision unit FIS-6, the false target bait delivery decision unit FIS-7 and the maneuver parameter solution unit FIS-8; The input of the escape party state processing unit FIS-1 is the position, speed, attitude, number of remaining false target decoys and control surface state of the escape party, and the output is the space orientation of the escape party, the control margin of the escape party, and whether the remaining false target decoys of the escape party are sufficient; The input of the chasing party motion state processing unit FIS-2 is the chasing party's position, speed and posture, and the output is the chasing party's spatial orientation; The input of the relative motion state processing unit FIS-3 is the relative distance, line of sight deflection angle and line of sight inclination of the two parties, and the output is the relative height advantage, relative energy advantage and relative direction of the two parties; The input of the first intermediate signal conversion unit FIS-4 is the control margin of the escaping party and whether the remaining false target bait of the escaping party is sufficient, and the output is the intention of the escaping party to release the false target bait and the intention of the escaping party to change the maneuver; The input of the second intermediate signal conversion unit FIS-5 is the spatial orientation of the chasing party, the relative height advantage, relative energy advantage and relative direction of the chasing and escaping parties, and the output is the threat level of the chasing party to the escaping party; The input of the maneuver planning and decision unit FIS-6 is the escaping party's intention to change the maneuver, the escaping party's spatial orientation, the chasing party's spatial orientation, the relative height advantage, relative energy advantage, relative orientation of the chasing and escaping parties, and the threat level of the chasing party to the escaping party. The output is the escaping party's maneuver type. The input of the false target decoy release decision unit FIS-7 is the threat level of the chasing party to the escaping party, the escaping party's willingness to release the false target decoy and the type of maneuver of the escaping party. The output is whether the escaping party releases the false target decoy and the release parameters of the false target decoy. The input of the maneuver parameter solving unit FIS-8 is the maneuver type and spatial orientation of the escaping party, and the output is the maneuver parameter of the escaping party.

5. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 1, characterized in that: The intelligent optimization algorithm described in step 5 is a genetic algorithm, a particle swarm algorithm, an ant colony algorithm or a simulated annealing algorithm.

6. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to claim 5 is characterized in that: The transfer training of the fuzzy inference tree in step 5 refers to regularizing the parameters of the fuzzy inference tree into a parameter sequence according to an arbitrary parameter order, and optimizing the parameter sequence by using a genetic algorithm, a particle swarm algorithm, an ant colony algorithm or a simulated annealing algorithm after binary encoding the parameter sequence.

7. The deep reinforcement learning strategy transfer method based on fuzzy inference tree according to any one of claims 1 to 6, characterized in that: The method for optimizing the structure of the fuzzy inference tree constructed in step 4 in step 7 is: increasing the number of divided fuzzy sets, reconstructing the fuzzification unit in each fuzzy inference unit in the fuzzy inference tree according to the current fuzzy set, and correspondingly increasing the number of fuzzy rules required for the fuzzy rule inference unit in each fuzzy inference unit, and / or decomposing at least one fuzzy inference unit in the fuzzy inference tree into multiple different fuzzy inference units according to the input signal.

Images