Game agent input and output boundary positioning method for time series data inversion

Abstract

The invention discloses a time series data inversion game agent input and output boundary positioning method, which comprises the steps of constructing a game environment, and obtaining input situation data and output maneuvering decision data of both parties and input and output characteristics of a decision network in our agent after a game task is finished; according to the input situation data, calculating an accumulated reward value of a maneuvering decision of the agent at each moment, determining a performance boundary moment set of situation transformation and performing inversion positioning to obtain performance boundary input situation data and performance boundary output maneuvering decision data; unsupervised clustering is carried out on the output features to obtain a network boundary moment set, inversion positioning is carried out, and network boundary input situation data and network boundary output maneuvering decision data of our party are obtained; and iteratively executing the process until the boundary feature coverage rate is greater than or equal to a feature coverage threshold value, and obtaining the final agent input and output boundary of our part. According to the invention, the real-time boundary interpretability of the game agent at the maneuvering strategy level is realized.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning game agents, and specifically relates to a method for locating input and output boundaries of game agents by inverting time series data. Background Art

[0002] In scenarios such as combat simulation, the game agent method based on deep reinforcement learning training is a common method. Its main purpose is to learn an optimal maneuver decision strategy so that the agent (such as our drone) can maximize the cumulative reward in the interaction with the real or simulated environment. However, the neural network decision model obtained by its training is a black box model with unknown internal execution logic. The mapping relationship between network input and output is complex, and the maneuver strategy is poorly interpretable. When the agent's autonomous decision network is installed on actual equipment, it is necessary to obtain the maneuver input and output boundaries of the game agent and construct failure boundary scenarios to enhance the reliability and safety of its intelligent decision-making.

[0003] At present, the definition of the autonomous input and output boundaries of game agents is usually based on hyperparameter sampling in the task-level environment. For example, in "Lu H, Wang S, Cheng S, et al. Surrogate-Based Decision Boundary Evolutionary Sampling Method for Autonomous Systems [J]. IEEE Transactions on Emerging", sampling is performed in the pre-known environmental variables such as battlefield scope, number of air defense equipment and firepower coverage, number of enemy aircraft and equipment, and task simulation is performed in the simulation system. The autonomous decision performance boundary of the agent is judged based on the success or failure of the task. However, even under the same environmental conditions, the simulated temporal autonomous decision and the task result in each task of the agent are likely to be different, and the boundary definition does not take into account the real-time temporal maneuver autonomous decision. The existing interpretable reinforcement learning game method is only aimed at the action decision of the intelligent agent. For example, "Air Combat Maneuver Decision Method Based on Interpretable Reinforcement Learning, Yang Shuheng, Zhang Dong, Xiong Wei, Ren Zhi, Tang Shuo" only designs the reward function and repairs the game agent strategy through linear regression, classifies the network decision of the intelligent agent into interpretable combination actions such as dive, hover, half roll, somersault, etc., and summarizes the output strategy of the model at the level of maneuver intention. Patent CN202410815001.5 converts the incomplete information dynamic game model into a complete information dynamic game model through the Hasani transformation to obtain the optimal maneuver strategy of the game agent, and also classifies and explains the maneuver decision output by the model in detail.

[0004] Due to the complexity and opacity of the model, the maneuver decision-making strategy generated by the network of deep reinforcement learning game agents is difficult to explain, the complex mapping of input and output features is difficult to model, and the generalization scenario usability of the strategy is poor. Existing methods for deep reinforcement learning interpretability mainly learn interpretable strategies by fitting a transition probability matrix, or interpret the reward function to convert the learning strategy into an interpretable form. In the autonomous decision-making of game agents, the existing methods simply classify the decision output of the network into pre-set combined actions, and do not have the ability to judge the rationality of the decision-making of the agent network under what battlefield situation, and cannot meet the boundary determinism requirements of high dynamics and strong real-time performance of air combat. For the boundary positioning of deep reinforcement learning agents, the existing technology simulates according to the preset environmental variable parameters of the sampling, and only locates the boundary of the adjustable hyperparameters according to the final task results. The shortcomings are as follows: (1) There is a lack of explanation for the maneuver decision variables that are strongly correlated with the game time sequence. The definition does not take into account the temporal cyclic relationship of the autonomous decision-making network, and does not combine the real-time battlefield situation for boundary judgment; (2) There are defects in the boundary definition method. Under the same environmental settings, the maneuver decisions and final results of the intelligent agents for each simulated game task will be largely inconsistent. Instead, the boundary positioning is based only on the success or failure of the simulation task. In terms of basic concepts, it does not meet the requirements of high real-time response of game agents. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method for locating the input and output boundaries of a game agent by time series data inversion. The technical problem to be solved by the present invention is achieved by the following technical solutions: A method for locating input and output boundaries of a game agent based on time series data inversion, comprising: S1, constructing a game environment for the confrontation between enemy and friendly aircraft, obtaining the input situation data and output maneuver decision data of the enemy and friendly aircraft after the game task is completed, as well as the input features and output features of the decision network based on deep reinforcement learning in our intelligent agent; wherein our aircraft is an intelligent agent; the input features of the decision network are obtained according to the input situation data of our intelligent agent, and the output maneuver decision data of our intelligent agent are obtained according to the output features of the decision network; S2, according to the input situation data, calculate the cumulative reward value of the maneuvering decision of our agent at each moment during the entire game task execution process; S3, according to the cumulative reward value of the maneuvering decision of our intelligent agent at each moment, determine the performance boundary moment set of our situation change; perform inversion positioning at each moment according to the performance boundary moment set to obtain our performance boundary input situation data and performance boundary output maneuvering decision data; S4, by performing unsupervised clustering on the output features, a network boundary time set corresponding to the boundary features of the decision network is obtained; inversion positioning is performed at each time according to the network boundary time set to obtain our network boundary input situation data and network boundary output maneuver decision data; S5, iteratively execute S1~S4, and calculate the boundary feature coverage rate based on the obtained performance boundary input situation data, performance boundary output maneuver decision data, network boundary input situation data and network boundary output maneuver decision data, and stop the iteration when the boundary feature coverage rate is greater than or equal to the preset feature coverage threshold to obtain our final intelligent agent input and output boundary.

[0006] The method for locating the input and output boundaries of game agents by inverting time series data provided in the embodiment of the present invention has for the first time constructed a method for locating the performance boundaries and network boundaries of game agents. It uses a feature-level unsupervised clustering interpretation method and a behavior-level real-time environmental situation awareness judgment, is driven by time series data of actual or simulated environments, and inverts the boundary scene input of the agent by boundary features, thus realizing the real-time boundary interpretability of game agents at the level of maneuver strategies. Specifically, it has the following beneficial effects: 1. Deeper decision-making explanation capability: This invention can go deep into the internal feature level of the agent's decision through the feature-level unsupervised clustering explanation method. By solving the first-order derivative and second-order derivative of the cumulative reward function, it can accurately locate the interval of the agent's situation reversal and reveal the internal logic of the agent's decision-making under different battlefield situations. This in-depth explanation capability goes beyond the explanation of the action decision level that the existing technology can usually only provide, making the decision-making process more transparent and easier for operators to understand and trust.

[0007] 2. Higher decision fidelity and credibility: Existing technologies are usually based on task-level boundary definition. Under the same environment setting, the simulation results of each task of the agent may be inconsistent, resulting in low fidelity and credibility of boundary positioning. The present invention provides a new method for positioning the performance boundary and network boundary of game agents through temporal maneuver input and output boundary positioning, which can more accurately reflect the performance of the agent in the actual game and provide a more comprehensive battlefield situation awareness, thereby improving the fidelity and credibility of decision-making.

[0008] 3. Stronger strategy generalization and adaptability: This invention uses boundary feature inversion to identify the boundary scene input of the agent, and can identify the battlefield situations in which the agent performs poorly, so as to optimize the strategy in a targeted manner. This capability makes the agent's strategy effective not only in the training environment, but also in a wider range of actual game environments, improving the generalization and adaptability of the strategy.

[0009] 4. More efficient real-time response capability: The present invention can identify the input and output boundaries of the intelligent agent in real time based on battlefield situation perception, which is crucial for highly dynamic gaming environments. In contrast, existing technologies usually require post-analysis and cannot provide real-time decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic flow chart of a method for locating the input and output boundaries of a gaming agent by time series data inversion provided by an embodiment of the present invention; Figure 2 A schematic diagram of the principle of a method for locating the input and output boundaries of a gaming agent by time series data inversion provided by an embodiment of the present invention; Figure 3 A schematic diagram of a battlefield situation function according to an embodiment of the present invention; Figure 4 It is a single air combat data intelligent agent performance boundary positioning feature map of an embodiment of the present invention; Figure 5 It is a single air combat data agent network boundary positioning feature map of an embodiment of the present invention; Figure 6 It is a characteristic graph of the boundary positioning of the intelligent agent performance according to an embodiment of the present invention; Figure 7 This is a feature map of the intelligent agent network boundary positioning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0012] The current method for locating the boundaries of game agents is to search for settable environmental variables to determine the success or failure of a task, and the adjacent variables of the environmental variables corresponding to success or failure are defined as boundaries. However, even under the same environmental settings, the simulation process and results of the agent each time may not be completely consistent, so the boundary definition based on the task level has poor fidelity and low credibility. The present invention proposes a new method for locating the performance boundaries and network boundaries of game agents based on the temporal maneuvering decision of the agent.

[0013] At the same time, most of the current game agents trained based on deep reinforcement learning have models that contain a large number of parameters and complex network structures, and their nonlinear characteristics of maneuvering decision-making mechanisms are difficult to explain. This invention is driven by time series data, locates the characteristic boundary of the agent's situation reversal at the level of maneuvering strategy characteristics, and constructs the agent's boundary scene through inversion.

[0014] Specifically, the embodiment of the present invention provides a method for locating the input and output boundaries of a game agent by inverting time series data. Figure 1 The method steps shown and Figure 2 The method principle shown is understood that the method may include the following steps: S1, build a game environment for the confrontation between enemy and friendly aircraft, obtain the input situation data and output maneuver decision data of both sides after the game task is completed, as well as the input features and output features of the decision network based on deep reinforcement learning in our intelligent agent; In an embodiment of the present invention, a game environment between the enemy and us can be constructed by real environment interaction or simulated environment interaction, for example, it can be constructed using a game simulation platform, but the game simulation platform used is not limited here. In this game environment, our side is the red side and the enemy side is the blue side, using their respective aircraft to play against each other, our aircraft is an intelligent agent, and whether the enemy aircraft is an intelligent agent is not limited. A well-trained decision network is deployed in our intelligent agent, and the decision network is a deep reinforcement learning network, which can be implemented using any existing intelligent agent game decision network based on deep reinforcement learning, and is not limited here.

[0015] The input features of the decision network are obtained based on the input situation data of our agent, and the output maneuver decision data of our agent is obtained based on the output features of the decision network.

[0016] The decision network can transform the input situation data perceived by our agent through environmental interaction into input features of the decision network, and obtain output features through deep learning decisions of the decision network, and then transform the output features into output maneuver decision data of our agent to guide the action execution of our agent, so as to conduct a confrontation game. For the concepts and interrelationships of our agent's input situation data, the input features of the decision network, the output features of the decision network, and the output maneuver decision data of our agent, please refer to the relevant content in the field of deep reinforcement learning game agents for understanding, and no detailed explanation will be given here.

[0017] For S1, first, the game environment of the enemy and ours is constructed using the game simulation platform. During the construction process, the input and output variables of our agent and the intelligent game simulation environment are mainly determined. Among them, the input variables of our agent usually include the agent's position information (such as longitude, latitude, altitude), state information (such as pitch angle, heading angle, roll angle), and observations (such as enemy position, speed, state, etc.), and the output variables include the agent's decision-making actions, such as controllable parameters such as the aileron, elevator, rudder, and throttle of the agent. The corresponding relationship between the input variables and the output variables is the nonlinear mapping of the agent's decision network. The game simulation environment usually includes an aerodynamic model (such as the JSBSim aerodynamic platform, etc.), a combat map, etc. The main function of the intelligent game simulation environment is to ensure that the decisions made by the agent during simulation meet the constraints of the real environment.

[0018] Secondly, start the game task for both parties. Specifically, start the timing, obtain input situation data through the interaction between our agent and the environment, update the state space of our agent, output decision data, and repeat until the game task is terminated and the timing ends. The termination of the game task can be, for example, reaching the maximum overload of the aircraft or being shot down; At the same time, with the help of the game simulation platform, the input situation data of both sides during the execution of the game task is obtained and saved. and output maneuver decision data , and the input features of our agent’s decision network and output features This part corresponds to Figure 2 The temporal input perception data and temporal output decision data in the game task are the input situation data at each moment in the game task execution process, and the temporal output decision data are the output maneuver decision data at each moment in the game task execution process.

[0019] Among them, the superscript Represents our data, superscript Represents enemy data, subscript and Represent input and output respectively. Input situation data and output situation data can be set according to the number of aircraft of both sides of the game, the scope of the game, the type of operation, etc. Among them, for the number of aircraft of both sides of the game, it can be a 1-on-1 game, a 2-on-2 game, a multi-agent collaborative game, etc. For the scope of the game, it can be a near-visual-range game, a mid-visual-range game, a beyond-visual-range game, etc. For the type of operation, it can be air combat, electronic warfare, air patrol reconnaissance, etc.

[0020] Taking a one-on-one game as an example, in an optional implementation, the input situation data may include the aircraft's altitude, longitude, latitude, heading angle, pitch angle, roll angle, and speed; the output maneuver decision data may include the aircraft's aileron state, elevator state, rudder state, throttle state, and whether to launch a missile. Of course, the input situation data and the output maneuver decision data include but are not limited to the above examples.

[0021] Furthermore, after obtaining the input situation data and output maneuver decision data of both the enemy and our party after the game task is completed, as well as the input features and output features of the decision network based on deep reinforcement learning in our agent, the method may also include: Determine the mapping relationship between the input features and the output features, as well as the corresponding relationship between the input features and the input situation data of our agent, and the corresponding relationship between the output maneuver decision data of our agent and the output features.

[0022] Among them, the input features of our agent's decision network are and output features The mapping relationship can be expressed as: , represents the decision network of our agent. This mapping relationship reflects the processing process of the decision network. Through this mapping relationship, we can and output features When one feature is present, another feature is solved.

[0023] Determining the correspondence between the input features and the input situation data of our intelligent agent, and the correspondence between the output maneuver decision data of our intelligent agent and the output features, is to be able to use the correspondence between the input features and the input situation data of our intelligent agent in the future to obtain the input features corresponding to the decision network for any input situation data of our intelligent agent, and to use the correspondence between the output maneuver decision data of our intelligent agent and the output features to obtain the output features corresponding to the decision network for any output maneuver decision data of our intelligent agent, for the positioning of the input and output boundaries of the intelligent agent. This part will be explained in detail later.

[0024] S2, according to the input situation data, calculate the cumulative reward value of the maneuvering decision of our agent at each moment during the entire game task execution process; This step is to calculate the real-time battlefield situation of our agent during the execution of the entire game task.

[0025] Specifically, the decision network of our intelligent agent is set with a reward function, which can calculate the cumulative reward value of our intelligent agent's mobile decision-making at each moment during the execution of the entire game task based on the input situation data at each moment and the reward function preset in the decision network.

[0026] In the actual calculation process, the cumulative reward value of the mobile decision at each moment is obtained by calculating the real-time reward value of the mobile decision at each moment using the preset reward function. That is to say, assuming that the entire game task execution process corresponds to 1~ moment, then for each of those moments , we can calculate the real-time reward value of our agent's maneuver decision at that moment , the real-time reward value The real-time reward value accumulated at all previous moments Then sum it up and we can get the time The cumulative reward value of our agent's maneuver decision , indicating 1~ The sum of the real-time reward values ​​at each moment. For the process of calculating the real-time reward value of the agent's maneuver decision at each moment given a reward function, please refer to the relevant technical understanding.

[0027] Taking a one-on-one game as an example, the preset reward function can be combined and set according to the position, posture, whether it is in the attack range, whether it is locked, whether it has entered the no-escape zone, whether it has been shot down, etc. of the aircraft of both parties, and there is no restriction here.

[0028] Without considering radar detection, in an optional implementation, for a 1-on-1 game, the preset reward function is obtained based on the sum of the distance reward, the posture advantage reward and the shootdown reward; for example, the sum of the three reward values ​​can be directly calculated. The distance reward is expressed as: ; in, represents said distance reward; Indicates the calculation of relative distance; Indicates the position of our aircraft; Indicates the position of the enemy aircraft; the position of the aircraft can be represented by the aircraft's altitude, accuracy, and latitude geographic coordinate system. The closer our aircraft is to the enemy aircraft, the greater the distance reward setting is, in order to make our agent actively contact the enemy agent.

[0029] The posture advantage bonus is expressed as: ; in, represents the said posture advantage bonus; Indicates the azimuth of our aircraft relative to the enemy aircraft; Indicates the azimuth of the enemy aircraft relative to our aircraft; when the sum of the azimuths of the enemy and our aircraft is 360°, it means that our agent is in a tail-chasing attack posture, and our angle posture advantage is the greatest at this time; conversely, when the sum of the azimuths of the enemy and our aircraft is 0°, it means that our agent is in a tail-chasing attack posture, and our angle posture advantage is the smallest at this time; The shootdown reward is It means that the shootdown reward reflects the result of the confrontation. A positive reward is given if our aircraft wins, a negative reward is given if the enemy aircraft wins, and a zero reward is given if the game is a draw or the game is not over.

[0030] Of course, the above reward function is only an example. When different methods are used to obtain battlefield situation awareness, the reward function can be different and can be set accordingly. For example, the corresponding battlefield situation assessment criteria can be replaced according to different game scenarios, such as increasing the radar detection distance, radar reflection interface, etc., and then the corresponding reward function that meets the needs can be designed.

[0031] S3, according to the cumulative reward value of the maneuvering decision of our intelligent agent at each moment, determine the performance boundary moment set of our situation change; perform inversion positioning at each moment according to the performance boundary moment set to obtain our performance boundary input situation data and performance boundary output maneuvering decision data; This step corresponds to Figure 2 Mid-time sequence battlefield situation - situation reversal positioning - performance boundary positioning and the corresponding inversion part.

[0032] In an optional implementation, calculating the cumulative reward value of the maneuvering decision of our agent at each moment during the entire game task execution process according to the input situation data may include: 1) Determine a cumulative reward function according to the cumulative reward value of the mobile decision-making of our agent at each moment; According to the 1~ time, and the cumulative reward value of our agent's mobile decision at each time , the cumulative reward value of the moment and maneuver decision can be determined by data fitting and other methods The functional relationship is defined as the cumulative reward function, expressed as: ; See also Figure 3 , Figure 3 This is a schematic diagram of the battlefield situation function of an embodiment of the present invention. The battlefield situation function is the cumulative reward function. The red curve represents the cumulative reward function of our intelligent agent. . Figure 3 The horizontal axis represents time, and the vertical axis represents the cumulative reward value of the maneuver decision at each moment.

[0033] 2) Calculate the first-order derivative and the second-order derivative of the cumulative reward function, determine the reversal time point when our situation changes from disadvantage to advantage according to the calculation results, obtain a first reversal time set, and determine the reversal time point when our situation changes from advantage to disadvantage, obtain a second reversal time set; Specifically, for the cumulative reward function Find the first derivative , and find the second-order derivative ; If the cumulative reward function Any point on the curve satisfies ,and , it can be understood that this point in the cumulative reward function The curve is an inflection point located at the crest. At this time, we determine that this point is the reversal point where our situation changes from disadvantage to advantage. The moment corresponding to this point is the reversal moment when our situation changes from disadvantage to advantage. Thus, all such points are combined into a set to obtain the performance boundary set. , the performance boundary set The corresponding time points form a set to obtain the first reversal time set; If the cumulative reward function Any point on the curve satisfies ,and , it can be understood that this point in the cumulative reward function The curve is an inflection point located at the trough. At this time, we determine that this point is the reversal point where our situation changes from advantage to disadvantage. The moment corresponding to this point is the reversal moment when our situation changes from advantage to disadvantage. Thus, all such points are combined into a set to obtain the performance boundary set. , the performance boundary set The corresponding time points form a set to obtain a second reversal time set; 3) Combining the first reversal moment set and the second reversal moment set to obtain a performance boundary moment set.

[0034] Set performance boundaries and Merge to get the performance boundary range ; All moments in the first reversal moment set and the second reversal moment set are combined in order to obtain a performance boundary moment set, that is, to obtain the corresponding time sequence coding interval of the boundary feature, which can be expressed as .in, It can be seen that the present invention proposes a method for perceiving and distinguishing the input and output boundary situations of game agents, which judges the battlefield situation information of the game agents according to the cumulative reward value, and obtains the situation reversal interval of the agent by combining the first-order derivative and the second-order derivative of the situation function (i.e., the cumulative reward function).

[0035] See also Figure 4 , Figure 4 This is a single air combat data intelligent agent performance boundary positioning feature diagram of an embodiment of the present invention, where red represents performance boundary points and the rest are green non-performance boundary points (normal situation points of the intelligent agent).

[0036] Next, the inversion positioning is performed at each moment according to the performance boundary moment set to obtain the performance boundary input situation data and the performance boundary output maneuver decision data. At each moment in the game, the input situation data of our agent stored in S1 and output maneuver decision data Inverse positioning is performed in the process, that is, data acquisition at the corresponding time is performed to obtain performance boundary input situation data Output maneuver decision data and performance boundary .

[0037] Get performance boundary input situation data Output maneuver decision data and performance boundary That is to complete our performance boundary positioning.

[0038] Furthermore, in order to facilitate visual positioning display, after performing inversion positioning at each moment according to the performance boundary moment set and obtaining our performance boundary input situation data and performance boundary output maneuver decision data, the method may further include: ① The performance boundary input situation data and the performance boundary output maneuver decision data are respectively expanded at a given time according to the performance boundary positioning visualization range to obtain the visualization input situation data and the visualization output situation data corresponding to the performance boundary; Since the performance boundary represented by the performance boundary input situation data and the performance boundary output maneuver decision data reflects the short-term change of our situation, in order to clearly display the specific battle scene before and after the situation change and understand the change process, the embodiment of the present invention can give a time period to represent the performance boundary positioning visualization range, which is used to expand the situation change moment. Indicates that the specific length can be set as needed, for example, 20 seconds.

[0039] Entering situational data for performance boundaries Output maneuver decision data and performance boundary , respectively locate the visualization range according to the given performance boundary Perform time expansion to obtain the visual input situation data corresponding to the performance boundary and visualize situation data , that is, each moment is increased before and after The visual input situation data corresponding to the obtained performance boundary and visualize situation data Can combine input and output data to visualize performance boundaries .

[0040] ② The visualized input situation data and the visualized output situation data corresponding to the performance boundary are reproduced through the game simulation platform used when constructing the game environment, and a performance boundary battlefield scene is generated.

[0041] Visualize performance boundaries in input and output data Through the reproduction on the game simulation platform, the performance boundary battlefield scene in the corresponding period can be visualized and dynamically reproduced, which helps users to intuitively understand and feel it. For example, for the boundary battlefield scene where our intelligent agent locks the enemy, if there is only a boundary moment set, the simulation platform can only reproduce the state of the enemy and our party at the current moment, and it is impossible to intuitively see the temporal dynamic decision-making of the enemy and our party before and after the lock. This problem can be solved by expanding the moment and then reproducing it on the game simulation platform. This part can be seen in Figure 2 Mid-performance boundary battlefield scenario understanding.

[0042] S4, obtaining a network boundary time set corresponding to the boundary features of the decision network by performing unsupervised clustering on the output features; performing inversion positioning at each time according to the network boundary time set to obtain network boundary input situation data and network boundary output maneuver decision data; This step corresponds to Figure 2 Unsupervised clustering - network boundary location and the corresponding inversion part.

[0043] Unsupervised clustering can adopt existing density clustering, principal component analysis, hierarchical clustering, spectral clustering, etc., which are not limited here, and the purpose is to obtain the boundary features in the output features. Through unsupervised clustering, the decision features of the intelligent agent can be clustered into two clusters, and the decision features corresponding to the cluster with a smaller number are defined as boundary features, which can be understood as features with poor generalization of the decision network.

[0044] In an optional implementation manner, obtaining a network boundary time set corresponding to the boundary features of the decision network by performing unsupervised clustering on the output features may include: 1) Using density clustering method on the output features to filter out the boundary feature set; Before executing this step, we first preprocess the output features of the decision network of our agent saved in S1, specifically removing the corresponding variables of missing values ​​and normalizing them to maintain consistent data dimensions.

[0045] The process of density clustering the obtained output features can include the following steps: Step a1: Input all output features of our agent decision network The output feature set and the preset clustering radius and the minimum number of clusters ; in, and All are greater than 0; Cluster radius It can be set as needed, for example, it can be 3.

[0046] In the boundary location of the 1-to-1 game, the minimum number of clusters Set to 2 to indicate whether an output feature is a boundary feature.

[0047] Step a2: for each output feature in the output feature set, calculate the output feature The number of points contained in the neighborhood is the number of output features, and the obtained number is used as the density of the output feature. ; and determine the density of the output feature Is it satisfied? , if so, determine the output feature as a core feature, otherwise, determine the output feature as a non-core feature; Through step a2, all core features can be screened out.

[0048] Step a3: for each core feature, find the All points in the neighborhood form a cluster; Step a4, for the core features For each point in the neighborhood, if it is also a core point, then The points in the neighborhood are added to the current cluster, and the process is repeated recursively until all the core points are The points in the neighborhood have been judged, and no new points can be added to the current cluster; Step a5: Output the two clusters generated by clustering. The one with a smaller number is defined as the boundary feature cluster, and the other is the regular feature cluster, so as to obtain the boundary feature set according to the boundary feature cluster. .

[0049] 2) According to the time corresponding to the boundary features in the boundary feature set, a network boundary time set is obtained.

[0050] The time points corresponding to the boundary features in the boundary feature set are combined to obtain a network boundary time point set, which represents the corresponding time sequence coding interval of the boundary features. express.

[0051] See also Figure 5 , Figure 5 This is a single air combat data agent network boundary positioning feature map of an embodiment of the present invention, where red represents the network boundary points and green represents the points corresponding to the remaining output features.

[0052] Next, the network boundary time set is used to perform inversion positioning at each time to obtain the network boundary input situation data and the network boundary output maneuver decision data. At each moment in the game, the input situation data of our agent stored in S1 and output maneuver decision data Inverse positioning is performed in the network, that is, data acquisition at the corresponding time is performed to obtain the network boundary input situation data. and network boundaries to output maneuver decision data .

[0053] Get network boundary input situation data and network boundaries to output maneuver decision data That completes the positioning of our network boundaries.

[0054] It can be seen that the present invention proposes a method for determining the generalization of the network boundary of a game agent, which forms equivalent clusters of the features with poor network generalization through unsupervised clustering, and generates game boundary scenarios by inversion positioning the input situation data and output decision data.

[0055] Furthermore, in order to facilitate visual positioning display, after performing inversion positioning at each moment according to the network boundary moment set and obtaining our network boundary input situation data and network boundary output maneuver decision data, the method may also include: ① The network boundary input situation data and the network boundary output maneuver decision data are respectively extended at a given time according to the given network boundary positioning visualization range to obtain the visualization input situation data and visualization output situation data corresponding to the network boundary; Since the network boundary represented by the network boundary input situation data and the network boundary output maneuver decision data reflects the short-term change of our situation, in order to clearly display the specific battle scene before and after the situation change and understand the change process, the embodiment of the present invention can give a time period to represent the network boundary positioning visualization range, which is used to expand the situation change time. Indicates that the specific length can be set as needed, for example, 20 seconds.

[0056] Input situational data to network boundaries and network boundaries to output maneuver decision data , respectively locate the visualization range according to the given network boundary Perform time expansion to obtain visual input situation data corresponding to the network boundary and visualize situation data , that is, each moment is increased before and after The obtained network boundary corresponds to the visual input situation data and visualize situation data Can be combined to visualize input and output data at the network boundary .

[0057] ② The visualized input situation data and the visualized output situation data corresponding to the network boundary are reproduced through the game simulation platform used when constructing the game environment, and a network boundary battlefield scene is generated.

[0058] Visualize network boundaries and input and output data Through the reproduction on the game simulation platform, the network boundary battlefield scene in the corresponding period can be visualized and dynamically reproduced, which helps users to intuitively understand and feel it. Similar to the previous performance boundary, for example, for the boundary battlefield scene where our agent decision network makes abnormal maneuver decisions, if there is only a network boundary moment set, the simulation platform can only reproduce the state of the enemy and us at the current moment, and it is impossible to intuitively see the temporal dynamic decision of the agent's abnormal maneuver decision. This problem can be solved by expanding the moment and then reproducing it on the game simulation platform. This part can be seen in Figure 2 Battlefield scenario understanding at the border of the medium network.

[0059] For S3 and S4, the present invention performs boundary analysis from the decision network feature level, locates and expands the time series visualization range through inversion, maps it to the input situation data and output decision data corresponding to the intelligent agent, and reproduces and generates the input and output boundary battlefield scenes through the game simulation platform, thereby visually reproducing and displaying the performance boundary and network boundary, which is conducive to intuitive understanding.

[0060] S5, iteratively execute S1~S4, and calculate the boundary feature coverage rate based on the obtained performance boundary input situation data, performance boundary output maneuver decision data, network boundary input situation data and network boundary output maneuver decision data, and stop the iteration when the boundary feature coverage rate is greater than or equal to the preset feature coverage threshold to obtain the final agent input and output boundary.

[0061] This step corresponds to Figure 2 The understanding of the content in the middle iteration and the box in the lower part reflects the input and output boundary positioning of the deep reinforcement learning agent.

[0062] By iteratively executing S1~S4 once, the performance boundary input situation data, performance boundary output maneuver decision data, network boundary input situation data and network boundary output maneuver decision data of the current iteration can be obtained.

[0063] In an optional implementation manner, the calculation of boundary feature coverage based on the obtained performance boundary input situation data, performance boundary output maneuver decision data, network boundary input situation data, and network boundary output maneuver decision data includes: 1) Merge the performance boundary input situation data and the network boundary input situation data obtained in the current iteration to obtain the merged input situation data of the current iteration; merge the performance boundary output situation data and the network boundary output situation data obtained in the current iteration to obtain the merged output situation data of the current iteration; For each iteration, two boundary input situation data are merged, and two boundary output situation data are merged to obtain merged input situation data and merged output situation data.

[0064] 2) Calculate the boundary input feature corresponding to the combined input situation data of the current iteration and the boundary output feature corresponding to the combined output situation data of the current iteration according to the determined correspondence between the input feature and the input situation data of our agent, and the correspondence between the output maneuver decision data of our agent and the output feature; As mentioned above, in the relevant steps of S1, the correspondence between the input features of the decision network of our intelligent body and the input situation data of our intelligent body, and the correspondence between the output maneuver decision data of our intelligent body and the output features of the decision network of our intelligent body have been determined. Then, the input features of the decision network corresponding to the combined input situation data of the current iteration, that is, the boundary input features, can be determined by using the correspondence between the input features of the decision network of our intelligent body and the input situation data of our intelligent body. The output features of the decision network corresponding to the combined output situation data of the current iteration, that is, the boundary output features, can be determined by using the correspondence between the output maneuver decision data of our intelligent body and the output features of the decision network of our intelligent body.

[0065] 3) According to the boundary input features and boundary output features obtained in the current iteration, and the boundary input features and boundary output features accumulated before the current iteration, the boundary feature coverage is obtained by calculating the intersection; The formula for calculating boundary feature coverage is as follows: ; in, represents the boundary feature coverage; Represents the boundary input features and boundary output features obtained in the current iteration; Represents the boundary input features and boundary output features accumulated before the current iteration; Indicates intersection.

[0066] The preset feature coverage threshold can be used express, Greater than 0. The specific value can be set as needed, for example, it can be 0.95.

[0067] like Then stop the iteration.

[0068] Specifically, the iteration is stopped until the boundary feature coverage is greater than or equal to the preset feature coverage threshold, and our final agent input and output boundaries are obtained, including: Based on the boundary input features and boundary output features obtained at the corresponding iteration when the iteration is stopped, we get our final agent input and output boundaries.

[0069] That is to say, when the iteration is stopped, the boundary input features and boundary output features obtained in the corresponding iteration are This is our final agent input and output boundary.

[0070] See also Figure 6 and Figure 7 understand, Figure 6 is a characteristic graph of the intelligent agent performance boundary positioning in an embodiment of the present invention, Figure 7 It is a characteristic map of the intelligent agent network boundary positioning in an embodiment of the present invention. Specifically, Figure 4 The corresponding single air combat data agent performance boundary positioning feature map is the result of executing S1-S3 once, and Figure 6 It is the result after executing S1-S3 multiple times until the boundary feature coverage requirement is met. Figure 5 The corresponding single air combat data agent network boundary positioning feature map is the result of executing S1-S4 once, and Figure 7 It is the result after executing S1-S4 multiple times until the boundary feature coverage requirement is met.

[0071] Aiming at the problem that the intention is difficult to understand, the strategy is difficult to explain, and the decision is difficult to trust, this paper proposes a method for locating the input and output boundaries of game agents based on time series data inversion. Figure 2 It is understood that the present invention obtains the input situation data and output maneuver decision data of our agent in an actual or simulated environment by giving a fully trained deep reinforcement learning decision network, and simultaneously locates the performance boundary and network boundary of the agent through the proposed maneuver input-output boundary method, and continuously analyzes and iterates to improve the obtained input-output boundary of the agent, thereby realizing a new positioning scheme for the maneuver decision network boundary and performance boundary of the game agent.

[0072] This method uses feature-level unsupervised clustering interpretation methods and behavior-level real-time environmental situation awareness and discrimination, driven by time series data of actual or simulated environments, and inverts the boundary scene input of the agent from boundary features, thus achieving real-time boundary interpretability of the game agent at the level of maneuver strategy. Among them, based on high-fidelity game data, network boundary positioning realizes clustering analysis of features with poor generalization of decision networks, performance boundary positioning realizes the transformation of maneuver strategies made by decision networks into real-time situation judgments, and constructs a situation reversal discrimination method.

[0073] Specifically, the method of the present invention has the following beneficial effects: 1. Deeper decision-making explanation capability: This invention can go deep into the internal feature level of the agent's decision through the feature-level unsupervised clustering explanation method. By solving the first-order derivative and second-order derivative of the cumulative reward function, it can accurately locate the interval of the agent's situation reversal and reveal the internal logic of the agent's decision-making under different battlefield situations. This in-depth explanation capability goes beyond the explanation of the action decision level that the existing technology can usually only provide, making the decision-making process more transparent and easier for operators to understand and trust.

[0074] 2. Higher decision fidelity and credibility: Existing technologies are usually based on task-level boundary definition. Under the same environment setting, the simulation results of each task of the agent may be inconsistent, resulting in low fidelity and credibility of boundary positioning. The present invention provides a new method for positioning the performance boundary and network boundary of game agents through temporal maneuver input and output boundary positioning, which can more accurately reflect the performance of the agent in the actual game and provide a more comprehensive battlefield situation awareness, thereby improving the fidelity and credibility of decision-making.

[0075] 3. Stronger strategy generalization and adaptability: This invention uses boundary feature inversion to identify the boundary scene input of the agent, and can identify the battlefield situations in which the agent performs poorly, so as to optimize the strategy in a targeted manner. This capability makes the agent's strategy effective not only in the training environment, but also in a wider range of actual game environments, improving the generalization and adaptability of the strategy.

[0076] 4. More efficient real-time response capability: The present invention can identify the input and output boundaries of the intelligent agent in real time based on battlefield situation perception, which is crucial for highly dynamic gaming environments. In contrast, existing technologies usually require post-analysis and cannot provide real-time decision support.

[0077] It should be noted that, in the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0078] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for locating the input and output boundaries of a game agent based on time series data inversion, characterized in that: include: S1, constructing a game environment for the confrontation between enemy and friendly aircraft, obtaining the input situation data and output maneuver decision data of both sides after the game task is completed, as well as the input features and output features of the decision network based on deep reinforcement learning in our intelligent agent; wherein our aircraft is an intelligent agent; the input features of the decision network are obtained according to the input situation data of our intelligent agent, and the output maneuver decision data of our intelligent agent are obtained according to the output features of the decision network; S2, according to the input situation data, calculate the cumulative reward value of the maneuvering decision of our agent at each moment during the entire game task execution process; S3, according to the cumulative reward value of the maneuvering decision of our intelligent agent at each moment, determine the performance boundary moment set of our situation change; perform inversion positioning at each moment according to the performance boundary moment set to obtain our performance boundary input situation data and performance boundary output maneuvering decision data; S4, by performing unsupervised clustering on the output features, a network boundary time set corresponding to the boundary features of the decision network is obtained; inversion positioning is performed at each time according to the network boundary time set to obtain our network boundary input situation data and network boundary output maneuver decision data; S5, iteratively execute S1~S4, and calculate the boundary feature coverage rate based on the obtained performance boundary input situation data, performance boundary output maneuver decision data, network boundary input situation data and network boundary output maneuver decision data, and stop the iteration when the boundary feature coverage rate is greater than or equal to the preset feature coverage threshold to obtain our final intelligent agent input and output boundary.

2. The method for locating the input and output boundaries of a game agent based on time series data inversion according to claim 1, characterized in that: The input situation data includes the aircraft's altitude, longitude, latitude, heading angle, pitch angle, roll angle and speed; the output maneuver decision data includes the aircraft's aileron status, elevator status, rudder status, throttle status and whether to launch a missile.

3. The method for locating the input and output boundaries of a game agent based on time series data inversion according to claim 1, characterized in that: After obtaining the input situation data and output maneuver decision data of both the enemy and our party after the game task is completed, as well as the input features and output features of the decision network based on deep reinforcement learning in our party's intelligent agent, the method further includes: Determine the mapping relationship between the input features and the output features, as well as the corresponding relationship between the input features and the input situation data of our agent, and the corresponding relationship between the output maneuver decision data of our agent and the output features.

4. The method for locating the input and output boundaries of a game agent based on time series data inversion according to claim 1, characterized in that: According to the input situation data, the cumulative reward value of the maneuvering decision of our agent at each moment during the execution of the entire game task is calculated, including: According to the input situation data at each moment and the reward function preset by the decision network, the cumulative reward value of the mobile decision-making of our agent at each moment during the execution of the entire game task is calculated.

5. The method for locating the input and output boundaries of a game agent based on time series data inversion according to claim 4 is characterized in that: For a 1-on-1 game, the preset reward function is obtained by adding the distance reward, the posture advantage reward and the shootdown reward; wherein, The distance reward is expressed as: ； in, represents said distance reward; Indicates the position of our aircraft; Indicates the position of the enemy aircraft; the closer our aircraft is to the enemy aircraft, the greater the distance reward setting; The posture advantage bonus is expressed as: ； in, represents the said posture advantage bonus; Indicates the azimuth of our aircraft relative to the enemy aircraft; Indicates the azimuth of the enemy aircraft relative to our aircraft; when the sum of the azimuths of the enemy and our aircraft is 360°, it means that our agent is in a tail-chasing attack posture, and our angle posture advantage is the greatest at this time; conversely, when the sum of the azimuths of the enemy and our aircraft is 0°, it means that our agent is in a tail-chasing attack posture, and our angle posture advantage is the smallest at this time; The kill reward is It means that a positive reward is given if our aircraft wins, a negative reward is given if the enemy aircraft wins, and a zero reward is given if the game is a draw or the game is not over.

6. The method for locating the input and output boundaries of a game agent based on time series data inversion according to claim 1, characterized in that: According to the accumulated reward value of the maneuver decision of our agent at each moment, the performance boundary moment set of our situation change is determined, including: Determine a cumulative reward function according to the cumulative reward value of the mobile decision-making of the agent at each moment; Calculate the first-order derivative and the second-order derivative of the cumulative reward function, determine the reversal time point when the situation of the party changes from disadvantage to advantage according to the calculation results, obtain a first reversal time set, and determine the reversal time point when the situation of the party changes from advantage to disadvantage, obtain a second reversal time set; The first reversal moment set and the second reversal moment set are combined to obtain a performance boundary moment set.

7. The method for locating the input and output boundaries of a game agent by time series data inversion according to claim 1 or 6, characterized in that: After performing inversion positioning at each moment according to the performance boundary moment set to obtain our performance boundary input situation data and performance boundary output maneuver decision data, the method further includes: The performance boundary input situation data and the performance boundary output maneuver decision data are respectively extended at a given time according to the given performance boundary positioning visualization range to obtain the visualization input situation data and the visualization output situation data corresponding to the performance boundary; The visualized input situation data and the visualized output situation data corresponding to the performance boundary are reproduced through the game simulation platform used when constructing the game environment, and a performance boundary battlefield scene is generated.

8. The method for locating the input and output boundaries of a game agent by time series data inversion according to claim 1, characterized in that: By performing unsupervised clustering on the output features, a network boundary time set corresponding to the boundary features of the decision network is obtained, including: Using density clustering method on the output features to filter out a boundary feature set; A network boundary moment set is obtained according to the moments corresponding to the boundary features in the boundary feature set.

9. The method for locating the input and output boundaries of a game agent by time series data inversion according to claim 1 or 8, characterized in that: After performing inversion positioning at each moment according to the network boundary moment set to obtain our network boundary input situation data and network boundary output maneuver decision data, the method further includes: The network boundary input situation data and the network boundary output maneuver decision data are respectively extended at a given time according to the given network boundary positioning visualization range to obtain the visualization input situation data and visualization output situation data corresponding to the network boundary; The visualized input situation data and the visualized output situation data corresponding to the network boundary are reproduced through the game simulation platform used when constructing the game environment, and a network boundary battlefield scene is generated.

10. The method for locating the input and output boundaries of a game agent by time series data inversion according to claim 3, characterized in that: The step of calculating the boundary feature coverage rate based on the obtained performance boundary input situation data, performance boundary output maneuver decision data, network boundary input situation data, and network boundary output maneuver decision data includes: The performance boundary input situation data and the network boundary input situation data obtained in the current iteration are combined to obtain the combined input situation data of the current iteration; the performance boundary output situation data and the network boundary output situation data obtained in the current iteration are combined to obtain the combined output situation data of the current iteration; According to the determined correspondence between the input features and the input situation data of our agent, and the correspondence between the output maneuver decision data of our agent and the output features, the boundary input features corresponding to the combined input situation data of the current iteration and the boundary output features corresponding to the combined output situation data of the current iteration are calculated; The boundary feature coverage is obtained by calculating the intersection of the boundary input features and boundary output features obtained in the current iteration and the boundary input features and boundary output features accumulated before the current iteration; The iteration stops when the boundary feature coverage is greater than or equal to the preset feature coverage threshold, and our final agent input and output boundaries are obtained, including: Based on the boundary input features and boundary output features obtained at the corresponding iteration when the iteration is stopped, we get our final agent input and output boundaries.

Images