Prediction type system performance evaluation method and system based on time sequence differential learning

Abstract

The invention discloses a predictive system performance evaluation method and system based on time sequence differential learning, and the method comprises the steps: constructing a heterogeneous graph neural network model, and extracting the interaction relation and interaction characteristics between nodes and component systems of a system; according to different heterogeneous unmanned platform cooperative action schemes, empirical state transition track samples under all possible conditions are obtained through mass simulation deduction; training the efficiency estimation model by adopting a time sequence difference learning and experience playback mode; according to the embodiment of the invention, massive simulation deduction data is fully utilized, future efficiency prediction of a given action scheme in various possible real-time states is provided, and an important reference basis is provided for management personnel.

Description

Technical Field

[0001] The present invention relates to the fields of performance evaluation and artificial intelligence technology, and in particular to a predictive system performance evaluation method and system based on temporal difference learning. Background Art

[0002] Heterogeneous unmanned collaborative systems involve a large number of unit nodes, among which task allocation, area coverage, reconnaissance and search, coordinated strike and other links all affect the effectiveness of the cluster and are difficult to evaluate. It is urgent to select appropriate evaluation methods. At present, each performance evaluation method has its scope of use and limitations for different task environments, task types and platform types.

[0003] In essence, traditional methods artificially quantify the impact of each link through weighted summation, and obtain weights through expert scoring or self-learning. On the one hand, methods such as hierarchical analysis method and ADC rely on expert scoring to construct indicators, decompose and refine the comprehensive and complex factors that affect the system effectiveness step by step, extract the key capabilities that constitute the system effectiveness, and refine the constituent indicators of each level of capabilities, which are highly subjective. Even if key quantitative parameters such as task radius and detection range are added in the indicator selection process, and the subjective and objective aspects are integrated as much as possible, it is still essentially unable to explain the nonlinear interaction relationship in the complex system, and it is impossible to effectively evaluate the comprehensive system effectiveness brought by high-level decision-making solutions. On the other hand, learning-based methods such as artificial neural networks have good self-organization and self-learning properties, but are currently mainly used to replace local effectiveness analysis, using subdivided indicators as input parameters of artificial neural networks and true value effectiveness as output parameters. When the indicator system is complete and the influence relationship between each subdivided indicator is clear, this method has the ability to learn and adjust autonomously, but it requires manual specification of the true value effectiveness analysis model. Complex networks can be described by graphs as topological relationships between component systems, which are simple and easy to construct. However, they are still based on highly subjective manual analysis and calculation of key characteristic quantities, and cannot truly reflect the effectiveness of heterogeneous unmanned collaborative systems.

[0004] The effectiveness evaluation of heterogeneous unmanned systems can be considered as the degree to which the unmanned equipment cluster can effectively complete the task within the specified time through intelligent collaboration under certain task conditions. In essence, the effectiveness of the system mainly comes from the occurrence of key events. It is only necessary to define the reward and loss functions based on the intermediate and final results. The fundamental goal of the system collaborative task is to maximize the overall return, and the performance evaluation process is not affected by human factors. The present invention uses heterogeneous graph neural networks as a system modeling and parameterized feature extraction tool, establishes a system "big model", obtains massive data through simulation and deduction, and trains the evaluation model offline, so as to quickly predict the future cumulative reward expectations under any state and with a given action plan online as a high-performance indicator. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the present invention proposes a predictive system effectiveness evaluation method based on temporal difference learning, which, for a given action plan, judges and predicts the pros and cons of the future comprehensive effectiveness of the collaborative system under the decision plan.

[0007] Another object of the present invention is to propose a predictive system performance evaluation system based on temporal difference learning.

[0008] To achieve the above objectives, the present invention proposes a predictive system performance evaluation method based on temporal difference learning, comprising:

[0009] Based on heterogeneous graph neural network, the implicit interactive information in edge features and node feature vectors is extracted, and the single scalar output by the multi-layer fully connected neural network is used as the estimated value to build the performance estimation model;

[0010] The reward function is designed by querying the total change of the target value of all heterogeneous unit nodes in two states;

[0011] Based on different heterogeneous unmanned platform collaborative action plans, empirical state transfer trajectory samples of the effectiveness evaluation model are obtained through simulation and calculation of the reward function;

[0012] Using the method of temporal difference learning and experience replay, and using empirical state transition trajectory samples to train the performance estimation model;

[0013] The trained performance estimation model is used to perform real-time performance prediction to compare the expected performance of different action plans and select the optimal plan.

[0014] The predictive system performance evaluation method based on temporal difference learning in the embodiment of the present invention may also have the following additional technical features:

[0015] In one embodiment of the present invention, before extracting the implicit interactive information in the edge features and node feature vectors based on the heterogeneous graph neural network, the method further includes constructing a system heterogeneous graph, including:

[0016] Constructing heterogeneous node features in complex systems; wherein the node features include speed, location, survival status, remaining resources, and remaining endurance;

[0017] Construct heterogeneous edge feature vectors between interacting heterogeneous nodes; wherein the heterogeneous edge feature vectors include: information interaction connection edges between the same camp, detection connection edges between different camps, and attack connection edges between different camps.

[0018] In one embodiment of the present invention, a performance estimation model is constructed:

[0019]

[0020] In one embodiment of the present invention, based on the fundamental goal of system performance, the instant reward for completing a given task, the reward for damaging the opponent's camp unit, and the penalty for losing one's own camp unit are designed; the reward function is designed by querying the total change of the target value of all heterogeneous unit nodes in two states:

[0021]

[0022] In one embodiment of the present invention, based on different heterogeneous unmanned platform collaborative action plans, the empirical state transition trajectory samples of the effectiveness evaluation model are obtained by simulation and calculation of the reward function, including:

[0023] Based on action plan π a Add various random perturbations to obtain initialization parameter combinations covering all scenarios;

[0024] The initialization parameter combination is used to perform simulation experiments or semi-physical experiments to obtain a time series deduction state trajectory set τ~π a ;

[0025] Get the reward value r for each discrete step by calculating the reward function t , and pair the calculated reward value with the time series deduction state trajectory set to obtain the experience sample data tuple Provide training sample and test sample data sets for the performance evaluation model.

[0026] In one embodiment of the present invention, system performance is defined as:

[0027]

[0028] Get the Bellman equation:

[0029]

[0030] The performance evaluation model contains a set of trained parameters ω, which approximates the true performance, namely:

[0031]

[0032] The training performance evaluation model based on time series difference and experience replay has a loss function of:

[0033]

[0034] In one embodiment of the present invention, a trained performance estimation model is used to perform real-time performance prediction to compare the estimated performances of different action plans and select the optimal plan, including:

[0035] Using the trained performance evaluation model, output the global state at the current moment during the actual task process To get the action plan π a The expected value of future rewards

[0036] Set an action plan b , based on the trained performance evaluation model, we get the action plan π b The expected value of future rewards

[0037] Comparison of action plans a and action plan π b The estimated value of the expected future reward is used to select the optimal solution based on the comparison results.

[0038] To achieve the above object, the present invention further provides a predictive system performance evaluation system based on temporal difference learning, comprising:

[0039] The performance estimation model building module is used to extract the implicit interactive information in the edge features and node feature vectors based on the heterogeneous graph neural network, and use the single scalar output by the multi-layer fully connected neural network as the estimated value to build the performance estimation model;

[0040] A reward function building module is used to design a reward function by querying the change in the sum of target values ​​of all heterogeneous unit nodes in two states;

[0041] A model training data acquisition module is used to obtain empirical state transfer trajectory samples of the effectiveness evaluation model through simulation and calculation of the reward function based on different heterogeneous unmanned platform collaborative action plans;

[0042] The performance estimation model training module is used to train the performance estimation model using the temporal difference learning and experience replay method and the experience state transition trajectory samples;

[0043] The real-time performance prediction output module is used to use the trained performance estimation model to perform real-time performance prediction to compare the expected performance of different action plans and select the optimal plan.

[0044] The predictive system performance evaluation method and system based on temporal difference learning of the embodiments of the present invention, through the pre-trained performance evaluation model, only requires one calculation when deployed to compare the advantages and disadvantages of different action plans in specific scenarios, thereby greatly improving the evaluation efficiency during task execution.

[0045] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0047] Figure 1 is a flow chart of a predictive system performance evaluation method based on temporal difference learning according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of heterogeneous graph network connections according to an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of the connection relationship of the red camp according to an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a Markov decision and effectiveness evaluation chain according to an embodiment of the present invention;

[0051] Figure 5 4 is a structural diagram of a predictive system performance evaluation system based on temporal difference learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0053] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0054] The following describes a predictive system performance evaluation method and system based on temporal difference learning according to an embodiment of the present invention with reference to the accompanying drawings.

[0055] Figure 1 is a flow chart of a predictive system performance evaluation method based on temporal difference learning according to an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0056] S1, based on heterogeneous graph neural network, extracts the implicit interactive information in edge features and node feature vectors, and uses the single scalar output by the multi-layer fully connected neural network as the estimated value to build a performance estimation model.

[0057] First, the embodiment of the present invention automatically constructs a system heterogeneous graph based on a general parameter table. It is necessary to process various types of heterogeneous nodes and various types of heterogeneous edges between these nodes. v and edge set ε The total number of categories is greater than 2. Figure 2 As shown, the red camp includes two types of nodes, A and B, and the blue camp includes two types of nodes, H and G.

[0058] Specifically, building a system heterogeneous graph includes:

[0059] Construct key unit node features in complex systems. Heterogeneous node features include corresponding key feature quantities, such as speed, location, survival status, remaining resources, remaining endurance, etc.

[0060] Construct the feature vector of heterogeneous edges between heterogeneous nodes that have interactions. Figure 2 and 3 For example, the edge types and their characteristic quantities are: 1) information interaction connection edges between the same camps, including characteristic quantities such as communication bandwidth and anti-interference ability; 2) detection connection edges between different camps, including characteristic quantities such as detection probability and tracking accuracy; 3) attack connection edges between different camps, including characteristic quantities such as damage probability and attack intensity.

[0061] Furthermore, based on heterogeneous graph neural networks and attention mechanisms, the implicit interactive information in edge features and node feature vectors is extracted, and a single scalar is output as the estimated value of the performance evaluation model through a multi-layer fully connected neural network or any other type of aggregation network:

[0062]

[0063] It is understandable that this part can be replaced by any heterogeneous graph neural network model. The present invention takes the heterogeneous edge enhanced graph attention model (HEAT) as an embodiment. The weights and bias parameters in the neural network are randomly initialized and will gradually converge through training.

[0064] S2, designs the reward function by querying the total change of the target value of all heterogeneous unit nodes under two states.

[0065] It is understandable that each node has different value attributes due to the heterogeneity of camps or categories. As the fundamental goal of system effectiveness, the instant rewards obtained by completing the given tasks, the rewards generated by the damage to the opponent's camp units, and the penalties (negative rewards) generated by the loss of one's own camp units are designed.

[0066] By querying the total change in target value of all heterogeneous unit nodes in two states, the reward function is designed:

[0067]

[0068] The embodiment of the present invention starts from the original goal of system performance and does not require artificial weight allocation and indicator system design. The only step involving human experience is the definition of the value table to distinguish the values ​​of different units and different tasks. The goal of system performance is to obtain the highest expected benefit in the shortest time.

[0069] S3, based on different collaborative action plans of heterogeneous unmanned platforms, obtains empirical state transition trajectory samples of the effectiveness evaluation model through simulation and calculation of reward functions.

[0070] like Figure 4 As shown, the present invention regards the system evolution process as a Markov decision process consisting of multiple discrete time state slices, and the decision schemes involved include but are not limited to real-time single-step decision-making based on reinforcement learning or predefined multi-step planning based on rules and optimization. Specifically, this step includes:

[0071] For a given action plan π a , add various random perturbations to obtain initialization parameter combinations that can cover all possible scenarios.

[0072] Then, according to the initialization parameter combination, the time series deduction state trajectory set τ~π is obtained through simulation experiments or semi-physical experiments a .

[0073] Finally, the reward value r of each discrete step is obtained by querying the value table, that is, calculating the reward function t , matching the time series deduction state trajectory set to be divided into empirical sample data tuples (Where done is True at the end of the task, otherwise it is False), providing the performance evaluation model with training samples and test sample data sets containing general rules.

[0074] S4, using the method of temporal difference learning and experience replay, and using the empirical state transition trajectory samples to train the performance estimation model.

[0075] Specifically, the present invention defines system performance as:

[0076]

[0077] From the current state Start by taking action on heterogeneous unmanned platforms a The expected value of the future cumulative discounted reward that the system can obtain under the condition of . Due to the existence of the discount factor γ∈(0,1), it naturally implies the goal of completing the task in the shortest possible time.

[0078] The Bellman equation can be obtained from the above definition:

[0079]

[0080] The heterogeneous network performance evaluation model defined in the previous steps contains a set of trainable parameters ω , hoping to approach the true performance, that is:

[0081]

[0082] The training performance evaluation model based on time series difference and experience replay has the following loss function:

[0083]

[0084] The stochastic gradient descent algorithm is used to train the heterogeneous graph neural network model. According to the Bellman contraction mapping theory, after sufficient training with massive data, the heterogeneous network performance evaluation model can better approximate the system performance definition, select the network with the highest accuracy on the verification data set, and thus accurately estimate the system performance.

[0085] S5, using the trained performance estimation model to perform real-time performance prediction to compare the expected performance of different action plans and select the optimal plan.

[0086] Specifically, this embodiment may include the following steps:

[0087] Using the trained performance evaluation model, in the actual task process, the global state of the current moment can be output Adopt action plan π a The expected value of future rewards

[0088] Similarly, given another action plan π b , and get the action plan π b The expected value of future rewards

[0089] Comparison of action plans a and action plan π bThe estimated value of the expected future reward is used to select the best option based on the comparison results. In this way, the expected effectiveness of different action plans is compared and the best option is selected.

[0090] Furthermore, indicators such as resilience, reliability, and reconfigurability can be expanded.

[0091] According to the centralized supervision method of network switches in the embodiment of the present invention, as the system nodes change dynamically, the performance evaluation model can still extract global state features, learn the highly complex potential interaction relationships between nodes, make full use of the massive data of simulation and experimental testing, fully learn the impact of various factors on the final system performance in offline calculations, and encode this potential impact into the weight, bias, and attention mechanism parameters of the neural network. It is also possible to use a pre-trained performance evaluation model, which only requires one calculation during deployment, to compare the advantages and disadvantages of different action plans in a specific scenario, greatly improving the evaluation efficiency during task execution.

[0092] In order to implement the above embodiment, Figure 5 As shown, this embodiment also provides a predictive system performance evaluation system 10 based on temporal difference learning, including:

[0093] The performance estimation model building module 100 is used to extract the implicit interactive information in the edge features and node feature vectors based on the heterogeneous graph neural network, and use the single scalar output by the multi-layer fully connected neural network as an estimated value to build a performance estimation model;

[0094] A reward function construction module 200, for designing a reward function by querying the total change of target values ​​of all heterogeneous unit nodes under two states;

[0095] The model training data acquisition module 300 is used to obtain the empirical state transfer trajectory samples of the performance evaluation model through simulation and calculation of reward functions based on different heterogeneous unmanned platform collaborative action plans;

[0096] The performance estimation model training module 400 is used to train the performance estimation model using the temporal difference learning and experience playback method and using the experience state transition trajectory samples;

[0097] The real-time performance prediction output module 500 is used to perform real-time performance prediction using the trained performance estimation model to compare the estimated performances of different action plans and select the optimal plan.

[0098] Furthermore, before the performance estimation model building module 100, a system heterogeneous graph building module is also included, which is used to:

[0099] Constructing heterogeneous node features in complex systems; wherein the node features include speed, location, survival status, remaining resources, and remaining endurance;

[0100] Construct heterogeneous edge feature vectors between interacting heterogeneous nodes; wherein the heterogeneous edge feature vectors include: information interaction connection edges between the same camp, detection connection edges between different camps, and attack connection edges between different camps.

[0101] Furthermore, a performance estimation model is constructed:

[0102]

[0103] According to the centralized monitoring system for network switches of the embodiment of the present invention, as the system nodes change dynamically, the performance evaluation model can still extract global state features, learn the highly complex potential interaction relationships between nodes, make full use of the massive data of simulation and experimental testing, fully learn the impact of various factors on the final system performance in offline calculations, and encode this potential impact into the weight, bias, and attention mechanism parameters of the neural network. It is also possible to use the pre-trained performance evaluation model, which only requires one calculation during deployment, to compare the advantages and disadvantages of different action plans in specific scenarios, greatly improving the evaluation efficiency during task execution.

[0104] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0105] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A predictive system performance evaluation method based on temporal difference learning, characterized in that: include: Based on heterogeneous graph neural network, the implicit interactive information in edge features and node feature vectors is extracted, and the single scalar output by the multi-layer fully connected neural network is used as the estimated value to build the performance estimation model; The reward function is designed by querying the total change of the target value of all heterogeneous unit nodes in two states; Based on different heterogeneous unmanned platform collaborative action plans, empirical state transfer trajectory samples of the effectiveness evaluation model are obtained through simulation and calculation of the reward function; Using the method of temporal difference learning and experience replay, and using empirical state transition trajectory samples to train the performance estimation model; The trained performance estimation model is used to perform real-time performance prediction to compare the expected performance of different action plans and select the optimal plan.

2. The method according to claim 1, characterized in that: Before extracting the implicit interactive information in the edge features and node feature vectors based on the heterogeneous graph neural network, the method further includes constructing a system heterogeneous graph, including: Constructing heterogeneous node features in complex systems; wherein the node features include speed, location, survival status, remaining resources, and remaining endurance; Construct heterogeneous edge feature vectors between interacting heterogeneous nodes; wherein the heterogeneous edge feature vectors include: information interaction connection edges between the same camp, detection connection edges between different camps, and attack connection edges between different camps.

3. The method according to claim 1, characterized in that Constructing the performance estimation model:

4. The method according to claim 3, characterized in that Based on the fundamental goal of system effectiveness, we design the instant rewards for completing the given tasks, the rewards for damaging the opponent's units, and the penalties for losing our own units. We design the reward function by querying the total change in the target value of all heterogeneous unit nodes in two states:

5. The method according to claim 1, characterized in that Based on different heterogeneous unmanned platform collaborative action plans, the empirical state transfer trajectory samples of the effectiveness evaluation model are obtained through simulation and calculation of the reward function, including: Based on action plan π a Add various random perturbations to obtain initialization parameter combinations covering all scenarios; The initialization parameter combination is used to perform simulation experiments or semi-physical experiments to obtain a time series deduction state trajectory set τ~π a ; Get the reward value r for each discrete step by calculating the reward function t , and pair the calculated reward value with the time series deduction state trajectory set to obtain the experience sample data tuple Provide training sample and test sample data sets for the performance evaluation model.

6. The method according to claim 4, characterized in that System performance is defined as: Obtain the Bellman equation: The performance evaluation model contains a set of trained parameters ω, which approximates the true performance, namely: The training performance evaluation model based on time series difference and experience replay has a loss function of:

7. The method according to claim 5, characterized in that Use the trained performance estimation model to perform real-time performance prediction to compare the expected performance of different action plans and select the optimal plan, including: Using the trained performance evaluation model, output the global state at the current moment during the actual task process To get the action plan π a The expected value of future rewards Set an action plan b , based on the trained performance evaluation model, we get the action plan π b The expected value of future rewards Comparison of action plans a and action plan π b The estimated value of the expected future reward is used to select the optimal solution based on the comparison results.

8. A predictive system performance evaluation system based on temporal difference learning, characterized in that: include: The performance estimation model building module is used to extract the implicit interaction information in the edge features and node feature vectors based on the heterogeneous graph neural network, and use the single scalar output by the multi-layer fully connected neural network as the estimated value to build the performance estimation model; A reward function building module is used to design a reward function by querying the change in the sum of target values ​​of all heterogeneous unit nodes in two states; A model training data acquisition module is used to obtain empirical state transfer trajectory samples of the effectiveness evaluation model through simulation and calculation of the reward function based on different heterogeneous unmanned platform collaborative action plans; The performance estimation model training module is used to train the performance estimation model using the temporal difference learning and experience replay method and the experience state transition trajectory samples; The real-time performance prediction output module is used to use the trained performance estimation model to perform real-time performance prediction to compare the expected performance of different action plans and select the optimal plan.

9. The system according to claim 8, characterized in that Before the performance estimation model building module, a system heterogeneous graph building module is also included for: Constructing heterogeneous node features in complex systems; wherein the node features include speed, location, survival status, remaining resources, and remaining endurance; Construct heterogeneous edge feature vectors between interacting heterogeneous nodes; wherein the heterogeneous edge feature vectors include: information interaction connection edges between the same camp, detection connection edges between different camps, and attack connection edges between different camps.

10. The system according to claim 8, characterized in that Constructing the performance estimation model:

Images