A prediction type system performance evaluation method and system based on time series difference learning

Abstract

This invention discloses a predictive system performance evaluation method and system based on temporal difference learning. The method includes constructing a heterogeneous graph neural network model to extract the interaction relationships and features between nodes and component systems of the system; obtaining empirical state transition trajectory samples under all possible conditions through massive simulations for different heterogeneous unmanned platform collaborative action schemes; and training the performance estimation model using temporal difference learning and experience replay. The embodiments of this invention fully utilize massive simulation data to provide future performance predictions for a given action scheme under various possible real-time conditions, providing important reference for managers.

Description

Technical Field

[0001] This invention relates to the fields of performance evaluation and artificial intelligence, and in particular to a predictive system performance evaluation method and system based on temporal difference learning. Background Technology

[0002] Heterogeneous unmanned collaborative systems involve a massive number of unit nodes. Multiple aspects, including task allocation, area coverage, reconnaissance and search, and coordinated strikes, all influence the cluster's effectiveness and are difficult to evaluate, necessitating the selection of appropriate assessment methods. Currently, each effectiveness assessment method has its own scope and limitations depending on the mission environment, mission type, and platform type.

[0003] Traditional methods essentially quantify the impact of each stage artificially through weighted summation, obtaining weights through expert scoring or self-learning. On one hand, methods like the Analytic Hierarchy Process (AHP) and Algorithm-Conversion (ADC) rely on expert scoring to construct indicators, progressively decomposing and refining the comprehensive and complex factors affecting system effectiveness, extracting key capabilities that constitute system effectiveness, and detailing the constituent indicators of each level of capability; however, this approach is highly subjective. Even when key quantitative parameters such as task radius and detection range are incorporated into the indicator selection process to try to integrate subjective and objective factors, it still cannot fundamentally explain the nonlinear interactions in complex systems and cannot effectively assess the comprehensive system effectiveness brought about by high-level decision-making schemes. On the other hand, learning-based methods, such as artificial neural networks, have good self-organization and self-learning capabilities, but are currently mainly used to replace local effectiveness analysis, using subdivided indicators as input parameters and true value effectiveness as output parameters. When the indicator system is complete and the influence relationships of each subdivided indicator are clear, this method has the ability to learn and adjust autonomously, but it still requires the manual specification of the true value effectiveness analysis model. Complex networks can describe the topological relationships between component systems through graphs, and are easy to construct. However, they are currently still based on subjective manual analysis and calculation of key features, and cannot truly reflect the effectiveness of heterogeneous unmanned collaborative systems.

[0004] Performance evaluation of heterogeneous unmanned systems can be considered as the degree to which an unmanned equipment cluster effectively completes a task within a specified time through intelligent collaboration under certain mission conditions. Essentially, the system's performance primarily stems from the occurrence of key events. It only requires defining reward and loss functions based on intermediate and final results. The fundamental goal of the system's collaborative tasks is to maximize overall reward, and the performance evaluation process is unaffected by human factors. This invention uses a heterogeneous graph neural network as a system modeling and parameterized feature extraction tool to establish a large-scale system model. Through simulation and deduction, massive amounts of data are acquired, and the evaluation model is trained offline. This allows for rapid online prediction of the expected cumulative reward under any given action plan in any state, serving as a high-performance indicator. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] To this end, this invention proposes a predictive system performance evaluation method based on temporal difference learning, which judges and predicts the future comprehensive performance of the collaborative system under a given action plan.

[0007] Another objective of this invention is to propose a predictive system performance evaluation system based on temporal difference learning.

[0008] To achieve the above objectives, this invention proposes a predictive system performance evaluation method based on temporal difference learning, comprising:

[0009] Based on heterogeneous graph neural networks, the interaction information implicit in edge features and node feature vectors is extracted, and the single scalar output by the multilayer fully connected neural network is used as the estimate to construct an efficiency estimation model.

[0010] The reward function is designed by querying the change in the total target value of all heterogeneous unit nodes under two states.

[0011] Based on different collaborative action schemes of heterogeneous unmanned platforms, empirical state transition trajectory samples of the effectiveness evaluation model are obtained through simulation and calculation of the reward function.

[0012] We utilize temporal difference learning and experience replay, and train the performance estimation model using empirical state transition trajectory samples.

[0013] A trained effectiveness estimation model is used to predict real-time effectiveness in order to compare the expected effectiveness of different action plans and select the optimal plan.

[0014] The predictive system performance evaluation method based on temporal difference learning in this invention may also have the following additional technical features:

[0015] In one embodiment of the present invention, before extracting the interaction information implicit in the edge features and node feature vectors based on the heterogeneous graph neural network, the method further includes constructing a system heterogeneous graph, including:

[0016] Constructing heterogeneous node characteristics in a complex system; wherein, the node characteristics include multiple factors such as speed, location, survival status, remaining resources, and remaining endurance;

[0017] Construct heterogeneous edge feature vectors between heterogeneous nodes that interact with each other; wherein, the heterogeneous edge feature vectors include: information interaction connection edges between the same faction, detection connection edges between different factions, and attack connection edges between different factions.

[0018] In one embodiment of the present invention, an effectiveness estimation model is constructed:

[0019]

[0020] In one embodiment of the present invention, based on the fundamental goal of system performance, the invention designs immediate rewards for completing a predetermined task, rewards for damaging enemy units, and penalties for losing friendly units; by querying the change in the total target value of all heterogeneous unit nodes in two states, a reward function is designed:

[0021]

[0022] In one embodiment of the present invention, based on different heterogeneous unmanned platform cooperative action schemes, empirical state transition trajectory samples of the effectiveness evaluation model are obtained through simulation and calculation of the reward function, including:

[0023] Based on action plan π a Add various random perturbations to obtain initialization parameter combinations covering all scenarios;

[0024] Using the aforementioned initialization parameter combination, simulation experiments or hardware-in-the-loop tests are conducted to obtain the time-series extrapolation state trajectory set τ~π. a ;

[0025] The reward value r for each step away from the walk is obtained by calculating the reward function. t The calculated reward value is then paired with the time-series deduced state trajectory set to obtain empirical sample data tuples. This provides training and testing datasets for the performance evaluation model.

[0026] In one embodiment of the present invention, the system performance is defined as:

[0027]

[0028] Obtaining the Bellman equation:

[0029]

[0030] The performance evaluation model contains a set of trained parameters ω that approximates the true performance, i.e.:

[0031]

[0032] The training performance evaluation model based on temporal difference and experience replay has the following loss function:

[0033]

[0034] In one embodiment of the present invention, a trained performance estimation model is used to perform real-time performance prediction to compare the predicted performance of different action plans and select the optimal plan, including:

[0035] Using a trained performance evaluation model, output the global state at the current moment during the actual task. To obtain action plan π a The estimated value of future reward expectations

[0036] Set an action plan π b Based on the trained performance evaluation model, the action plan π is obtained. b The estimated value of future reward expectations

[0037] Comparison of action plan π a and action plan π b The estimated value of the expected future reward is used to select the optimal solution based on the comparison results.

[0038] To achieve the above objectives, another aspect of the present invention proposes a predictive system performance evaluation system based on temporal difference learning, comprising:

[0039] The efficiency estimation model construction module is used to extract the implicit interaction information in the edge features and node feature vectors based on heterogeneous graph neural networks, and to construct the efficiency estimation model by using the single scalar output by the multilayer fully connected neural network as the estimate.

[0040] The reward function construction module is used to design a reward function by querying the change in the total target value of all heterogeneous unit nodes in two states.

[0041] The model training data acquisition module is used to obtain empirical state transition trajectory samples of the effectiveness evaluation model through simulation and calculation of the reward function based on different collaborative action schemes of heterogeneous unmanned platforms.

[0042] The performance estimation model training module is used to train the performance estimation model using temporal difference learning and experience playback, and by using empirical state transition trajectory samples.

[0043] The real-time performance prediction output module is used to perform real-time performance prediction using a trained performance estimation model to compare the expected performance of different action plans and select the optimal plan.

[0044] The predictive system performance evaluation method and system based on temporal difference learning in this invention can compare the advantages and disadvantages of different action plans in a specific scenario by using a pre-trained performance evaluation model. This greatly improves the evaluation efficiency during task execution.

[0045] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0046] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 This is a flowchart of a predictive system performance evaluation method based on temporal difference learning according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of heterogeneous graph network connections according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the connection relationship of the red team according to an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of a Markov decision-making and performance evaluation chain according to an embodiment of the present invention;

[0051] Figure 5 This is a structural diagram of a predictive system performance evaluation system based on temporal difference learning according to an embodiment of the present invention. Detailed Implementation

[0052] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0053] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0054] The following describes, with reference to the accompanying drawings, a predictive system performance evaluation method and system based on temporal difference learning according to embodiments of the present invention.

[0055] Figure 1 This is a flowchart of a predictive system performance evaluation method based on temporal difference learning according to an embodiment of the present invention, as follows: Figure 1 As shown, the method includes:

[0056] S1 extracts the implicit interaction information in the edge features and node feature vectors based on heterogeneous graph neural networks, and uses the single scalar output by the multi-layer fully connected neural network as the estimate to construct an efficiency estimation model.

[0057] First, embodiments of the present invention automatically construct system heterogeneity diagrams based on a general parameter table. It is necessary to process various types of heterogeneous nodes and various types of heterogeneous edges between these nodes, and the node set. v Sum of edges ε The total number of categories is greater than 2. For example... Figure 2 As shown, the red team consists of two types of nodes, A and B, while the blue team consists of two types of nodes, H and G.

[0058] Specifically, constructing the system heterogeneity diagram includes:

[0059] Construct key unit node characteristics in complex systems. Heterogeneous node characteristics include corresponding key feature quantities, such as speed, location, survival status, remaining resources, and remaining endurance.

[0060] Construct feature vectors for heterogeneous edges between interacting heterogeneous nodes, in order to Figure 2 and 3 For example, the types of edges and their features are: 1) Information interaction connection edges between the same camp, including features such as communication bandwidth and anti-interference capability; 2) Detection connection edges between different camps, including features such as detection probability and tracking accuracy; 3) Attack connection edges between different camps, including features such as damage probability and attack intensity.

[0061] Furthermore, based on heterogeneous graph neural networks and attention mechanisms, the implicit interaction information in edge features and node feature vectors is extracted, and a single scalar is output as the performance evaluation model estimate through a multi-layer fully connected neural network or any other type of aggregation network.

[0062]

[0063] It is understood that this part can be replaced with any heterogeneous graph neural network model; this invention uses the Heterogeneous Edge Augmented Graph Attention (HEAT) model as an example. The weights and bias parameters in the neural network are randomly initialized and will gradually converge through training.

[0064] S2, design the reward function by querying the change in the total target value of all heterogeneous unit nodes in the two states.

[0065] Understandably, each node has different value attributes due to the heterogeneity of its faction or category. As the fundamental goal of system effectiveness, the design includes immediate rewards for completing predetermined tasks, rewards for destroying enemy faction units, and penalties (negative rewards) for losing friendly faction units.

[0066] By querying the change in the total target value of all heterogeneous unit nodes under two states, a reward function is designed:

[0067]

[0068] This invention starts from the original goal of system effectiveness, without the need for manual weight allocation and indicator system design. The only step involving human experience is the definition of the value table to distinguish the value of different units and different tasks. The goal of system effectiveness is to obtain the highest expected benefit in the least amount of time.

[0069] S3, based on different collaborative action schemes of heterogeneous unmanned platforms, obtains empirical state transition trajectory samples of the effectiveness evaluation model through simulation and calculation of reward function.

[0070] like Figure 4 As shown, this invention views the system evolution process as a Markov decision process composed of multiple discrete-time state slices. The decision-making schemes involved include, but are not limited to, real-time single-step decision-making based on reinforcement learning or predefined multi-step planning based on rules and optimization. Specifically, the steps include:

[0071] For a given action plan π a Add various random perturbations to obtain initialization parameter combinations that can cover all possible scenarios.

[0072] Then, based on the initial parameter combination, the time-series deduced state trajectory set τ~π is obtained through simulation experiments or semi-physical experiments. a .

[0073] Finally, the reward value r for each walk is obtained by querying the value table and calculating the reward function. t The matching time-series deduction state trajectory set is divided into empirical sample data tuples. (Where done is True at the end of the task, otherwise it is False), providing a training and test dataset containing general patterns for the performance evaluation model.

[0074] S4 utilizes temporal difference learning and experience replay, and trains the performance estimation model using empirical state transition trajectory samples.

[0075] Specifically, the present invention defines system performance as:

[0076]

[0077] That is, from the current state Departure: Action Plan π on a Heterogeneous Unmanned Platform a Under the given conditions, this represents the expected value of the cumulative future discount reward that the system can obtain. The existence of the discount factor γ∈(0,1) naturally implies the goal of minimizing task completion time.

[0078] The Bellman equation can be obtained from the above definition:

[0079]

[0080] The heterogeneous network performance evaluation model defined in the preceding steps includes a set of trainable parameters. ω The goal is to approximate the true performance, that is:

[0081]

[0082] The training performance evaluation model based on temporal difference and experience replay has the following loss function:

[0083]

[0084] The heterogeneous graph neural network model is trained using the stochastic gradient descent algorithm. Based on the Bellman contraction map theory, after sufficient training on massive amounts of data, the heterogeneous network performance evaluation model can approximate the system performance definition well. By selecting the network with the highest accuracy on the validation dataset, the system performance can be accurately estimated.

[0085] S5 utilizes a trained effectiveness estimation model to perform real-time effectiveness prediction in order to compare the expected effectiveness of different action plans and select the optimal plan.

[0086] Specifically, this embodiment may include the following steps:

[0087] Using a trained performance evaluation model, in the course of a real-world task, it can output the global state at the current moment. Adopt action plan π a The estimated value of future reward expectations

[0088] Similarly, given another course of action π b , thus obtaining action plan π b The estimated value of future reward expectations

[0089] Comparison of action plan π a and action plan π bThe estimated future reward is used to select the optimal solution based on the comparison results. This allows for a comparison of the projected effectiveness of different action plans and the selection of the optimal solution.

[0090] Furthermore, it can be extended to obtain indicators such as resilience, reliability, and reconfigurability.

[0091] According to the centralized monitoring method for network switches of this invention, as the system nodes dynamically change, the performance evaluation model can still extract global state features, learn highly complex potential interaction relationships between nodes, and fully utilize massive amounts of data from simulations and experiments. In offline computation, it fully learns the impact of various factors on the final system performance and encodes this potential impact into the weights, biases, and attention mechanism parameters of the neural network. Furthermore, through a pre-trained performance evaluation model, only one computation is needed during deployment to compare the merits of different action plans in specific scenarios, greatly improving the evaluation efficiency during task execution.

[0092] To achieve the above embodiments, such as Figure 5 As shown, this embodiment also provides a predictive system performance evaluation system 10 based on temporal difference learning, including:

[0093] The efficiency estimation model construction module 100 is used to extract the implicit interaction information in the edge features and node feature vectors based on the heterogeneous graph neural network, and to use the single scalar output by the multilayer fully connected neural network as the estimate to construct the efficiency estimation model.

[0094] The reward function construction module 200 is used to design a reward function by querying the change in the total target value of all heterogeneous unit nodes in two states.

[0095] The model training data acquisition module 300 is used to obtain empirical state transition trajectory samples of the effectiveness evaluation model through simulation and calculation of reward function based on different collaborative action schemes of heterogeneous unmanned platforms.

[0096] The performance estimation model training module 400 is used to train the performance estimation model using temporal difference learning and experience playback, and using empirical state transition trajectory samples.

[0097] The real-time performance prediction output module 500 is used to perform real-time performance prediction using a trained performance estimation model to compare the expected performance of different action plans and select the optimal plan.

[0098] Furthermore, prior to the efficiency estimation model construction module 100, a system heterogeneity diagram construction module is also included, used for:

[0099] Constructing heterogeneous node characteristics in a complex system; wherein, the node characteristics include multiple factors such as speed, location, survival status, remaining resources, and remaining endurance;

[0100] Construct heterogeneous edge feature vectors between heterogeneous nodes that interact with each other; wherein, the heterogeneous edge feature vectors include: information interaction connection edges between the same faction, detection connection edges between different factions, and attack connection edges between different factions.

[0101] Furthermore, an efficacy estimation model is constructed:

[0102]

[0103] According to embodiments of the present invention, the centralized monitoring system for network switches, despite the dynamic changes in system nodes, can still extract global state features through its performance evaluation model. It learns the highly complex potential interactions between nodes, fully utilizes massive amounts of data from simulations and experiments, and comprehensively learns the impact of various factors on the final system performance in offline computation. This potential impact is then encoded into the weights, biases, and attention mechanism parameters of the neural network. Furthermore, through a pre-trained performance evaluation model, only one computation is needed during deployment to compare the merits of different action plans in specific scenarios, significantly improving the evaluation efficiency during task execution.

[0104] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0105] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A predictive system performance evaluation method based on temporal difference learning, characterized in that, include: Based on heterogeneous graph neural networks, the interaction information implicit in edge features and node feature vectors is extracted, and the single scalar output by the multilayer fully connected neural network is used as the estimate to construct an efficiency estimation model. The reward function is designed by querying the change in the total target value of all heterogeneous unit nodes under two states. Based on different collaborative action schemes of heterogeneous unmanned platforms, empirical state transition trajectory samples of the effectiveness evaluation model are obtained through simulation and calculation of the reward function. We utilize temporal difference learning and experience replay, and train the performance estimation model using empirical state transition trajectory samples. The trained effectiveness estimation model is used to predict real-time effectiveness in order to compare the expected effectiveness of different action plans and select the optimal plan. Before extracting the implicit interaction information in the edge features and node feature vectors based on the heterogeneous graph neural network, the method further includes constructing a system heterogeneous graph, including: Constructing heterogeneous node characteristics in a complex system; wherein, the node characteristics include multiple factors such as speed, location, survival status, remaining resources, and remaining endurance; Construct heterogeneous edge feature vectors between heterogeneous nodes that interact with each other; wherein, the heterogeneous edge feature vectors include: information interaction connection edges between the same faction, detection connection edges between different factions, and attack connection edges between different factions; Constructing an efficiency estimation model: ； Based on the fundamental goal of system effectiveness, the design includes immediate rewards for completing a given task, rewards for destroying enemy units, and penalties for losing friendly units. A reward function is designed by querying the change in the total target value of all heterogeneous unit nodes under two states. 。 2. The method according to claim 1, characterized in that, Based on different collaborative action schemes of heterogeneous unmanned platforms, empirical state transition trajectory samples of the effectiveness evaluation model are obtained through simulation and calculation of the reward function, including: Based on action plan Add various random perturbations to obtain initialization parameter combinations covering all scenarios; The time-series state trajectory set is obtained by conducting simulation experiments or semi-physical experiments using the aforementioned initialization parameter combination. ; The reward value for each walk is obtained by calculating the reward function. The calculated reward value is then paired with the time-series deduced state trajectory set to obtain empirical sample data tuples. This is to provide training and testing datasets for the performance evaluation model.

3. The method according to claim 1, characterized in that, System effectiveness is defined as: Obtaining the Bellman equation: The performance evaluation model includes a set of trained parameters. Approaching true performance, that is: The training performance evaluation model based on temporal difference and experience replay has the following loss function: 。 4. The method according to claim 2, characterized in that, Real-time performance prediction is performed using a trained performance estimation model to compare the projected performance of different action plans and select the optimal plan, including: Using a trained performance evaluation model, output the global state at the current moment during the actual task. In order to obtain an action plan The estimated value of future reward expectations ; Set an action plan Action plans are derived based on the trained performance evaluation model. The estimated value of future reward expectations ; Comparison of action plans and action plan The estimated value of the expected future reward is used to select the optimal solution based on the comparison results.

5. A predictive system performance evaluation system based on temporal difference learning using the method described in claim 1, characterized in that, include: The efficiency estimation model construction module is used to extract the implicit interaction information in the edge features and node feature vectors based on heterogeneous graph neural networks, and to construct the efficiency estimation model by using the single scalar output by the multilayer fully connected neural network as the estimate. The reward function construction module is used to design a reward function by querying the change in the total target value of all heterogeneous unit nodes in two states. The model training data acquisition module is used to obtain empirical state transition trajectory samples of the effectiveness evaluation model through simulation and calculation of the reward function based on different collaborative action schemes of heterogeneous unmanned platforms. The performance estimation model training module is used to train the performance estimation model using temporal difference learning and experience playback, and by using empirical state transition trajectory samples. The real-time performance prediction output module is used to perform real-time performance prediction using a trained performance estimation model to compare the expected performance of different action plans and select the optimal plan.

6. The system according to claim 5, characterized in that, Before the performance estimation model building module, a system heterogeneity diagram building module is also included, which is used for: Constructing heterogeneous node characteristics in a complex system; wherein, the node characteristics include multiple factors such as speed, location, survival status, remaining resources, and remaining endurance; Construct heterogeneous edge feature vectors between heterogeneous nodes that interact with each other; wherein, the heterogeneous edge feature vectors include: information interaction connection edges between the same faction, detection connection edges between different factions, and attack connection edges between different factions.

7. The system according to claim 5, characterized in that, Constructing an efficiency estimation model: 。

Images