An intelligent unmanned system decision reliability evaluation and optimization method based on reinforcement learning

Abstract

The application provides an intelligent unmanned system decision reliability evaluation and optimization method based on reinforcement learning, and belongs to the technical field of intelligent unmanned systems. The application adopts a multi-layer operation network model based on a coloring graph theory to represent individual autonomy and functional diversity, overcomes the limitation of a single-layer network, and provides a basis for decision analysis. An intelligent unmanned system distributed decision model based on an enhanced actor-critic architecture is adopted to improve the strategy performance and learning stability of individual decision. An intelligent unmanned system decision reliability evaluation model oriented to an operation loop is adopted. The model integrates the proposed reliability indexes into the individual learning process to improve the effectiveness of evaluation. The model successfully quantifies the influence of individual decision on system reliability. A decision reliability optimization model based on a cooperative multi-agent deep deterministic policy gradient algorithm is adopted. The proposed cooperative reward is integrated into the policy gradient optimization process to optimize the decision reliability.

Description

Technical Field

[0001] This invention belongs to the field of intelligent unmanned systems technology, and relates to distributed collaborative decision-making reliability analysis technology, specifically to a method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning. Background Technology

[0002] Intelligent unmanned systems are complex, task-oriented systems comprised of intelligent agents (such as drones, unmanned vehicles, and unmanned surface vessels) that collaborate through decision-making and operational networks. Distributed collaborative decision-making, acting as the brain of the intelligent unmanned system, has a crucial impact on both the system's internal operational stability and external task performance. Unreliable decisions can lead to cascading failures, task conflicts, and resource waste. Therefore, although intelligent unmanned systems have been applied in many critical mission scenarios (e.g., cooperative coverage, task allocation, control optimization, and game theory strategies), these scenarios place extremely high demands on the reliability of their decision-making.

[0003] Decision reliability refers to the ability of an intelligent unmanned system to complete a designated task through distributed collaborative decision-making under disturbances. This concept involves the reliability of decisions and their results. Decision-making is a distributed collaboration among intelligent agents through local interactions, directly determining the execution efficiency of the task outcome. However, the individual autonomy, functional diversity, and emergent behavior of individuals in intelligent unmanned systems increase the uncertainty and unreliability of decision-making. For example, in collaborative reconnaissance missions, rapid autonomous decision-making based on local information, while improving agility, may also lead to system-wide miscoordination, induce task conflicts, reduce overall performance, and waste resources. Therefore, the analysis of decision reliability is crucial in application scenarios, and the impact of individual decisions on system reliability must be quantified to ensure stable operation and effective decision-making.

[0004] Despite significant progress in current research, some limitations remain: for example, current network models struggle to accurately depict the operational logic centered on decision-making, and existing decision-making models fail to maintain the stability of individual decisions under the constraint of global task performance. Furthermore, research on decision reliability still cannot effectively quantify the specific impact of individual decisions on the reliability of the entire intelligent unmanned system. Summary of the Invention

[0005] In view of the defects and shortcomings of the existing technology, the purpose of this invention is to provide a method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning. This method addresses the shortcomings of existing intelligent unmanned system network models in terms of decision-oriented operational logic, the difficulty in maintaining the stability of individual decisions with global task effectiveness, and the inability of decision reliability research to quantify the impact of individual decisions on system reliability.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning, specifically including the following steps: Step 1: Input the agent parameters and task parameters and initialize them. Then visualize the initialized parameters to obtain a multi-layer running network model based on coloring graph theory.

[0007] Specifically, in step one, the agent parameters include: agent type, number of agents, agent coordinates, agent resources, and agent cost; the task parameters include: number of tasks and task coordinates.

[0008] Specifically, step one includes: abstracting each intelligent agent with corresponding functions into a node; classifying intelligent agents into perceptual intelligent agents, decision-making intelligent agents, and action intelligent agents according to their functions; abstracting each task to be executed into a node; having interactive relationships between nodes, using lines to represent the interactive relationships between nodes; defining perceptual intelligent agents, decision-making intelligent agents, and action intelligent agents as different node colors, defining tasks as one color, and defining the lines between nodes and the lines between nodes and tasks as different edge colors.

[0009] Specifically, in step one, Matplotlib 3.7.1 is used to visualize the initialized parameters.

[0010] Step 2: Define and input the state space, observation space, action space, and state transition rules to obtain an intelligent unmanned system environment.

[0011] Specifically, in step two, the state space is defined using a coordinate matrix, an adjacency matrix, a resource vector, and a cost matrix.

[0012] Specifically, in step two, the observation space includes: a perception observation space, a decision observation space, and an action observation space; the perception observation space is defined using the coordinate vector, adjacency matrix, resource vector, and cost vector of the perception agent facing the decision agent; the decision observation space is defined using the coordinate vector, adjacency matrix, resource vector, and cost vector of the decision agent facing the decision agent; and the action observation space is defined using the coordinate vector, adjacency matrix, resource vector, and cost vector of the action agent facing the decision agent.

[0013] Specifically, in step two, the action space includes: the perception action space, the decision action space, and the action space; the perception action space is defined by the set of actions of all perceptual agents toward each decision agent; the decision action space is defined by the set of actions of all decision agents toward each decision agent; and the action action space is defined by the set of actions of all action agents toward each decision agent.

[0014] Specifically, in step two, the state transition rules include: a single agent obtains a set of actions for a certain type of agent or task based on the current observation, updates the edges between agents and the edges between agents and tasks according to the set of actions (i.e., updates the adjacency matrix). Since the adjacency matrix is ​​a state space element, the global cooperative state of the intelligent unmanned system environment is also updated (i.e., transitions from the current state to the state at the next moment).

[0015] Step 3: Build an agent decision-making model using an enhanced actor-critic architecture and obtain sample information: Step 3.1: Output the agent's actions using the actor network: Sequentially encode, extract, and compress the values ​​observed by a single agent to finally output the actions.

[0016] Specifically, in step 3.1, feature encoding includes: inputting the observations of a single agent into the feature encoding layer of the actor network, and introducing weight vectors, bias vectors, and mask vectors, and then... The activation function performs non-linear processing to obtain the encoded features.

[0017] Specifically, in step 3.1, feature extraction includes: inputting the encoded features of a single agent into the feature extraction layer of the actor network, and then introducing weight vectors, bias vectors, and mask vectors, and then... The activation function performs non-linear processing to obtain the extracted features.

[0018] Specifically, in step 3.1, feature compression includes: inputting the extracted features of a single agent into the feature compression layer of the actor network, and then introducing weight vectors, bias vectors, and mask vectors, and then... The activation function performs non-linear processing to obtain compressed features.

[0019] Specifically, in step 3.1, the output action includes: inputting the compressed features of a single agent into the action output layer of the feature compression layer of the actor network, and introducing weight vectors, bias vectors, and mask vectors, and then... The activation function performs non-linear processing to obtain the action output of a single agent.

[0020] Step 3.2: Output evaluation value using the critic network: Combine the values ​​observed by a single agent with the action output of a single agent to form an action pair, then input the above action pair into the critic network, perform vector concatenation and extraction fitting in sequence, and output the evaluation value.

[0021] Specifically, in step 3.2, vector concatenation includes: inputting the action pairs of individual agents into the input layer of the critic network, by introducing weight vectors and bias vectors and then... The activation function performs non-linear processing to obtain the concatenated features.

[0022] Specifically, in step 3.2, the extraction and fitting includes: inputting the concatenated features of individual agents into the hidden layer of the critic network, and introducing weight vectors and bias vectors and then... The activation function performs non-linear processing to obtain the extracted and fitted features.

[0023] Specifically, in step 3.2, the output evaluation value includes: inputting the extracted and fitted features of a single agent into the output layer of the critic network, and introducing weight vectors and bias vectors, and then... The activation function performs non-linear processing to obtain an evaluation value.

[0024] Step four: Assess the reliability of the intelligent unmanned system's decision-making. Step 4.1: Based on the actions output by the actor network in Step 3.1, determine whether a connection is formed between the node and the task; after the node and the task are connected by the connection, a closed arrival chain from task to task is formed; after multiplying the adjacency matrices of all perceptual agents to tasks, the adjacency matrices of all perceptual agents, the adjacency matrices of all decision agents, and the adjacency matrices of all action agents, the arrival chain matrix from task to task is obtained.

[0025] Step 4.2: Calculate the total number of values ​​on the diagonal of the task-to-task arrival chain matrix, which is the number of task running cycles.

[0026] Step 4.3: Calculate the number of running chains formed for a certain task in the task-to-task arrival chain matrix, which is the number of task running loops that the task participates in.

[0027] Step 4.4: For each task in the intelligent unmanned system, the system determines success based on the actual number of running loops formed for a single task and the formation time of each running loop, combined with the preset thresholds for the number of running loops required to complete a single task and the time required to form a running loop; the task is determined to be successful if and only if both the number and the time required to form a running loop meet the requirements.

[0028] Step 4.5: Count the number of successful tasks at each time step in each run, and calculate the task success rate at each time step in a single run based on the number of successful tasks and the total number of tasks; after multiple independent simulation runs, perform statistical processing on the task success rate of all simulation runs to obtain the overall task success rate of the intelligent unmanned system.

[0029] Step 4.6: Based on the overall task success rate and the preset task success rate threshold, calculate the task completion rate of the intelligent unmanned system. The task completion rate is used to quantify the overall effectiveness and decision-making effect of the intelligent unmanned system for the task.

[0030] Step 5: Integrate the rewards of all intelligent agents and the rewards for cooperation among intelligent agents to establish a cooperative reward calculation model for the intelligent unmanned system, and calculate and obtain the total reward of the intelligent unmanned system based on this model.

[0031] Specifically, the cooperative reward calculation model for intelligent unmanned systems is shown in Equation 29 below: Equation 29.

[0032] In the formula: This represents the total reward of the intelligent unmanned system; Indicates rewards for cooperation among intelligent agents; , , All are weighting coefficients, with values ​​ranging from [0.3, 0.5]. Indicates a penalty for connection costs; , , These represent the rewards generated after the perception, decision-making, and action agents perform actions, respectively. Indicates the first i The actions of a sensing intelligent agent; Indicates the first j The actions of a decision-making intelligent agent.

[0033] Step 6: Optimize the decision-making reliability of the intelligent unmanned system using a cooperative multi-agent deep deterministic policy gradient algorithm model: Step 6.1: Configure the parameters of the cooperative multi-agent deep deterministic policy gradient algorithm model. The parameters include: actor learning rate, critic learning rate, discount factor, soft update coefficient, batch size, experience buffer pool size, and hidden layer dimension.

[0034] Step 6.2: Input the critic network parameters, actor network parameters, sample information, number of rounds and time step into the cooperative multi-agent deep deterministic policy gradient algorithm model for reinforcement learning, and update the actor network and critic network according to the output values.

[0035] Specifically, step 6.2 includes the following steps: Step 6.2.1: Store the sample information obtained in Step 3 into the experience buffer pool of each type of agent; the sample information includes: the value observed by the agent itself, the action output of the agent, the observation value of the agent at the next moment, and the total reward of the agent.

[0036] Step 6.2.2: The next moment observation of the agent is input into the actor networks of the perceptual agent, decision-making agent, and action agent respectively, and the actor networks generate the corresponding next action. The next moment observation and the next action generated by the actor networks are combined to form a state-action pair, which is input into the critic network of the corresponding agent. Each critic network calculates and outputs the corresponding predicted evaluation value (i.e., Q value). Based on the predicted evaluation value and the preset discount factor of the corresponding agent, the target evaluation value of each agent is calculated according to the predetermined target evaluation value calculation function.

[0037] Step 6.2.3: Based on the target evaluation value and the current evaluation value, calculate the loss value of the critic network corresponding to the perception, decision-making and action agents respectively.

[0038] Step 6.2.4: Calculate the gradients of the parameters of each actor network and critic network using the backpropagation algorithm, and update the parameters of each online network based on the gradients.

[0039] Step 6.2.5: Using a soft update method, synchronize the parameters of each online actor network and online critic network to their corresponding target actor network and target critic network.

[0040] Specifically, intelligent agents include drones, unmanned vehicles, and unmanned boats.

[0041] Compared with the prior art, the present invention has the following beneficial technical effects: This invention employs a multi-layer operational network model based on coloring graph theory to characterize individual autonomy and functional diversity, overcoming the limitations of single-layer networks and providing a foundation for decision analysis. It utilizes a distributed decision-making model for intelligent unmanned systems based on an enhanced actor-critic architecture, where actor and critic modules enable individuals to balance local and global decisions, improving the policy performance and learning stability of individual decisions. Furthermore, it employs an operational-loop-oriented decision reliability assessment model for intelligent unmanned systems, integrating proposed reliability indicators into the individual learning process to enhance assessment effectiveness. This model successfully quantifies the impact of individual decisions on system reliability. Finally, it proposes a decision reliability optimization model based on the Cooperative Multi-Agent Deep Deterministic Policy Gradient (C-MADDPG) algorithm, incorporating proposed cooperative rewards into the policy gradient optimization process to optimize decision reliability. Attached Figure Description

[0042] Figure 1 A multi-layered running network graph with node and edge coloring.

[0043] Figure 2 This is a schematic diagram illustrating the interaction between an intelligent agent and its environment through distributed decision-making.

[0044] Figure 3 A schematic diagram of an enhanced actor network architecture design.

[0045] Figure 4 This is a schematic diagram of an enhanced critic network architecture design.

[0046] Figure 5 A flowchart for the reliability assessment of decision-making in intelligent unmanned systems.

[0047] Figure 6 Flowchart for optimizing the reliability of decision-making in intelligent unmanned systems.

[0048] Figure 7 This is a simulation diagram of a multi-layered running network model based on coloring graph theory.

[0049] Figure 8 This is a graph showing the change in total reward during the training process of an intelligent unmanned system.

[0050] Figure 9 A graph showing the change in loss for the enhanced actor-critic network.

[0051] Figure 10 The graph shows the total reward change curves of the proposed method and the comparison algorithm.

[0052] Figure 11 The proposed method and the comparison algorithm are shown in the loss variation curves.

[0053] Figure 12 This is a curve showing the operational cycle of an intelligent unmanned system under disturbed conditions.

[0054] Figure 13 This is a graph showing the change in the success rate of intelligent unmanned systems under disturbed environments.

[0055] Figure 14 This is a graph showing the change in the task completion rate of an intelligent unmanned system under disturbed conditions.

[0056] Figure 15 This is a stability analysis chart for the success rate of intelligent unmanned systems under disturbed environments.

[0057] The technical solution of the present invention will be further described below with reference to the embodiments. Detailed Implementation

[0058] It should be noted that all hardware and software used in this invention, unless otherwise specified, are those known in the art. Specifically, the simulation environment comprises both hardware and software components. The hardware environment is implemented on a workstation with a Core i7-13700 processor and 64GB of DDR4 memory, providing sufficient computing power for large-scale intelligent agents. All algorithms are implemented in Python 3.10, specifically including using Gymnasium 0.28.1 to build the reinforcement learning environment, Torch 1.13.1 for deep learning, NumPy 1.24.3 for numerical computation, and Matplotlib 3.7.1 for visualization.

[0059] The technical concept of this invention is as follows: First, each intelligent agent with corresponding functions is uniformly abstracted as a node. Its role and interaction topology are represented by three types of nodes: perception, decision-making, and action, and the connection relationships between nodes. Each functional role corresponds to a node color, and each cooperative relationship corresponds to an edge color, thereby constructing a multi-layer operating network model based on coloring graph theory.

[0060] Secondly, in order to improve the reliability of decision-making, considering the autonomy and cooperation of the agent, a distributed intelligent unmanned system decision-making model based on an enhanced actor-critic architecture is proposed. Specifically, a four-layer actor network and a three-layer critic network are used to improve the reliability of the agent's decision-making in environmental interaction.

[0061] Then, to ensure the reliability of decision-making results, a decision reliability assessment model for intelligent unmanned systems oriented towards the operational cycle is proposed. Metrics such as the operational cycle, task success rate, and task completion rate are integrated into the individual learning process to improve the assessment effect. These metrics are used to quantify the decision-making efficiency of intelligent unmanned systems.

[0062] Finally, with the goal of maximizing individual benefits and improving task efficiency, the decision reliability is optimized by using the Cooperative Multi-Agent Deep Deterministic Gradient Policy (C-MADDPG) algorithm. The cooperative reward is integrated into the policy gradient optimization process of the agents to improve the decision reliability of the intelligent unmanned system.

[0063] Based on the above strategy, intelligent agents can be effectively directed to complete various tasks. For example, when this invention is applied to drones striking targets, both the drone and the target can be abstracted as nodes; when a target participates in multiple task cycles, it can be precisely struck according to the above strategy.

[0064] Following the above technical ideas and solutions, the following are specific embodiments of the present invention. It should be noted that the present invention is not limited to the following specific embodiments, and all equivalent modifications made based on the technical solutions of this application fall within the protection scope of the present invention.

[0065] Example 1 This embodiment provides a method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning. Taking a collaborative scenario as an example, the correctness and superiority of the reinforcement learning-based method for evaluating and optimizing the decision reliability of intelligent unmanned systems are verified. The specific steps include the following: Step 1: Construct a multi-layered running network model based on coloring graph theory: Step 1.1: Each intelligent agent with corresponding functions is uniformly abstracted as a node. Functions include perception, decision-making, and action, with each node possessing one or two of these functions. Furthermore, the tasks performed by the intelligent agents are also abstracted as nodes. Interactions exist between nodes, represented by lines. For a more intuitive display, each function is defined as a node color, each task as a color, and the lines between nodes and between nodes and tasks are defined as edge colors. To generate a multi-layered operational network model based on coloring graph theory, the following parameters need to be input (or generated): agent type, number of agents, and number of tasks. These three parameters are generally defined by the user and are fixed parameters. Additionally, agent coordinates, resources, costs, and task coordinates also need to be input. These parameters can be randomly generated or defined by the user. The set of intelligent agents within the intelligent unmanned system is shown in Equations 1 and 2 below: Formula 1.

[0066] Equation 2.

[0067] In the formula: It represents the set of intelligent agents within an intelligent unmanned system. Represents a set of perceptual intelligent agents. Represents a set of decision-making intelligent agents. Represents a set of intelligent agents in action. A colored multi-layered operational network representing an intelligent unmanned system. Represents a set of nodes. Represents the set of cooperative edges. This represents the set of superedges, where each superedge represents a cycle. Represents the node coloring vector. This represents the edge coloring vector.

[0068] In this embodiment, some parameters are shown in Table 1 below (the agent coordinates, resources, costs, and task coordinates are all randomly generated and are not shown in Table 1).

[0069] Table 1. Input parameters of the multilayer running network model based on coloring graph theory in Example 1

[0070] Step 1.2: Initialize the above parameters using NumPy 1.24.3, then visualize the initialized parameters using Matplotlib 3.7.1, finally outputting a multi-layer running network model based on coloring graph theory (e.g., Figure 7 (As shown).

[0071] Step 2: Establish the environment for the intelligent unmanned system: Using Gymnasium, define and input the state space, observation space, action space, and state transition rules to obtain an intelligent unmanned system environment (i.e., the environment where the agent resides and the environment required for interaction). The interaction relationship between the agent and the environment is as follows: Figure 2 As shown.

[0072] As a specific embodiment, the state space is defined using the following equation 3: Formula 3.

[0073] In the formula: Representing the state space, Represents a coordinate matrix. Represents the adjacency matrix. Represents a resource vector. This represents the cost matrix.

[0074] As a specific embodiment, the observation space is defined using the following equations 4 to 7: Formula 4.

[0075] Formula 5.

[0076] Formula 6.

[0077] Formula 7.

[0078] In the formula: Represents the observation space. These represent the observation spaces for perception, decision-making, and action, respectively. , , These represent the observation spaces of individual agents in the perception, decision-making, and action categories, respectively. Let these represent the coordinate vector, adjacency matrix, resource vector, and cost vector of a single perceptual agent facing the decision-making agent, respectively. Let these represent the coordinate vector, adjacency matrix, resource vector, and cost vector of a single decision-making agent facing the action agent, respectively. These represent the coordinate vector, adjacency matrix, resource vector, and cost vector of a single action agent facing the target, respectively.

[0079] As a specific embodiment, the action space is defined as follows: Equations 8 to 11: Formula 8.

[0080] Formula 9.

[0081] Formula 10.

[0082] Formula 11.

[0083] In the formula: Represents the action space, These represent the spaces for perception, decision-making, and action, respectively. , , These represent the action spaces of individual agents in the perception, decision-making, and action categories, respectively. These represent the actions of a single perceptual agent towards each decision-making agent. These represent the actions of a single decision-making agent towards each action agent. These represent the target-oriented actions of a single agent.

[0084] As a specific embodiment, the state transition rules are defined as follows: Equations 12 to 15: Equation 12.

[0085] Formula 13.

[0086] Equation 14.

[0087] Formula 15.

[0088] In the formula: Represents intelligent agents and Establish an edge between them. Represents intelligent agents and Establish an edge between them. Represents intelligent agents With the task Establish an edge between them. express t At time +1, the action space of a single perceptual agent is directed towards the decision-making agent. The action, expresst At time +1, the action space of a single decision-making agent for the action agent The action, express t At time +1, the action space of a single agent targeting the target The action, express t The state at time +1 is determined by... t The state at any given moment and the action space after the change are obtained.

[0089] In this embodiment, the parameter configuration of the intelligent unmanned system environment is shown in Table 2.

[0090] Table 2. Parameter configuration of the intelligent unmanned system environment in Example 1

[0091] In Table 1, taking the perception state dimension “30*16*2” as an example, 30 represents the number of perception agents, 16 represents the number of decision agents, and “2” represents the two action states of “0 or 1”.

[0092] Step 3: Construct an Enhanced Actor-Critic Architecture (Enhanced-AC) agent decision-making model, which includes a four-layer actor network (e.g., Figure 3 (as shown) and a three-layered critic network (such as...) Figure 4 (As shown). The actor network outputs the agent's actions; the critic network outputs evaluation values.

[0093] Step 3.1, the values ​​observed by a single agent itself (i.e. , or The input is fed into the actor network, and through calculations such as Equation 16 (feature encoding layer), Equation 17 (feature extraction layer), Equation 18 (feature compression layer), and Equation 19 (action output layer), the output action (i.e.,...) is generated. , or ): Formula 16.

[0094] Formula 17.

[0095] Formula 18.

[0096] Formula 19.

[0097] In the formula: , , These represent the features obtained by the actor network after encoding the observations of the perception, decision-making, and action agents, respectively; , , These represent the weight vectors used by the actor network to encode the perception, decision-making, and action agents, respectively. , , These represent the bias vectors used by the actor network to encode the perceptual, decision-making, and action agents, respectively. , , These represent the mask vectors used by the actor network to encode the perception, decision-making, and action agents, respectively.

[0098] , and These represent the features extracted from the encoded features of the perceptual, decision-making, and action agents by the actor network, respectively. , and These represent the weight vectors used by the actor network to extract the encoded features of the perceptual, decision-making, and action agents, respectively. , , These represent the bias vectors used by the actor network to extract the encoded features of the perceptual, decision-making, and action agents, respectively. , , These represent the mask vectors used by the actor network to extract the encoded features of the perceptual, decision-making, and action agents, respectively.

[0099] , and These represent the compressed features extracted by the actor network from the perceptual, decision-making, and action agents, respectively. , and These represent the weight vectors used by the actor network to compress the features extracted by the perceptual, decision-making, and action agents, respectively. , , and represent the bias vectors used by the actor network to compress the extracted features of the perceptual, decision-making, and action agents, respectively.

[0100] , and These represent the weight vectors of the actor network when outputting compressed features of the perception, decision-making, and action agents, respectively. , , These represent the bias vectors when the actor network outputs compressed features of the perception, decision-making, and action agents, respectively. , , Represents the action output of a single sensing, decision-making, and action-oriented intelligent agent; and Both represent activation functions.

[0101] Step 3.2: Combine the values ​​observed by a single agent with the action output of a single agent to form an action pair (i.e., ... , or Then, the above actions are input into the critic network, and are calculated sequentially through Equation 20 (input layer), Equation 21 (hidden layer), and Equation 22 (output layer) to output the evaluation value (Q value, i.e., ...). , or ): Formula 20.

[0102] Equation 21.

[0103] Equation 22.

[0104] In the formula: , , These represent the concatenated features of the commentator network's observation pairs of the perceptual, decision-making, and action agents; , , These represent observation-action pairs of the perceptual, decision-making, and action agents, respectively. , , These represent the weight vectors of the commentator network when concatenating the observation action pairs of the perception, decision-making, and action agents; , , and represent the bias vectors used by the commentator network to concatenate the observation pairs of the perception, decision-making, and action agents, respectively.

[0105] , , These represent the features extracted and fitted by the critic network after concatenating the features of the perception, decision-making, and action agents; , , , respectively, represent the weight vectors used by the critic network to extract and fit the features concatenated from the perception, decision-making, and action agents; , , and represent the bias vectors used by the critic network to extract and fit the features of the concatenated perception, decision-making, and action agents, respectively.

[0106] , , These represent the Q-values ​​of the perception, decision-making, and action agents, respectively. , , These represent the weight vectors of the critic network when it extracts and outputs the fitted features of the perception, decision-making, and action agents, respectively. , , represent the bias vectors of the critic network when it extracts and outputs the fitted features of the perception, decision-making, and action agents, respectively. This represents the activation function.

[0107] Step four: Evaluate the decision-making reliability of the intelligent unmanned system (the process is as follows). Figure 5 (as shown) Step 4.1: Based on the actions output by the actor network in Step 3.1, determine whether a connection is formed between the node and the task. After the node and the task are connected by a connection, a chain can be formed "starting from the task, passing through the node, and ending at the task" (hereinafter referred to as the task-to-task arrival chain), and the task-to-task arrival chain matrix shown in Equation 23 is constructed as follows: Equation 23.

[0108] In the formula: This represents the closed-loop arrival matrix from task to task; This represents the adjacency matrix of all perceptual agents to the task. This represents the adjacency matrix of all perceptual agents. Represents the adjacency matrix of all decision-making agents. This represents the adjacency matrix of all acting agents; the adjacency matrix is ​​mentioned in Equation 3 above.

[0109] Step 4.2: Based on the task-to-task arrival chain matrix, obtain the task execution cycle (i.e., the starting and ending tasks of the arrival chain are the same task, such as...). Figure 1 As shown in the diagram, the light blue dashed lines indicate the number of task execution loops. Specifically, the total number of values ​​on the diagonal of the matrix shown in Equation 23 represents the number of task execution loops, expressed as: Equation 24.

[0110] In the formula: This indicates the number of task operation loops formed in an intelligent unmanned system. This represents the total number of values ​​on the diagonal of the task-to-task arrival chain matrix.

[0111] Step 4.3, use the following formula 25 to obtain the number of task execution cycles in which a certain task participates: Formula 25.

[0112] In the formula: Indicates task The number of task execution loops involved. This represents the arrival chain matrix from task to task for the first task. l The number of running chains formed by each task.

[0113] Step 4.4 sets the threshold for the number of task loops linked to a given task and the time required to form a task loop. The task is considered successful when the following condition (equation 26) is met: Equation 26.

[0114] In the formula: express t The number of tasks successfully completed in the current intelligent unmanned system. A threshold representing the number of task loops a single task participates in. This represents the threshold time required to form a task execution loop. This indicates the actual time required to form a task execution loop; Indicates an indicator function, This indicates that the task completion time does not exceed the time threshold. ; This indicates that when the number of runtime loops formed for a task exceeds the required threshold, .

[0115] Step 4.5, Substitute into Equation 27 below to calculate and obtain the task success rate: Equation 27.

[0116] In the formula: This indicates the mission success rate of the intelligent unmanned system. This represents the total number of simulations. Indicates the first i The simulation was successful. This indicates the total number of tasks (25 in this example).

[0117] Step 4.6: Calculate and obtain the task completion rate using the following formula 28: Equation 28.

[0118] In the formula: This indicates the task completion rate of the intelligent unmanned system. This represents the total number of simulations. Indicates the first i The simulation was successful. This represents the threshold for task success rate.

[0119] Step 5: Integrate the total reward of all agents and the cooperative reward of agents to establish a cooperative reward calculation model for the intelligent unmanned system, and calculate and obtain the total reward of the intelligent unmanned system based on this model. , This will be used for the next step of optimization learning.

[0120] As a specific embodiment, the cooperative reward calculation model for the intelligent unmanned system is shown in Equation 29 below: Equation 29.

[0121] In the formula: This represents the total reward of the intelligent unmanned system; Indicates rewards for cooperation among intelligent agents; , , These are all weighting coefficients, and their values ​​generally range from [0.3, 0.5]. Indicates a penalty for connection costs; , , These represent the rewards generated after the perception, decision-making, and action agents perform actions, respectively. This represents the action of the i-th perceptual agent; This represents the action of the j-th decision-making agent; This represents the action of the k-th agent.

[0122] As a specific embodiment, the reward for a single agent is calculated and obtained using the following formulas 30 to 32: Formula 30.

[0123] Equation 31.

[0124] Equation 32.

[0125] In the formula: This indicates the number of perceptual agents. Indicates the number of decision-making agents; Indicates the number of intelligent agents in action; Indicates the number of tasks (as mentioned above) "Indicates the total number of tasks". and They are numerically equal. Indicates that the i-th perceptual agent is targeting the i-th... j The actions of a decision-making intelligent agent; Indicates the first j The decision-making agent targets the first... k The actions of an intelligent agent; Indicates the first k The first action agent targets the first l Actions targeting a specific objective.

[0126] As a specific embodiment, the agent cooperation reward is calculated and obtained using the following equations 33 to 35: Equation 33.

[0127] Equation 34.

[0128] Formula 35.

[0129] In the formula: This represents the reward generated by forming a running loop; This indicates the reward for successfully completing the task.

[0130] Step six: Optimize the decision reliability of the intelligent unmanned system using the Cooperative Multi-Agent Deep Deterministic Policy Gradient (C-MADDPG) algorithm model. The specific process is as follows: Figure 6 As shown.

[0131] Step 6.1: First, configure the parameters of the C-MADDPG algorithm model. In this embodiment, the specific parameters are shown in Table 3.

[0132] Table 3. Parameter configuration of the C-MADDPG algorithm model in Example 1

[0133] Step 6.2: Train the intelligent unmanned system using the C-MADDPG algorithm model. During training, the network parameters of the commentator are used. Actor network parameters Playback buffer Number of rounds and time step As input, update the actor network and critic network based on the output values.

[0134] Step 6.2.1: Store the sample information obtained in Step 3 into the various experience buffer pools of the playback buffer, as shown below: Formula 35.

[0135] In the formula, , , This represents the data structure stored in the buffer for each type of agent, and a batch of samples with a size of [missing information] is sampled from the experience replay buffer. Z experience; , , These represent the values ​​observed by a single sensing, decision-making, and action agent, respectively. , , Represents the action output of a single sensing, decision-making, and action-oriented intelligent agent; These represent the next moment's observation values ​​for the three types of agents, respectively.

[0136] Step 6.2.2: Using the following formula 36, ​​calculate and obtain the evaluation values ​​corresponding to the next observation and the next action of the sample: Equation 36.

[0137] In the formula, , , This represents the target Q-value for each type of agent. Indicates the sample index. Indicates actor network, This refers to the network of critics. , , The first agent represents the intelligent agent. z The next step for each sample , , The first agent represents the intelligent agent. z The Q-value corresponding to the next observation and the next action for each sample. , , This represents the discount factor.

[0138] Step 6.2.3, use the following formula 37 to calculate the minimized loss: Equation 37.

[0139] In the formula, , , Indicates the network loss of various intelligent agents. , , This represents the current Q value.

[0140] Step 6.2.4, calculate the gradient using the following formula 38: Equation 38.

[0141] In the formula, , These represent the gradients of actors and critics, respectively.

[0142] Step 6.2.5, update the target network parameters using the following formula 39: Equation 39.

[0143] In the formula, , These represent the parameters of the target actor network and the target critic network, respectively. , represent the soft update values ​​of the target actor network and the target critic network parameters, respectively. , These represent the learning parameters of the actor and critic networks, respectively.

[0144] Verification of the effect of Example 1: (A) Train the intelligent agent in the intelligent unmanned system and obtain the total reward through interaction with the environment, such as Figure 8 As shown in the figure, the number of training rounds and the step size were 4000 and 200, respectively. The blue line represents the reward for each round. The red line represents the average reward over the past 50 rounds. Compared to the blue line, the red line is smoother and better illustrates the overall trend of training. In the red line, the reward reaches an inflection point at round 145 and then quickly stabilizes, indicating that the proposed augmented actor-critic architecture has better convergence and stable learning.

[0145] (B) The network loss curves for the actors and critics of the intelligent agent are as follows: Figure 9 As shown in (a) and (b), the critic loss rises rapidly in the early stages of training, then drops sharply, and then gradually decreases until it reaches a stable low value close to zero. This indicates that the critic network has learned an accurate and stable value estimation function. Figure 9 (c) shows the total variation curves of actor loss and critic loss. Since the critic network loss is significantly greater than the actor network loss, the actor loss is represented by a straight line. Figure 9 (d) shows the loss curves for each type of agent, with the order of loss convergence being decision, perception, and action.

[0146] (C) The Enhanced Actor-Critic Architecture (Enhanced-AC) agent decision-making model is compared with several variants derived from the actor-critic architecture, including Residual Networks (ResNet), Multilayer Perceptrons (MLP), attention mechanisms, and regularization. Simulations of these methods were performed under the same conditions. The reward curves and specific values ​​for the compared methods are shown in Figure [Figure number missing]. Figure 10As shown in Table 6.

[0147] A careful analysis of these variant methods reveals some limitations: (1) ResNet-AC ( Figure 10 (a)), ResNet-AC-MLP ( Figure 10 (b) and AC-MLP Figure 10 (d) It exhibits an oscillating upward trend at the inflection point, followed by continuous fluctuations. These variant methods based on ResNet and MLP result in insufficient stability of the decision-making process. (2) Atten-AC Figure 10 (c)), AC ( Figure 10 (e) and Dropout-AC Figure 10 (g) It exhibits significant volatility at the inflection point. Both of these variant methods lead to insufficient stability in decision-making. (3) Dropout-AC-MLP Figure 10 (f) and Enhanced-AC-MLP Figure 10 (h) exhibits a declining reward curve in the later stages of training, revealing the instability introduced by the MLP-based architecture. Compared to the methods mentioned above, the proposed Enhanced-AC ( Figure 10 (i) received a higher reward of 4.7636 × 10⁶. Furthermore, Enhanced-AC ( Figure 10 (i) and Enhanced-AC-MLP Figure 10 The reward curve (h) shows an almost right angle rise, indicating that the proposed enhancement significantly accelerates convergence.

[0148] Table 4. Performance Comparison Analysis of Various Algorithms

[0149] As can be seen from Table 4, the Enhanced-AC proposed in this invention performs best in all performance aspects. It achieves a score of 4.7636 × 10⁻⁶. 6 Its number of running loops remained at 4.7681 × 10⁻⁶. 5 Its connection rate reached 99.3361%, and it converged in only 145 rounds, demonstrating the high learning efficiency of the proposed method. In contrast, the AC method has a lower reward (3.5697 × 10⁻⁶). 6 The connection rate was lower (74.4529%), and the convergence speed was slower (2604 truns). Notably, the regularized variants exhibited better generalization, for example, Dropout-AC achieved 89.5952%, and Dropout-AC-MLP achieved 87.2492%. This confirms that regularization helps prevent overfitting. Furthermore, although Atten-AC converged faster (995 truns), its final performance was weaker.

[0150] The network loss curves for the above comparison methods are as follows: Figure 11 As shown, the critic loss initially rises rapidly during the early stages of training, then drops sharply, then gradually decreases, eventually reaching a stable low value close to zero. All critic networks learn an accurate and stable value estimation function, indicating that the networks constructed using these methods are correct.

[0151] (D) Based on Monte Carlo simulation, the number of simulations and the time step are both 100. The agent pre-trained based on the C-MADDPG algorithm is loaded into a perturbed environment. A failure model is constructed to simulate the effects of perturbation on the agent. The agent's failure follows the parameters... The Poisson distribution is followed. If an agent fails, its interaction edges are broken, and its resources are reset to zero. The surviving agents then make decisions through an enhanced actor-critic network, interacting with the environment to obtain rewards, and collecting data such as rewards at each time step, the number of running cycles, and task success rates to evaluate decision-making performance and stability.

[0152] The reliability of decision-making is evaluated using three metrics: number of operational loops, mission success rate, and mission completion rate. The number of operational loops measures the ability to maintain and reconstruct coordinated strike capabilities under disturbances, forming a closed operational loop from reconnaissance to decision-making, action, and target acquisition. Mission success rate is quantified as the ratio of completed missions to total missions, used to evaluate mission effectiveness at a given time point. Mission completion rate is quantified as the ratio of the number of simulations where the mission success rate exceeds a threshold to the total number of simulations, describing the robustness of the intelligent unmanned system in maintaining mission success rate across multiple disturbance simulations.

[0153] Figure 12 This shows the evolution trend of the number of running loops under perturbation under different methods. At the initial time ( t =0) The proposed method, which combines Enhanced-AC with the C-MADDPG algorithm, generates more running cycles than other comparative methods, directly inheriting its performance advantage from the training phase. Although the number of running cycles under perturbation decreases over time for all methods, the proposed method consistently leads, demonstrating its superior resilience in resisting perturbations and maintaining system functional integrity.

[0154] Figure 13 The evolution trend of task success rate under perturbation is shown for different methods. Over time, the advantages of the proposed C-MADDPG algorithm become increasingly apparent. Although the initial performance of C-MADDPG is comparable to other methods, it exhibits stronger perturbation resistance over time. Its task success rate curve is superior to other methods, and the performance gap widens over time, demonstrating its effectiveness under long-term perturbation.

[0155] Figure 14 The evolution trend of task completion under perturbation is shown. Although the proposed method occasionally matches other methods at certain time steps, it maintains a significant lead over most of the time, demonstrating consistent decision stability and overall performance advantage.

[0156] Figure 15 A 95% confidence interval for the task success rate is given to verify the stability of the proposed method. The width of the confidence interval for the task success rate ranges from [0.022, 0.0528]. This indicates that the performance output of the proposed method is relatively stable with small random fluctuations, further verifying the stability of the method in maintaining efficient and reliable decision-making under perturbations.

Claims

1. A method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning, characterized in that, Specifically, the steps include the following: Step 1: Input the agent parameters and task parameters and initialize them. Then visualize the initialized parameters to obtain a multi-layer running network model based on coloring graph theory. Step 2: Define and input the state space, observation space, action space, and state transition rules to obtain an intelligent unmanned system environment; Step 3: Obtain sample information using an enhanced actor-critic architecture agent decision-making model: Step 3.1: Output the agent's actions using the actor network: Sequentially encode, extract, and compress the values ​​observed by a single agent to finally output the actions; Step 3.2, using the critic network to output evaluation values: combine the values ​​observed by a single agent and the action output of a single agent to form action pairs, then input the above action pairs into the critic network, perform vector concatenation and extraction fitting in sequence, and output the evaluation values; Step four: Assess the reliability of the intelligent unmanned system's decision-making. Step 5: Combine the total reward of all intelligent agents and the cooperative reward of intelligent agents to establish a cooperative reward calculation model for the intelligent unmanned system, and calculate and obtain the total reward of the intelligent unmanned system based on the model. Step six: Optimize the decision reliability of the intelligent unmanned system using a cooperative multi-agent deep deterministic policy gradient algorithm model.

2. The method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning as described in claim 1, characterized in that, In step one, the agent parameters include: agent type, number of agents, agent coordinates, agent resources, and agent cost; the task parameters include: number of tasks and task coordinates.

3. The method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning as described in claim 1, characterized in that, Step one includes: abstracting each intelligent agent with corresponding functions into a node; classifying intelligent agents into perceptual agents, decision-making agents, and action agents according to their functions; abstracting each task to be executed into a node; having interactive relationships between nodes, using lines to represent these relationships; defining perceptual agents, decision-making agents, and action agents with different node colors, defining tasks with one color, and defining the lines between nodes and the lines between nodes and tasks with different edge colors.

4. The method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning as described in claim 1, characterized in that, In step two, the state space is defined using a coordinate matrix, an adjacency matrix, a resource vector, and a cost matrix; In step two, the observation space includes: the perception observation space, the decision observation space, and the action observation space. The perception observation space is defined using the coordinate vector, adjacency matrix, resource vector, and cost vector of the perception agent facing the decision agent. The decision observation space is defined using the coordinate vector, adjacency matrix, resource vector, and cost vector of the decision agent facing the decision agent. The action observation space is defined using the coordinate vector, adjacency matrix, resource vector, and cost vector of the action agent facing the decision agent. In step two, the action space includes: the perception action space, the decision action space, and the action action space; the perception action space is defined by the set of actions of all perceptual agents toward each decision agent; the decision action space is defined by the set of actions of all decision agents toward each decision agent; and the action action space is defined by the set of actions of all action agents toward each decision agent. In step two, the state transition rules include: a single agent obtains a set of actions for a certain type of agent or task based on the current observation, updates the edges between agents and the edges between agents and tasks according to the set of actions, and since the adjacency matrix is ​​a state space element, the global cooperative state of the intelligent unmanned system environment is also updated.

5. The method for evaluating and optimizing the reliability of decision-making in intelligent unmanned systems based on reinforcement learning as described in claim 1, characterized in that, In step 3.1, feature encoding includes: inputting the observations of a single agent into the feature encoding layer of the actor network, and introducing weight vectors, bias vectors, and mask vectors, and then... The activation function performs non-linear processing to obtain the encoded features; In step 3.1, feature extraction includes: inputting the encoded features of a single agent into the feature extraction layer of the actor network, and introducing weight vectors, bias vectors, and mask vectors, and then... The activation function performs non-linear processing to obtain the extracted features; In step 3.1, feature compression includes: inputting the extracted features of a single agent into the feature compression layer of the actor network, and introducing weight vectors, bias vectors, and mask vectors, and then... The activation function performs non-linear processing to obtain compressed features; In step 3.1, the output action includes: inputting the compressed features of a single agent into the action output layer of the feature compression layer of the actor network, and introducing weight vectors, bias vectors, and mask vectors, and then... The activation function performs linear processing to obtain the action output of a single agent.

6. The method for evaluating and optimizing the reliability of decision-making in intelligent unmanned systems based on reinforcement learning as described in claim 1, characterized in that, In step 3.2, vector concatenation includes: inputting the action pairs of individual agents into the input layer of the critic network, by introducing weight vectors and bias vectors, and then... The activation function performs non-linear processing to obtain the concatenated features; In step 3.2, the extraction of the fit includes: inputting the concatenated features of individual agents into the hidden layer of the critic network, and introducing weight vectors and bias vectors through... The activation function performs non-linear processing to obtain the extracted and fitted features; In step 3.2, the output evaluation value includes: inputting the extracted and fitted features of a single agent into the output layer of the critic network, and introducing weight vectors and bias vectors through... The activation function performs non-linear processing to obtain an evaluation value.

7. The method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning as described in claim 1, characterized in that, Step four specifically includes the following steps: Step 4.1: Based on the actions output by the actor network in Step 3.1, determine whether a connection is formed between the node and the task; after the node and the task are connected by the connection, a task-to-task arrival chain is formed; after multiplying the adjacency matrices of all perceptual agents to the task, the adjacency matrices of all perceptual agents, the adjacency matrices of all decision agents, and the adjacency matrices of all action agents, the task-to-task arrival chain matrix is ​​obtained. Step 4.2: Calculate the total number of values ​​on the diagonal of the task-to-task arrival chain matrix, which is the number of task running cycles; Step 4.3: Calculate the number of running chains formed for a certain task in the task-to-task arrival chain matrix, which is the number of task running loops that the task participates in. Step 4.4: For each task in the intelligent unmanned system, the system determines success based on the actual number of running loops formed for a single task and the formation time of each running loop, combined with the preset thresholds for the number of running loops required to complete a single task and the time required to form a running loop; the task is determined to be successful if and only if both the number and the time required to form a running loop meet the thresholds. Step 4.5: Count the number of successful tasks at each time step in each run, and calculate the task success rate at each time step in a single run based on the number of successful tasks and the total number of tasks; after multiple independent simulation runs, perform statistical processing on the task success rate of all simulation runs to obtain the overall task success rate of the intelligent unmanned system. Step 4.6: Based on the overall task success rate and the preset task success rate threshold, calculate the task completion rate of the intelligent unmanned system. The task completion rate is used to quantify the overall effectiveness and decision-making effect of the intelligent unmanned system for the task.

8. The method for evaluating and optimizing the reliability of decision-making in intelligent unmanned systems based on reinforcement learning as described in claim 1, characterized in that, Step six specifically includes the following steps: Step 6.1: Configure the parameters of the cooperative multi-agent deep deterministic policy gradient algorithm model; Step 6.2: Input the critic network parameters, actor network parameters, sample information, number of rounds and time step into the cooperative multi-agent deep deterministic policy gradient algorithm model for reinforcement learning, and update the actor network and critic network according to the output values.

9. The method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning as described in claim 8, characterized in that, In step 6.1, the parameters include: actor learning rate, critic learning rate, discount factor, soft update coefficient, batch size, experience buffer pool size, and hidden layer dimension.

10. The method for evaluating and optimizing the decision reliability of intelligent unmanned systems based on reinforcement learning as described in claim 8, characterized in that, Step 6.2 includes the following steps: Step 6.2.1: Store the sample information obtained in Step 3 into the experience buffer pool of each type of agent in the replay buffer; the sample information includes: the value observed by the agent itself, the action output of the agent, the observation value of the agent at the next moment, and the total reward of the agent; Step 6.2.2: The next moment observation of the agent is input into the actor networks of the perceptual agent, decision-making agent, and action agent respectively, and the actor networks generate the corresponding next action; the next moment observation and the next action generated by the actor networks are combined to form a state-action pair, which is input into the critic network of the corresponding agent respectively, and each critic network calculates and outputs the corresponding predicted evaluation value; based on the predicted evaluation value and the preset discount factor of the corresponding agent, the target evaluation value of each agent is calculated according to the predetermined target evaluation value calculation function. Step 6.2.3: Based on the target evaluation value and the current evaluation value, calculate the loss value of the critic network corresponding to the perception, decision-making and action agents respectively; Step 6.2.4: Calculate the gradients of the parameters of each actor network and critic network using the backpropagation algorithm, and update the parameters of each online network based on the gradients; Step 6.2.5: Using a soft update method, synchronize the parameters of each online actor network and online critic network to their corresponding target actor network and target critic network.

Images