A Reinforcement Learning-Based Integrated Scheduling Analysis Method for Avionics Systems

Abstract

The application provides a comprehensive avionics system schedulable analysis method based on reinforcement learning, comprising the following steps: designing a Markov decision process according to a comprehensive avionics system schedulable problem, and constructing a virtual environment; combining a value-based reinforcement learning algorithm with a multi-head attention mechanism, interacting with the virtual environment through an agent, and learning an optimal state-action value function; when the agent interacts with the environment, using the optimal state-action value function to obtain the value of each actionable action under the current state; selecting the actionable action with the highest value and executing the actionable action; obtaining the next state and reward, and accumulating the reward; or combining a policy-based reinforcement learning algorithm with a multi-head attention mechanism, learning an optimal policy function; using the optimal policy function to obtain a plurality of preselected actions, and executing the candidate action sequence with the highest score.

Description

Technical Field

[0001] This invention belongs to the field of safety-critical software technology, specifically a comprehensive avionics system schedulability analysis method based on reinforcement learning. Background Technology

[0002] In the past, aircraft systems employed a federated architecture: each avionics function was integrated onto a single computing resource, creating a "one function equals one computer" model. This architecture's implicit application isolation facilitated runtime fault isolation, but at the cost of excessive weight, size, and cabling to provide sufficient power and connectivity for all modules. With the continuous evolution of avionics systems, modern aircraft incorporate far more intelligent functions than their predecessors. The number of new functions added to modern systems has increased exponentially compared to older architectures, rendering the "one function equals one computer" model and its required computing resources, weight, and power consumption unfeasible.

[0003] To address this challenge, the Integrated Modular Avionics (IMA) architecture emerged. Under the IMA architecture, multiple avionics functions can share computing resources on the same computer. IMA not only effectively reduces the space, weight, and power consumption required by the aircraft, but also, by integrating multiple functions into a single module, enables the sharing of components such as processing resources, power supply, and I / O management services, further reducing system heat dissipation and fuel consumption. Furthermore, another major advantage of the IMA architecture is its versatility, which facilitates the design phase by ensuring the independence of specific applications from other applications and hardware platforms, thereby promoting system flexibility and scalability.

[0004] Currently, the management and scheduling of avionics systems are mostly performed manually, without effectively considering the time delays that may occur due to resource contention in a multi-core processor environment. This means that once the system changes or needs to be upgraded, the existing task allocation and scheduling procedures must be replanned, a process that is extremely time-consuming and labor-intensive. Summary of the Invention

[0005] Purpose of the invention: To address the problems of low efficiency and high manual and time costs in task scheduling of integrated avionics systems on multi-core processors, this invention proposes a schedulable analysis method for integrated avionics systems based on reinforcement learning. By employing reinforcement learning based on a multi-head attention mechanism, the expected scheduling results can be obtained automatically, safely, and optimally, reducing the manual iteration process and improving the efficiency of task scheduling generation.

[0006] Technical solution: A method for schedulable analysis of integrated avionics systems based on reinforcement learning, comprising the following steps:

[0007] Step 1: Design a Markov decision process based on the schedulability problem of integrated avionics systems and construct a virtual environment;

[0008] Step 2: Combine value-based reinforcement learning algorithms with multi-head attention mechanisms, and learn the optimal state-action value function by having the agent interact with the virtual environment;

[0009] Step 3: When the agent interacts with the environment, use the optimal state-action value function obtained in Step 2 to obtain the value of each action in the current state; select the action with the highest value and execute it; obtain the next state and reward, and accumulate the reward.

[0010] Furthermore, the design of a Markov decision process based on the schedulability problem of integrated avionics systems and the construction of a virtual environment specifically include:

[0011] Modeling the schedulability problem of integrated avionics systems;

[0012] Based on the schedulable model of integrated avionics systems, a Markov decision process is constructed, and a virtual environment is built.

[0013] Furthermore, the modeling of the schedulability problem of integrated avionics systems specifically includes:

[0014] The time budget required for each partition to execute tasks on each core is calculated using the following formula, expressed as:

[0015]

[0016] In the formula, Indicates partition p i The time budget required for tasks assigned to core C, C SW This represents the overhead of context switching from the completion of a task to the next scheduled task;

[0017]

[0018] R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, where i represents the partition code, k represents the frame in which the partition is located, and P i T represents the period of the partition. j Indicates the duration of the task;

[0019] Compare the time budgets of all partitions on the core, and take the maximum value as the total time budget for the partition. The specific calculation formula is as follows:

[0020]

[0021] In the formula, Indicates partition p i The total time budget for the k-th frame, where c represents the core and N c Indicates the number of cores;

[0022] The MAF (Main Time Frame) is the primary time frame, and the MIF (Minimum Time Frame) is the secondary time frame. The MAF is divided into frames of equal length, with one MIF constituting one frame. The number of MIFs is nFrames. The lengths of both the MAF and MIF are related to the period, which is always an integer multiple of the MIF length, expressed as:

[0023]

[0024]

[0025] In the formula, p i This represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor.

[0026] Furthermore, the construction of a Markov decision process and the creation of a virtual environment based on the schedulable model of the integrated avionics system specifically includes the following operations:

[0027] In solving the scheduling problem, the available actions in each state are to allocate a start time to the current partition, and the action space is defined as: a t ={v}, v∈[0,MAF]; where v represents the start time of the current partition allocation;

[0028] To determine the start time of each partition, a global action array is introduced in the state definition. This action array is used to store the action value selected each time during the solution process, which indicates the start time of each partition.

[0029] The state transition process can be described as a quadruple: (s t ,a t ,R t ,s t+1 In the formula, s t a represents the state at time step t. t Indicates that in state s t The agent selects the action to execute, R t This indicates that the agent is in state s. t Next, execute action a t The instant reward obtained afterward, s t+1 This indicates the next state the environment enters after an action is performed;

[0030] The reward function is expressed as:

[0031]

[0032] In the formula, n is the number of tasks in the scheduling problem, items_taken is the global action array, and cpu_time[i] represents the time budget of the partition;

[0033] This allows us to construct a Markov decision process and build a virtual environment.

[0034] Furthermore, the combination of value-based reinforcement learning algorithms with multi-head attention mechanisms, through the interaction between the agent and the virtual environment, learns the optimal state-action value function, specifically including:

[0035] The value-based reinforcement learning algorithm is the DDQN algorithm. During the learning process, each agent constructs an evaluation network and a target network. The evaluation network is used to generate a scheduling policy for the task. The input of the evaluation network is the state s, and the output is the scheduling value function Q(s,a,ω) calculated based on the state, where ω represents the parameters of the evaluation network. The target network is used to evaluate the scheduling value function Q(s',a',ω') of the next state. The scheduling value function Q(s',a',ω') of the next state will be used as the target value for the evaluation network to learn, where ω' is the parameter of the target network.

[0036] The DDQN algorithm includes:

[0037] First, initialize the experience replay pool and the parameters ω of the evaluation network for all agents, as well as the parameters ω' of the target network; the experience replay pool is used to store data generated by the interaction between agents and the environment, which exists in the form of quadruples, represented as (s t ,a t ,R t ,s t+1 );

[0038] Subsequently, each agent interacts with the environment. For any agent, it receives the current state s as input, selects action a using the ε-greedy exploration strategy, and executes action a. The ε-greedy exploration strategy is expressed as:

[0039]

[0040] In the formula, a represents the action, argmaxQ(s,a) represents choosing the action with the largest Q value in the current state s, p represents the probability of selection, and ε represents the minimum exploration rate;

[0041] When selecting the next action, the evaluation network is used to find the action that maximizes the Q value based on the current parameter ω, a' = argmaxQ(s',a',ω), where a' represents the maximum Q value of all possible actions under the next state s', s' represents the next state, and ω represents the parameters of the policy network.

[0042] Then, observe the environment and obtain the new state s. t+1 and instant rewards R t ;

[0043] After all agents have completed one round of interaction with the environment, a batch of quadruplets is randomly selected from the experience replay pool. The agents use the evaluation network to calculate and output the scheduling value function Q(s,a,ω) of the current state, and use the target network to calculate the scheduling value function Q(s',a',ω') of the next state. Subsequently, the agents construct a loss function based on these calculation results and update the evaluation network parameters through backpropagation and gradient descent. After a certain number of iterations, the evaluation network parameters are copied to the target network, and the parameters of the target network are updated. This series of steps is repeated until the preset maximum number of iterations or the task termination condition is reached.

[0044] The agent constructs a loss function based on these calculation results, and updates and evaluates the network parameters through backpropagation and gradient descent, expressed as:

[0045] The time difference error is used to update the parameters of the scheduling value function, and the formula is: L(θ)=|Q e (s t ,a,θ)-Q ti |;In the formula, θ represents the set of parameters of the network, s t Indicates the input state, 'a' represents the action, and 'Q' represents the input state. e (s t (a, θ) represents the expression used to estimate the state s in a given state. t Q is the Q value when performing action a under parameter θ. ti The desired target Q value is expressed as:

[0046] Q ti =R t +λQ(s t+1 ,argmaxQ(s t+1 ,a t+1 ,ω),ω')

[0047] In the formula, λ represents the discount factor;

[0048] A multi-head attention mechanism is introduced in both the evaluation network and the target network.

[0049] Furthermore, both the evaluation network and the target network include an encoder and a decoder;

[0050] The encoder consists of two ISAB modules, used to encode the input data and capture the relationships between elements in the set;

[0051] The decoder performs regularization through a Dropout layer, pooling through a PMA module, extracts a fixed number of seeds, performs regularization again through a Dropout layer, and finally maps the pooled result to the output dimension through a fully connected layer.

[0052] Each ISAB module includes two MAB modules. The first MAB module is used to handle the relationship between the induction point set and the input set, and the second MAB module is used to handle the relationship between the updated input set and the induction point set. The PMA module is a pooling module implemented through the MAB module. It is used to extract a fixed number of seeds from the input data and interact these seeds with the input data through a multi-head attention mechanism.

[0053] The MAB module is the most basic attention block. It receives a query Q, a key K, and a value V, and calculates the output through a multi-head attention mechanism.

[0054] The calculation of output through a multi-head attention mechanism includes:

[0055] The query Q, key K, and value V are split into multiple heads along the last feature dimension;

[0056] For each head, the dot product of query Q and key K is computed independently, and then the attention weight A is obtained by applying the softmax function;

[0057] The value V is weighted and summed using the attention weight A to obtain the output of each head;

[0058] The outputs of all heads are concatenated in the last dimension, and a linear transformation is applied to perform a dimension transformation to obtain the final multi-head attention output.

[0059] This invention discloses a schedulable analysis method for integrated avionics systems based on reinforcement learning, comprising the following steps:

[0060] Step 1: Design a Markov decision process based on the schedulability problem of integrated avionics systems and construct a virtual environment;

[0061] Step 2: Combine policy-based reinforcement learning algorithms with multi-head attention mechanisms, and learn the optimal policy function by having the agent interact with the virtual environment;

[0062] Step 3: When the agent interacts with the environment, it uses the optimal policy function obtained in Step 2 to obtain several pre-selected actions. Based on the predicted probability distribution, it generates a candidate action sequence for each pre-selected action and records the corresponding state transition and reward update. It sorts all candidate action sequences in descending order of score, records and returns the candidate action sequence with the highest score and its corresponding reward, and executes the candidate action sequence with the highest score.

[0063] Furthermore, the design of a Markov decision process based on the schedulability problem of integrated avionics systems and the construction of a virtual environment specifically include:

[0064] Modeling the schedulability problem of integrated avionics systems involves the following steps:

[0065] The time budget required for each partition to execute tasks on each core is calculated using the following formula, expressed as:

[0066]

[0067] In the formula, Indicates partition p i The time budget required for tasks assigned to core C, C SW This represents the overhead of context switching from the completion of a task to the next scheduled task;

[0068]

[0069] R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, where i represents the partition code, k represents the frame in which the partition is located, and P i T represents the period of the partition. j Indicates the duration of the task;

[0070] Compare the time budgets of all partitions on the core, and take the maximum value as the total time budget for the partition. The specific calculation formula is as follows:

[0071]

[0072] In the formula, Indicates partition p i The total time budget for the k-th frame, where c represents the core and N c Indicates the number of cores;

[0073] The MAF (Main Time Frame) is the primary time frame, and the MIF (Minimum Time Frame) is the secondary time frame. The MAF is divided into frames of equal length, with one MIF constituting one frame. The number of MIFs is nFrames. The lengths of both the MAF and MIF are related to the period, which is always an integer multiple of the MIF length, expressed as:

[0074]

[0075]

[0076] In the formula, p i This represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor;

[0077] Based on the schedulable model of integrated avionics systems, a Markov decision process is constructed, and a virtual environment is built. Specific operations include:

[0078] In solving the scheduling problem, the available actions in each state are to allocate a start time to the current partition, and the action space is defined as: a t ={v}, v∈[0,MAF]; where v represents the start time of the current partition allocation;

[0079] To determine the start time of each partition, a global action array is introduced in the state definition. This action array is used to store the action value selected each time during the solution process, which indicates the start time of each partition.

[0080] The state transition process can be described as a quadruple: (s t ,a t ,R t ,s t+1 In the formula, s t a represents the state at time step t. t Indicates that in state s t The agent selects the action to execute, R t This indicates that the agent is in state s. t Next, execute action a t The instant reward obtained afterward, s t+1 This indicates the next state the environment enters after an action is performed;

[0081] The reward function is expressed as:

[0082]

[0083] In the formula, n is the number of tasks in the scheduling problem, items_taken is the global action array, and cpu_time[i] represents the time budget of the partition;

[0084] This allows us to construct a Markov decision process and build a virtual environment.

[0085] Furthermore, the PPO algorithm is used as a policy-based reinforcement learning algorithm.

[0086] Furthermore, the PPO algorithm specifically includes:

[0087] First, the experience replay buffer, Actor network, and Critic network are initialized. The Actor network acts as a policy function, responsible for interacting with the task scheduling environment and generating scheduling actions. The Actor network receives the environment state as input and outputs actions after calculation. The Critic network evaluates the actions output by the Actor network. During training, the Critic network receives the state matrix as input and calculates the maximum reward value obtained by taking the best action in the current state to help the Actor network update its parameters.

[0088] Subsequently, each agent interacts with the environment, executes the current policy within the environment, and collects training samples (s). t ,a t ,R t ,s t+1 The collected training samples are stored in the experience replay cache.

[0089] The importance sampling ratio r is calculated according to the following formula. t (θ), represented as:

[0090]

[0091] Where, π θ (a t |s t π represents the probability of the agent taking action in state s at time t, given the current policy. θold (a t |s t ) represents the probability of the action taken under the previous strategy;

[0092] The dominance function is calculated using the following formula, expressed as:

[0093]

[0094] In the formula, Denotes the dominance function, δ t Let t represent the TD error at time step t, γ represent the discount factor, λ represent the hyperparameters based on generalized advantage estimation, and T represent the scheduling end time.

[0095] in:

[0096] δ t =R t +γV(s t+1 )-V(s t )

[0097] In the formula, V(·) represents the value function estimate of the state;

[0098] In each iteration, training samples are sampled from the experience replay buffer;

[0099] The objective function L is calculated according to the following formula. CLIP (θ):

[0100]

[0101] Here, ε represents the truncation hyperparameter, and clip(·) represents the truncation function. The truncation function clip(·) is used to constrain the truncation hyperparameter ε to ensure the scaling factor r. t The value is within the interval [1-ε, 1+ε]; at the same time, the smaller value between the untruncated and truncated targets is selected as the lower limit by the min function;

[0102] For the Critic network, the mean squared error L(μ) is used as its loss function to update the network parameters, expressed as:

[0103]

[0104] In the formula, E t This represents the expectation for time step t;

[0105] Update the network parameters of the Actor network and Critic network to minimize L. CLIP (θ) and L(μ);

[0106] The algorithm terminates when the preset maximum number of iterations or the task termination condition is reached, and outputs the trained network; a multi-head attention mechanism is introduced in both the Actor network and the Critic network.

[0107] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0108] (1) The method of the present invention considers the impact of multi-core processors and partitions on the scheduling results after the task has been assigned. It models the system, defines the required variables, and gives the Markov decision process modeling of the avionics system task scheduling problem. Based on the DDQN algorithm and PPO algorithm, it trains the neural network using randomly generated scheduling problem instances and introduces a multi-head attention mechanism to optimize the algorithm. It constructs a set of scheduling algorithms based on reinforcement learning to automatically, safely and optimally generate partition scheduling results.

[0109] (2) The method of the present invention provides a task scheduling method for avionics systems in a multi-core scenario. It uses reinforcement learning to replace the manual iteration process, which improves efficiency and saves a lot of manpower and time costs, while obtaining the expected results more safely and in an optimized manner.

[0110] (3) The method of the present invention uses a reinforcement learning method based on multi-head attention mechanism to solve the scheduling problem of avionics system. By introducing multi-head attention mechanism, the correlation between tasks can be focused on, thereby improving the scheduling effect. Attached Figure Description

[0111] Figure 1 This is a flowchart illustrating the overall schedulable analysis method for integrated avionics systems based on value-based reinforcement learning algorithms proposed in this invention.

[0112] Figure 2 This is a flowchart illustrating the overall schedulable analysis method for integrated avionics systems based on a policy-based reinforcement learning algorithm proposed in this invention.

[0113] Figure 3 The reward curves for the DQN, DDQN, and MADDQN algorithms are shown.

[0114] Figure 4 The reward curves for the PPO and MAPPO algorithms are shown. Detailed Implementation

[0115] The objects and functions of the present invention, as well as the methods for achieving these objects and functions, will be clarified by referring to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below; it can be implemented in various forms. The purpose of this specification is merely to help those skilled in the art to comprehensively understand the specific details of the invention.

[0116] Embodiments of the present invention will now be described with reference to the accompanying drawings.

[0117] Example 1:

[0118] Suppose 20 tasks are randomly generated and distributed across 8 different partitions. These tasks are then randomly assigned to different partitions on 4 cores. The time budget allocated to each partition should be determined based on the execution duration of the tasks within it, ensuring that tasks within each partition can complete on time. The time budget required for each partition is the maximum of the time budgets required for all cores.

[0119] This embodiment discloses a comprehensive avionics system schedulability analysis method based on value-based reinforcement learning algorithms, which is now combined with... Figure 1 and Figure 3 The method of this embodiment will be further described. It mainly includes the following steps:

[0120] Step 1: Model the schedulability problem of integrated avionics systems, design a Markov decision process based on the schedulability problem of integrated avionics systems, and construct a virtual environment;

[0121] The state is defined as a tuple consisting of a static part and a dynamic part, denoted as: s t =(Q, s i In the formula, s t The definition of the state is given by Q, which represents the static part and remains unchanged throughout the decision-making process. It represents a specific instance of the task scheduling problem currently under consideration. Q can be further refined into a quintuple, Q = (C, T, D, P, K), where C represents the array of task execution times, T represents the array of task periods, D represents the array of task deadlines, P represents the array of partitions in which the task resides, and K represents the array of cores in which the task resides. These arrays are all n-dimensional and describe in detail the key parameters from task 0 to task n. i This represents the dynamic part, which changes according to the current time step, indicating which partition's start time should be assigned in the current state.

[0122] In this embodiment, the time budget required for each core to run the partition is calculated based on 20 tasks to ensure that each core has enough time to complete the partition tasks, thereby ensuring that all tasks can be completed before the deadline. The formula is as follows:

[0123]

[0124] In the formula, Indicates partition p i The time budget required for tasks assigned to core C, C SW This represents the overhead of context switching from the completion of a task to the next scheduled task;

[0125]

[0126] R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, where i represents the partition code, k represents the frame in which the partition is located, and P i T represents the period of the partition. j Indicates the duration of the task.

[0127] The maximum time budget across all cores is taken as the total time budget for the partition. The specific calculation formula is as follows:

[0128]

[0129] In the formula, Indicates partition p i The total time budget for the k-th frame, where c represents the core and N c Indicates the number of cores.

[0130] In solving the scheduling problem, the available actions in each state are to allocate a start time to the current partition, and the action space is defined as: a t ={v},v∈[0,MAF]; where v represents the start time of the current partition allocation.

[0131] The MAF (Main Time Frame) is the primary time frame, and the MIF (Minimum Time Frame) is the secondary time frame. The MAF is divided into equal-length frames, with one MIF constituting one frame. The number of MIFs is nFrames. The lengths of both the MAF and MIF are related to the period, which is always an integer multiple of the MIF length. This is represented as:

[0132]

[0133]

[0134] In the formula, p i This represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor.

[0135] To record all actions taken before reaching the termination state, i.e., to determine the start time of each partition, a global action array is introduced into the state definition. This action array stores the action value selected at each step during the solution process, which is the start time of each partition.

[0136] The state transition process can be described as a quadruple: (s t ,a t ,R t ,s t+1 In the formula, s t a represents the state at time step t. t Indicates that in state s t The agent selects the action to execute, R t This indicates that the agent is in state s. t Next, execute action a t The instant reward obtained afterward, s t+1 This indicates the next state the environment enters after an action is performed.

[0137] The reward function is constructed as follows:

[0138]

[0139] In the formula, n is the number of tasks in the problem instance, items_taken is the global action array, the start time of adjacent partitions must be ensured not to be within the running time period of the previous partition, so as to ensure that the two partitions do not overlap in running, and cpu_time[i] represents the time budget of the partition.

[0140] Step 2: As Figure 1As shown, a value-based reinforcement learning algorithm is combined with a multi-head attention mechanism, and the optimal state-action value function is learned by the agent interacting with the virtual environment.

[0141] Now combined Figure 1 This paper provides a detailed explanation of the algorithm that combines value-based reinforcement learning with multi-head attention mechanism.

[0142] First, the experience replay pool and the evaluation network parameters ω and target network parameters ω' of all agents are initialized. During the learning process, each agent constructs an evaluation network and a target network, and a multi-head attention mechanism is introduced into both networks. The main function of the evaluation network is to generate a scheduling policy for the task. The input of the evaluation network is the task state space s, and its output is the scheduling value function Q(s,a,ω) calculated based on these states. The target network represents the scheduling value function Q(s',a',ω') for the next state. This value function will serve as the target value for the evaluation network to learn, and the weight parameter of the target network is ω'.

[0143] An experience replay pool is used to store experiential data generated by the agent's interactions with the environment. This experiential data is typically stored in the form of quadruples, i.e., (s t ,a t ,R t ,s t+1 ).

[0144] Subsequently, each agent interacts with the environment. For any agent, it receives the current state s as input, selects action a using the ε-greedy exploration strategy, and executes action a.

[0145] This embodiment introduces an ε-greedy exploration strategy to prevent the agent from getting stuck in local optima by focusing only on the currently known best action during training. This strategy helps the agent explore the environment more comprehensively during training and discover potentially better solutions. The formula is as follows:

[0146]

[0147] In the formula, a represents the action chosen by the agent, argmaxQ(s,a) represents choosing the action with the largest Q value in the current state s, p represents the probability of selection, and ε represents the minimum exploration rate.

[0148] When selecting the next action, it uses the evaluation network to find the action that maximizes the Q value based on the current parameter ω, a' = argmaxQ(s',a',ω), where a' represents the maximum Q value of all possible actions under the next state s', s' represents the next state, and ω represents the policy network parameters.

[0149] Then, observe the environment and obtain the new state s. t+1 and instant rewards R t The experience replay pool is used to store data generated through interactions.

[0150] After all agents have completed a round of interaction with the environment, a batch of quadruplets is randomly selected from the experience replay pool.

[0151] Then, the agent uses the evaluation network to calculate and output the scheduling value function Q(s,a,ω) for the current state, and uses the target network to calculate the scheduling value function Q(s',a',ω') for the next state. Subsequently, the agent constructs a loss function based on these calculations and updates the evaluation network parameters through backpropagation and gradient descent. This value function serves as the target value for the evaluation network's learning, and the weight parameters of the target network are ω'. After a certain number of iterations, the algorithm copies the evaluation network parameters to the target network and updates the target network parameters. This series of steps is repeated until the preset maximum number of iterations is reached or the task termination condition is met. Specifically, the parameters of the scheduling value function are updated using temporal difference error, with the formula: L(θ)=|Q e (s t ,a,θ)-Q ti |;In the formula, θ represents the set of parameters of the network, s t The input state is represented by 'a', and the selected action value is represented by 'Q'. e (s t (a, θ) represents the expression used to estimate the state s in a given state. t Q is the Q value when performing action a with parameter θ. ti This represents the desired target Q value, which is typically calculated based on the observed reward and the Q value estimate for the next state.

[0152] The formula for calculating the target Q value is:

[0153]

[0154] In the formula, R represents the Q value corresponding to taking action a in state s. t λ represents the reward value obtained by taking this action, and s represents the discount factor. t+1 ω' represents the next state, and ω' represents the target network parameters.

[0155] The evaluation network and target network used in this embodiment both consist of two main parts: an encoder and a decoder. The encoder comprises two ISAB modules used to encode the input data, capturing the relationships between elements in the set. The ISAB includes two MAB modules: the first MAB handles the relationship between the induced point set and the input set, and the second MAB handles the relationship between the updated input set and the induced point set. The decoder performs regularization through a Dropout layer, pooling through a PMA module to extract a fixed number of seeds, regularization again through a Dropout layer, and finally maps the pooled result to the output dimension through a fully connected layer. The PMA is a pooling module implemented through an MAB, used to extract a fixed number of seeds from the input data and interact these seeds with the input data through a multi-head attention mechanism.

[0156] MAB is the most basic attention block, which receives a query (Q), a key (K), and a value (V) and computes the output through a multi-head attention mechanism.

[0157] The computational process of multi-head attention mechanisms can be summarized into four steps:

[0158] The Q, K, and V components are divided into multiple heads along the last feature dimension.

[0159] For each head, the dot product of Q and K is calculated independently, and then the attention weights A are obtained by applying the softmax function.

[0160] We use the attention weights A to perform a weighted summation on V to obtain the output of each head;

[0161] The outputs of all heads are concatenated in the last dimension, and a linear transformation is applied to perform a dimension transformation to obtain the final multi-head attention output.

[0162] In this embodiment, the neural network is trained after a 2000-step exploration process; the learning rate of the current value network is set to 0.0001, and the target value network is updated every 500 iterations; the discount rate of future rewards is set to 0.99 during the calculation of the Q value; the amount of data in each training batch is set to 64 samples; in the ε-greedy exploration strategy, the ε-greedy parameter of the network is set to a minimum exploration rate ε of 0.1, and the target value network is updated every 500 steps.

[0163] Step 3: When the agent interacts with the environment, it uses the obtained optimal state-action value function to obtain the value of each action in the current state; selects the action with the highest value and executes it; obtains the next state and reward, and accumulates the reward; in this embodiment, the current state is converted into neural network input through the environment, the model predicts the value of each legal action, and the action with the largest value is selected for execution.

[0164] The environment returns the next state and reward based on the selected action, updates the state and total reward, until the termination condition is met, that is, all partition decision phases have been completed or the total reward value is negative.

[0165] Figure 3 Experimental results are shown, with curves illustrating how the algorithm's reward value changes with increasing training rounds. The results demonstrate that the reinforcement learning scheduling algorithm based on multi-head attention proposed in this invention achieves better performance than the DDQN and DQN algorithms.

[0166] Example 2:

[0167] Suppose 20 tasks are randomly generated and distributed across 8 different partitions. These tasks are then randomly assigned to different partitions on 4 cores. The time budget allocated to each partition should be determined based on the execution duration of the tasks within it, ensuring that tasks within each partition can complete on time. The time budget required for each partition is the maximum value within the budget required for each core.

[0168] This embodiment discloses a comprehensive avionics system schedulability analysis method based on policy-driven reinforcement learning algorithms, combined with... Figure 2 and Figure 4 The method of this embodiment will be further described. It mainly includes the following steps:

[0169] Step 1: Model the schedulability problem of integrated avionics systems, design a Markov decision process based on the schedulability problem of integrated avionics systems, and construct a virtual environment;

[0170] The state is defined as a tuple consisting of a static part and a dynamic part, denoted as: S t =(Q, S) i In the formula, S tThe definition of the state is given by Q, which represents the static part and remains unchanged throughout the decision-making process. It represents a specific instance of the task scheduling problem currently under consideration. Q can be further refined into a quintuple, Q = (C, T, D, P, K), where C represents the array of task execution times, T represents the array of task periods, D represents the array of task deadlines, P represents the array of partitions in which the task resides, and K represents the array of cores in which the task resides. These arrays are all n-dimensional and describe in detail the key parameters from task 0 to task n. i This represents the dynamic part, which changes according to the current time step, indicating which partition's start time should be assigned in the current state.

[0171] In this embodiment, the time budget required for each core to run the partition is calculated based on 20 tasks to ensure that each core has enough time to complete the partition tasks, thereby ensuring that all tasks can be completed before the deadline. The formula is as follows:

[0172]

[0173] In the formula, Indicates partition p i The time budget required for tasks assigned to core C, C SW This represents the overhead of context switching from the completion of a task to the next scheduled task;

[0174]

[0175] R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, where i represents the partition code, k represents the frame in which the partition is located, and P i T represents the period of the partition. j Indicates the duration of the task.

[0176] The maximum time budget across all cores is taken as the total time budget for the partition. The specific calculation formula is as follows:

[0177]

[0178] In the formula, Indicates partition p i The total time budget for the k-th frame, where c represents the core and N c Indicates the number of cores.

[0179] In solving the scheduling problem, the available actions in each state are to allocate a start time to the current partition, and the action space is defined as: a t={v},v∈[0,MAF]; where v represents the start time of the current partition allocation.

[0180] The MAF (Main Time Frame) is the primary time frame, and the MIF (Minimum Time Frame) is the secondary time frame. The MAF is divided into equal-length frames, with one MIF constituting one frame. The number of MIFs is nFrames. The lengths of both the MAF and MIF are related to the period, which is always an integer multiple of the MIF length. This is represented as:

[0181]

[0182]

[0183] In the formula, p i This represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor.

[0184] To record all actions taken before reaching the termination state, i.e., to determine the start time of each partition, a global action array is introduced into the state definition. This action array stores the action value selected each time during the solution process, which is the start time of each partition.

[0185] The state transition process can be described as a quadruple: (s t ,a t ,R t ,s t+1 In the formula, s t a represents the state at time step t. t Indicates that in state s t The agent selects the action to execute, R t This indicates that the agent is in state s. t Next, execute action a t The instant reward obtained afterward, s t+1 This indicates the next state the environment enters after an action is performed.

[0186] The reward function is constructed as follows:

[0187]

[0188] In the formula, n is the number of tasks in the problem instance, items_taken is the global action array, the start time of adjacent partitions must be ensured not to be within the running time period of the previous partition, so as to ensure that the two partitions do not overlap in running, and cpu_time[i] represents the time budget of the partition.

[0189] Step 2: As Figure 2 As shown, a policy-based reinforcement learning algorithm is combined with a multi-head attention mechanism, and the optimal policy function is learned by the agent interacting with the virtual environment.

[0190] This embodiment uses a trial-and-error approach for network training. The algorithm explores by executing a series of actions within a given Markov decision process, and obtains corresponding rewards through these actions. PPO, as a reinforcement learning algorithm, uses a quadruple (s) as the basis for its exploration process at different time steps t. t ,a t ,R t ,s t+1 The PPO algorithm uses the transition pairs θ to represent these transitions and stores this information in the algorithm's experience replay buffer M. Afterward, the PPO algorithm periodically samples these transition pairs from the experience replay buffer and uses them to update the neural network's parameters θ, gradually approximating the optimal policy π for task scheduling. Its core objective is to find a policy π that maximizes the expected reward value through policy gradient descent. In this process, the PPO algorithm continuously adjusts the policy parameters θ to achieve higher cumulative rewards in its interactions with the environment.

[0191] Now combined Figure 2 This paper provides a detailed explanation of the algorithm that combines policy-based reinforcement learning with multi-head attention mechanism.

[0192] First, the experience replay buffer, Actor network, and Critic network are initialized. The Actor network acts as the policy function, responsible for interacting with the task scheduling environment and generating scheduled actions. It receives the environment state as input and outputs the start time of the current partition execution after calculation. The Actor network can directly generate the target action, while the Critic network evaluates the actions output by the Actor network, acting as the value function. During training, the Critic network receives the state matrix as input and calculates the maximum reward value that can be obtained by taking the best action in the current state, to help the Actor network update its parameters.

[0193] Subsequently, each agent interacts with the environment, executes the current policy within the environment, and collects training samples (s). t ,a t ,R t ,s t+1 ).

[0194] The collected training samples are stored in the experience replay buffer. The importance sampling ratio r is calculated. t (θ), this importance sampling ratio is used to measure the difference between the old and new strategies, and is expressed as:

[0195]

[0196] Where, π θ (a t |st π represents the probability of the agent taking action in state s at time t, given the current policy. θold (a t |s t () represents the probability of the action taken under the previous strategy. If the difference between the old and new strategies is significant and has a large advantage function value, it is advisable to appropriately increase the update amplitude; when r t The closer the ratio is to 1, the smaller the difference between the old and new strategies.

[0197] The advantage function is calculated, which measures the quality of taking a certain action relative to the average performance in a given state. It is expressed as:

[0198]

[0199] In the formula, Denotes the dominance function, δ t Let represent the TD error at time step t, γ represent the discount factor, λ represent the hyperparameters based on generalized advantage estimation, and T represent the scheduling end time.

[0200] in:

[0201] δ t =R t +γV(s t+1 )-V(s t )

[0202] In the formula, R t V represents the reward obtained at time step t, and V(·) represents the value function estimate of the state.

[0203] In each iteration, training samples are sampled from the experience replay buffer.

[0204] Calculate the objective function L CLIP (θ), the objective function combines the ideas of importance sampling ratio and shearing mechanism, and is expressed as:

[0205]

[0206] Here, ε represents the truncation hyperparameter, and clip(·) represents the truncation function. The truncation function clip(·) is used to constrain the truncation hyperparameter ε to ensure the scaling factor r. t The value is taken in the interval [1-ε, 1+ε] to ensure convergence; at the same time, the smaller value of the untrunculated and trunculated target is selected as the lower limit by the min function.

[0207] For the Critic network, the mean squared error L(μ) is used as its loss function to update the network parameters. The iterative process aims to enable the Critic network to more accurately estimate the expected cumulative reward in a given state, thereby supporting the Actor network to learn more effectively. The formula is as follows:

[0208] L(μ)=E t [(R t -V(s t )) 2 ]

[0209] In the formula, E t This represents the expectation for time step t.

[0210] Update the network parameters of the Actor network and Critic network to minimize L. CLIP (θ) and L(μ). The algorithm terminates and outputs the trained network when the preset maximum number of iterations or the task termination condition is reached.

[0211] This embodiment uses both the Actor and Critic networks, each comprising an encoder and a decoder. The encoder consists of two ISAB modules, used to encode the input data and capture the relationships between elements in the set. The ISAB includes two MAB modules: the first MAB handles the relationship between the induced point set and the input set, and the second MAB handles the relationship between the updated input set and the induced point set. The decoder performs regularization through a Dropout layer, pooling through a PMA module to extract a fixed number of seeds, regularization again through a Dropout layer, and finally mapping the pooled result to the output dimension through a fully connected layer. The PMA is a pooling module implemented through the MAB, used to extract a fixed number of seeds from the input data and interact these seeds with the input data through a multi-head attention mechanism.

[0212] The Multi-Head Attention Block (MAB) is the most basic attention block. It receives a query (Q), a key (K), and a value (V), and computes the output using a multi-head attention mechanism. The computation process of the multi-head attention mechanism can be summarized into four steps:

[0213] The Q, K, and V components are divided into multiple heads along the last feature dimension.

[0214] For each head, the dot product of Q and K is calculated independently, and then the attention weights A are obtained by applying the softmax function.

[0215] We use the attention weights A to perform a weighted summation on V to obtain the output of each head;

[0216] The outputs of all heads are concatenated in the last dimension, and a linear transformation is applied to perform a dimension transformation to obtain the final multi-head attention output.

[0217] Step 3: When the agent interacts with the environment, it obtains several pre-selected actions using the optimal policy function. Based on the predicted probability distribution, a candidate action sequence is generated for each pre-selected action, and the corresponding state transitions and reward updates are recorded. All candidate sequences are sorted in descending order of score, and the sequence with the highest score and its corresponding reward are recorded and returned. In this embodiment, the initial state is obtained from the environment, and a sequence list `sequences` is initialized to store the state, actions, and scores during the exploration process.

[0218] For each sequence in the current sequence list, generate an action probability distribution based on the current state. For each possible action, calculate the new state and reward after performing the action, and generate candidate sequences. Sort all candidate sequences according to their scores, and select the four highest-scoring sequences as the basis for the next round of exploration.

[0219] From the final selected sequences, choose the sequences with valid actions and calculate the total reward for each sequence. Select the sequence with the highest total reward as the optimal solution.

[0220] Figure 4 Experimental results are shown, with curves illustrating how the algorithm's reward value changes with the number of training rounds. The results demonstrate that the reinforcement learning scheduling algorithm based on multi-head attention proposed in this invention achieves better performance than the PPO algorithm.

Claims

1. A method for schedulability analysis of a synthetic avionics system based on reinforcement learning, characterized in that: The method comprises the following steps: Step 1: design a Markov decision process according to the schedulable problem of the integrated avionics system, and build a virtual environment; Step 2: combine a value-based reinforcement learning algorithm with a multi-head attention mechanism, interact with the virtual environment through an agent, and learn an optimal state-action value function; Step 3: when the agent interacts with the environment, use the optimal state-action value function obtained in step 2 to obtain the value of each actionable action in the current state; select the actionable action with the highest value and execute the actionable action; obtain the next state and reward, and accumulate the reward; The Markov decision process designed according to the schedulable problem of the integrated avionics system and the virtual environment are specifically as follows: Modeling the schedulable problem of the integrated avionics system; According to the schedulable model of the integrated avionics system, a Markov decision process is constructed, and a virtual environment is built; The modeling of the schedulable problem of the integrated avionics system specifically comprises: According to the following formula, the time budget required for the partition execution task on each core is calculated, which is represented as: wherein denotes the partition p i the time budget required by the task allocated on the core c, C SW denotes the overhead corresponding to the context switch from the task just completed its execution to the next scheduled task; R j denotes the execution time of task j, pid j denotes the partition code to which the task belongs, i denotes the partition code, k denotes the frame in which the partition is located, P i denotes the period of the partition, T j denotes the period of the task; Compare the time budgets of all partitions on the core, and take the maximum value as the total time budget of the partition, and the specific calculation formula is as follows: In the formula, denotes the partition p i total time budget at the kth frame, c denotes the core where c denotes the number of cores; MAF is the main time frame, and MIF is the secondary time frame, MAF is divided into frames of equal length, one MIF is a frame, the number of MIFs is nFrames, the length of MAF and MIF is related to the period, and the period is always an integer multiple of the length of MIF, which is represented as: In the formula, p i represents the partition identification, lcm represents the least common multiple, and gcd represents the greatest common divisor.

2. The method of claim 1, wherein: The Markov decision process constructed according to the schedulable model of the integrated avionics system and the virtual environment specifically comprises: In the solution process of the scheduling problem, the optional action in each state is to assign a start time for the current partition, and the action space is defined as: t = {v}, v ∈ [0, MAF]; where v represents the start time assigned to the current partition. In order to determine the start time of each partition, a global action array is introduced in the state definition, which is used to store the selected action value in the solving process, and the action value refers to the start time of each partition; A state transition is described as a four-tuple: (s t ,a t ,R t ,s t+1 ); where s t represents the state at time step t, a t represents the action chosen by the agent to perform in state s t , R t represents the immediate reward obtained by the agent after performing action a t in state s t , and s t+1 represents the next state entered by the environment after performing the action; The reward function is represented as: In the formula, n is the number of tasks in the scheduling problem, items_taken is the global action array, and cpu_time[i] represents the time budget of the partition; The Markov decision process is constructed, and the virtual environment is built.

3. The method of claim 2, wherein: The value-based reinforcement learning algorithm is combined with the multi-head attention mechanism, the agent interacts with the virtual environment, and the optimal state-action value function is learned, specifically comprising: The value-based reinforcement learning algorithm is a DDQN algorithm, in the learning process, each agent will build an evaluation network and a target network, the evaluation network is used to generate a scheduling policy for the task, the input of the evaluation network is the state s, and the output is the scheduling value function Q(s, a, ω) calculated based on the state, ω represents the parameters of the evaluation network, and the target network is used to evaluate the scheduling value function Q(s', a', ω') of the next state, the scheduling value function Q(s', a', ω') of the next state will be used as the target value of the evaluation network learning, and ω' is the parameter of the target network; The DDQN algorithm comprises: First, the parameters ω of the evaluation network and the parameters ω' of the target network of all agents are initialized, and an experience replay pool is initialized, which is used to store the data generated by the interaction between the agent and the environment, which is in the form of a four-tuple, denoted as (s t ,a t ,R t ,s t+1 ). Subsequently, each agent interacts with the environment, for any agent, receives the current state s as input, selects an action a using an ε-greedy exploration strategy, and executes the action a; the ε-greedy exploration strategy is represented as: In the formula, a represents an action, arg max Q(s,a) represents selecting an action with the maximum Q value under the current state s, p represents a selection probability, and ε represents a minimum exploration rate; When selecting the next action, the evaluation network is used to find the action a' = arg max Q(s',a',ω) with the maximum Q value based on the current parameters ω, in which a' represents the maximum Q value of all possible actions under the next state s', s' represents the next state, and ω represents the parameters of the policy network; Then, the environment is observed and a new state s is acquired t+1 and the immediate reward R t ; After all agents complete a round of interaction with the environment, a batch of four-tuples is randomly selected from the experience replay pool; the agent calculates the scheduling value function Q(s,a,ω) of the current state using the evaluation network and outputs it, and calculates the scheduling value function Q(s',a',ω') of the next state using the target network; subsequently, the agent constructs a loss function based on these calculation results, updates the evaluation network parameters through backpropagation and gradient descent; after a certain number of iterations, the evaluation network parameters are copied to the target network to update the parameters of the target network, and this series of steps is executed in a loop until a preset maximum number of iterations or a task termination condition is reached; The agent constructs a loss function based on these calculation results, updates the evaluation network parameters through backpropagation and gradient descent, and is represented as: The time difference error is used to update the parameters of the scheduling value function, and the formula is: L(θ)=|Q e (s t ,a,θ)-Q ti |;In the formula, θ represents the set of parameters of the network, s t Indicates the input state, 'a' represents the action, and 'Q' represents the input state. e (s t (a, θ) represents the expression used to estimate the state s in a given state. t Q is the Q value when performing action a under parameter θ. ti The desired target Q value is expressed as: Q ti = R t + λQ(s t+1 , argmaxQ(s t+1 , a t+1 , ω), ω') In the formula, λ represents a discount factor; A multi-head attention mechanism is introduced into the evaluation network and the target network.

4. The method of claim 3, wherein: The evaluation network and the target network both include an encoder and a decoder; The encoder is composed of two ISAB modules and is used to encode the input data and capture the relationships between elements in the set; The decoder is regularized by a Dropout layer, pooled by a PMA module to extract a fixed number of seeds, regularized again by a Dropout layer, and finally mapped to the output dimension by a fully connected layer; Each ISAB module includes two MAB modules, the first MAB module is used to process the relationship between the induced point set and the input set, and the second MAB module is used to process the relationship between the updated input set and the induced point set; the PMA module is a pooling module implemented by a MAB module, which is used to extract a fixed number of seeds from the input data and interact with the input data through a multi-head attention mechanism; The MAB module is the most basic attention block, which receives query Q, key K, and value V and calculates the output through a multi-head attention mechanism; The output is calculated through a multi-head attention mechanism, which includes: Divide the query Q, key K, and value V into multiple heads in the last feature dimension; Independently calculate the dot product of the query Q and the key K for each head, and then apply the softmax function to obtain the attention weight A; Weighted sum the value V using the attention weight A to obtain the output of each head; The outputs of all heads are concatenated along the last dimension, dimensionally transformed by a linear transformation, and the final multi-head attention output is obtained.

5. A method for schedulability analysis of a synthetic avionics system based on reinforcement learning, characterized in that: The method comprises the following steps: Step 1: design a Markov decision process according to the schedulability problem of the integrated avionics system, and construct a virtual environment; Step 2: combine a policy-based reinforcement learning algorithm with a multi-head attention mechanism, interact with the virtual environment through an agent, and learn an optimal policy function; Step 3: when the agent interacts with the environment, use the optimal policy function obtained in step 2 to obtain a plurality of preselected actions, generate a candidate action sequence for each preselected action according to the predicted probability distribution, and record the corresponding state transition and reward update; sort all candidate action sequences in descending order of scores, record and return the candidate action sequence with the highest score and the corresponding reward, and execute the candidate action sequence with the highest score; The Markov decision process is designed according to the schedulability problem of the integrated avionics system, and the virtual environment is constructed, and the specific operations include: Modeling the schedulability problem of the integrated avionics system, the specific operations include: The time budget required for partition execution tasks on each core is calculated according to the following formula, denoted as: wherein denotes the partition p i the time budget required by the task allocated on the core c, C SW denotes the overhead corresponding to the context switch from the task just completed its execution to the next scheduled task; R j denotes the execution time of task j, pid j denotes the partition code to which the task belongs, i denotes the partition code, k denotes the frame in which the partition is located, P i denotes the period of the partition, T j denotes the period of the task; Compare the time budgets of all partitions on the core, and take the maximum value as the total time budget of the partition, and the specific calculation formula is as follows: In the formula, denotes the partition p i total time budget at the kth frame, c denotes the core where c denotes the number of cores; MAF is the main time frame, and MIF is the secondary time frame, MAF is divided into frames of equal length, one MIF is a frame, the number of MIFs is nFrames, the length of MAF and MIF is related to the period, and the period is always an integer multiple of the length of MIF, denoted as: In the formula, p i represents the partition identification, lcm represents the least common multiple, and gcd represents the greatest common divisor. According to the schedulability model of the integrated avionics system, a Markov decision process is constructed, and a virtual environment is constructed, and the specific operations include: In the solution process of the scheduling problem, the optional action in each state is to assign a start time for the current partition, and the action space is defined as: t = {v}, v ∈ [0, MAF]; where v represents the start time assigned to the current partition. In order to determine the start time of each partition, a global action array is introduced in the state definition, which is used to store the selected action value in the solving process, and the action value refers to the start time of each partition; A state transition is described as a four-tuple: (s t ,a t ,R t ,s t+1 ) ; where s t represents the state at time step t, a t represents the action chosen by the agent to perform in state s t , R t represents the immediate reward obtained by the agent after performing action a t in state s t , and s t+1 represents the next state entered by the environment after performing the action; The reward function is represented as: In the formula, n is the number of tasks in the scheduling problem, items_taken is a global action array, and cpu_time[i] represents the time budget of the partition; A Markov decision process is constructed, and a virtual environment is constructed.

6. The method of claim 5, wherein: The PPO algorithm is used as the policy-based reinforcement learning algorithm.

7. The method of claim 6, wherein: The PPO algorithm specifically includes: First, initialize the experience replay buffer and the Actor network and Critic network, the Actor network plays the role of the policy function, is responsible for interacting with the task scheduling environment and generating scheduling actions, the Actor network receives the environment state as input, and outputs the action after calculation, and the Critic network evaluates the action output by the Actor network; during the training process, the Critic network receives a state matrix as input, and calculates the maximum reward value obtained by taking the optimal action in the current state, to help the Actor network update parameters; Subsequently, each agent interacts with the environment, executes the current policy in the environment, collects training samples (s t ,a t , R t ,s t+1 ) and stores the collected training samples into an experience replay buffer; The importance sampling ratio r is calculated according to the following formula t (θ), is expressed as: wherein, π θ (a t |s t ) represents the behavior probability of the current policy when the agent is in state s at time t, π θold (a t |s t ) represents the behavior probability of the previous policy; The advantage function is calculated according to the following formula, denoted as: wherein represents the advantage function, δ t represents the TD error at time step t, γ represents the discount factor, λ represents a hyperparameter based on generalized advantage estimation, and T represents the time at which the schedule ends. Wherein: delta t = R t + gamma V(s t+1 ) - V(s t ) In the formula, V(·) represents the value function estimate of the state; In each iteration, a training sample is sampled from the experience replay buffer; The objective function L is calculated according to the following formula CLIP (θ): wherein ε denotes a clipping hyper-parameter, and clip(·) denotes a clipping function, and the clipping function clip(·) is used to limit the clipping hyper-parameter ε, to ensure that the proportion r t is in the interval [1-ε, 1+ε]; at the same time, the minimum function is used to select the smaller value between the unclipped target and the clipped target as the lower limit; For the Critic network, the mean square error L(μ) is used as its loss function to update the network parameters, denoted as: L(μ) = E t [(R t -V(s t )) 2 ] In the formula, E t denotes denotes the expectation over time step t; updating network parameters of the Actor network and the Critic network to minimize L CLIP (θ) and L(μ); When the preset maximum number of iterations or the task termination condition is reached, the algorithm ends and outputs the trained network; the multi-head attention mechanism is introduced into both the Actor network and the Critic network.

Images