Scheduling analysis method for integrated avionics system based on reinforcement learning

Abstract

The invention provides an integrated avionics system schedulability analysis method based on reinforcement learning, and the method comprises the steps: designing a Markov decision process according to an integrated avionics system schedulability problem, and constructing a virtual environment; combining a value-based reinforcement learning algorithm with a multi-head attention mechanism, interacting with a virtual environment through an intelligent agent, and learning to obtain an optimal state-action value function; when the intelligent agent interacts with the environment, obtaining the value of each feasible action in the current state by using the optimal state-action value function; selecting the feasible action with the highest value, and executing the feasible action; acquiring a next state and rewards, and accumulating the rewards; or combining a strategy-based reinforcement learning algorithm with a multi-head attention mechanism, and learning to obtain an optimal strategy function; and obtaining a plurality of pre-selected actions by using the optimal strategy function, and executing a candidate action sequence with the highest score.

Description

Technical Field

[0001] The present invention belongs to the technical field of safety-critical software, and in particular is a schedulable analysis method for an integrated avionics system based on reinforcement learning. Background Art

[0002] In the past, aircraft systems have adopted a federated architecture: each avionics function was integrated on a separate computing resource, forming a "one function equals one computer" model. The application isolation implied by this architecture promotes safe fault isolation at runtime, but at the cost of embedding too much weight, volume, and cables in the aircraft to provide sufficient power and connectivity for all modules. As avionics systems continue to evolve, modern aircraft have embedded more intelligent functions than earlier aircraft. Compared with the old architecture, the number of new functions added to modern systems has increased exponentially, making the "one function equals one computer" model and its required computing resources, weight, and power consumption no longer feasible.

[0003] To meet this challenge, the Integrated Modular Avionics (IMA) architecture was born. Under the IMA architecture, multiple avionics functions can share computing resources on the same computer. IMA not only effectively reduces the space, weight and power consumption required for the aircraft, but also integrates multiple functions into one module to share components such as processing resources, power supply and I / O management services, further reducing the system's heat dissipation and fuel consumption. In addition, another major advantage of the IMA architecture is its versatility, which is conducive to achieving the independence of specific applications from other applications and hardware platforms during the design phase, thereby promoting the flexibility and scalability of the system.

[0004] Currently, the management and scheduling of avionics systems are mostly done manually, without effectively considering the time delays that may occur due to resource competition in a multi-core processor environment. As a result, once the system changes or needs to be upgraded, the existing task allocation and scheduling procedures must be replanned, which is a very time-consuming and labor-intensive process. Summary of the invention

[0005] Purpose of the invention: In order to solve the problems of low efficiency, large amount of manpower and time cost in task scheduling of integrated avionics systems on multi-core processors, the present invention proposes a schedulable analysis method for integrated avionics systems based on reinforcement learning. It adopts reinforcement learning based on multi-head attention mechanism to automatically, safely and optimally obtain the expected scheduling results, reduce the manual iteration process, and improve the efficiency of task scheduling generation.

[0006] Technical solution: A comprehensive avionics system schedulability analysis method based on reinforcement learning, comprising the following steps:

[0007] Step 1: Design a Markov decision process based on the schedulability problem of the integrated avionics system and build a virtual environment;

[0008] Step 2: Combine the value-based reinforcement learning algorithm with the multi-head attention mechanism, and learn the optimal state-action value function through the interaction between the intelligent agent and the virtual environment;

[0009] Step 3: When the agent interacts with the environment, use the optimal state-action value function obtained in step 2 to obtain the value of each action in the current state; select the action with the highest value and execute it; obtain the next state and reward, and accumulate the rewards.

[0010] Furthermore, the Markov decision process is designed according to the schedulable problem of the integrated avionics system, and the virtual environment is constructed, which specifically includes:

[0011] Modeling the schedulability problem of integrated avionics systems;

[0012] According to the schedulable model of integrated avionics system, a Markov decision process is constructed and a virtual environment is built.

[0013] Furthermore, the modeling of the schedulability problem of the integrated avionics system specifically includes:

[0014] According to the following formula, the time budget required for partition execution tasks on each core is calculated, which is expressed as:

[0015]

[0016] In the formula, Represents partition p i The time budget required for the task assigned to core c, C SW represents the cost of context switching from the task that just completed its execution to the next scheduled task;

[0017]

[0018] R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, i indicates the partition code, k indicates the frame where the partition is located, P i represents the partition period, T j Indicates the period of the task;

[0019] Compare the time budgets of all partitions on the core and take the maximum value as the total time budget of the partition. The specific calculation formula is as follows:

[0020]

[0021] In the formula, Represents partition p i The total time budget in the kth frame, c represents the core, N c Indicates the number of cores;

[0022] MAF is the main time frame, MIF is the secondary time frame, MAF is divided into frames of equal length, one MIF is one frame, the number of MIFs is nFrames, the length of MAF and MIF is related to the period, the period is always an integer multiple of the length of MIF, expressed as:

[0023]

[0024]

[0025] In the formula, p i Represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor.

[0026] Furthermore, the Markov decision process is constructed based on the integrated avionics system schedulable model, and the virtual environment is constructed. The specific operations include:

[0027] In the process of solving the scheduling problem, the optional action in each state is to assign a start time to the current partition, and the action space is defined as: a t ={v},v∈[0,MAF]; where v represents the start time of the current partition allocation;

[0028] In order to determine the start time of each partition, a global action array is introduced in the state definition. The action array is used to store the action value selected each time during the solution process. The action value refers to the start time of each partition.

[0029] The state transition process is described as a four-tuple: (s t ,a t ,R t ,s t+1 );where s t represents the state at time step t, a t Indicates that in state s t The next agent chooses the action to perform, R t Indicates that the agent is in state s t Next, perform action a t The immediate reward after t+1 Indicates the next state the environment enters after executing the action;

[0030] The reward function is expressed as:

[0031]

[0032] Where n is the number of tasks in the scheduling problem, items_taken is the global action array, and cpu_time[i] represents the time budget of the partition;

[0033] In this way, a Markov decision process is constructed and a virtual environment is built.

[0034] Furthermore, the value-based reinforcement learning algorithm is combined with the multi-head attention mechanism, and the optimal state-action value function is learned through the interaction between the intelligent agent and the virtual environment, which specifically includes:

[0035] The value-based reinforcement learning algorithm is a DDQN algorithm. During the learning process of this algorithm, each agent will build an evaluation network and a target network. The evaluation network is used to generate a scheduling strategy for the task. The input of the evaluation network is the state s, and the output is a scheduling value function Q(s,a,ω) calculated based on the state. ω represents the parameter of the evaluation network. The target network is used to evaluate the scheduling value function Q(s',a',ω') of the next state. The scheduling value function Q(s',a',ω') of the next state will be used as the target value of the evaluation network learning, and ω' is the parameter of the target network.

[0036] The DDQN algorithm includes:

[0037] First, the experience replay pool and the parameters ω of the evaluation network and ω' of the target network of all agents are initialized; the experience replay pool is used to store the data generated by the interaction between the agent and the environment. The data exists in the form of a four-tuple, which is expressed as (s t ,a t ,R t ,s t+1 );

[0038] Subsequently, each agent interacts with the environment. For any agent, the current state s is received as input, and an action a is selected using the ε-greedy exploration strategy, and the action a is executed. The ε-greedy exploration strategy is expressed as:

[0039]

[0040] In the formula, a represents the action, argmaxQ(s,a) represents the action with the largest Q value in the current state s, p represents the probability of selection, and ε represents the minimum exploration rate;

[0041] When selecting the next action, use the evaluation network to find the action a'=argmaxQ(s',a',ω) that maximizes the Q value based on the current parameter ω, where a' represents the maximum Q value of all possible actions in the next state s', s' represents the next state, and ω represents the parameters of the policy network;

[0042] Then, observe the environment and obtain the new state s t+1 and instant reward R t ;

[0043] When all agents complete a round of interaction with the environment, a batch of quadruplets are randomly selected from the experience replay pool; the agent uses the evaluation network to calculate the scheduling value function Q(s, a, ω) of the current state and outputs it, and uses the target network to calculate the scheduling value function Q(s', a', ω') of the next state; then, the agent constructs a loss function based on these calculation results, and updates the evaluation network parameters through back propagation and gradient descent; after a certain number of iterations, the evaluation network parameters are copied to the target network, and the parameters of the target network are updated. This series of steps is executed repeatedly until the preset maximum number of iterations or the task termination condition is reached;

[0044] The agent constructs a loss function based on these calculation results and updates the evaluation network parameters through back propagation and gradient descent, which is expressed as:

[0045] The time difference error is used to update the parameters of the scheduling value function, and its formula is: L(θ)=|Q e (s t ,a,θ)-Q ti |; where θ represents the parameter set of the network, s t represents the input state, a represents the action, Q e (s t ,a,θ) represents the value used to estimate the t and parameter θ, the Q value when executing action a, Q ti It represents the target Q value that we expect to achieve, expressed as:

[0046] Q ti =R t +λQ(s t+1 ,argmaxQ(s t+1 ,a t+1 ,ω),ω')

[0047] In the formula, λ represents the discount coefficient;

[0048] A multi-head attention mechanism is introduced in both the evaluation network and the target network.

[0049] Furthermore, the evaluation network and the target network both include an encoder and a decoder;

[0050] The encoder consists of two ISAB modules to encode the input data and capture the relationship between elements in the collection;

[0051] The decoder is regularized by a Dropout layer, pooled by a PMA module, extracts a fixed number of seeds, regularized again by a Dropout layer, and finally maps the pooled result to the output dimension by a fully connected layer;

[0052] Each ISAB module includes two MAB modules, the first MAB module is used to process the relationship between the induction point set and the input set, and the second MAB module is used to process the relationship between the updated input set and the induction point set; the PMA module is a pooling module implemented by the MAB module, which is used to extract a fixed number of seeds from the input data and interact these seeds with the input data through a multi-head attention mechanism;

[0053] The MAB module is the most basic attention block, which receives the query Q, key K and value V, and calculates the output through the multi-head attention mechanism;

[0054] The output is calculated by the multi-head attention mechanism, including:

[0055] Split the query Q, key K, and value V into multiple heads along the last feature dimension;

[0056] For each head, the dot product of the query Q and the key K is calculated independently, and then the softmax function is applied to obtain the attention weight A;

[0057] Use the attention weight A to weight the value V and get the output of each head;

[0058] The outputs of all heads are concatenated in the last dimension and transformed through a linear transformation to obtain the final multi-head attention output.

[0059] The present invention discloses a schedulable analysis method for an integrated avionics system based on reinforcement learning, comprising the following steps:

[0060] Step 1: Design a Markov decision process based on the schedulability problem of the integrated avionics system and build a virtual environment;

[0061] Step 2: Combine the policy-based reinforcement learning algorithm with the multi-head attention mechanism, and learn the optimal policy function through the interaction between the agent and the virtual environment;

[0062] Step 3: When the agent interacts with the environment, use the optimal policy function obtained in step 2 to obtain several pre-selected actions. According to the predicted probability distribution, generate a candidate action sequence for each pre-selected action, and record the corresponding state transition and reward update; sort all candidate action sequences in descending order by score, record and return the candidate action sequence with the highest score and its corresponding reward, and execute the candidate action sequence with the highest score.

[0063] Furthermore, the Markov decision process is designed according to the schedulable problem of the integrated avionics system, and the virtual environment is constructed, which specifically includes:

[0064] Modeling the schedulability problem of an integrated avionics system includes:

[0065] According to the following formula, the time budget required for partition execution tasks on each core is calculated, which is expressed as:

[0066]

[0067] In the formula, Represents partition p i The time budget required for the task assigned to core c, C SW represents the cost of context switching from the task that just completed its execution to the next scheduled task;

[0068]

[0069] R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, i indicates the partition code, k indicates the frame where the partition is located, P i represents the partition period, T j Indicates the period of the task;

[0070] Compare the time budgets of all partitions on the core and take the maximum value as the total time budget of the partition. The specific calculation formula is as follows:

[0071]

[0072] In the formula, Represents partition p i The total time budget in the kth frame, c represents the core, N c Indicates the number of cores;

[0073] MAF is the main time frame, MIF is the secondary time frame, MAF is divided into frames of equal length, one MIF is one frame, the number of MIFs is nFrames, the length of MAF and MIF is related to the period, the period is always an integer multiple of the length of MIF, expressed as:

[0074]

[0075]

[0076] In the formula, p i represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor;

[0077] According to the schedulable model of the integrated avionics system, a Markov decision process is constructed and a virtual environment is built. The specific operations include:

[0078] In the process of solving the scheduling problem, the optional action in each state is to assign a start time to the current partition, and the action space is defined as: a t ={v},v∈[0,MAF]; where v represents the start time of the current partition allocation;

[0079] In order to determine the start time of each partition, a global action array is introduced in the state definition. The action array is used to store the action value selected each time during the solution process. The action value refers to the start time of each partition.

[0080] The state transition process is described as a four-tuple: (s t ,a t ,R t ,s t+1 );where s t represents the state at time step t, a t Indicates that in state s t The next agent chooses the action to perform, R t Indicates that the agent is in state s t Next, perform action a t The immediate reward after t+1 Indicates the next state the environment enters after executing the action;

[0081] The reward function is expressed as:

[0082]

[0083] Where n is the number of tasks in the scheduling problem, items_taken is the global action array, and cpu_time[i] represents the time budget of the partition;

[0084] In this way, a Markov decision process is constructed and a virtual environment is built.

[0085] Furthermore, the PPO algorithm is used as a policy-based reinforcement learning algorithm.

[0086] Furthermore, the PPO algorithm specifically includes:

[0087] First, initialize the experience replay buffer as well as the Actor network and the Critic network. The Actor network plays the role of a policy function, responsible for interacting with the task scheduling environment and generating scheduling actions. The Actor network receives the environment state as input and outputs actions after calculation. The Critic network evaluates the actions output by the Actor network. During the training process, the Critic network receives the state matrix as input and calculates the maximum reward value obtained by taking the best action in the current state to help the Actor network update parameters.

[0088] Subsequently, each agent interacts with the environment, executes the current strategy in the environment, and collects training samples (s t ,a t ,R t ,s t+1 ), storing the collected training samples in the experience replay buffer;

[0089] The importance sampling ratio r is calculated according to the following formula t (θ), expressed as:

[0090]

[0091] Among them, π θ (a t |s t ) represents the probability of the action taken by the agent in state s at time t, π θold (a t |s t ) represents the probability of the action taken by the previous strategy;

[0092] The advantage function is calculated as follows:

[0093]

[0094] In the formula, represents the advantage function, δ t represents the TD error at time step t, γ represents the discount factor, λ represents the hyperparameter based on generalized advantage estimation, and T represents the time when the scheduling ends;

[0095] in:

[0096] δ t =R t +γV(s t+1 )-V(s t )

[0097] In the formula, V(·) represents the value function estimate of the state;

[0098] In each iteration, training samples are sampled from the experience replay buffer;

[0099] The objective function L is calculated as follows: CLIP (θ):

[0100]

[0101] Among them, ε represents the truncation hyperparameter, clip(·) represents the truncation function, and the truncation function clip(·) is used to limit the truncation hyperparameter ε to ensure that the ratio r t The value is in the interval [1-ε,1+ε]; at the same time, the smaller value of the untruncated and truncated target is selected as the lower limit through the min function;

[0102] For the Critic network, the mean square error L(μ) is used as its loss function to update the network parameters, which is expressed as:

[0103]

[0104] In the formula, E t represents the expectation at time step t;

[0105] Update the network parameters of the Actor network and the Critic network to minimize L CLIP (θ) and L(μ);

[0106] When the preset maximum number of iterations or the task termination condition is reached, the algorithm ends and outputs the trained network; a multi-head attention mechanism is introduced in both the Actor network and the Critic network.

[0107] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0108] (1) After the task has been assigned, the method of the present invention considers the impact of multi-core processors and partitions on the scheduling results, models the system, defines the required variables, and provides a Markov decision process modeling of the avionics system task scheduling problem. Based on the DDQN algorithm and the PPO algorithm, the neural network is trained using randomly generated scheduling problem instances, and the multi-head attention mechanism is introduced to optimize the algorithm. A scheduling algorithm based on reinforcement learning is constructed to automatically, safely, and optimally generate partition scheduling results.

[0109] (2) The method of the present invention provides a task scheduling method in a multi-core scenario for an avionics system, which uses reinforcement learning to replace the manual iteration process, thereby improving efficiency and saving a lot of manpower and time costs, while achieving the expected results more safely and optimally;

[0110] (3) The method of the present invention uses a reinforcement learning method based on a multi-head attention mechanism to solve the avionics system scheduling problem. The introduction of a multi-head attention mechanism can focus on the correlation between tasks and improve the scheduling effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Figure 1 The overall flow chart of the integrated avionics system schedulability analysis method based on a value-based reinforcement learning algorithm proposed by the present invention;

[0112] Figure 2 The overall flow chart of the integrated avionics system schedulability analysis method based on a policy-based reinforcement learning algorithm proposed by the present invention;

[0113] Figure 3 The reward curves for the DQN algorithm, DDQN algorithm, and MADDQN algorithm are shown below.

[0114] Figure 4 This is the reward curve diagram of the PPO algorithm and the MAPPO algorithm. DETAILED DESCRIPTION

[0115] By referring to the exemplary embodiments, the objects and functions of the present invention and the methods for achieving these objects and functions will be explained. However, the present invention is not limited to the exemplary embodiments disclosed below; it can be implemented in different forms. The essence of the description is only to help those skilled in the relevant art to comprehensively understand the specific details of the present invention.

[0116] Embodiments of the present invention will now be described with reference to the accompanying drawings.

[0117] Embodiment 1:

[0118] Assume that 20 tasks are randomly generated and distributed in 8 different partitions. These tasks are randomly assigned to different partitions on 4 cores. The time budget allocated to each partition should be determined based on the execution time of the tasks within it, and it is important to ensure that the tasks in each partition can be completed on time. The time budget required for a partition is the maximum value of the time budget required for each core.

[0119] This embodiment discloses a comprehensive avionics system schedulability analysis method based on a value-based reinforcement learning algorithm. Figure 1 and Figure 3 The method of this embodiment is further described. It mainly includes the following steps:

[0120] Step 1: Model the schedulability problem of the integrated avionics system, design a Markov decision process based on the schedulability problem of the integrated avionics system, and construct a virtual environment;

[0121] The state is defined as a tuple consisting of a static part and a dynamic part, represented as: t =(Q,s i );where s t Represents the definition of state, Q represents the static part and remains unchanged throughout the decision-making process. It represents a specific instance of the task scheduling problem currently being considered. Q can be further refined into a five-tuple, namely Q = (C, T, D, P, K), where C represents the array of task execution time, T represents the array of task cycles, D represents the array of task deadlines, P represents the array of task partitions, and K represents the array of task cores. These arrays are all n-dimensional and describe in detail the key parameters from task 0 to task n. i Represents the dynamic part, which changes according to the current time step and indicates which partition's start time should be assigned in the current state.

[0122] In this embodiment, the time budget required for each core to run the partition is calculated based on 20 tasks to ensure that each core has enough time to complete the partition task, thereby ensuring that all tasks can be completed before the deadline. The formula is as follows:

[0123]

[0124] In the formula, Represents partition p i The time budget required for the task assigned to core c, C SW represents the cost of context switching from the task that just completed its execution to the next task to be scheduled;

[0125]

[0126] R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, i indicates the partition code, k indicates the frame where the partition is located, P i represents the partition period, T j Indicates the period of the task.

[0127] Compare the time budgets of all cores and take the maximum value as the total time budget of the partition. The specific calculation formula is as follows:

[0128]

[0129] In the formula, Represents partition p i The total time budget in the kth frame, c represents the core, N c Indicates the number of cores.

[0130] In the process of solving the scheduling problem, the optional action in each state is to assign a start time to the current partition, and the action space is defined as: a t ={v},v∈[0,MAF]; where v represents the start time assigned to the current partition.

[0131] MAF is the main time frame, MIF is the secondary time frame, MAF is divided into frames of equal length, one MIF is one frame, the number of MIFs is nFrames, the length of MAF and MIF is related to the period, and the period is always an integer multiple of the length of MIF. It can be expressed as:

[0132]

[0133]

[0134] In the formula, p i Represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor.

[0135] In order to record all actions taken before reaching the terminal state, that is, to determine the start time of each partition, a global action array is introduced in the state definition. This action array is used to store the action value selected each time during the solution process, that is, the start time of each partition.

[0136] The state transition process can be described as a four-tuple: (s t ,a t ,R t ,s t+1 );where s t represents the state at time step t, a t Indicates that in state s t The next agent chooses the action to perform, R t Indicates that the agent is in state s t Next, perform action a t The immediate reward after t+1 Represents the next state the environment enters after executing the action.

[0137] The specific construction of the reward function is:

[0138]

[0139] Where n is the number of tasks in the problem instance, items_taken is the global action array, the start time of adjacent partitions must be ensured not to be within the running time period of the previous partition to ensure that the two partitions do not overlap, and cpu_time[i] represents the time budget of the partition.

[0140] Step 2: If Figure 1As shown in the figure, the value-based reinforcement learning algorithm is combined with the multi-head attention mechanism, and the optimal state-action value function is learned through the interaction between the intelligent agent and the virtual environment;

[0141] Now combined Figure 1 , a detailed explanation of the algorithm that combines the value-based reinforcement learning algorithm with the multi-head attention mechanism is given.

[0142] First, initialize the experience replay pool and the evaluation network parameters ω and target network parameters ω' of all agents. During the learning process of this algorithm, each agent will build an evaluation network and a target network, and introduce a multi-head attention mechanism in these two networks. The main function of the evaluation network is to generate a scheduling strategy for the task. The input of the evaluation network is the task state space s, and its output is the scheduling value function Q(s,a,ω) calculated based on these states. The role of the target network is to represent the scheduling value function Q(s',a',ω') of the next state. This value function will serve as the target value for the evaluation network learning, and the weight parameter of the target network is ω'.

[0143] The experience replay pool is used to store the experience data generated by the interaction between the agent and the environment. These experience data usually exist in the form of four tuples, namely (s t ,a t ,R t ,s t+1 ).

[0144] Subsequently, each agent interacts with the environment. For any agent, it receives the current state s as input, selects action a using the ε-greedy exploration strategy, and executes the action a.

[0145] In order to prevent the agent from falling into the local optimal solution by focusing only on the currently known best action during the training process, this embodiment introduces the ε-greedy exploration strategy to help the agent explore the environment more comprehensively during the training process and find possible better solutions. The formula is as follows:

[0146]

[0147] In the formula, a represents the action selected by the agent, argmaxQ(s,a) means selecting the action with the largest Q value in the current state s, p represents the probability of selection, and ε represents the minimum exploration rate.

[0148] When selecting the next action, it uses the evaluation network to find the action a'=argmaxQ(s',a',ω) that maximizes the Q value based on the current parameters ω, where a' represents the maximum Q value of all possible actions in the next state s', s' represents the next state, and ω represents the policy network parameters.

[0149] Then, observe the environment and obtain the new state s t+1 and instant reward R t The experience replay pool is used to store data generated by interactions.

[0150] When all agents complete a round of interaction with the environment, a batch of quads are randomly selected from the experience replay pool.

[0151] Then, the agent uses the evaluation network to calculate the scheduling value function Q(s,a,ω) of the current state and outputs it, and uses the target network to calculate the scheduling value function Q(s',a',ω') of the next state. Subsequently, the agent constructs a loss function based on these calculation results, and updates the evaluation network parameters through back propagation and gradient descent. This value function will serve as the target value for learning the evaluation network, and the weight parameter of the target network is ω'. After a certain number of iterations, the algorithm copies the evaluation network parameters to the target network and updates the target network parameters. This series of steps is executed in a loop until the preset maximum number of iterations or the task termination condition is reached. Specifically, the time difference error is used to update the parameters of the scheduling value function, and the formula is: L(θ)=|Q e (s t ,a,θ)-Q ti |; where θ represents the parameter set of the network, s t represents the input state, a represents the selected action value, Q e (s t ,a,θ) represents the value used to estimate the t The Q value when executing action a under the condition of and parameter θ. ti represents the target Q-value that we want to achieve, which is usually calculated based on the actual observed reward and the Q-value estimate of the next state.

[0152] The calculation formula for the target Q value is:

[0153]

[0154] In the formula, represents the Q value corresponding to taking action a in state s, R t represents the reward value obtained by taking this action, λ represents the discount factor, and s t+1 represents the next state, and ω' represents the target network parameters.

[0155] The evaluation network and the target network used in this embodiment both include two main parts: an encoder and a decoder. The encoder consists of two ISAB modules, which are used to encode the input data and capture the relationship between the elements in the set. ISAB includes two MAB modules, the first MAB is used to process the relationship between the induction point set and the input set, and the second MAB is used to process the relationship between the updated input set and the induction point set. The decoder is regularized by a Dropout layer, pooled by a PMA module, extracts a fixed number of seeds, regularized again by a Dropout layer, and finally maps the pooled results to the output dimension by a fully connected layer. Among them, PMA is a pooling module implemented by MAB, which is used to extract a fixed number of seeds from the input data and interact these seeds with the input data through a multi-head attention mechanism.

[0156] MAB is the most basic attention block, which receives query (Q), key (K), and value (V) and computes the output through a multi-head attention mechanism.

[0157] The calculation process of the multi-head attention mechanism can be summarized into 4 steps:

[0158] Split Q, K, and V into multiple heads along the last feature dimension;

[0159] Calculate the dot product of Q and K for each head independently, and then apply the softmax function to get the attention weight A;

[0160] Use the attention weight A to perform weighted summation on V to get the output of each head;

[0161] The outputs of all heads are concatenated in the last dimension and transformed through a linear transformation to obtain the final multi-head attention output.

[0162] In this embodiment, the neural network training is started after setting a 2000-step exploration process; the learning rate of the current value network is set to 0.0001, and the target value network is updated every 500 iterations; in the process of calculating the Q value, the discount rate of future rewards is set to 0.99; the amount of data in each training batch is set to 64 samples; in the ε-greedy exploration strategy, the ε-greedy parameter of the network sets a minimum exploration rate ε of 0.1, and the target value network is updated every 500 steps.

[0163] Step 3: When the agent interacts with the environment, the optimal state-action value function is used to obtain the value of each available action in the current state; the available action with the highest value is selected and executed; the next state and reward are obtained, and the rewards are accumulated; in this embodiment, the current state is converted into a neural network input through the environment, the model is used to predict the value of each legal action, and the action with the largest value is selected for execution.

[0164] The environment returns the next state and reward based on the chosen action, updating the state and total reward until the termination condition is reached, when all partitions' decision stages have been completed or the total reward value is negative.

[0165] Figure 3 The experimental results are shown, and the curve describes how the reward value of the algorithm changes with the increase of training rounds. The results show that the reinforcement learning scheduling algorithm based on the multi-head attention mechanism proposed in the present invention achieves better performance than the DDQN algorithm and the DQN algorithm.

[0166] Embodiment 2:

[0167] Assume that 20 tasks are randomly generated and distributed in 8 different partitions. These tasks are randomly assigned to different partitions on 4 cores. The time budget allocated to each partition should be determined based on the execution time of the tasks within it, and it is important to ensure that the tasks in each partition can be completed on time. The time budget required for a partition is the maximum value of the budget required by each core.

[0168] This embodiment discloses a comprehensive avionics system schedulability analysis method based on a policy-based reinforcement learning algorithm, combined with Figure 2 and Figure 4 The method of this embodiment is further described. It mainly includes the following steps:

[0169] Step 1: Model the schedulability problem of the integrated avionics system, design a Markov decision process based on the schedulability problem of the integrated avionics system, and construct a virtual environment;

[0170] The state is defined as a tuple consisting of a static part and a dynamic part, represented as: S t =(Q, S i );where S tRepresents the definition of the state, Q represents the static part and remains unchanged throughout the decision-making process. It represents a specific instance of the task scheduling problem currently being considered. Q can be further refined into a five-tuple, namely Q = (C, T, D, P, K), where C represents the array of task execution time, T represents the array of task cycles, D represents the array of task deadlines, P represents the array of task partitions, and K represents the array of task cores. These arrays are all n-dimensional and describe in detail the key parameters from task 0 to task n. S i Represents the dynamic part, which changes according to the current time step and indicates which partition's start time should be assigned in the current state.

[0171] In this embodiment, the time budget required for each core to run the partition is calculated based on 20 tasks to ensure that each core has enough time to complete the partition task, thereby ensuring that all tasks can be completed before the deadline. The formula is as follows:

[0172]

[0173] In the formula, Represents partition p i The time budget required for the task assigned to core c, C SW represents the cost of context switching from the task that just completed its execution to the next task to be scheduled;

[0174]

[0175] R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, i indicates the partition code, k indicates the frame where the partition is located, P i represents the partition period, T j Indicates the period of the task.

[0176] Compare the time budgets of all cores and take the maximum value as the total time budget of the partition. The specific calculation formula is as follows:

[0177]

[0178] In the formula, Represents partition p i The total time budget in the kth frame, c represents the core, N c Indicates the number of cores.

[0179] In the process of solving the scheduling problem, the optional action in each state is to assign a start time to the current partition, and the action space is defined as: a t={v},v∈[0,MAF]; where v represents the start time assigned to the current partition.

[0180] MAF is the main time frame, MIF is the secondary time frame, MAF is divided into frames of equal length, one MIF is one frame, the number of MIFs is nFrames, the length of MAF and MIF is related to the period, and the period is always an integer multiple of the length of MIF. It can be expressed as:

[0181]

[0182]

[0183] In the formula, p i Represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor.

[0184] In order to record all actions taken before reaching the terminal state, that is, to determine the start execution time of each partition, a global action array is introduced in the state definition. This action array is used to store the action value selected each time during the solution process, that is, the start time of each partition.

[0185] The state transition process can be described as a four-tuple: (s t ,a t ,R t ,s t+1 );where s t represents the state at time step t, a t Indicates that in state s t The next agent chooses the action to perform, R t Indicates that the agent is in state s t Next, perform action a t The immediate reward after t+1 Represents the next state the environment enters after executing the action.

[0186] The specific construction of the reward function is:

[0187]

[0188] Where n is the number of tasks in the problem instance, items_taken is the global action array, the start time of adjacent partitions must be ensured not to be within the running time period of the previous partition to ensure that the two partitions do not overlap, and cpu_time[i] represents the time budget of the partition.

[0189] Step 2: If Figure 2 As shown in the figure, the policy-based reinforcement learning algorithm is combined with the multi-head attention mechanism, and the optimal policy function is learned through the interaction between the intelligent agent and the virtual environment.

[0190] This example uses a trial and error method to train the network. The algorithm is explored by executing a series of actions in a given Markov decision process, and the corresponding rewards are obtained through these actions. PPO is a reinforcement learning algorithm. Its exploration process at different time steps t is based on a four-tuple (s t ,a t ,R t ,s t+1 ) and stores this information in the algorithm's experience replay buffer M. After that, the PPO algorithm will periodically sample these transfer pairs from the experience replay buffer and use them to update the parameters θ of the neural network, gradually approaching the optimal strategy π for task scheduling. Its core goal is to find a strategy π through the policy gradient descent method to maximize the expected reward value. In this process, the PPO algorithm continuously adjusts the policy parameters θ in order to achieve higher cumulative rewards in the interaction with the environment.

[0191] Now combined Figure 2 , a detailed explanation of the algorithm that combines the policy-based reinforcement learning algorithm with the multi-head attention mechanism is given.

[0192] First, initialize the experience replay buffer as well as the Actor network and the Critic network. The Actor network plays the role of a policy function, responsible for interacting with the task scheduling environment and generating scheduling actions. It receives the environment state as input and outputs the start time of the partition execution for this scheduling after calculation. The Actor network can directly generate the target action, while the Critic network evaluates the action output by the Actor network and acts as a value function. During the training process, the Critic network receives the state matrix as input and calculates the maximum reward value that can be obtained by taking the best action in the current state to help the Actor network update parameters.

[0193] Subsequently, each agent interacts with the environment, executes the current strategy in the environment, and collects training samples (s t ,a t ,R t ,s t+1 ).

[0194] Store the collected training samples in the experience replay buffer. Calculate the importance sampling ratio r t (θ), the importance sampling ratio is used to measure the difference between the new and old strategies, expressed as:

[0195]

[0196] Among them, π θ (a t |st ) represents the probability of the action taken by the agent in state s at time t, π θold (a t |s t ) represents the probability of the behavior taken by the previous strategy. If the difference between the new and old strategies is significant and has a large advantage function value, you can consider increasing the update amplitude appropriately; when r t The closer the ratio is to 1, the smaller the difference between the new and old strategies is.

[0197] Calculate the advantage function, which is used to measure how good an action is relative to the average performance in a given state, expressed as:

[0198]

[0199] In the formula, represents the advantage function, δ t represents the TD error at time step t, γ represents the discount factor, λ represents the hyperparameter based on generalized advantage estimation, and T represents the time when the scheduling ends.

[0200] in:

[0201] δ t =R t +γV(s t+1 )-V(s t )

[0202] In the formula, R t represents the reward obtained at time step t, and V(·) represents the value function estimate of the state.

[0203] In each iteration, training examples are sampled from the experience replay buffer.

[0204] Calculate the objective function L CLIP (θ), the objective function combines the ideas of importance sampling ratio and shearing mechanism, expressed as:

[0205]

[0206] Among them, ε represents the truncation hyperparameter, clip(·) represents the truncation function, and the truncation function clip(·) is used to limit the truncation hyperparameter ε to ensure that the ratio r t The value is taken in the interval [1-ε,1+ε] to ensure convergence; at the same time, the smaller value of the untruncated and truncated target is selected as the lower limit through the min function.

[0207] For the Critic network, the mean square error L(μ) is used as its loss function to update the network parameters. The iterative process aims to enable the Critic network to more accurately estimate the expected cumulative reward under a given state, thereby supporting the Actor network to learn more effectively. The formula is as follows:

[0208] L(μ)=E t [(R t -V(s t )) 2 ]

[0209] In the formula, E t Denotes the expectation at time step t.

[0210] Update the network parameters of the Actor network and the Critic network to minimize L CLIP (θ) and L(μ). When the preset maximum number of iterations or the task termination condition is reached, the algorithm ends and the trained network is output.

[0211] The Actor network and Critic network used in this embodiment both include an encoder and a decoder. The encoder consists of two ISAB modules, which are used to encode the input data and capture the relationship between the elements in the set. ISAB includes two MAB modules, the first MAB is used to process the relationship between the induction point set and the input set, and the second MAB is used to process the relationship between the updated input set and the induction point set. The decoder is regularized by a Dropout layer, pooled by a PMA module, extracts a fixed number of seeds, regularized again by a Dropout layer, and finally maps the pooled results to the output dimension by a fully connected layer. PMA is a pooling module implemented by MAB, which is used to extract a fixed number of seeds from the input data and interact these seeds with the input data through a multi-head attention mechanism.

[0212] MAB is the most basic attention block, which receives query (Q), key (K) and value (V) and calculates the output through the multi-head attention mechanism. The calculation process of the multi-head attention mechanism can be summarized into 4 steps:

[0213] Split Q, K, and V into multiple heads along the last feature dimension;

[0214] Calculate the dot product of Q and K for each head independently, and then apply the softmax function to get the attention weight A;

[0215] Use the attention weight A to perform weighted summation on V to get the output of each head;

[0216] The outputs of all heads are concatenated in the last dimension and transformed through a linear transformation to obtain the final multi-head attention output.

[0217] Step 3: When the agent interacts with the environment, it uses the obtained optimal policy function to obtain several pre-selected actions, generates a candidate action sequence for each pre-selected action based on the predicted probability distribution, and records the corresponding state transitions and reward updates. Sort all candidate sequences in descending order of score, record and return the sequence with the highest score and its corresponding reward; in this embodiment, the initial state is obtained from the environment, and a sequence list sequences is initialized to store the state, action, and score during the exploration process.

[0218] For each sequence in the current sequence list, generate an action probability distribution based on the current state. For each possible action, calculate the new state and reward after executing the action, and generate candidate sequences. Sort all candidate sequences by score, and select the 4 sequences with the highest scores as the basis for the next round of exploration.

[0219] From the final screened sequences, select the sequences of valid actions and calculate the total reward of each sequence. The sequence with the highest total reward is selected as the best solution.

[0220] Figure 4 The experimental results are shown, and the curve describes how the reward value of the algorithm changes with the increase of training rounds. The results show that the reinforcement learning scheduling algorithm based on the multi-head attention mechanism proposed in the present invention achieves better performance than the PPO algorithm.

Claims

1. A comprehensive avionics system schedulability analysis method based on reinforcement learning, characterized by: The following steps are involved: Step 1: Design a Markov decision process based on the schedulability problem of the integrated avionics system and build a virtual environment; Step 2: Combine the value-based reinforcement learning algorithm with the multi-head attention mechanism, and learn the optimal state-action value function through the interaction between the intelligent agent and the virtual environment; Step 3: When the agent interacts with the environment, use the optimal state-action value function obtained in step 2 to obtain the value of each action in the current state; select the action with the highest value and execute it; obtain the next state and reward, and accumulate the rewards.

2. The method for schedulability analysis of integrated avionics systems based on reinforcement learning according to claim 1, characterized in that: The Markov decision process is designed according to the schedulable problem of the integrated avionics system, and the virtual environment is constructed, which specifically includes: Modeling the schedulability problem of integrated avionics systems; According to the schedulable model of integrated avionics system, a Markov decision process is constructed and a virtual environment is built.

3. The method for schedulable analysis of integrated avionics systems based on reinforcement learning according to claim 2, characterized in that: The modeling of the schedulability problem of the integrated avionics system specifically includes: According to the following formula, the time budget required for partition execution tasks on each core is calculated, which is expressed as: In the formula, Represents partition p i The time budget required for the task assigned to core c, C SW represents the cost of context switching from the task that just completed its execution to the next scheduled task; R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, i indicates the partition code, k indicates the frame where the partition is located, P i represents the partition period, T j Indicates the period of the task; Compare the time budgets of all partitions on the core and take the maximum value as the total time budget of the partition. The specific calculation formula is as follows: In the formula, Represents partition p i The total time budget in the kth frame, c represents the core, N c Indicates the number of cores; MAF is the main time frame, MIF is the secondary time frame, MAF is divided into frames of equal length, one MIF is one frame, the number of MIFs is nFrames, the length of MAF and MIF is related to the period, the period is always an integer multiple of the length of MIF, expressed as: In the formula, p i Represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor.

4. The method for schedulability analysis of integrated avionics systems based on reinforcement learning according to claim 3, characterized in that: The Markov decision process is constructed based on the schedulable model of the integrated avionics system, and the virtual environment is constructed. The specific operations include: In the process of solving the scheduling problem, the optional action in each state is to assign a start time to the current partition, and the action space is defined as: a t ={v},v∈[0,MAF]; where v represents the start time of the current partition allocation; In order to determine the start time of each partition, a global action array is introduced in the state definition. The action array is used to store the action value selected each time during the solution process. The action value refers to the start time of each partition. The state transition process is described as a four-tuple: (s t ,a t ,R t ,s t+1 );where s t represents the state at time step t, a t Indicates that in state s t The next agent chooses the action to perform, R t Indicates that the agent is in state s t Next, perform action a t The immediate reward after t+1 Indicates the next state the environment enters after executing the action; The reward function is expressed as: Where n is the number of tasks in the scheduling problem, items_taken is the global action array, and cpu_time[i] represents the time budget of the partition; In this way, a Markov decision process is constructed and a virtual environment is built.

5. The method for schedulability analysis of integrated avionics systems based on reinforcement learning according to claim 4, characterized in that: The above-mentioned combination of the value-based reinforcement learning algorithm and the multi-head attention mechanism, through the interaction between the intelligent agent and the virtual environment, learns the optimal state-action value function, specifically including: The value-based reinforcement learning algorithm is a DDQN algorithm. During the learning process of this algorithm, each agent will build an evaluation network and a target network. The evaluation network is used to generate a scheduling strategy for the task. The input of the evaluation network is the state s, and the output is a scheduling value function Q(s,a,ω) calculated based on the state. ω represents the parameter of the evaluation network. The target network is used to evaluate the scheduling value function Q(s',a',ω') of the next state. The scheduling value function Q(s',a',ω') of the next state will be used as the target value of the evaluation network learning, and ω' is the parameter of the target network. The DDQN algorithm includes: First, the experience replay pool and the parameters ω of the evaluation network and ω' of the target network of all agents are initialized; the experience replay pool is used to store the data generated by the interaction between the agent and the environment. The data exists in the form of a four-tuple, which is expressed as (s t ,a t ,R t ,s t+1 ); Subsequently, each agent interacts with the environment. For any agent, the current state s is received as input, and an action a is selected using the ε-greedy exploration strategy, and the action a is executed. The ε-greedy exploration strategy is expressed as: In the formula, a represents the action, arg maxQ(s,a) represents the action with the largest Q value in the current state s, p represents the probability of selection, and ε represents the minimum exploration rate; When selecting the next action, use the evaluation network to find the action a'=argmaxQ(s',a',ω) that maximizes the Q value based on the current parameter ω, where a' represents the maximum Q value of all possible actions in the next state s', s' represents the next state, and ω represents the parameters of the policy network; Then, observe the environment and obtain the new state s t+1 and instant reward R t ; When all agents complete a round of interaction with the environment, a batch of quadruplets are randomly selected from the experience replay pool; the agent uses the evaluation network to calculate the scheduling value function Q(s, a, ω) of the current state and outputs it, and uses the target network to calculate the scheduling value function Q(s', a', ω') of the next state; then, the agent constructs a loss function based on these calculation results, and updates the evaluation network parameters through back propagation and gradient descent; after a certain number of iterations, the evaluation network parameters are copied to the target network, and the parameters of the target network are updated. This series of steps is executed repeatedly until the preset maximum number of iterations or the task termination condition is reached; The agent constructs a loss function based on these calculation results and updates the evaluation network parameters through back propagation and gradient descent, which is expressed as: The time difference error is used to update the parameters of the scheduling value function, and its formula is: L(θ)=|Q e (s t ,a,θ)-Q ti |; where θ represents the parameter set of the network, s t represents the input state, a represents the action, Q e (s t ,a,θ) represents the value used to estimate the t and parameter θ, the Q value when executing action a, Q ti It represents the target Q value that we expect to achieve, expressed as: Q ti =R t +λQ(s t+1 ,arg maxQ(s t+1 ,a t+1 ,oh),oh') In the formula, λ represents the discount coefficient; A multi-head attention mechanism is introduced in both the evaluation network and the target network.

6. The method for schedulability analysis of integrated avionics systems based on reinforcement learning according to claim 5, characterized in that: The evaluation network and the target network both include an encoder and a decoder; The encoder consists of two ISAB modules to encode the input data and capture the relationship between elements in the collection; The decoder is regularized by a Dropout layer, pooled by a PMA module, extracts a fixed number of seeds, regularized again by a Dropout layer, and finally maps the pooled result to the output dimension by a fully connected layer; Each ISAB module includes two MAB modules, the first MAB module is used to process the relationship between the induction point set and the input set, and the second MAB module is used to process the relationship between the updated input set and the induction point set; the PMA module is a pooling module implemented by the MAB module, which is used to extract a fixed number of seeds from the input data and interact these seeds with the input data through a multi-head attention mechanism; The MAB module is the most basic attention block, which receives the query Q, key K and value V, and calculates the output through the multi-head attention mechanism; The output is calculated by the multi-head attention mechanism, including: Split the query Q, key K, and value V into multiple heads along the last feature dimension; For each head, the dot product of the query Q and the key K is calculated independently, and then the softmax function is applied to obtain the attention weight A; Use the attention weight A to weight the value V and get the output of each head; The outputs of all heads are concatenated in the last dimension and transformed through a linear transformation to obtain the final multi-head attention output.

7. A method for schedulability analysis of integrated avionics systems based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Design a Markov decision process based on the schedulability problem of the integrated avionics system and build a virtual environment; Step 2: Combine the policy-based reinforcement learning algorithm with the multi-head attention mechanism, and learn the optimal policy function through the interaction between the agent and the virtual environment; Step 3: When the agent interacts with the environment, use the optimal policy function obtained in step 2 to obtain several pre-selected actions. According to the predicted probability distribution, generate a candidate action sequence for each pre-selected action, and record the corresponding state transition and reward update; sort all candidate action sequences in descending order by score, record and return the candidate action sequence with the highest score and its corresponding reward, and execute the candidate action sequence with the highest score.

8. The method for schedulability analysis of integrated avionics systems based on reinforcement learning according to claim 7, characterized in that: The Markov decision process is designed according to the schedulable problem of the integrated avionics system, and the virtual environment is constructed, which specifically includes: Modeling the schedulability problem of an integrated avionics system includes: According to the following formula, the time budget required for partition execution tasks on each core is calculated, which is expressed as: In the formula, Represents partition p i The time budget required for the task assigned to core c, C SW represents the cost of context switching from the task that just completed its execution to the next scheduled task; R j Indicates the execution time of task j, pid j Indicates the partition code to which the task belongs, i indicates the partition code, k indicates the frame where the partition is located, P i represents the partition period, T j Indicates the period of the task; Compare the time budgets of all partitions on the core and take the maximum value as the total time budget of the partition. The specific calculation formula is as follows: In the formula, Represents partition p i The total time budget in the kth frame, c represents the core, N c Indicates the number of cores; MAF is the main time frame, MIF is the secondary time frame, MAF is divided into frames of equal length, one MIF is one frame, the number of MIFs is nFrames, the length of MAF and MIF is related to the period, the period is always an integer multiple of the length of MIF, expressed as: In the formula, p i represents the partition identifier, lcm represents the least common multiple, and gcd represents the greatest common factor; According to the schedulable model of the integrated avionics system, a Markov decision process is constructed and a virtual environment is built. The specific operations include: In the process of solving the scheduling problem, the optional action in each state is to assign a start time to the current partition, and the action space is defined as: a t ={v},v∈[0,MAF]; where v represents the start time of the current partition allocation; In order to determine the start time of each partition, a global action array is introduced in the state definition. The action array is used to store the action value selected each time during the solution process. The action value refers to the start time of each partition. The state transition process is described as a four-tuple: (s t ,a t ,R t ,s t+1 );where s t represents the state at time step t, a t Indicates that in state s t The next agent chooses the action to perform, R t Indicates that the agent is in state s t Next, perform action a t The immediate reward after t+1 Indicates the next state the environment enters after executing the action; The reward function is expressed as: Where n is the number of tasks in the scheduling problem, items_taken is the global action array, and cpu_time[i] represents the time budget of the partition; In this way, a Markov decision process is constructed and a virtual environment is built.

9. The method for schedulability analysis of integrated avionics systems based on reinforcement learning according to claim 7, characterized in that: The PPO algorithm is used as the policy-based reinforcement learning algorithm.

10. The method for schedulability analysis of integrated avionics systems based on reinforcement learning according to claim 9, characterized in that: The PPO algorithm specifically includes: First, initialize the experience replay buffer as well as the Actor network and the Critic network. The Actor network plays the role of a policy function, responsible for interacting with the task scheduling environment and generating scheduling actions. The Actor network receives the environment state as input and outputs actions after calculation. The Critic network evaluates the actions output by the Actor network. During the training process, the Critic network receives the state matrix as input and calculates the maximum reward value obtained by taking the best action in the current state to help the Actor network update parameters. Subsequently, each agent interacts with the environment, executes the current strategy in the environment, and collects training samples)s t ,a t ,R t ,s t+1 ), storing the collected training samples in the experience replay buffer; The importance sampling ratio r is calculated according to the following formula t (θ), expressed as: Among them, π θ (a t |s t ) represents the probability of the action taken by the agent in state s at time t, π θold (a t |s t ) represents the probability of the action taken by the previous strategy; The advantage function is calculated as follows: In the formula, represents the advantage function, δ t represents the TD error at time step t, γ represents the discount factor, λ represents the hyperparameter based on generalized advantage estimation, and T represents the time when the scheduling ends; in: δ t =R t +γV(s t+1 )-V(s t ) In the formula, V(·) represents the value function estimate of the state; In each iteration, training samples are sampled from the experience replay buffer; The objective function L is calculated as follows: CLIP (θ): Among them, ε represents the truncation hyperparameter, clip(·) represents the truncation function, and the truncation function clip(·) is used to limit the truncation hyperparameter ε to ensure that the ratio r t The value is in the interval [1-ε,1+ε]; at the same time, the smaller value of the untruncated and truncated target is selected as the lower limit through the min function; For the Critic network, the mean square error L(μ) is used as its loss function to update the network parameters, which is expressed as: L(μ)=E t [(R t -V(s t )) 2 ] In the formula, E t represents the expectation at time step t; Update the network parameters of the Actor network and the Critic network to minimize L CLIP (θ) and L(μ); When the preset maximum number of iterations or the task termination condition is reached, the algorithm ends and outputs the trained network; a multi-head attention mechanism is introduced in both the Actor network and the Critic network.

Images