Control model optimization method and device, equipment, storage medium and product

Abstract

The invention relates to the technical field of machine learning, and particularly provides a control model optimization method and device, equipment, a storage medium and a product. The method comprises the following steps: constructing a constraint equation of a control model according to an environment state, equipment action and expected discount cost at each moment in environment interaction data, solving an optimal solution of a target function of the control model under the condition of the constraint equation according to the environment interaction data, and obtaining a target model parameter, in this way, the expected discount reward and the expected discount cost are combined, the model is optimized, the control performance of the target control model is improved, and then the flight safety and stability of the flight equipment can be guaranteed.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and specifically to a method, device, equipment, storage medium and product for controlling model optimization. Background Art

[0002] In the flight control scenario of a flying device (such as an aircraft), a control system with a control model is usually provided to control the flying device to fly stably under various interference conditions, and to control the attitude of the flying device and then correct the trajectory to ensure the accuracy of the track.

[0003] However, due to the complexity of the flight environment, the control effect of the control model is usually not good, and it is difficult to ensure the safety and stability of the flight equipment during flight. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, equipment, storage medium and product for optimizing a control model, so as to improve the control performance of the control model and ensure the safety and stability of the flight equipment.

[0005] On the one hand, an embodiment of the present application provides a method for optimizing a control model, comprising:

[0006] Obtaining environmental interaction data of the flight equipment operating in the target environment; the environmental interaction data includes the environmental state of the target environment at each time, the equipment actions of the flight equipment at each time, and their corresponding expected discount rewards and expected discount costs;

[0007] According to the environmental state, equipment action and expected discounted cost at each moment in the environmental interaction data, the constraint equation of the control model is constructed; the control model is used to determine the equipment action performed by the flight equipment according to the environmental state; the constraint equation is used to constrain the cost corresponding to the equipment action;

[0008] According to the environmental interaction data and the constraint equation, the optimal solution of the objective function of the control model is solved, so that in the case of the optimal solution, the result of the objective function is optimal and satisfies the constraints of the constraint equation; the result of the objective function is the expected discounted cumulative reward of the control model; the optimal solution is the solved target model parameters;

[0009] According to the optimal solution, the parameters of the control model are updated to obtain the target control model.

[0010] In one implementation, the constraint equation includes a target cost constraint and a target divergence constraint;

[0011] According to the environmental state, device action and expected discounted cost at each moment in the environmental interaction data, the constraint equations of the control model are constructed, including:

[0012] Get the cost cap and discount factor at each moment;

[0013] According to the discount factor and expected discounted cost at each moment, as well as the cost upper limit, the normalized safety status of the flight equipment at each moment is determined; the normalized safety status is negatively correlated with the discount factor, the expected discounted cost, and the cost upper limit;

[0014] Determine the advantage estimate at each moment based on the environmental status and equipment actions at each moment;

[0015] Construct target cost constraints based on environmental interaction data, control models, cost caps, and normalized safety states, advantage estimates, and discount factors at each moment;

[0016] A target divergence constraint condition is constructed based on a specified control model and a control model using a divergence function; the specified control model is obtained by replacing model parameters of the control model with specified model parameters.

[0017] In one implementation, a target cost constraint is constructed based on the environmental interaction data, the cost cap, the control model, and the normalized safety state, advantage estimate, and discount factor at each moment, including:

[0018] The normalized security state at each moment is input into the activation function respectively to obtain the mapping value of the normalized security state at each moment;

[0019] Generate a first update factor at each moment according to each mapping value;

[0020] Determine the Lagrange multiplier based on the environmental interaction data, the control model, and the advantage estimate at each time;

[0021] According to the speed control coefficient and the Lagrange multiplier, a second update factor is determined; the second update factor is positively correlated with the speed control coefficient and the Lagrange multiplier;

[0022] The target cost constraint is constructed based on the environmental interaction data, the control model, the cost upper limit, the second update factor, and the first update factor, the expected discounted cost and the discount factor at each moment.

[0023] In one implementation, generating a first update factor at each time point according to each mapping value includes:

[0024] Determine the minimum value between each mapped value and the first specified value;

[0025] The sum between each minimum value and the second specified value is determined respectively to obtain the first update factor at each moment.

[0026] In one implementation, a target cost constraint condition is constructed based on the environmental interaction data, the control model, the cost upper limit, the second update factor, the first update factor at each moment, and the discount factor, including:

[0027] Determine the expected discounted cumulative cost based on the expected discounted cost at each moment;

[0028] Construct the initial target cost constraint based on the environmental interaction data, the expected discounted cumulative cost, the cost upper limit, the second update factor, and the first update factor and discount factor at each moment;

[0029] Perform first-order approximation on the initial target cost constraint to obtain the constraint gradient;

[0030] Calculate the difference between the expected discounted cumulative cost and the cost cap to obtain the residual cost;

[0031] Calculate the difference between the current model parameters of the control model and the specified model parameters to obtain the parameter change;

[0032] The target cost constraint is constructed based on the remaining cost, constraint gradient, and parameter change.

[0033] In one implementation, a divergence function is used to construct a target divergence constraint condition based on a specified control model and a control model, including:

[0034] Using a divergence function, constructing a divergence expression based on a specified control model and a control model;

[0035] According to the divergence expression, construct the initial divergence constraint condition;

[0036] Perform a second-order approximation on the initial divergence constraint to obtain the Hessian matrix;

[0037] According to the Hessian matrix and the residual cost, the target divergence constraint is constructed.

[0038] In one implementation, solving the optimal solution of the objective function of the control model based on the environmental interaction data and the constraint equation includes:

[0039] Determine the dual problem of the objective function; the dual problem includes a first variable and a second variable;

[0040] Solve the dual problem according to the environmental interaction data and the constraint equation to obtain a first variable value of the first variable and a second variable value of the second variable;

[0041] An optimal solution is determined based on initial model parameters of the control model, constraint gradients, a Hessian matrix, a first variable value, and a second variable value.

[0042] In one embodiment, the method further comprises:

[0043] If it is determined that the dual problem fails to be solved, the optimal solution is determined based on the initial model parameters, constraint gradients and Hessian matrix of the control model.

[0044] In one implementation, after updating the parameters of the control model according to the optimal solution to obtain the target control model, the method further includes:

[0045] If the target control model does not meet the set training conditions, the following steps are executed repeatedly until a target control model that meets the set training conditions is obtained:

[0046] New environmental interaction data is obtained, and the constraint equation is updated according to the new environmental interaction data. The optimal solution of the objective function is solved again according to the new environmental interaction data and the new constraint equation, and the target control model is updated according to the new optimal solution.

[0047] In one implementation, after updating the parameters of the control model according to the optimal solution to obtain the target control model, the method further includes:

[0048] When it is determined that the meta-learning conditions are met, obtaining a task environment interaction dataset corresponding to at least one flight task;

[0049] By using at least one task environment interaction data set, the target control model is trained in sequence to obtain the trained flight task models corresponding to each flight task and the model losses corresponding to each flight task;

[0050] According to the sum of the losses of each model, the target control model is fine-tuned to obtain a fine-tuned target control model.

[0051] In one implementation, after updating the parameters of the control model according to the optimal solution to obtain the target control model, the method further includes:

[0052] When it is determined that the set state conditions are met, a mission environment interaction data set of the current flight mission is obtained, and the mission environment interaction data set is divided into support data and test data;

[0053] Determine the test loss of the target control model for executing the current flight mission based on the supporting data;

[0054] According to the test loss, the target model parameters of the target control model are adjusted to obtain a test model;

[0055] Determine the model loss of the test model in executing the current flight mission based on the test data of the flight mission;

[0056] According to the model loss, the target control model is fine-tuned to obtain a new target control model.

[0057] On the one hand, an embodiment of the present application provides a device for optimizing a control model, comprising:

[0058] An acquisition unit is used to acquire environmental interaction data of the flight equipment operating in the target environment; the environmental interaction data includes the environmental state of the target environment at each time, and the equipment action of the flight equipment at each time, and the expected discount reward and expected discount cost corresponding thereto;

[0059] A construction unit is used to construct a constraint equation of a control model according to the environmental state, device action and expected discounted cost at each moment in the environmental interaction data; the control model is used to determine the device action performed by the flight device according to the environmental state; the constraint equation is used to constrain the cost corresponding to the device action;

[0060] A solving unit is used to solve the optimal solution of the objective function of the control model according to the environmental interaction data and the constraint equation, so that in the case of the optimal solution, the result of the objective function is optimal and satisfies the constraints of the constraint equation; the result of the objective function is the expected discounted cumulative reward of the control model; the optimal solution is the solved target model parameters;

[0061] The updating unit is used to update the parameters of the control model according to the optimal solution to obtain the target control model.

[0062] In one implementation, the constraint equation includes a target cost constraint and a target divergence constraint;

[0063] Building blocks are used to:

[0064] Get the cost cap and discount factor at each moment;

[0065] According to the discount factor and expected discounted cost at each moment, as well as the cost upper limit, the normalized safety status of the flight equipment at each moment is determined; the normalized safety status is negatively correlated with the discount factor, the expected discounted cost, and the cost upper limit;

[0066] Determine the advantage estimate at each moment based on the environmental status and equipment actions at each moment;

[0067] Construct target cost constraints based on environmental interaction data, control models, cost caps, and normalized safety states, advantage estimates, and discount factors at each moment;

[0068] A target divergence constraint condition is constructed based on a specified control model and a control model using a divergence function; the specified control model is obtained by replacing model parameters of the control model with specified model parameters.

[0069] In one embodiment, the building block is used to:

[0070] The normalized security state at each moment is input into the activation function respectively to obtain the mapping value of the normalized security state at each moment;

[0071] Generate a first update factor at each moment according to each mapping value;

[0072] Determine the Lagrange multiplier based on the environmental interaction data, the control model, and the advantage estimate at each time;

[0073] According to the speed control coefficient and the Lagrange multiplier, a second update factor is determined; the second update factor is positively correlated with the speed control coefficient and the Lagrange multiplier;

[0074] The target cost constraint is constructed based on the environmental interaction data, the control model, the cost upper limit, the second update factor, and the first update factor, the expected discounted cost and the discount factor at each moment.

[0075] In one embodiment, the building block is used to:

[0076] Determine the minimum value between each mapped value and the first specified value;

[0077] The sum between each minimum value and the second specified value is determined respectively to obtain the first update factor at each moment.

[0078] In one embodiment, the building block is used to:

[0079] Determine the expected discounted cumulative cost based on the expected discounted cost at each moment;

[0080] Construct the initial target cost constraint based on the environmental interaction data, the expected discounted cumulative cost, the cost upper limit, the second update factor, and the first update factor and discount factor at each moment;

[0081] Perform first-order approximation on the initial target cost constraint to obtain the constraint gradient;

[0082] Calculate the difference between the expected discounted cumulative cost and the cost cap to obtain the residual cost;

[0083] Calculate the difference between the current model parameters of the control model and the specified model parameters to obtain the parameter change;

[0084] The target cost constraint is constructed based on the remaining cost, constraint gradient, and parameter change.

[0085] In one embodiment, the building block is used to:

[0086] Using a divergence function, constructing a divergence expression based on a specified control model and a control model;

[0087] According to the divergence expression, construct the initial divergence constraint condition;

[0088] Perform a second-order approximation on the initial divergence constraint to obtain the Hessian matrix;

[0089] According to the Hessian matrix and the residual cost, the target divergence constraint is constructed.

[0090] In one implementation, the solving unit is used to:

[0091] Determine the dual problem of the objective function; the dual problem includes a first variable and a second variable;

[0092] Solve the dual problem according to the environmental interaction data and the constraint equation to obtain a first variable value of the first variable and a second variable value of the second variable;

[0093] An optimal solution is determined based on initial model parameters of the control model, constraint gradients, a Hessian matrix, a first variable value, and a second variable value.

[0094] In one implementation, the solving unit is further configured to:

[0095] If it is determined that the dual problem fails to be solved, the optimal solution is determined based on the initial model parameters, constraint gradients and Hessian matrix of the control model.

[0096] In one implementation, the solving unit is further configured to:

[0097] If the target control model does not meet the set training conditions, the following steps are executed repeatedly until a target control model that meets the set training conditions is obtained:

[0098] New environmental interaction data is obtained, and the constraint equation is updated according to the new environmental interaction data. The optimal solution of the objective function is solved again according to the new environmental interaction data and the new constraint equation, and the target control model is updated according to the new optimal solution.

[0099] In one implementation, the solving unit is further configured to:

[0100] When it is determined that the meta-learning conditions are met, obtaining a task environment interaction dataset corresponding to at least one flight task;

[0101] By using at least one task environment interaction data set, the target control model is trained in sequence to obtain the trained flight task models corresponding to each flight task and the model losses corresponding to each flight task;

[0102] According to the sum of the losses of each model, the target control model is fine-tuned to obtain a fine-tuned target control model.

[0103] In one implementation, the solving unit is further configured to:

[0104] When it is determined that the set state conditions are met, a mission environment interaction data set of the current flight mission is obtained, and the mission environment interaction data set is divided into support data and test data;

[0105] Determine the test loss of the target control model for executing the current flight mission based on the supporting data;

[0106] According to the test loss, the target model parameters of the target control model are adjusted to obtain a test model;

[0107] Determine the model loss of the test model in executing the current flight mission based on the test data of the flight mission;

[0108] According to the model loss, the target control model is fine-tuned to obtain a new target control model.

[0109] On the one hand, an embodiment of the present application provides an electronic device, including:

[0110] Processor; and

[0111] A memory stores computer instructions, wherein the computer instructions are used to enable a processor to execute the steps of the method provided in any of the various optional implementations of the control model optimization described above.

[0112] On the one hand, an embodiment of the present application provides a computer-readable storage medium storing computer instructions, which are used to enable a computer to execute the steps of the method provided in any of the various optional implementations of any of the above-mentioned control model optimization.

[0113] On the one hand, a computer program product is provided in an embodiment of the present application, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the steps of the method provided in any of the various optional implementations of the control model optimization described above.

[0114] The method for optimizing the control model in the embodiment of the present application includes constructing the constraint equation of the control model according to the environmental state, equipment action and expected discount cost at each moment in the environmental interaction data, and solving the optimal solution of the objective function of the control model under the condition of the constraint equation according to the environmental interaction data, obtaining the target model parameters, and then obtaining the optimized target control model. In this way, the model is optimized by combining the expected discount reward and the expected discount cost, thereby improving the control performance of the target control model, thereby ensuring the safety and stability of the flight of the flight equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1It is a flow chart of a method for controlling model optimization in an embodiment of the present application.

[0116] Figure 2 It is a schematic diagram of an offline model training framework in an embodiment of the present application.

[0117] Figure 3 This is a framework diagram of an ESB-CPO algorithm in an embodiment of the present application.

[0118] Figure 4 It is a flow chart of a method for fine-tuning a control model based on meta-learning in an embodiment of the present application.

[0119] Figure 5 It is a flowchart of an example of fine-tuning a control model in an embodiment of the present application.

[0120] Figure 6 It is a schematic diagram of an online model optimization framework in an embodiment of the present application.

[0121] Figure 7 It is a structural block diagram of a device for controlling model optimization in an embodiment of the present application.

[0122] Figure 8 It is a structural schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0123] The technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described implementation methods are part of the implementation methods of the present application, rather than all of the implementation methods. Based on the implementation methods in the present application, all other implementation methods obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In addition, the technical features involved in the different implementation methods of the present application described below can be combined with each other as long as they do not conflict with each other.

[0124] Traditionally, the control model is usually trained and optimized based on reinforcement learning. Among them, reinforcement learning is to generate and control the flight equipment to perform corresponding equipment actions through perceived environmental changes, and train the control model with the goal of maximizing rewards.

[0125] Taking into account that the control performance of traditional control models is usually not high and it is difficult to ensure the safety and stability of flight equipment during flight, a method, device, equipment, storage medium and product for optimizing a control model are provided in the embodiments of the present application, aiming to improve the control performance of the control model to ensure the safety and stability of flight equipment during flight.

[0126] In an embodiment of the present application, a method for optimizing a control model is provided, which can be applied to electronic equipment for controlling the operation of flight equipment. For example, the flight equipment can be an aircraft, and the control model can be applied to the control system of the flight equipment to control the flight of the aircraft through the control system. The present application does not limit the type of electronic equipment, which can be any type of equipment suitable for implementation, such as a terminal device and a server, etc., and the present application will not elaborate on this.

[0127] In the embodiments of the present application, only a single update of the control model is used as an example for illustration. In actual applications, in order to obtain a control model with better performance, the control model can also be updated multiple times based on principles similar to the single update of the control model. Each round of model update is a model training to obtain a target control model that meets the control requirements.

[0128] See also Figure 1 As shown, it is a flow chart of a method for optimizing a control model in an embodiment of the present application. Figure 1 The method is described below. The specific implementation process of the method is as follows:

[0129] Step 101: Obtain environmental interaction data of the flight equipment operating in the target environment; the environmental interaction data includes the environmental state of the target environment at each time, and the equipment actions of the flight equipment at each time, and their corresponding expected discount rewards and expected discount costs.

[0130] In one embodiment, the control model is used to control the flight equipment to perform corresponding equipment actions according to the environmental state of the target environment, and the environmental state and equipment actions are periodically sampled, and the expected discounted reward and expected discounted cost at each moment are determined. For example, the target environment can be an offline training environment.

[0131] The expected discount reward and the expected discount cost may be determined according to the environment state and the device action. Optionally, the expected discount reward may be a reward after executing the device action, which may include an intrinsic reward and an extrinsic reward. The expected discount cost may be the cost of executing the device action.

[0132] In the embodiment of the present application, the expected discounted reward can be a core concept in reinforcement learning, which is used to quantify the current value of future rewards. The goal of reinforcement learning is to find a strategy (i.e., a control model) that enables the agent (i.e., a control system provided with a control model) to maximize the total expected return obtained by interacting with the environment. This total expected return is obtained by calculating the cumulative sum of a series of discounted rewards. The discount factor is a hyperparameter used in reinforcement learning to adjust the impact of future rewards. When the discount factor is close to 0, the agent pays more attention to short-term returns, and when the discount factor is close to 1, the agent takes into account more distant rewards. This design allows the agent to weigh long-term and short-term interests. Specifically, in reinforcement learning, the value of a state is equal to the expected value of the cumulative sum of the rewards on all possible paths from the state to the end multiplied by the corresponding discount factor.

[0133] In the embodiment of the present application, expected discounted cost can also be a concept in reinforcement learning, which refers to the cumulative calculation of the discount of immediate costs according to the discount rate when considering future costs. In reinforcement learning, the goal of the agent is usually to maximize the total reward in the future, while in some scenarios, the agent may need to minimize the total cost, such as consumed resources, time or energy. The concept of expected discounted cost is similar to the expected discounted reward mentioned above, except that the focus is shifted from reward to cost.

[0134] When considering costs, the agent may adopt strategies to reduce the long-term total cost rather than just optimizing the immediate cost. By applying a discount factor, the reinforcement learning algorithm can weigh the impact of costs at different time points and make more comprehensive decisions. This design reflects the potential of reinforcement learning to achieve cost control and resource optimization in multiple fields, such as reducing energy consumption and reducing task execution time.

[0135] Expected discounted cost in reinforcement learning is an important concept used to evaluate the total cost that an agent may incur by adopting a strategy when considering long-term effects. It is similar to the concept of expected discounted reward, but focuses on minimizing costs. By setting the discount rate appropriately, short-term and long-term costs can be effectively balanced, helping agents make more economical and efficient decisions in complex environments.

[0136] Step 102: Construct the constraint equation of the control model according to the environmental state, device action and expected discounted cost at each moment in the environmental interaction data; the control model is used to determine the device action performed by the flight device according to the environmental state; the constraint equation is used to constrain the cost corresponding to the device action.

[0137] The constraint equations include target cost constraints and target divergence constraints.

[0138] In one implementation, when executing step 102, the following steps may be used:

[0139] S102-1: Obtain the cost upper limit and the discount factor at each moment.

[0140] The cost upper limit may be preset, and the cost upper limit for each model optimization may be the same or different.

[0141] S102-2: Determine the normalized safety status of the flight equipment at each moment based on the discount factor and expected discounted cost at each moment, as well as the cost upper limit; the normalized safety status is negatively correlated with the discount factor, expected discounted cost, and cost upper limit.

[0142] Among them, the normalized safety state is an internal state based on the environmental interaction data, which is used to represent the safety of the flight equipment at time t.

[0143] Optionally, the following formula may be used to determine the normalized safety status:

[0144]

[0145] Among them, z iθ (s t ) represents the normalized safety state at time t in the i-th training, i represents the number of training times of the control model, θ represents the model parameters of the control model, s represents the environment state, t represents time, and d i represents the cost upper limit at the i-th training, l represents the time point at [0, t], a represents the device action, γ represents the discount factor, C i (s l ,a l ,s l+1 ) is the expected discounted cost at time l in the i-th training.

[0146] When the total cost is greater than the cost upper limit, z iθ (s t ) is less than 0. Before t = 0, z iθ (s t ) has an initial value of 1, so z iθ (s t ) can be updated to:

[0147]

[0148] S102-3: Determine the advantage estimate at each moment based on the environmental status and device operation at each moment.

[0149] Among them, advantage estimation, namely Lyapunov Advantage Estimation (LAE), is a technique for estimating the advantage function, which takes into account the dynamic characteristics and stability of the system. In this step, the LAE corresponding to each time step is calculated.

[0150] In one embodiment, the advantage estimate at a certain time is determined When , the following formula can be used:

[0151]

[0152] in, represents the advantage estimate at a certain moment in the i-th training, s is the environment state at a certain moment, a is the device action at a certain moment, represents the environment state-device action cost function, Represents the cost function of the i-th training.

[0153] In one implementation, determining the advantage estimate for the next moment When , the following formula can be used:

[0154]

[0155] in, represents the advantage estimate of the next moment in the i-th training, s′ is the environmental state at the next moment, E() is the expected function, is the distribution of the next environmental state of (s,a), α is the first adaptive factor, β is the second adaptive factor, V θ represents the value function, C i is the expected discounted cost of the i-th training.

[0156] S102-4: Construct target cost constraints based on environmental interaction data, control model, cost cap, and normalized safety status, advantage estimate, and discount factor at each moment.

[0157] In one implementation, when S102-4 is executed, the following steps may be performed:

[0158] S102-41: Input the normalized security state at each moment into the activation function respectively to obtain the mapping value of the normalized security state at each moment.

[0159] S102-42: Generate a first update factor at each moment according to each mapping value.

[0160] In one implementation, the minimum value between each mapping value and the first specified value (eg, 0) is determined respectively; the sum between each minimum value and the second specified value (eg, 1) is determined respectively to obtain the first update factor at each moment.

[0161] Optionally, when determining the first update factor, the following formula may be used:

[0162] β iθ (S t )=1+min(tamh(z iθ (s t )),0);

[0163] Among them, β iθ (s t ) represents the first update factor at time t in the i-th training. The tanh function is an activation function used to map input values ​​to continuous values ​​ranging from -1 to 1. iθ (s t ) is less than 0, indicating that it is in an unsafe environment state, β iθ (s t ) is reduced to 0.

[0164] S102-43: Determine the Lagrange multiplier based on the environmental interaction data, the control model, and the advantage estimate at each time.

[0165] In one implementation, based on the policy gradient of the constraint function (i.e., the gradient of the constraint equation), a Lagrange multiplier is introduced to calculate the second update factor. As an example, based on the advantage estimate and the Lagrange multiplier λ, a local optimization problem is constructed, where the local optimization problem can be expressed as:

[0166]

[0167]

[0168] in, represents the optimization function value, which is the gradient of the constraint equation, λ i is the Lagrange multiplier of the i-th training, To specify the model parameters, ρ θ represents the environmental state distribution of s, π θ represents the distribution of device actions output by the control model, represents the output of the specified control model, π θ (a|s) represents the output of the control model.

[0169] The dual problem of this local optimization problem can be expressed as:

[0170]

[0171] st i ≥0;

[0172] Based on this dual problem and the first step length η, λ can be i Updated to:

[0173]

[0174] S102-44: Determine a second update factor according to the speed control coefficient and the Lagrange multiplier; the second update factor is positively correlated with the speed control coefficient and the Lagrange multiplier.

[0175] Optionally, when determining the second update factor, the following formula may be used:

[0176]

[0177] Among them, α iθ is the second update factor during the i-th training, k i is the speed control coefficient of the i-th training, which is used to globally control α during the i-th training process iθ The rate of decrease of α iθ Changes from 1 to 0.

[0178] The first update factor and the second update factor are used to adjust the emphasis on security constraints during the policy update process.

[0179] S102-4: Construct target cost constraints based on environmental interaction data, control model, cost upper limit, second update factor, and first update factor, expected discounted cost and discount factor at each moment.

[0180] In one implementation, when S102-4 is executed, the following steps may be performed:

[0181] S102-41: Determine the expected discounted cumulative cost at each moment based on the expected discounted cost at each moment.

[0182] Optionally, the following formula may be used to determine the expected discounted cumulative cost:

[0183]

[0184] in, represents the expected discounted cumulative cost, τ~π θ Represents the control model π θ The trajectory adopted.

[0185] S102-42: Construct an initial target cost constraint based on the environmental interaction data, the expected discounted cumulative cost, the cost upper limit, the second update factor, and the first update factor and discount factor at each moment.

[0186] Optionally, the initial target cost constraint can be expressed as follows:

[0187]

[0188]

[0189] in, Represents the control model π θ Update to the specified control model The trend of i is the upper limit of the cost of the i-th training.

[0190] because and The following relationship exists:

[0191]

[0192]

[0193]

[0194] Therefore, the initial target cost constraint can also be expressed as:

[0195]

[0196] in, is the first gap, is the second gap. If the first gap and the second gap are negative, the constraint will be relaxed, otherwise, the constraint will be tightened.

[0197] S102-42: Perform first-order approximation on the initial target cost constraint to obtain the constraint gradient b.

[0198] S102-43: Calculate the difference between the expected discounted cumulative cost and the cost upper limit to obtain the remaining cost c.

[0199] Optionally, determine the residual cost c of the i-th training i When , the following formula can be used:

[0200]

[0201] S102-44: Calculate the difference between the current model parameters of the control model and the specified model parameters to obtain the parameter change; construct the target cost constraint condition according to the remaining cost, the constraint gradient and the parameter change.

[0202] Optionally, the following formula may be used to determine the target cost constraint:

[0203]

[0204] Among them, b i is the constraint gradient of the i-th training, is the parameter variation.

[0205] S102-5: Using a divergence function, based on a specified control model and a control model, construct a target divergence constraint condition; the specified control model is obtained by replacing model parameters of the control model with specified model parameters.

[0206] In one implementation, when executing S102-5, the following steps may be used:

[0207] S102-51: construct a specified control model based on specified model parameters; use a divergence function to construct a divergence expression based on the specified control model and the control model.

[0208] S102-52: Construct initial divergence constraints based on the divergence expression.

[0209] Optionally, the initial divergence constraint can be expressed as:

[0210]

[0211] D KL () represents the divergence expression, is the divergence expression, Represents the specified control model under the environmental state s The output of θ (·|s)) represents the control model π under the environmental state s θ The output of , δ is the second step length.

[0212] S102-53: Perform a second-order approximation on the initial divergence constraint to obtain the Hessian matrix; construct the target divergence constraint based on the Hessian matrix and the residual cost.

[0213] As an example, the target divergence constraint can be expressed as:

[0214]

[0215] Where H is the Hessian matrix of the KL-divergence.

[0216] Step 103: According to the environmental interaction data and the constraint equation, the optimal solution of the objective function of the control model is solved, so that in the case of the optimal solution, the result of the objective function is optimal and satisfies the constraints of the constraint equation; the result of the objective function is the expected discounted cumulative reward of the control model.

[0217] In one implementation, when executing step 103, the following steps may be used:

[0218] S1031: Determine the dual problem of the objective function; the dual problem includes a first variable and a second variable.

[0219] In one implementation, an initial objective function is constructed based on the expected discounted reward and the discount factor in the environmental interaction data.

[0220] Optionally, the initial objective function can be expressed as:

[0221]

[0222] Among them, J R (θ) represents the expected discounted cumulative reward.

[0223] Furthermore, the initial objective function can be transformed to obtain the objective function, which is the following expression:

[0224]

[0225]

[0226] Among them, Q θ (s,a) represents the environment state-device action value function, V θ (s) represents the value function.

[0227] Furthermore, to facilitate the solution, the objective function is first-order approximated to obtain the target gradient g, and the objective function can also be transformed into:

[0228]

[0229] Then the dual problem of the objective function is:

[0230]

[0231] Among them, c=[c0,c1,...], B=[b0,b1,...], μ1 is the first variable, and μ2 is the second variable.

[0232] S1032: Solve the dual problem according to the environmental interaction data and the constraint equation to obtain a first variable value of the first variable and a second variable value of the second variable.

[0233] In one implementation, the dual problem based on the objective function and the objective constraint equation are solved to obtain the first variable value and the value of the second variable

[0234] S1033: Determine the optimal solution according to the initial model parameters, constraint gradients, Hessian matrix, first variable values, and second variable values ​​of the control model.

[0235] In one embodiment, the optimal solution is determined, i.e., the target model parameters When , the following formula can be used:

[0236]

[0237] Furthermore, if it is determined that the dual problem fails to be solved, the optimal solution is determined based on the initial model parameters, constraint gradients, and Hessian matrix of the control model.

[0238] In one implementation, the following formula may also be used to determine the optimal solution:

[0239]

[0240] Among them, σ is the training step size.

[0241] Step 104: According to the optimal solution, update the parameters of the control model to obtain the target control model.

[0242] Furthermore, if the target control model does not meet the set training conditions, steps 101-104 can be executed repeatedly to obtain the target control model that meets the set training conditions. The set training conditions can be determined based on a threshold of the number of training times, or can be determined based on the model loss, or can be set according to the actual application scenario, which is not limited here. In other words, new environmental interaction data is obtained, and the constraint equations are updated according to the new environmental interaction data, and the optimal solution of the objective function is solved again according to the new environmental interaction data and the new constraint equations, and the target control model is updated according to the new optimal solution. I will not go into details here.

[0243] Combine the following Figure 2 This paper describes the application scenarios of the control model optimization method. Figure 2 As shown in Figure 1, it is a schematic diagram of an offline model training framework. Taking the target environment as an offline training environment as an example, Figure 2 It includes an offline training environment, a control model, and a model optimization module, and the model optimization module is used to optimize the control model through the control model optimization method of the above embodiment.

[0244] The model optimization module can obtain environmental interaction data based on the interaction between the control model and the offline training environment, that is, it can obtain the environmental state and equipment action at each moment, and can determine the expected discount reward and expected discount cost of the flight equipment based on the environmental state and equipment action. Then, the model optimization module can perform offline training on the control model based on the environmental interaction data to obtain a trained target control model.

[0245] See also Figure 3 As shown in FIG. 1 , a framework diagram of a Constrained Policy Optimization with Extra Safety Budge (ESB-CPO) algorithm based on an additional safety budget is shown in FIG. In the embodiment of the present application, the technical idea of ​​the control model optimization method is based on the ESB-CPO design. Figure 3 , the ESB-CPO algorithm is explained. Figure 3 The system includes a replay buffer, an ESB-CPO algorithm module, and an output module. The ESB-CPO algorithm module includes a first update factor module, a second update factor module, an objective function, and a constraint equation module.

[0246] Among them, the Replay Buffer is used to store environmental interaction data. The first update factor module is used to determine the first update factor according to the environmental interaction data. The second update factor module is used to determine the second update factor according to the environmental interaction data. The objective function and constraint equation module is used to construct the constraint equation according to the first update factor and the second update factor, and solve the optimal solution of the objective function based on the environmental interaction data and the constraint equation, and according to the optimal solution, update the parameters of the control model to obtain the target control model. The output module is used to output the target control model.

[0247] Among them, the constraint equation is constructed based on the trust region method, and the advantage estimate consists of stability and safety values, which can partially amplify the gap between safe transition and unsafe transition through the safety value. When the safety value part is regarded as an additional safety budget, the constraints of unsafe transition can be relaxed in the early stage, while the safety constraints can be gradually maintained to avoid flight accidents or mission failures caused by online learning.

[0248] in, Figure 2 as well as Figure 3 For the specific implementation steps of each module, please refer to the above steps 101 to 104, which will not be repeated here.

[0249] In this way, after obtaining the target control model, the operation of the flight equipment can be controlled through the target control model.

[0250] Furthermore, the target control model can be further fine-tuned online during the operation of the flight equipment.

[0251] In one implementation, meta-learning is used to fine-tune the target control model online. Meta-learning, also known as learning to learn, has the greatest feature that the trained deep model structure is not designed to complete a flight mission in a specific scenario, but rather to quickly adapt to and complete new flight missions in different scenarios with only a few training samples and only one or several iterations. This fully reflects the idea of ​​letting machines learn to learn, so that the network can be trained with a small set of training samples to quickly obtain the network model parameters corresponding to the flight mission.

[0252] Combine the following Figure 4 , the method of fine-tuning the control model based on meta-learning is explained, Figure 4 The flowchart of a method for fine-tuning a control model based on meta-learning is shown below. The method includes:

[0253] Step 401: When it is determined that the meta-learning conditions are met, a task environment interaction dataset corresponding to at least one flight task is obtained; the task environment interaction dataset includes the environment state and equipment action when executing the flight task.

[0254] Among them, the mission environment interaction data set is the environment interaction data corresponding to the flight mission. Each mission environment interaction data set is divided into support data (Support Set) and test data (Query Set). The meta-learning condition can be to determine when a multi-task training request is received, or when a task is updated. In actual applications, the meta-learning conditions can be set according to the actual application scenario. For example, the operating status of the flight equipment can also be obtained, such as the attitude angle error. If it is determined that the attitude angle error is higher than the error threshold, it is determined that the meta-learning conditions are met, where the attitude angle error is the absolute value of the difference between the actual attitude angle and the predicted attitude angle.

[0255] Step 402: Train the target control model in sequence through at least one mission environment interaction data set to obtain the trained flight mission models corresponding to each flight mission and their corresponding model losses.

[0256] Step 403: fine-tune the target control model according to the sum of the losses of each model to obtain a fine-tuned target control model.

[0257] Combine the following Figure 5 , for an example of the above-mentioned method of fine-tuning the control model based on meta-learning, see Figure 5 , which is a flowchart of an example of control model fine-tuning. Figure 5In the target control model, the initial target model parameter is φ, and the sum of the losses of each model is determined by L(f φ ), the following formula can be used:

[0258]

[0259] Where n is the total number of flight missions, Represents the model loss corresponding to task j. When determining the latest target model parameters, the following formula can be used:

[0260]

[0261] Figure 5 In the example, the target control model is trained in turn based on each flight task. The initial target model parameter of the target control model is φ. After training based on the task environment interaction dataset 1 of flight task 1, the update gradient of the parameter change is the first gradient (which can be expressed as ), the target model parameters are updated to θ 1 Then, based on the mission environment interaction dataset 2 of flight mission 2, continue training, and the parameter change gradient is the second gradient (which can be expressed as ), the target model parameters are updated to θ 2 ,Finally, according to the sum of the model losses of the target model after the two updates, the ,initial target model parameter φ is fine-tuned to obtain the fine-tuned φ and ,then the fine-tuned target control model is obtained.

[0262] Furthermore, in the process of executing the current flight mission, when fine-tuning the model based only on the mission environment interaction dataset of the current flight mission, the following steps can be adopted:

[0263] When it is determined that the set state conditions are met, a mission environment interaction data set of the current flight mission is obtained, and the mission environment interaction data set is divided into support data and test data; based on the support data, the test loss of the target control model to perform the current flight mission is determined; based on the test loss, the target model parameters of the target control model are adjusted to obtain the test model; based on the test data of the flight mission, the model loss of the test model to perform the current flight mission is determined; based on the model loss, the target control model is fine-tuned to obtain a new target control model.

[0264] In actual applications, the set state condition can be set according to the actual application scenario. For example, if it is determined that the attitude angle error is higher than the error threshold, it is determined that the set state condition is met, and no limitation is made here.

[0265] Optionally, determine the model parameters θ of the test model for the current flight mission j ′, the following formula can be used:

[0266]

[0267] Among them, the target control model corresponding to the current flight mission The target model parameter is θ j , α′ is the first flight mission coefficient, is the test loss.

[0268] Optionally, determine the adjusted target model parameters When , the following formula can be used:

[0269]

[0270] Among them, β is the second flight mission coefficient, is the model loss.

[0271] Furthermore, the value function V can be updated every time the model is trained and fine-tuned. θ , where the value function V θ It is generated by the intrinsic value function and the extrinsic value function. Therefore, based on the updated intrinsic value function and the extrinsic value function, a new value function can be obtained, and then the model training can be carried out through the updated value function during the next training.

[0272] In one implementation, the intrinsic return at each moment can be determined based on the intrinsic expected discounted reward at each moment, and the intrinsic loss can be determined based on the intrinsic return at each moment, and the parameter φ of the intrinsic value function can be updated based on the intrinsic loss. i .

[0273] Optionally, determine intrinsic losses When , the following formula can be used:

[0274]

[0275] Among them, D is the environment interaction data of the i-th training, is the intrinsic value of the i-th training, R i (s) is the intrinsic reward of the i-th training.

[0276] In one implementation, the extrinsic reward at each moment can be determined based on the extrinsic expected discounted reward at each moment, and the extrinsic loss can be determined based on the extrinsic reward at each moment, and the parameter φ of the extrinsic value function can be updated based on the extrinsic loss. e .

[0277] Optionally, determine external losses When , the following formula can be used:

[0278]

[0279] in, is the external value, R e (s) is the external reward.

[0280] Combine the following Figure 6 Describes the online model optimization framework. Figure 6 , which is a schematic diagram of an online model optimization framework. Figure 6 It includes online learning environment, target control model and online learning algorithm module.

[0281] Among them, the online learning environment is used to obtain online environment interaction data, namely the task environment interaction dataset, such as Figure 3 Multiple flight mission environments and their corresponding mission environment interaction datasets, such as Figure 4 Supporting data and test data in .

[0282] The online learning algorithm module is used to adopt a meta-learning approach to fine-tune the target control model online based on the mission environment interaction dataset of one or more flight missions when the trigger conditions are met.

[0283] In the embodiment of the present application, an offline training phase and an online optimization phase are included. In the offline training phase, the ESB-CPO algorithm is used to train and optimize the control model based on rewards and costs, and the flight equipment (such as the aircraft in the simulation) is driven to perform the flight mission by training the optimized target control model. Among them, the target control model is a policy function, which can also be called a control law. Its input is the environmental state, and its output is the equipment action of the flight equipment, that is, it is used to determine the equipment action that the flight equipment needs to perform according to the current environmental state in order to perform the flight mission. Rewards include intrinsic rewards (i.e., intrinsic expected discounted rewards) and extrinsic rewards (i.e., extrinsic expected discounted rewards). Intrinsic incentives can be used to solve reinforcement learning tasks with sparse or lacking external rewards. In a typical sparse continuous control task such as attitude control of flight equipment, considering that the efficiency of model training cannot be improved by relying solely on external rewards, the combination of constrained intrinsic incentives and constraint strategy optimization technology with additional safety budget can ensure the safety and stability of flight equipment. For example, in the aircraft control task, the deflection of the aircraft's rudder surface can be made as smooth as possible, reducing energy consumption while maintaining the aircraft's safe and stable flight, allowing the aircraft to perceive the characteristics of the environment more effectively, complete the flight mission by mastering the spatial characteristics, and ensure the safety of the aircraft's flight.

[0284] In the online optimization stage, the target control model is fine-tuned based on the task environment interaction dataset when the target control model performs each flight mission. Meta-learning is applied to flight control. When the trigger conditions are met, the target control model is fine-tuned. The target control model can be fine-tuned once each time a task environment interaction dataset is constructed. The target control model can also be fine-tuned based on the task environment interaction datasets of multiple tasks. In this way, the target control model is dynamically optimized and adjusted through active learning, which improves the adaptability of the target control model and can adapt to various environmental changes.

[0285] Based on the same inventive concept, a device for optimizing a control model is also provided in an embodiment of the present application. Since the principle of solving the problem by the above-mentioned device and equipment is similar to a method for optimizing a control model, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. The present application does not limit the type of electronic device. It can be any type of device suitable for implementation, such as terminal devices and servers, etc. The present application will not repeat this. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for operation.

[0286] See also Figure 7 , which is a structural block diagram of a device for optimizing a control model in an embodiment of the present application. In some embodiments, the device for optimizing a control model in an example of the present application includes:

[0287] The acquisition unit 701 is used to acquire the environmental interaction data of the flight device operating in the target environment; the environmental interaction data includes the environmental state of the target environment at each time, the device action of the flight device at each time, and the expected discount reward and expected discount cost corresponding thereto;

[0288] A construction unit 702 is used to construct a constraint equation of a control model according to the environmental state, device action and expected discounted cost at each moment in the environmental interaction data; the control model is used to determine the device action performed by the flight device according to the environmental state; the constraint equation is used to constrain the cost corresponding to the device action;

[0289] A solving unit 703 is used to solve the optimal solution of the objective function of the control model according to the environmental interaction data and the constraint equation, so that in the case of the optimal solution, the result of the objective function is optimal and satisfies the constraints of the constraint equation; the result of the objective function is the expected discounted cumulative reward of the control model; the optimal solution is the solved target model parameters;

[0290] The updating unit 704 is used to update the parameters of the control model according to the optimal solution to obtain the target control model.

[0291] In one implementation, the constraint equation includes a target cost constraint and a target divergence constraint;

[0292] The construction unit 702 is used to:

[0293] Get the cost cap and discount factor at each moment;

[0294] According to the discount factor and expected discounted cost at each moment, as well as the cost upper limit, the normalized safety status of the flight equipment at each moment is determined; the normalized safety status is negatively correlated with the discount factor, the expected discounted cost, and the cost upper limit;

[0295] Determine the advantage estimate at each moment based on the environmental status and equipment actions at each moment;

[0296] Construct target cost constraints based on environmental interaction data, control models, cost caps, and normalized safety states, advantage estimates, and discount factors at each moment;

[0297] A target divergence constraint condition is constructed based on a specified control model and a control model using a divergence function; the specified control model is obtained by replacing model parameters of the control model with specified model parameters.

[0298] In one implementation, the construction unit 702 is used to:

[0299] The normalized security state at each moment is input into the activation function respectively to obtain the mapping value of the normalized security state at each moment;

[0300] Generate a first update factor at each moment according to each mapping value;

[0301] Determine the Lagrange multiplier based on the environmental interaction data, the control model, and the advantage estimate at each time;

[0302] According to the speed control coefficient and the Lagrange multiplier, a second update factor is determined; the second update factor is positively correlated with the speed control coefficient and the Lagrange multiplier;

[0303] The target cost constraint is constructed based on the environmental interaction data, the control model, the cost upper limit, the second update factor, and the first update factor, the expected discounted cost and the discount factor at each moment.

[0304] In one implementation, the construction unit 702 is used to:

[0305] Determine the minimum value between each mapped value and the first specified value;

[0306] The sum between each minimum value and the second specified value is determined respectively to obtain the first update factor at each moment.

[0307] In one implementation, the construction unit 702 is used to:

[0308] Determine the expected discounted cumulative cost based on the expected discounted cost at each moment;

[0309] Construct the initial target cost constraint based on the environmental interaction data, the expected discounted cumulative cost, the cost upper limit, the second update factor, and the first update factor and discount factor at each moment;

[0310] Perform first-order approximation on the initial target cost constraint to obtain the constraint gradient;

[0311] Calculate the difference between the expected discounted cumulative cost and the cost cap to obtain the residual cost;

[0312] Calculate the difference between the current model parameters of the control model and the specified model parameters to obtain the parameter change;

[0313] The target cost constraint is constructed based on the remaining cost, constraint gradient, and parameter change.

[0314] In one implementation, the construction unit 702 is used to:

[0315] Using a divergence function, constructing a divergence expression based on a specified control model and a control model;

[0316] According to the divergence expression, construct the initial divergence constraint condition;

[0317] Perform a second-order approximation on the initial divergence constraint to obtain the Hessian matrix;

[0318] According to the Hessian matrix and the residual cost, the target divergence constraint is constructed.

[0319] In one implementation, the solving unit 703 is used to:

[0320] Determine the dual problem of the objective function; the dual problem includes a first variable and a second variable;

[0321] Solve the dual problem according to the environmental interaction data and the constraint equation to obtain a first variable value of the first variable and a second variable value of the second variable;

[0322] An optimal solution is determined based on initial model parameters of the control model, constraint gradients, a Hessian matrix, a first variable value, and a second variable value.

[0323] In one implementation, the solving unit 703 is further configured to:

[0324] If it is determined that the dual problem fails to be solved, the optimal solution is determined based on the initial model parameters, constraint gradients and Hessian matrix of the control model.

[0325] In one implementation, the solving unit 703 is further configured to:

[0326] If the target control model does not meet the set training conditions, the following steps are executed repeatedly until a target control model that meets the set training conditions is obtained:

[0327] New environmental interaction data is obtained, and the constraint equation is updated according to the new environmental interaction data. The optimal solution of the objective function is solved again according to the new environmental interaction data and the new constraint equation, and the target control model is updated according to the new optimal solution.

[0328] In one implementation, the solving unit 703 is further configured to:

[0329] When it is determined that the meta-learning conditions are met, obtaining a task environment interaction dataset corresponding to at least one flight task;

[0330] By using at least one task environment interaction data set, the target control model is trained in sequence to obtain the trained flight task models corresponding to each flight task and the model losses corresponding to each flight task;

[0331] According to the sum of the losses of each model, the target control model is fine-tuned to obtain a fine-tuned target control model.

[0332] In one implementation, the solving unit 703 is further configured to:

[0333] When it is determined that the set state conditions are met, a mission environment interaction data set of the current flight mission is obtained, and the mission environment interaction data set is divided into support data and test data;

[0334] Determine the test loss of the target control model for executing the current flight mission based on the supporting data;

[0335] According to the test loss, the target model parameters of the target control model are adjusted to obtain a test model;

[0336] Determine the model loss of the test model in executing the current flight mission based on the test data of the flight mission;

[0337] According to the model loss, the target control model is fine-tuned to obtain a new target control model.

[0338] The method for optimizing the control model in the embodiment of the present application includes constructing the constraint equation of the control model according to the environmental state, equipment action and expected discount cost at each moment in the environmental interaction data, and solving the optimal solution of the objective function of the control model under the condition of the constraint equation according to the environmental interaction data, obtaining the target model parameters, and then obtaining the optimized target control model. In this way, the model is optimized by combining the expected discount reward and the expected discount cost, thereby improving the control performance of the target control model, thereby ensuring the safety and stability of the flight of the flight equipment.

[0339] In an embodiment of the present application, an electronic device is further provided, including:

[0340] Processor; and

[0341] The memory stores computer instructions, where the computer instructions are used to enable the processor to execute the method of any of the above embodiments.

[0342] In an embodiment of the present application, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a computer to execute a method in any of the above-mentioned embodiments.

[0343] An embodiment of the present application also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements any of the above-mentioned methods.

[0344] Figure 8 FIG. 8 is a schematic diagram showing the structure of an electronic device 8000. Figure 8 As shown, the electronic device 8000 includes: a processor 8010 and a memory 8020 , and optionally, may also include a power supply 8030 , a display unit 8040 , and an input unit 8050 .

[0345] The processor 8010 is the control center of the electronic device 8000. It connects various components using various interfaces and lines, and performs various functions of the electronic device 8000 by running or executing software programs and / or data stored in the memory 8020, thereby monitoring the electronic device 8000 as a whole.

[0346] In the embodiment of the present application, the processor 8010 executes the various steps in the above embodiment when calling the computer program stored in the memory 8020.

[0347] Optionally, the processor 8010 may include one or more processing units; preferably, the processor 8010 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 8010. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips.

[0348] The memory 8020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, various applications, etc.; the data storage area may store data created according to the use of the electronic device 8000, etc. In addition, the memory 8020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices, etc.

[0349] The electronic device 8000 also includes a power source 8030 (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 8010 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0350] The display unit 8040 may be used to display information input by a user or information provided to a user, various menus of the electronic device 8000, etc. In the embodiment of the present application, it is mainly used to display the display interface of each application in the electronic device 8000 and objects such as text and pictures displayed in the display interface. The display unit 8040 may include a display panel 8041. The display panel 8041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0351] The input unit 8050 may be used to receive information such as numbers or characters input by the user. The input unit 8050 may include a touch panel 8051 and other input devices 8052. The touch panel 8051, also known as a touch screen, may collect user touch operations on or near it (such as operations performed by the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 8051).

[0352] Specifically, the touch panel 8051 can detect the user's touch operation, detect the signal brought by the touch operation, convert these signals into touch point coordinates, send them to the processor 8010, and receive and execute the command sent by the processor 8010. In addition, the touch panel 8051 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 8052 can include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0353] Of course, the touch panel 8051 can cover the display panel 8041. When the touch panel 8051 detects a touch operation on or near it, it transmits it to the processor 8010 to determine the type of touch event, and then the processor 8010 provides corresponding visual output on the display panel 8041 according to the type of touch event. Figure 8 In the embodiment, the touch panel 8051 and the display panel 8041 are used as two independent components to implement the input and output functions of the electronic device 8000, but in some embodiments, the touch panel 8051 and the display panel 8041 can be integrated to implement the input and output functions of the electronic device 8000.

[0354] The electronic device 8000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 8000 may also include other components such as a camera. Since these components are not the key components used in the embodiments of the present application, Figure 8 It is not shown and will not be described in detail.

[0355] Those skilled in the art will understand that Figure 8 The electronic device is merely an example and does not limit the electronic device, and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0356] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

Claims

1. A method for optimizing a control model, characterized in that: The method comprises: Acquire environmental interaction data of the flight device operating in the target environment; the environmental interaction data includes the environmental state of the target environment at each time, and the device action of the flight device at each time, and the expected discount reward and expected discount cost corresponding thereto; According to the environmental state, device action and expected discounted cost at each moment in the environmental interaction data, a constraint equation of a control model is constructed; the control model is used to determine the device action performed by the flight device according to the environmental state; the constraint equation is used to constrain the cost corresponding to the device action; According to the environmental interaction data and the constraint equation, the optimal solution of the objective function of the control model is solved, so that in the case of the optimal solution, the result of the objective function is optimal and satisfies the constraint of the constraint equation; the result of the objective function is the expected discounted cumulative reward of the control model; the optimal solution is the solved target model parameters; According to the optimal solution, the parameters of the control model are updated to obtain a target control model.

2. The method according to claim 1, characterized in that The constraint equation includes a target cost constraint condition and a target divergence constraint condition; The step of constructing the constraint equation of the control model according to the environmental state, device action and expected discounted cost at each moment in the environmental interaction data includes: Get the cost cap and discount factor at each moment; Determining a normalized safety state of the flight equipment at each moment according to the discount factor and the expected discounted cost at each moment, as well as the cost upper limit; the normalized safety state is negatively correlated with the discount factor, the expected discounted cost, and the cost upper limit; Determine the advantage estimate at each moment based on the environmental status and equipment actions at each moment; Constructing the target cost constraint condition according to the environmental interaction data, the control model, the cost upper limit, and the normalized safety state, advantage estimate and discount factor at each moment; The target divergence constraint condition is constructed based on a specified control model and the control model by using a divergence function; the specified control model is obtained by replacing the model parameters of the control model with the specified model parameters.

3. The method according to claim 2, characterized in that The target cost constraint condition is constructed according to the environmental interaction data, the cost upper limit, the control model, and the normalized safety state, advantage estimate and discount factor at each moment, including: The normalized security state at each moment is input into the activation function respectively to obtain the mapping value of the normalized security state at each moment; Generate a first update factor at each moment according to each mapping value; Determining Lagrange multipliers based on the environmental interaction data, the control model, and the advantage estimates at each time; Determine a second update factor according to the speed control coefficient and the Lagrange multiplier; the second update factor is positively correlated with the speed control coefficient and the Lagrange multiplier; The target cost constraint condition is constructed according to the environmental interaction data, the control model, the cost upper limit, the second update factor, and the first update factor, the expected discounted cost and the discount factor at each moment.

4. The method according to claim 3, characterized in that The step of generating a first update factor at each moment according to each mapping value includes: Determine the minimum value between each mapped value and the first specified value; The sum between each minimum value and the second specified value is determined respectively to obtain the first update factor at each moment.

5. The method according to claim 3, characterized in that: The target cost constraint condition is constructed according to the environmental interaction data, the control model, the cost upper limit, the second update factor, the first update factor at each moment, and the discount factor, including: Determine the expected discounted cumulative cost based on the expected discounted cost at each moment; Constructing an initial target cost constraint condition according to the environmental interaction data, the expected discounted cumulative cost, the cost upper limit, the second update factor, and the first update factor and discount factor at each moment; Performing a first-order approximation on the initial target cost constraint condition to obtain a constraint gradient; Calculate the difference between the expected discounted cumulative cost and the cost upper limit to obtain the remaining cost; Calculating the difference between the current model parameters of the control model and the specified model parameters to obtain the parameter change; The target cost constraint condition is constructed according to the remaining cost, the constraint gradient and the parameter change.

6. The method according to claim 5, characterized in that The adopting of the divergence function, based on the specified control model and the control model, constructing the target divergence constraint condition, comprises: Using a divergence function, constructing a divergence expression based on the specified control model and the control model; According to the divergence expression, construct an initial divergence constraint condition; Performing a second-order approximation on the initial divergence constraint condition to obtain a Hessian matrix; The target divergence constraint condition is constructed according to the Hessian matrix and the residual cost.

7. The method according to claim 6, characterized in that Solving the optimal solution of the objective function of the control model according to the environmental interaction data and the constraint equation includes: Determine a dual problem of the objective function; the dual problem includes a first variable and a second variable; Solving the dual problem according to the environmental interaction data and the constraint equation to obtain a first variable value of the first variable and a second variable value of the second variable; The optimal solution is determined according to the initial model parameters of the control model, the constraint gradient, the Hessian matrix, the first variable value and the second variable value.

8. The method according to claim 7, characterized in that The method further comprises: If it is determined that the dual problem fails to be solved, the optimal solution is determined based on the initial model parameters of the control model, the constraint gradient and the Hessian matrix.

9. The method according to any one of claims 1 to 8, characterized in that: After updating the parameters of the control model according to the optimal solution to obtain the target control model, the method further includes: If the target control model does not meet the set training conditions, the following steps are executed repeatedly until a target control model meeting the set training conditions is obtained: Acquire new environmental interaction data, and update the constraint equation according to the new environmental interaction data, and solve the optimal solution of the objective function again according to the new environmental interaction data and the new constraint equation, and update the target control model according to the new optimal solution.

10. The method according to any one of claims 1 to 8, characterized in that: After updating the parameters of the control model according to the optimal solution to obtain the target control model, the method further includes: When it is determined that the meta-learning condition is met, obtaining a task environment interaction dataset corresponding to at least one flight task; The target control model is trained in sequence through the at least one task environment interaction data set to obtain the trained flight task models corresponding to the respective flight tasks and the model losses corresponding to the respective flight tasks; According to the sum of the losses of each model, the target control model is fine-tuned to obtain a fine-tuned target control model.

11. The method according to any one of claims 1 to 8, characterized in that: After updating the parameters of the control model according to the optimal solution to obtain the target control model, the method further includes: When it is determined that the set state condition is met, a mission environment interaction data set of the current flight mission is obtained, where the mission environment interaction data set is divided into support data and test data; Determining, based on the supporting data, a test loss of the target control model in executing a current flight mission; According to the test loss, adjusting the target model parameters of the target control model to obtain a test model; Determining, based on the test data of the flight mission, a model loss of the test model in executing the current flight mission; According to the model loss, the target control model is fine-tuned to obtain a new target control model.

12. A device for controlling model optimization, characterized in that: The device comprises: an acquisition unit, configured to acquire environmental interaction data of the flight device operating in a target environment; the environmental interaction data includes an environmental state of the target environment at each time, and device actions of the flight device at each time, and their corresponding expected discount rewards and expected discount costs; A construction unit, configured to construct a constraint equation of a control model according to the environmental state, device action and expected discounted cost at each moment in the environmental interaction data; the control model is configured to determine the device action performed by the flight device according to the environmental state; and the constraint equation is configured to constrain the cost corresponding to the device action; A solving unit, used for solving the optimal solution of the objective function of the control model according to the environmental interaction data and the constraint equation, so that in the case of the optimal solution, the result of the objective function is optimal and satisfies the constraint of the constraint equation; the result of the objective function is the expected discounted cumulative reward of the control model; the optimal solution is the solved target model parameters; An updating unit is used to update the parameters of the control model according to the optimal solution to obtain a target control model.

13. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device performs the method according to any one of claims 1 to 11.

Images