A method and device for training satellite attitude control intelligent agent based on offline reinforcement learning

Abstract

The embodiments of the present application provide a method and device for training a satellite attitude control agent based on offline reinforcement learning, the method comprising: training a constructed environment model according to a data set to obtain a target environment model, wherein the input of the environment model is the sum of the dimensions of the attitude and action in the offline data, and the output is the attitude of the next step and a reward and punishment value; training the agent through the target environment model to obtain the target agent. On the one hand, the environment model constructed by some embodiments of the present application enables the agent in training to learn the dynamic model of satellite attitude control even if it is free from interaction with the real environment. On the other hand, the unfamiliar state discriminator constructed by some embodiments of the present application ensures the safety and generalization ability of the trained agent.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning. Specifically, embodiments of the present application relate to a method and device for training a satellite attitude control intelligent agent based on offline reinforcement learning. Background Art

[0002] like Figure 1 As shown in the figure, the traditional satellite attitude control uses PID closed-loop control to adjust the satellite attitude. By calculating the error with the target attitude and the proportional (P), differential (D), and integral (I) parameters, the control amount for the next adjustment is obtained, which is transmitted to the satellite and then the attitude is adjusted. If the error between the actual attitude and the target attitude is not within the range, the adjustment continues until the target attitude is reached. The effect of the control is affected by the P, I, and D parameters, which are set before deployment based on experience and calculation.

[0003] like Figure 2 As shown in the figure, traditional reinforcement learning methods are based on interaction for training. The intelligent agent interacts with the environment through decision-making and obtains new data. The data can be used to learn the environmental model, store or directly update the strategy and value function. If the learning of the environmental model is completed, the learning of the strategy can be assisted by generating false data, planning trajectories and other techniques. This type of method is model-based reinforcement learning. Deep reinforcement learning is based on stored data for training. The trained intelligent agent completes the target task by adaptively adjusting the strategy. If the interaction with the environment is abandoned, that is, learning the strategy directly from the data, this method is called offline reinforcement learning.

[0004] With the development of satellite Internet technology, fast, large-scale and low-cost satellite research and development have become the main requirements. From the perspective of satellite attitude control, the most suitable intelligent algorithm is the deep reinforcement learning method. The trained strategy model has a strong adaptive adjustment ability and can meet the development requirements of general attitude control algorithms. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for training a satellite attitude control agent based on offline reinforcement learning. On the one hand, the environmental model constructed by some embodiments of the present application enables the agent under training to learn the dynamic model of satellite attitude control even if it is free from interaction with the real environment. On the other hand, the unfamiliar state discriminator constructed by some embodiments of the present application ensures the safety and generalization ability of the trained agent.

[0006] In a first aspect, some embodiments of the present application provide a method for training a satellite attitude control agent based on offline reinforcement learning, the method comprising: training a constructed environment model according to a data set to obtain a target environment model, wherein the input of the environment model is the sum of the dimensions of the attitude and action in the offline data, and the output is the attitude of the next step and a reward or punishment value; and training the agent through the target environment model to obtain the target agent.

[0007] Some embodiments of the present application construct an environment model through a reinforcement learning method, and subsequently train the intelligent agent based on the constructed environment model, thereby enabling reinforcement learning to be completed without the intelligent agent interacting with the actual environment.

[0008] In some embodiments, the number of environmental models is k, and the target environmental model includes k trained environmental models, wherein k is an integer greater than 1, wherein training the agent through the target environmental model to obtain the target agent includes: constructing an unfamiliar state discriminator based on the k trained environmental models, wherein the unfamiliar state discriminator is used to determine whether an input state-action pair belongs to an unfamiliar state-action pair; and training the agent through the unfamiliar state discriminator to obtain the target agent.

[0009] Some embodiments of the present application not only expand the data that is not in the training set but may be generated in practice by constructing an unfamiliar state discriminator, but also ensure that the expanded data will not transition to an unfamiliar state, thereby ensuring the safety of the trained intelligent agent.

[0010] In some embodiments, the training of the agent through the unfamiliar state discriminator to obtain the target agent includes: obtaining a threshold, wherein the threshold is determined by the average of the difference values ​​of the reference i+1 state and the reference i+1 reward and punishment values ​​inferred by k-1 trained environment models according to the input i-th state action pair, the k-1 trained environment models constitute a target group environment model, and a reference i+1 state and a reference i+1 reward and punishment value are the results inferred by a trained environment model in the target group environment model; obtaining the verification environment model The i+1th state and the i+1th reward or punishment value are inferred according to the i+1th state-action pair, wherein the verification environment model is a trained environment model included in the target environment model excluding the target group environment model; the second difference between the i+1th state and the i+1th reward or punishment value and each reference i+1th state and the reference i+1th reward or punishment value is calculated; if the second difference value exceeds the threshold value for more than k / 2 times, it is determined that the i+1th state-action pair will transfer to an unfamiliar state, and the trained intelligent agent is punished accordingly to prevent the intelligent agent from making this decision again.

[0011] The embodiment of the present application starts with offline expert data of satellite attitude control, uses data to train multiple environmental models to integrate together, and proposes a method for distinguishing unfamiliar states based on the integrated environmental model. This method can prevent the environment from transitioning to an unfamiliar and dangerous state, thus ensuring safety in an offline environment. Since the unfamiliar state discriminator does not simply use the presence or absence of a record as a criterion for judgment, but uses the reasoning connection between multiple models, there will be some states that do not appear in the data set but do not belong to unfamiliar states, which proves that the method also has a certain generalization ability.

[0012] In some embodiments, the corresponding punishment to the trained agent includes: determining a penalty value according to a lower limit value of a reward function, wherein the penalty value is less than the lower limit value; and punishing the trained agent according to the penalty value.

[0013] Some embodiments of the present application determine the penalty for actions that may transition to unfamiliar states based on the lower limit of the reward function, thereby giving the current decision a severe penalty value to prevent the trained agent from making such a decision next time.

[0014] In some embodiments, the difference between the penalty value and the lower limit value is greater than a set threshold value. This is because the penalty value should be as large as possible so that the agent will not make a decision to transfer to this state.

[0015] Some embodiments of the present application increase the penalty by a very small value to improve the training effect on the intelligent agent.

[0016] In some embodiments, the training of the constructed environment model according to the data set to obtain the target environment model includes: obtaining the data set, and obtaining k groups of different training data subsets according to the data set, wherein the value of k is an integer greater than 1; constructing k regression models; training the i-th regression model according to a reward function and the i-th group of training data subset to obtain the i-th trained environment model, wherein the target environment model includes k trained environment models, and the value of i is greater than or equal to 1 and less than or equal to k.

[0017] Some embodiments of the present application perform reinforcement learning based on a regression model to obtain an environment model.

[0018] In some embodiments, the regression model includes: a neural network model, a Gaussian process model, or a Bayesian model.

[0019] In some embodiments, the neural network model includes three layers, wherein the data input to the first layer is the sum of the dimensions of the posture and action in the offline data, the middle layer is set to N dimensions, and the output of the last layer is the posture and reward and punishment value of the next step, and the posture and action in the offline data are normalized data.

[0020] Some embodiments of the present application perform reinforcement learning through a three-layer network model to obtain an environment model.

[0021] In some embodiments, the reward function designed corresponding to the neural network is a function related to a main value and an auxiliary value, wherein the Euclidean distance between the current posture of the three axes and the target posture is used as the main value, and the numerical values ​​of the angular velocity and input torque of the three axes are used as the auxiliary values.

[0022] In some embodiments, the reward function is:

[0023]

[0024] The design of this reward function will make the penalty smaller the closer the current posture is to the target posture, and eventually maintain it at around zero. At this point, it can be determined that the target posture has been reached, and the posture control task has been completed.

[0025] In some embodiments, the agent or the target agent makes decisions through a PPO network decision function.

[0026] The embodiments of the present application can further improve data utilization by adopting a proximal policy optimization (PPO) network decision function.

[0027] In a second aspect, some embodiments of the present application provide a method for deploying an intelligent agent, the method comprising: acquiring a target intelligent agent obtained by the method of any embodiment of the first aspect; and injecting a model and parameters related to the target intelligent agent into a satellite.

[0028] In a third aspect, some embodiments of the present application provide a device for training a satellite attitude control agent based on offline reinforcement learning, the device comprising: an environment model construction module, configured to perform reinforcement learning according to a data set to obtain an environment model, wherein the input of the environment model is the sum of the dimensions of the attitude and action in the offline data, and the output is the attitude of the next step and a reward or punishment value; a target agent acquisition module, configured to train the agent through the environment model to obtain a target agent.

[0029] Some embodiments of the present application utilize limited satellite offline data and, without interacting with the real environment, can still train an effective environment model, and the reinforcement learning agent interacts with the model, completing the training using unlimited virtual data.

[0030] In a fourth aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0031] In a fifth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the method described in any embodiment of the first aspect or the second aspect can be implemented.

[0032] In a sixth aspect, some embodiments of the present application provide a computer program product, wherein the computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method described in any embodiment of the first aspect or the second aspect can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of the process of using PID closed-loop control to adjust the satellite attitude is provided for related technologies;

[0035] Figure 2 Flowchart of elements of reinforcement learning methods provided for related technologies;

[0036] Figure 3 A flowchart of a method for training a satellite attitude control agent based on offline reinforcement learning provided in an embodiment of the present application;

[0037] Figure 4 A block diagram of the composition of the unfamiliar state discriminator provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of known states and unknown states provided for an embodiment of the present application;

[0039] Figure 6 A block diagram of the composition of a device for training a satellite attitude control agent based on offline reinforcement learning provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0042] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0043] like Figure 1 To achieve a perfect effect with the PID control method, it is necessary to design the appropriate parameters in advance, which requires prior knowledge, parameter adjustment experience and labor costs. Even satellites of the same batch and model will have some steady-state errors, and to overcome these errors, the integral parameters must be adjusted. There is no range for parameter adjustment and it depends entirely on experience and trial and error. Therefore, each time a new satellite is designed, the process must be restarted. Therefore, it is more difficult to meet the rapid development needs of satellite Internet.

[0044] like Figure 2 The deep reinforcement learning method can obtain a policy model with adaptive capabilities through training to complete the attitude control target task, but the biggest factor affecting the training effect is data. In a virtual environment, the amount of data is unlimited, but in real situations, it is not the case. For example, in real satellite application scenarios, first of all, the amount of data is limited, and secondly, the data set cannot be expanded at any time. It is very dangerous to use a policy that does not work well in the early stage of training to deploy on the satellite to collect new data. In other words, the data set is static.

[0045] In response to the problem of difficulty in adjusting PID control parameters, a method of adaptive control based on reinforcement learning has been proposed. However, traditional reinforcement learning is based on interaction, so strategies that do not work well in the early stages of training will make some unreasonable decisions, which may cause danger. Some embodiments of the present application start with a static data set, aiming to complete the training of the reinforcement learning method without interaction, and obtain a better strategy under offline conditions before deploying it on the satellite.

[0046] The primary problem that needs to be solved by the related offline reinforcement learning method is the emergence of unfamiliar states. Some embodiments of the present application use limited training data to learn a target environment model, and use the target environment model to simulate the entire satellite attitude control mission environment.

[0047] It is understandable that in order to complete the training of the policy model under a limited static data set, an offline deep reinforcement learning method with high data utilization is required, which can not only complete convergence under limited data, but also obtain good results and meet the requirements of deployment on satellites. The embodiments of the present application can at least achieve these technical purposes. Some embodiments of the present application abandon the interactive process of traditional deep reinforcement learning, achieve efficient data utilization and better adaptive effect.

[0048] For example, some embodiments of the present application start with offline expert data of satellite attitude control (i.e., data in the data set involved in S101 below), use the data to train multiple environmental models, and integrate the trained multiple environmental models together, and propose a method for distinguishing unfamiliar states based on the integrated environmental model. This method can avoid the environment from transitioning to an unfamiliar and dangerous state, so that safety in an offline environment can be guaranteed. Since the unfamiliar state discriminator of some embodiments of the present application is not simply based on whether the state appears as a judgment criterion, but through the reasoning connection between multiple environmental models, there will be some states that do not appear in the data set, but do not belong to the phenomenon of unfamiliar states, which proves that the method still has a certain generalization ability. Therefore, the embodiments of the present application can still learn the dynamic model of satellite attitude control under the premise of giving up interaction with the real environment, while ensuring safety and generalization ability.

[0049] Please see Figure 3 , Figure 3 A method for training a satellite attitude control agent based on offline reinforcement learning is provided for some embodiments of the present application, the method comprising: S101, training a constructed environment model according to a data set to obtain a target environment model, wherein the input of the environment model is the sum of the dimensions of the attitude and action in the offline data, and the output is the attitude and reward and punishment value of the next step; and S102, training the agent through the target environment model to obtain a target agent.

[0050] The implementation process of each of the above steps is described below by way of example.

[0051] In some embodiments of the present application, the data in the data set involved in S101 are offline satellite attitude control expert data. Executing S101 is to use the data provided by the offline satellite attitude control expert to train the environment model. It can be understood that generating virtual data based on the trained environment model can greatly improve the utilization rate of the data.

[0052] It should be noted that the target environment model may include one trained environment model or multiple trained environment models.

[0053] The following is an exemplary description of the process of obtaining a trained environment model. It can be understood that the environment model is a regression model, and the regression model at least includes: a neural network model, a Gaussian process model or a Bayesian model.

[0054] The following describes the process of training an environment model to obtain each trained environment model included in the target environment model with reference to a specific neural network model example.

[0055] In some embodiments of the present application, the neural network model includes three layers, wherein the data input to the first layer is the sum of the dimensions of the posture and action in the offline data, the middle layer is set to N dimensions, and the output of the last layer is the posture of the next step and a reward or punishment value, and the posture and action in the offline data are normalized data.

[0056] For example, the environmental model uses a simple three-layer neural network model, the input is the sum of the dimensions of the posture and action in the offline data, the middle layer is set to 512 dimensions, and the output is the posture and reward and punishment value of the next step; in data processing, the posture and action in the offline data should be normalized, and the corresponding mean and variance should be recorded for subsequent processing of new states and verification of reasoning (for example, when testing on a satellite). In the design of the reward function of some embodiments of the present application, with reference to the inverted pendulum model, the Euclidean distance between the current posture of the three axes and the target posture is used as the main value, and the angular velocity and input torque of the three axes are used as auxiliary. The meaning of the main value is the value to reach the target first, that is, the Euler angle must first transition to the target posture, and then limit the auxiliary value (such as angular velocity and torque) on this basis, so that it can remain stable and not shake at the target position. The formula corresponding to the specific reward function is as follows:

[0057]

[0058] Among them, γ, θ, φ represent the current posture of the agent in the three-axis Euler angle, and γ target ,θ target ,φ target Represents the target posture in three-axis Euler angle, γ dt ,θ dt ,φ dt Represents the current three-axis angular velocity of the agent, action γ ,action θ ,action φ It is the PWM value of the three-axis motor, that is, the torque value.

[0059] The design of this reward function will make the penalty smaller the closer the current posture is to the target posture, and eventually maintain it around zero. At this point, it can be determined that the target posture has been reached, that is, the posture control task has been completed. When constructing a data set, it is necessary to normalize the data state, action, reward, and the change value between the next state and the current state, so as to provide a certain guarantee for convergence during training.

[0060] It should be noted that, for training under static data sets, imitation learning is also used in the prior art for training, but imitation learning requires a large amount of expert data. If the size of the data set is limited, the effect will also be affected. Some embodiments of the present application, under the premise of offline operation, propose a reinforcement learning method with high data utilization to ensure effectiveness and generalization, so as to complete the adaptive control of satellite attitude control.

[0061] At least in order to further improve the accuracy of the unfamiliar state judgment and improve the defects of offline model training, in some embodiments of the present application, an unfamiliar state discriminator is also constructed based on multiple environmental models obtained through training. For example, if there are k environmental models, the target environmental model includes k trained environmental models.

[0062] The following first illustrates the process of obtaining k trained environment models. For example, in some embodiments of the present application, the number of environment models is k, and the target environment model involved in S102 includes k trained environment models, where k is an integer greater than 1, and S102 exemplarily includes: S310, constructing an unfamiliar state discriminator based on the k trained environment models, where the unfamiliar state discriminator is used to determine whether the input state-action pair belongs to an unfamiliar state-action pair; S320, training the agent through the unfamiliar state discriminator to obtain the target agent.

[0063] For example, in some embodiments of the present application, the process of obtaining k trained models involved in S310 exemplarily includes:

[0064] The first step is to obtain the above data set, and obtain k different training data subsets based on the data set, where the value of k is an integer greater than 1. For example, the data set includes 1000 different data (for example, decision data from different time points of the same task, or data collected by expert strategies with different parameters), and each 100 data are used as a group of training data to train an environment model respectively, and finally 10 trained environment models can be obtained.

[0065] The second step is to build k regression models. This is because the combination of the next state and the reward and punishment value is a continuous quantity, so a regression model is needed to fit it.

[0066] It should be noted that the regression model includes: a neural network model, a Gaussian process model or a Bayesian model. For example, k neural network models are constructed, and the structure of each neural network model can be the same or different. For example, if the regression model is a neural network model, the neural network model includes three layers, wherein the data input to the first layer is the sum of the dimensions of the posture and action in the offline data, the middle layer is set to N dimensions (for example, 512 dimensions), and the output of the last layer is the posture and reward and punishment value of the next step, and the posture and action in the offline data are normalized data.

[0067] The third step is to train the i-th regression model according to a reward function and the i-th group of training data subset to obtain the i-th trained environment model, wherein the target environment model includes k trained environment models, and the value of i is greater than or equal to 1 and less than or equal to k.

[0068] It should be noted that the reward function designed corresponding to the neural network provided in some embodiments of the present application is a function related to a main value and an auxiliary value, wherein the Euclidean distance between the current posture of the three axes and the target posture is used as the main value, and the numerical values ​​of the angular velocity and input torque of the three axes are used as the auxiliary values.

[0069] The following is an exemplary description of the unfamiliar state determination process performed by the constructed unfamiliar state discriminator.

[0070] In some embodiments of the present application, S320 exemplarily includes: obtaining a threshold value, wherein the threshold value is determined by the average of the difference values ​​of the reference i+1 state and the reference i+1 reward and punishment value inferred by k-1 trained environment models according to the input i-th state-action pair, wherein the k-1 trained environment models constitute a target group environment model, and a reference i+1 state and a reference i+1 reward and punishment value are the results inferred by a trained environment model in the target group environment model; obtaining the i+1 state and the i+1 reward and punishment value inferred by the verification environment model according to the i-th state-action pair, wherein the verification environment model The environment model is a trained environment model included in the target environment model except the target group environment model; the second difference between the i+1th state and the i+1th reward and punishment value and the reference i+1th state and the reference i+1th reward and punishment value is calculated; if the second difference value exceeds the threshold value for more than k / 2 times, it is determined that the i-th state-action pair will transfer to an unfamiliar state, and the trained intelligent agent is punished accordingly (this is because if the state is determined to be an unfamiliar state, that is, a state that is impossible to appear in the real world, the intelligent agent needs to be punished below the normal reward and punishment range) to prevent the intelligent agent from making this decision again.

[0071] It should be noted that the agent interacts with the trained environment model, and combines the current state and the decision made to form the i-th state-action pair, which specifically includes the state part composed of the three-axis Euler angle information, the three-axis angular velocity information, the three-axis motor speed value, the current three-axis motor PWM value, and the action part of the three-axis motor PWM value of the current decision.

[0072] It can be understood that the most important thing about offline reinforcement learning is to ensure the effectiveness of learning, that is, to process the states that have never appeared in the data set and can be reached in the real world while also ensuring that the state is avoided from being transferred to the state that cannot appear in the real world. At least in order to achieve the technical purpose as described above, some embodiments of the present application design an unfamiliar state discriminator. In some embodiments, the unfamiliar state discriminator performs the following operations: Assuming that a total of k environment models are trained (that is, k trained environment models are obtained), each time a new state-action pair is input, k-1 trained environment models are randomly selected as a group, and the mean of the difference values ​​of the next state and reward inferred by the group of models is recorded as the threshold, and then the remaining trained model is used to calculate the difference value of the next state and reward with the k-1 trained models. If the difference value exceeds the threshold more than k / 2 times, it means that the new state-action pair will transfer to an unfamiliar state. At this time, a set data (for example, -50) penalty is given to the decision to avoid making such a decision next time.

[0073] In some embodiments of the present application, the judgment threshold is determined by the following formula:

[0074]

[0075] s t ,a t represents the current state and action (i.e., the i-th state-action pair), P i represents the environment model after the i-th training, and Represents the state transition result of the model, Corresponding to the reference i+1 state, Corresponding to the reference i+1th reward or penalty value.

[0076]

[0077] Distance calculation function diff(P i ,P j ) represents the calculation of the difference value of the current state transition between the i-th trained environment model and the j-th trained environment model. This difference value is calculated based on the Euclidean distance between the next state and the reward value function. Based on the definition of the environment model P and the distance calculation function diff, the specific representation of the threshold thresold can be listed:

[0078] It should be noted that the corresponding punishment to the trained agent includes: determining a penalty value according to a lower limit value of the reward function, wherein the penalty value is less than the lower limit value; and punishing the trained agent according to the penalty value. For example, the difference between the penalty value and the lower limit value is greater than a set threshold.

[0079] In some embodiments of the present application, the reward function designed corresponding to the neural network is a function related to a main value and an auxiliary value, wherein the Euclidean distance between the current posture of the three axes and the target posture is used as the main value, and the numerical values ​​of the angular velocity and input torque of the three axes are used as the auxiliary value.

[0080] Combine the following Figure 4 The composition of the unfamiliar state discriminator is explained by way of example.

[0081] like Figure 4 As shown, the unfamiliar state discriminator 10 includes a data set 100 (i.e., the data set of S101), k adopted data sets (i.e., the data set of S101 is divided into multiple different sampling data sets, such as Figure 4 The first sampling data set 111, the second sampling data set 112, ... the k-th adopted data set 11k), k environmental models (i.e. Figure 4 The first environment model 211, the second environment model 212 ... until the kth environment model 21k), the target group environment model 311 and the verification environment model 312. Figure 4 It can be seen that the input of the unfamiliar state discriminator 10 is the new state action pair 101, which specifically includes the three-axis Euler angle, the three-axis angular velocity, the three-axis motor speed, the three-axis PWM value of the current motor and the three-axis PWM value information of this decision, and the result output by the unfamiliar state discriminator is to determine whether the input belongs to an unfamiliar state.

[0082] Combined with the above description, we can see that firstly, the data set 100 is divided into k different data sets (corresponding to k training data subsets), and then Figure 4 The first sampling data set 111, the second sampling data set 112, ..., until the kth sampling data set 11k. Then, k environment models are trained based on different sampling data sets, such as Figure 4As shown, the first environment model 211 is trained according to the first sampling data set 111, the second environment model 212 is trained according to the second sampling data set 112, ..., and so on, the kth environment model 21k is trained according to the kth sampling data set, and k trained environment models are obtained after the training. Then, k-1 of the k trained environment models are randomly selected to form the target group environment model 311, and the remaining trained environment model is used as the verification environment model 312. Finally, the target group environment model obtains the threshold 313 according to the input new action state pair 101, and the verification environment model 312 obtains the next action according to the input new state action pair, and determines whether the input new state action pair belongs to an unfamiliar state according to the threshold 313 and the next action.

[0083] The target group environment model is specifically configured to obtain the target group environment model by random sampling, and all the trained environment models included in the target group environment model are respectively for the current state action pair (s t ,a t ) makes a prediction, namely P i (s t ,a t ), and then calculate the mean of the prediction differences between different models as the threshold, which is the calculation method of the thresold function.

[0084] Verify the environment model (i.e. the remaining environment model, set as P z ) type is specifically configured to (s t ,a t ) makes a prediction to obtain the compared prediction result, and then calculates the difference between the compared prediction result and the prediction results of each model in the target group environment model to obtain the second difference value, that is, calculates the diff function, and compares the calculation result with the value of thresold. If the number of times greater than the threshold is greater than Then the current state-action pair is judged to be an unfamiliar state.

[0085] After the above operations, the environment model becomes a restricted Markov decision process, but the neural network has a certain generalization ability, such as Figure 5 The known states shown are states that the model determines may occur in the real world, that is, safe states; the non-point areas are the embodiment of generalization; that is, after the unfamiliar state is judged, it can still cover some states that do not appear in the data set but are reasonable in the real world, thereby improving generalization. The unknown state is a collection of states that are judged as unfamiliar states. The model believes that it is impossible to appear in the real world. Once a decision is made to transfer to an unknown state, no matter what decision is made, the unknown state will be transferred to the Halt endpoint and a large penalty will be given, and the trajectory will be declared over.

[0086] It should be noted that the intelligent agent or the target intelligent agent described in some embodiments of the present application makes decisions through the PPO network decision function.

[0087] Using some embodiments of the present application provided by Figure 4 The safe environment model shown is equivalent to having unlimited false data. At this time, the traditional deep reinforcement learning method can be used for interactive training of strategies. In some embodiments of the present application, the network decision function uses the PPO method, which limits the amount of change in the gradient to make the loss function monotonically increasing. At the same time, the idea of ​​off-policy is introduced, and the importance sampling and clip techniques are used to utilize asynchronous data, further improving data utilization. The PPO model obtained after training convergence is an intelligent agent that can complete satellite attitude control. Compared with the method of directly interacting with the satellite environment for training, it has strong control capabilities at the beginning, and only needs a small amount of training adjustment according to different situations to adapt to new satellite equipment and new environments.

[0088] In summary, some embodiments of the present application can train a safe environment model based on a certain amount of offline data, which can be used to generate false data to facilitate the training of deep reinforcement learning methods, and obtain a satellite attitude control agent with good initial effects, which has high data utilization and adaptability while ensuring effectiveness.

[0089] For example, in some embodiments of the present application, starting from the offline expert data of satellite attitude control, multiple environmental models are integrated together using data training, and a method for distinguishing unfamiliar states is proposed based on the integrated model. This method can prevent the environment from transitioning to an unfamiliar and dangerous state, thereby ensuring safety in an offline environment. Since the unfamiliar state discriminator of the embodiment of the present application does not simply use the record of whether the state appears as a judgment criterion, but uses the reasoning connection between multiple models, there will be some states that do not appear in the data set, but do not belong to the unfamiliar state. This proves that the method still has a certain generalization ability. Therefore, the embodiment of the present application can still learn the dynamic model of satellite attitude control under the premise of giving up interaction with the environment, while ensuring safety and generalization ability.

[0090] Some embodiments of the present application use trained environment models and unfamiliar state discriminators to establish a Markov decision process, so that mature deep reinforcement learning algorithms can be used to complete the training of the control policy network. The PPO algorithm used in some embodiments of the present application will use an immature policy network to collect data in the early stage. In this process, it is inevitable to transition to an unfamiliar state. The experience replay pool in the PPO algorithm can record these decisions and then continuously learn by sampling. That is to say, the recorded trajectory may be used more than once. From this perspective, the PPO algorithm has a higher data utilization rate than other similar algorithms.

[0091] After the above two steps, the training of the satellite attitude control strategy network can be completed. That is to say, as long as the target attitude is given, the strategy network can make a decision to transition the current attitude to the target attitude.

[0092] The embodiments of the present application have been verified to be able to converge in the environmental model training phase. At the same time, the use of the methods provided by some embodiments of the present application can make the overall reward of each iteration rise and finally converge to a stable value. Therefore, the technical effect of the methods provided by the embodiments of the present application is theoretically guaranteed. For example, the method provided by some embodiments of the present application uses 20,000 time steps of data, which only takes 1 minute to collect in the real world. Only using this order of magnitude of data can complete the training and have good results. It can be seen that this method has a very high data utilization rate.

[0093] It should be noted that, in the learning stage of the environment model, some embodiments of the present application use a neural network model, but Gaussian process, Bayesian model, etc. can also be used to learn the dynamic model of satellite attitude control. At the same time, in the selection of the strategy network, in addition to the PPO method used in some embodiments of the present application, any other deep learning method can be selected, because at this time there is already a learned environment model, which is equivalent to having a gym environment. PPO is selected because it can further improve the utilization of data.

[0094] Some embodiments of the present application utilize offline satellite attitude control expert data to train an environmental model, and generating virtual data based on the trained model can greatly improve data utilization. Some embodiments of the present application propose an unfamiliar state discrimination method, which can ensure both safety and generalization when training an environmental model. Some embodiments of the present application combine a trained environmental model with an unfamiliar state discrimination method to establish a new Markov decision process, based on which a policy network can be trained. In the selection of a policy network in some embodiments of the present application, the PPO method is selected to utilize the off-policy characteristics to further improve data utilization.

[0095] Some embodiments of the present application also provide a method for deploying an intelligent agent, the method comprising: obtaining a target intelligent agent obtained by any of the above-mentioned training methods; and injecting a model and parameters related to the target intelligent agent into a satellite.

[0096] Please refer to Figure 6 , Figure 6 The apparatus for training satellite attitude control agents based on offline reinforcement learning provided in the embodiment of the present application is shown. It should be understood that the apparatus is similar to the above-mentioned Figure 3 The method embodiment corresponds to the method embodiment, and can execute each step involved in the above method embodiment. The specific functions of the device can refer to the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in the memory in the form of software or firmware or solidified in the operating system of the device. The device for training satellite attitude control agents based on offline reinforcement learning includes: an environment model construction module 601 and a target agent acquisition module 602.

[0097] The environment model construction module 601 is configured to train the constructed environment model according to the data set to obtain a target environment model, wherein the input of the environment model is the sum of the dimensions of the posture and action in the offline data, and the output is the posture of the next step and a reward or punishment value.

[0098] The target agent acquisition module 602 is configured to train the agent through the target environment model to obtain the target agent.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0100] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the above-mentioned method for training a satellite attitude control agent based on offline reinforcement learning.

[0101] Some embodiments of the present application provide a computer program product, which includes a computer program, wherein the computer program, when executed by a processor, can implement the above-mentioned method for training a satellite attitude control agent based on offline reinforcement learning.

[0102] like Figure 7As shown, some embodiments of the present application provide an electronic device 500, including a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 can implement the above-mentioned method for training a satellite attitude control agent based on offline reinforcement learning when reading and executing the program through a bus 530.

[0103] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0104] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code to implement some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the present disclosure embodiment may be used to execute the instructions in the memory 510 to implement Figure 3 The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.

[0105] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0107] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0108] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0109] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0110] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for training a satellite attitude control agent based on offline reinforcement learning, characterized in that: The method comprises: The constructed environment model is trained according to the data set to obtain a target environment model, wherein the input of the environment model is the sum of the dimensions of the posture and action in the offline data, and the output is the posture and reward and punishment value of the next step; The target intelligent agent is obtained by training the intelligent agent through the target environment model; The number of the environment models is k, and the target environment model includes k trained environment models, where k is an integer greater than 1, wherein, The step of training the agent through the target environment model to obtain the target agent comprises: Constructing an unfamiliar state discriminator according to the k trained environment models, wherein the unfamiliar state discriminator is used to determine whether an input state-action pair belongs to an unfamiliar state-action pair; The target intelligent agent is obtained by training the intelligent agent through the unfamiliar state discriminator; The step of training the agent by using the unfamiliar state discriminator to obtain the target agent includes: Obtaining a threshold value, wherein the threshold value is determined by the average of the difference values ​​of the reference i+1 state and the reference i+1 reward and punishment value inferred by k-1 trained environment models according to the input i-th state action pair, wherein the k-1 trained environment models constitute a target group environment model, and a reference i+1 state and a reference i+1 reward and punishment value are the results inferred by a trained environment model in the target group environment model; Obtaining the i+1th state and the i+1th reward and penalty value inferred by the verification environment model according to the i-th state-action pair, wherein the verification environment model is a trained environment model included in the target environment model except the target group environment model; Calculate a second difference between the i+1th state and the i+1th reward or penalty value and each reference i+1th state and reference i+1th reward or penalty value; If the second difference value exceeds the threshold value more than k / 2 times, it is determined that the i-th state-action pair will be transferred to an unfamiliar state, and a penalty is imposed on the trained agent to prevent the agent from making this decision again; The target intelligent agent is obtained by training the intelligent agent through the unfamiliar state discriminator; The step of training the constructed environment model according to the data set to obtain the target environment model includes: Obtain the data set, and obtain k different training data subsets according to the data set, where the value of k is an integer greater than 1; Construct k regression models; The i-th regression model is trained according to a reward function and the i-th group of training data subset to obtain the i-th trained environment model, wherein the target environment model includes k trained environment models, and the value of i is greater than or equal to 1 and less than or equal to k.

2. The method according to claim 1, characterized in that The corresponding punishment to the trained agent includes: Determining a penalty value according to a lower limit value of the reward function, wherein the penalty value is less than the lower limit value; The trained agent is punished according to the penalty value.

3. The method according to claim 2, characterized in that The difference between the penalty value and the lower limit value is greater than a set threshold.

4. The method according to claim 1, characterized in that The regression model includes: a neural network model, a Gaussian process model or a Bayesian model.

5. The method according to claim 4, characterized in that The neural network model includes three layers, wherein the data input to the first layer is the sum of the dimensions of the posture and action in the offline data, the middle layer is set to N dimensions, and the output of the last layer is the posture and reward and punishment value of the next step. The posture and action in the offline data are normalized data.

6. The method according to claim 5, characterized in that The reward function designed corresponding to the neural network is a function related to a main value and an auxiliary value, wherein the Euclidean distance between the current posture of the three axes and the target posture is used as the main value, and the numerical values ​​of the angular velocity and input torque of the three axes are used as the auxiliary value.

7. The method according to claim 6, characterized in that The reward function is: Among them, γ, θ, φ represent the current posture of the intelligent agent in the three-axis Euler angle, γ target ,θ target ,φ target They represent the target posture under the three-axis Euler angle, γ dt ,θ dt ,φ dt Represents the current three-axis angular velocity of the agent, action γ ,action θ ,action φ They are the torque values ​​of the three-axis motors respectively.

8. The method according to claim 1, characterized in that The agent or the target agent makes a decision through a PPO network decision function.

9. A method for deploying an agent, characterized in that: The method comprises: Obtain a target agent obtained by any method according to claims 1 to 8; The model and parameters related to the target agent are uploaded to the satellite.

10. A device for training a satellite attitude control agent based on offline reinforcement learning, characterized in that: The device comprises: An environment model construction module is configured to train the constructed environment model according to the data set to obtain a target environment model, wherein the input of the environment model is the sum of the dimensions of the posture and action in the offline data, and the output is the posture of the next step and a reward and punishment value; A target agent acquisition module is configured to obtain a target agent by training the agent through the target environment model; The number of the environment models is k, and the target environment model includes k trained environment models, where k is an integer greater than 1, wherein, The step of training the agent through the target environment model to obtain the target agent comprises: Constructing an unfamiliar state discriminator according to the k trained environment models, wherein the unfamiliar state discriminator is used to determine whether an input state-action pair belongs to an unfamiliar state-action pair; The step of training the agent by using the unfamiliar state discriminator to obtain the target agent includes: Obtaining a threshold value, wherein the threshold value is determined by the average of the difference values ​​of the reference i+1 state and the reference i+1 reward and punishment value inferred by k-1 trained environment models according to the input i-th state action pair, wherein the k-1 trained environment models constitute a target group environment model, and a reference i+1 state and a reference i+1 reward and punishment value are the results inferred by a trained environment model in the target group environment model; Obtaining the i+1th state and the i+1th reward and penalty value inferred by the verification environment model according to the i-th state-action pair, wherein the verification environment model is a trained environment model included in the target environment model except the target group environment model; Calculate a second difference between the i+1th state and the i+1th reward or penalty value and each reference i+1th state and reference i+1th reward or penalty value; If the second difference value exceeds the threshold value more than k / 2 times, it is determined that the i-th state-action pair will be transferred to an unfamiliar state, and a penalty is imposed on the trained agent to prevent the agent from making this decision again; The target intelligent agent is obtained by training the intelligent agent through the unfamiliar state discriminator; The step of training the constructed environment model according to the data set to obtain the target environment model includes: Obtain the data set, and obtain k different training data subsets according to the data set, where the value of k is an integer greater than 1; Construct k regression models; The i-th regression model is trained according to a reward function and the i-th group of training data subset to obtain the i-th trained environment model, wherein the target environment model includes k trained environment models, and the value of i is greater than or equal to 1 and less than or equal to k.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 9 can be implemented.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method described in any one of claims 1 to 9 can be implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, wherein the computer program can implement the method according to any one of claims 1 to 9 when executed by a processor.

Images