A method and device for satellite attitude control based on multi-flywheel array

Abstract

The embodiments of the present application provide a method and device for satellite attitude control based on a multi-flywheel array, the method comprising: obtaining a current attitude attribute value of a satellite; confirming adjustment of the attitude of the satellite according to the current attitude attribute value and a target attitude value; inputting the current attitude attribute value and the target attitude value into a controller, and obtaining a target speed adjustment value corresponding to each flywheel through the controller, wherein the controller is set corresponding to each flywheel in a flywheel array, and the flywheel array is distributed on each axis in a multi-axis. Some embodiments of the present application provide a flywheel array satellite attitude control method based on multi-agent hierarchical reinforcement learning.

Description

Technical Field

[0001] The present application relates to the field of satellite attitude control. Specifically, embodiments of the present application relate to a method and device for satellite attitude control based on a multi-flywheel array. Background Art

[0002] Satellite attitude control mostly uses aerospace-grade devices but they are expensive, so related technologies propose using industrial-grade attitude control component flywheels instead of aerospace-grade devices to reduce costs.

[0003] Flywheels have the characteristics of small angular momentum and small output torque, and can achieve high-precision attitude control of small satellites. They are widely used in modern small satellites. The traditional satellite attitude control system sets a flywheel on the x-axis, y-axis, z-axis and auxiliary axis respectively, and uses a PID controller to control each flywheel. For example, the relevant technology mainly uses a PID controller, which can directly obtain attitude information from the sensor and obtain the control rate through proportional, integral and differential operations according to the target attitude information.

[0004] PID controllers are simple to implement, require little computation, and are highly reliable. They are generally suitable for linear systems, but are very sensitive to sensor noise, require filters, lack environmental adaptability, and require professional background knowledge to adjust parameters. The parameters of different satellites are often different, which consumes a lot of manual work to adjust the coefficients. In the complex environment of nonlinear, strongly coupled multi-flywheel arrays, PID controllers cannot complete high-precision control tasks. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for satellite attitude control based on a multi-flywheel array. In order to solve the problem that traditional control algorithms are unable to complete high-precision control tasks in the complex environment of nonlinear and strongly coupled multi-flywheel arrays, some embodiments of the present application decompose the multi-axis flywheel array into several single-axis multi-flywheel control problems, that is, some embodiments of the present application provide a flywheel array satellite attitude control method based on multi-agent hierarchical reinforcement learning.

[0006] In a first aspect, some embodiments of the present application provide a method for satellite attitude control based on a multi-flywheel array, the method comprising: obtaining a current attitude attribute value of the satellite; confirming an adjustment to the attitude of the satellite based on the current attitude attribute value and a target attitude value; inputting the current attitude attribute value and the target attitude value into a controller, and obtaining a target speed adjustment value corresponding to each flywheel through the controller, wherein the controller is arranged corresponding to each flywheel in the flywheel array, and the flywheel array is distributed on each axis in the multi-axis; wherein the controller is obtained by training the flywheel strategy network using the obtained axis adjustment torque of each axis as a training target.

[0007] Some embodiments of the present application provide an upper-level task (i.e., obtaining the axis adjustment torque of each axis) and a lower-level task (obtaining the speed adjustment value of each flywheel after obtaining the axis adjustment torque of each axis) of a multi-agent hierarchical reinforcement learning satellite attitude control algorithm, that is, decomposing a multi-axis divided-wheel array into several single-axis multi-flywheel control problems, which effectively solves the problem that the traditional control algorithm cannot complete high-precision control tasks in the complex environment of a nonlinear and strongly coupled multi-flywheel array.

[0008] In some embodiments, the multiple axes include an x-axis, a y-axis, and a z-axis constructed based on a body coordinate system of the satellite, wherein the x-axis, the y-axis, and the z-axis are perpendicular to each other.

[0009] In some embodiments of the present application, a flywheel array is arranged on three mutually perpendicular axes to realize satellite attitude control. For example, in some embodiments of the present application, multiple flywheels are arranged side by side, in parallel or in a stacked manner to form a group of flywheel arrays to realize satellite attitude control.

[0010] In some embodiments of the present application, before obtaining the target speed adjustment value corresponding to each flywheel through the controller, the method also includes: obtaining the shaft adjustment torque of each shaft in the multi-axis; and training the flywheel strategy network according to the shaft adjustment torque to obtain the controller.

[0011] Some embodiments of the present application use the obtained axis adjustment torque of each axis as an auxiliary target for training the next layer of intelligent agents, thereby improving the performance of the obtained intelligent agent.

[0012] In some embodiments, the axis adjustment torque is obtained through a first intelligent agent, wherein, before obtaining the axis adjustment torque of each axis in the multi-axis, the method also includes: training a policy network according to a first reward function to obtain the first intelligent agent, wherein the first reward function is positively correlated with the distance between the current posture attribute value and the target posture value and negatively correlated with the time difference from the moment when posture control is started to the current moment.

[0013] Some embodiments of the present application design a reward function required for training a first intelligent agent, through which the trained first intelligent agent has the required target performance.

[0014] In some embodiments, the formula of the first reward function is as follows:

[0015] R1(q t , t x , t y , t z )=-α1D(q t+1 ,q')-β1T

[0016] Among them, q i Used to represent the current posture attribute value obtained during training, q t+1 , represents the next posture attribute value predicted by the strategy network, t x Characterizes the axis adjustment torque of the x-axis, t y Characterizes the axis adjustment torque of the y-axis, t z Characterizes the axis adjustment torque of the z-axis, D(q t+1 ,q') represents the distance between the next posture attribute value and the target posture value, T represents the time difference from the moment of starting posture control to the current moment, and α1 and β1 are hyperparameters.

[0017] Some embodiments of the present application design a specific reward function, through which the trained first intelligent agent has the desired target performance.

[0018] In some embodiments, the first agent is obtained by training the policy network according to the first reward function, including: according to the initial satellite attitude value q0, obtaining the axis adjustment torque a0 of each axis in the three axes through the policy network = (t x ,t y ,t z ); add random interference torque and obtain the first adjusted posture q1 through the satellite dynamics model and the satellite kinematics model, substitute the first adjusted posture q1 and the target posture value q′ into the first reward function to obtain the global reward value r0, repeat this step to obtain a trajectory q0,a0,r0,q1,a1,r1,……,q T-1 ,a T-1 ,r T-1 ,q T ; Put all the data corresponding to the trajectory into the experience pool, and complete the training according to the data in the experience pool; repeat the above process until the training process of the policy network is completed to obtain the first intelligent agent.

[0019] In some embodiments, the training of the strategy network according to the axis adjustment torque to obtain the controller includes: training the flywheel strategy network according to a second reward function to obtain the controller, wherein the second reward function is negatively correlated with the error between the torque obtained by the flywheel strategy network and the expected torque, negatively correlated with the time difference from the moment when posture control is started to the current moment, and negatively correlated with the magnitude of the adjusted angular velocity value.

[0020] Some embodiments of the present application design a reward function required for training a controller, through which the trained controller has the desired target performance.

[0021] In some embodiments, the formula of the second reward function is as follows:

[0022]

[0023] Among them, t′ j is the total torque required on the j-axis after the target is decomposed according to the current posture, j∈{X,Y,Z}, It is the combined adjustment speed obtained by combining the outputs of the strategy networks of each flywheel on the j-axis. The adjustment speed indicates the amount by which the speed of each flywheel increases or decreases. To calculate the total torque generated by all flywheels on the j-axis through the flywheel dynamics model after the flywheel speed is adjusted, t′ j represents the desired torque on the three axes, It represents the error between the torque obtained by the flywheel strategy network and the expected torque. The smaller the difference, the greater the reward value. j It represents the time difference from the start of attitude control to the current time. The shorter the time to complete attitude control, the greater the reward value. To smoothly control the attitude of the satellite, the change in speed cannot be too drastic. The smaller the change in speed, the greater the reward value. j , β j , and μ j are hyperparameters, all greater than 0, measuring the impact on the reward.

[0024] Some embodiments of the present application design a specific reward function, through which the trained first intelligent agent has the desired target performance.

[0025] In some embodiments, the flywheel array includes n flywheels, wherein the controller is obtained by training the policy network according to the second reward function, including: t Input into the flywheel strategy network to obtain the joint action of n control bodies corresponding to any axis The combined action is used to characterize the combined speed change; the combined speed at the next moment can be obtained by the combined speed change and the current speed. The combined speed The updated torque is passed into the flywheel dynamics model Will and the current j-axis required torque t′ j Substitute the second reward function to get the reward value r, and then use the satellite dynamics model and satellite kinematics model to get the satellite attitude q at the next moment i+1 , repeat the process until t>T, and t starts from 0.

[0026] The embodiment of the present application obtains a controller by training a policy network through a value function.

[0027] In some embodiments, the satellite dynamics model is established through dynamics equations, and the satellite kinematics model is established through kinematics equations. The satellite dynamics model is used to obtain the attitude angular velocity vector of the satellite based on the braking torque, and the satellite kinematics model is used to obtain the updated attitude attribute value based on the attitude angular velocity vector.

[0028] In some embodiments, the current posture attribute value and the target posture value are both represented in the form of quaternion, direction cosine matrix or Euler angle.

[0029] In some embodiments, the interference torque is represented by a random function; or, the interference torque is obtained through an interference torque model, wherein the interference torque acquisition model uses a neural network to simulate space interference.

[0030] In a second aspect, some embodiments of the present application provide a device for satellite attitude control based on a multi-flywheel array, the device comprising: a current attitude attribute value acquisition module, configured to obtain the current attitude attribute value of the satellite; a first judgment module, configured to confirm the adjustment of the satellite attitude based on the current attitude attribute value and the target attitude value; a controller, configured to first obtain the axis adjustment torque of the axis belonging to the controller according to the current attitude attribute value, and then obtain the target speed adjustment value of the flywheel set corresponding to the controller according to the axis adjustment torque.

[0031] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0032] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement a method as described in any embodiment of the first aspect when executing the program. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A method flow for satellite attitude control based on a multi-flywheel array provided in an embodiment of the present application;

[0035] Figure 2 A diagram of the posture adjustment system architecture during the training process provided in an embodiment of the present application;

[0036] Figure 3 A network architecture diagram of the target decomposition training phase provided in an embodiment of the present application;

[0037] Figure 4 An architecture diagram for training a model for obtaining adjustment values ​​of the speed of each flywheel provided in an embodiment of the present application;

[0038] Figure 5 A block diagram of the composition of a device for satellite attitude control based on a multi-flywheel array provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0041] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0042] The inventor of the present application has found through research that in order to reduce the impact caused by the reduced reliability of a single device, a flywheel array can be used to increase the reliability of satellite attitude adjustment. For example, a plurality of flywheels are arranged side by side, in parallel or stacked to form a group of flywheel arrays, and the group of flywheel arrays is arranged on one axis. It is understandable that in the complex environment of a nonlinear, strongly coupled multi-flywheel array such as a flywheel array, it is impossible to use a PID controller of the related art to complete high-precision satellite attitude control.

[0043] Each flywheel has a single-chip microcomputer. The traditional PID algorithm only uses it as a controller for the actuator and does not fully utilize its computing power. Some embodiments of the present application place the policy network (i.e., controller) trained based on the multi-agent reinforcement learning algorithm on the single-chip microcomputer, so that each flywheel can make autonomous decisions and form bottom-up group intelligence.

[0044] It can be understood that the control algorithm based on multi-agent reinforcement learning in some embodiments of the present application does not require prior knowledge of the environment, and does not require precise modeling of the controlled object. Through the interaction between the agent and the environment, the agent actively adjusts the strategy through the feedback given by the environment, and autonomously learns to cooperate to complete complex control tasks. It can reduce a lot of manual parameter adjustment time, has good real-time and adaptability, and uses a centralized training and decentralized execution algorithm framework to avoid the dimensional disaster caused by the exponential growth of the action space with the number of agents. Some embodiments of the present application divide complex problems into several simple sub-problems based on hierarchical reinforcement learning and solve them separately, converting the multi-axis attitude control problem into a single-axis multi-agent control problem, reducing the complexity of the problem and greatly reducing the scale of the training network. For example, when training the flywheel strategy network, some embodiments of the present application obtain the axis adjustment torque of each axis through the first agent (that is, according to the current satellite attitude q t After the target is decomposed, the total torque required for any axis j is obtained. Then, the axis adjustment torque is used as the auxiliary training target of the intelligent agent corresponding to the controller, and the intelligent agent is trained to obtain the controller. In specific applications, the current attitude attribute value of the collected satellite is directly input into the controller, and the controller will output the speed adjustment value of each flywheel that meets the requirements.

[0045] In some embodiments of the present application, the flywheels of the multi-flywheel array are distributed in three-dimensional orthogonality and are respectively installed on the x-axis, y-axis and z-axis of the satellite body coordinate system. There are multiple flywheels on each axis (multiple flywheels form a flywheel array), and there are influences between the flywheels. The purpose of the control algorithm provided in the embodiments of the present application is to ensure that the flywheel array can work together to complete complex attitude control tasks, including the deployment of other actuators when the flywheel array becomes saturated and cannot continue to complete satellite attitude control.

[0046] In recent years, reinforcement learning has achieved good results in the fields of drones, autonomous driving, robot cluster control, and strategic games. It can complete multiple complex tasks end-to-end without human intervention and has great potential in the field of control. Multi-agent reinforcement learning enables multiple agents to work together to complete various complex tasks in a real-time dynamic environment, in cooperative or competitive scenarios.

[0047] like Figure 1As shown, some embodiments of the present application provide a method for satellite attitude control based on a multi-flywheel array, the method comprising: S101, obtaining a current attitude attribute value of the satellite; S102, confirming adjustment of the attitude of the satellite according to the current attitude attribute value and the target attitude value; S103, inputting the current attitude attribute value and the target attitude value into a controller, and obtaining a target speed adjustment value corresponding to each flywheel through the controller, wherein the controller is arranged corresponding to each flywheel in the flywheel array, and the flywheel array is distributed on each axis in the multi-axis; wherein the controller is obtained by training the flywheel strategy network using the obtained axis adjustment torque of each axis as a training target.

[0048] It is understandable that in order to enable the controller to have the above functions, the policy network needs to be trained first to obtain the controller. The following is an exemplary description of the process of training the policy network to obtain the controller.

[0049] For example, in order to obtain a controller, in some embodiments of the present application, before obtaining the target speed adjustment value corresponding to each flywheel through the controller, the method also includes: obtaining the shaft adjustment torque of each shaft in the multi-axis; training the strategy network according to the shaft adjustment torque to obtain the controller.

[0050] That is to say, some embodiments of the present application provide upper-level tasks (i.e., obtaining the axis adjustment torque of each axis) and lower-level tasks (obtaining the speed adjustment value of each flywheel after obtaining the axis adjustment torque of each axis) of a multi-agent hierarchical reinforcement learning satellite attitude control algorithm, that is, decomposing the multi-axis divided wheel array into several single-axis multi-flywheel control problems, which effectively solves the problem that the traditional control algorithm cannot complete high-precision control tasks in the complex environment of a nonlinear and strongly coupled multi-flywheel array.

[0051] It should be noted that in order to obtain the controller, the strategy network model must first be trained so that the strategy network model has the function of performing target decomposition according to the current posture attribute value and the target posture value (that is, obtaining the axis adjustment torque corresponding to each axis) and has the function of obtaining the change in the speed of each flywheel in the flywheel array of each axis according to the axis adjustment torque of each axis (that is, obtaining the speed adjustment value).

[0052] In some embodiments of the present application, the multi-axis includes an x-axis, a y-axis, and a z-axis constructed based on the satellite's body coordinate system, wherein the x-axis, the y-axis, and the z-axis are perpendicular to each other. In some embodiments of the present application, a flywheel array is arranged on three mutually perpendicular axes to realize satellite attitude control. For example, in some embodiments of the present application, multiple flywheels are arranged side by side, in parallel, or stacked to form a group of flywheel arrays to realize satellite attitude control.

[0053] In some embodiments of the present application, the first agent (not shown in the figure, the agent is included in the controller) realizes the target decomposition function. The following is an exemplary description of the process of obtaining the first agent through training. The shaft adjustment torque of each shaft can be obtained through the first agent, and then the flywheel strategy network is trained according to the shaft adjustment torque to obtain the controller.

[0054] That is to say, in some embodiments of the present application, the axis adjustment torque is obtained by a first agent, wherein, before obtaining the axis adjustment torque of each axis in the multi-axis, it includes: training the policy network according to a first reward function to obtain the first agent, wherein the first reward function is positively correlated with the distance between the current posture attribute value and the target posture value and negatively correlated with the time difference from the moment when posture control is started to the current moment. Some embodiments of the present application design a reward function required for training the first agent, and the reward function is used to make the trained first agent have the desired target performance.

[0055] For example, in some embodiments of the present application, the formula of the first reward function is as follows:

[0056] R1(q t , t x , t y , t z )=-α1D(q t+1 ,q')-β1T

[0057] Among them, q t Used to represent the current posture attribute value obtained during training, q t+1 , represents the next posture attribute value predicted by the strategy network, t x Characterizes the axis adjustment torque of the x-axis, t y Characterizes the axis adjustment torque of the y-axis, t z Characterizes the axis adjustment torque of the z-axis, D(q t+1 ,q') represents the distance between the next posture attribute value and the target posture value, T represents the time difference from the moment of starting posture control to the current moment, and α1 and β1 are hyperparameters.

[0058] Some embodiments of the present application design a specific reward function, through which the trained first intelligent agent has the desired target performance.

[0059] For example, in some embodiments of the present application, the first agent is obtained by training the policy network according to the first reward function, including: according to the initial satellite attitude value q0, obtaining the axis adjustment torque a0 of each axis in the three axes through the policy network = (t x ,ty ,t z ); add random interference torque and obtain the first adjusted posture q1 through the satellite dynamics model and the satellite kinematics model, substitute the first adjusted posture q1 and the target posture value q′ into the first reward function to obtain the global reward value r0, repeat this step to obtain a trajectory q0,a0,r0,q1,a1,r1,……,q T-1 ,a T-1 ,r T-1 ,q T ; Put all the data corresponding to the trajectory into the experience pool, and complete the training according to the data in the experience pool; repeat the above process until the training process of the policy network is completed to obtain the first intelligent agent.

[0060] The following is an example of the process of training the flywheel strategy network to obtain a controller by adjusting the shaft torque.

[0061] In some embodiments of the present application, the controller is obtained by training the flywheel strategy network according to the axis adjustment torque: the controller is obtained by training the strategy network according to a second reward function, wherein the second reward function is negatively correlated with the error between the torque obtained by the strategy network and the expected torque, negatively correlated with the time difference from the moment when the posture control is started to the current moment, and negatively correlated with the magnitude of the adjusted angular velocity value. Some embodiments of the present application design a reward function required for training the controller, through which the trained controller has the desired target performance.

[0062] For example, in some embodiments of the present application, the formula of the second reward function is as follows:

[0063]

[0064] Among them, t′ j is the total torque required on the j-axis after the target is decomposed according to the current posture, j∈{X,Y,Z}, It is the combined adjustment speed obtained by combining the outputs of the strategy networks of each flywheel on the j-axis. The adjustment speed indicates the amount by which the speed of each flywheel increases or decreases. To calculate the total torque generated by all flywheels on the j-axis through the flywheel dynamics model after the flywheel speed is adjusted, t′ j represents the desired torque on the three axes, represents the error between the torque obtained by the flywheel strategy network and the expected torque. The smaller the difference, the greater the reward value. xIt represents the time difference from the start of attitude control to the current time. The shorter the time to complete attitude control, the greater the reward value. To smoothly control the attitude of the satellite, the change in speed cannot be too drastic. The smaller the change in speed, the greater the reward value. j , β j , and μ j are hyperparameters, all greater than 0, measuring the impact on the reward.

[0065] Some embodiments of the present application design a specific reward function, through which the trained first intelligent agent has the desired target performance.

[0066] For example, in some embodiments of the present application, the flywheel array includes n flywheels, wherein the flywheel strategy network is trained according to the second reward function to obtain the controller, including: t Input into the flywheel strategy network to obtain the joint action of n control bodies corresponding to any axis j∈{X,Y,Z}, wherein the joint action is used to characterize the change in joint speed; the joint speed at the next moment can be obtained by the change in joint speed and the current speed The combined speed The updated torque is passed into the flywheel dynamics model Will and the current j-axis required torque t′ j Substitute the second reward function to get the reward value r, and then use the satellite dynamics model and satellite kinematics model to get the satellite attitude q at the next moment t+1 , repeat the process until t>T, and t starts from 0.

[0067] In some embodiments, the dynamic model is established through dynamic equations, and the satellite kinematic model is established through kinematic equations. The satellite dynamic model is used to obtain the attitude angular velocity vector of the satellite according to the braking torque, and the satellite kinematic model is used to obtain the updated attitude attribute value according to the attitude angular velocity vector.

[0068] In some embodiments, the current posture attribute value and the target posture value are both represented in the form of quaternion, direction cosine matrix or Euler angle.

[0069] In some embodiments, the interference torque is represented by a random function; or, the interference torque is obtained through an interference torque model, wherein the interference torque acquisition model uses a neural network to simulate space interference.

[0070] Combine the following Figure 2-Figure 4 The training process of the present application is described exemplarily.

[0071] Please see Figure 2 , Figure 2 This is a schematic diagram of some embodiments of the present application. Figure 2 The architecture includes: multiple flywheels arranged on three axes, a controller arranged for each flywheel, a sensor for obtaining satellite attitude attribute values, a dynamic model of the flywheel, a dynamic model of the satellite, and a kinematic model of the satellite. For example, Figure 2 The sensor collects the current attitude attribute value of the satellite, and each controller obtains the speed adjustment value of the corresponding flywheel (i.e., obtains the target speed adjustment value) according to the target attitude value and the current attitude attribute value. In the process of training the controller, it is necessary to judge whether the training process of the controller has been completed according to the speed adjustment value. The process of obtaining the controller by training the policy network according to the reward function will be described in detail below, and no further details will be given here to avoid repetition.

[0072] In order to solve the problem that traditional control algorithms cannot complete high-precision control tasks in the complex environment of nonlinear and strongly coupled multi-flywheel arrays, some embodiments of the present application decompose the multi-axis flywheel array into several single-axis multi-flywheel control problems. Some embodiments of the present application provide a flywheel array satellite attitude control method based on multi-agent hierarchical reinforcement learning. The method includes the following steps:

[0073] Step 1: Get the current satellite attitude q from the satellite sensor t (i.e., obtaining the current posture attribute value);

[0074] Step 2: Determine whether the satellite target attitude is reached (i.e., the target attitude value is reached). If D(q t+1 ,q')<ε, the target posture control is completed and the posture control ends. Otherwise, continue with steps 3-7;

[0075] Step 3: Set the current satellite attitude (i.e., the current satellite attitude attribute value) q t The satellite target attitude (i.e., target attitude value) q′ is transmitted to the controller of each flywheel, and the change in the rotational speed of each flywheel (i.e., the target rotational speed adjustment) can be obtained.

[0076] Step 4: Add a continuous disturbance torque (represented by a random function), the interference torque is used to simulate the normal interference environmental torque in space, such as gravity gradient torque, sunlight pressure torque, aerodynamic torque, etc.

[0077] Step 5: Pass the various rotational speeds of the flywheel into the flywheel's dynamic model to obtain the three-axis torque t generated by the flywheel.

[0078] Step 6: Add the interference torque to the three-axis torque generated by the flywheel The total torque t' is obtained as the input of the satellite's dynamic model, and the attitude angular velocity ω generated by the multi-axis flywheel array is calculated;

[0079] Step 7: Using ω as the input of the satellite’s kinematic model, we can derive the satellite’s attitude q at the next moment. t+1 , pass it into the sensor, update the current satellite attitude, and continue with step 1.

[0080] The above seven steps are explained one by one below.

[0081] In step 1 above, in attitude control problems, quaternions are used because of their small number of parameters and simple calculation process.

[0082]

[0083] Widely used. Quaternion representation q = [q0, q1, q2, q3] T , defined as:

[0084] Where, e=[e x , e y , e z ] T Euler axis, Φ is the rotation angle, and the quaternion parameters need to meet the conditions

[0085] The attitude matrix is ​​defined as follows:

[0086]

[0087] Based on the attitude matrix, the attitude of the satellite can be represented by the attitude quaternion.

[0088] Since some embodiments of the present application need to conduct experiments in a satellite simulation environment, it is first necessary to build a satellite dynamic model and a satellite kinematic model.

[0089] Considering the satellite as a rigid body, according to the dynamic equation of the rigid body, the dynamic equation of the satellite can be obtained as follows:

[0090] H=Jω

[0091]

[0092] Combining the two formulas, we get

[0093]

[0094] Among them, H represents the angular momentum of the satellite, J represents the moment of inertia of the satellite, ω represents the attitude angular velocity vector of the satellite, u represents the three-axis braking torque vector, and d represents the interference torque acting on the satellite.

[0095] The kinematic model of the satellite represents the law of change of attitude quaternion with angular velocity, as follows:

[0096]

[0097] In the above step 2, according to the current satellite attitude q and the target attitude q′, the distance between the current attitude and the target attitude is calculated to determine whether D(q,q') is less than ∈. If it is less than ∈, it means that the satellite attitude control is completed and the current attitude control task is ended. Otherwise, continue with the following steps to adjust the satellite attitude to achieve the target attitude.

[0098] In step 3 above, the problem of controlling multiple flywheels on three axes is first simplified to controlling a single flywheel on three axes. The multiple flywheels on each axis are regarded as a whole, and the learning target is decomposed. The input is the current satellite attitude q t and the satellite target attitude q′, the required torques (t x ,t y ,t z )(i.e., the axis adjustment torque of the three axes is obtained). After the target is decomposed, the flywheel of each axis has the decomposed target of the corresponding axis, i.e., the required torque t i ,i∈{x,y,z}, and a total goal, the final target posture q' that needs to be achieved, the control algorithm of the flywheel is trained according to the designed reward function of each flywheel, and finally the speed change of each flywheel can be obtained according to the current posture.

[0099] First, we need to design the target decomposition reward function (i.e. the first reward function):

[0100] R1(q t ,t x ,t y ,t z )=-α1D(q t+1 ,q')-β1T

[0101] Among them, D1(q t+1 ,q') represents the distance between the current posture and the target posture. The closer the distance, the greater the reward value. x Characterizes the axis adjustment torque of the x-axis, t y Characterizes the axis adjustment torque of the y-axis, t z Characterizes the axis adjustment torque of the z-axis, T represents the time difference from the moment of starting posture control to the current moment, and the shorter the time to complete posture control, the greater the reward value. α1 and β1 are hyperparameters, both greater than 0, which measure the impact on the reward.

[0102] Experience Pool (store the collected trajectory), k represents the number of network training iterations, taking values ​​of 0, 1, 2, 3, ..., K, the step length of each round of network iteration is T, and the batch size is B.

[0103] a) Randomly initialize the policy network Sum value network {φ0}, or pre-initialize the network using transfer learning;

[0104] b) Initialize the target value network

[0105] c) Use a joint strategy Collect a track;

[0106] According to the initial satellite attitude value q0, the axis adjustment torque a0 of each axis in the three axes is obtained through the strategy network. x ,t y ,t z ); add random interference torque and obtain the first adjusted posture q1 through the satellite dynamics model and the satellite kinematics model, substitute the first adjusted posture q1 and the target posture value q′ into the first reward function to obtain the global reward value r0, repeat this step to obtain a trajectory q0,a0,r0,q1,a1,r1,……,q T-1 ,a T-1 ,r T-1 ,q T .

[0107] d) Add to experience pool middle;

[0108] e) From the experience pool B strips are randomly sampled from the middle;

[0109] f) According to the TD algorithm, update V(q t );

[0110] V(q t )=r t +γV(q t+1 )

[0111] Where γ represents the discount factor γ∈[0,1]

[0112] Target Value Network Enter q t+1 get

[0113]

[0114] Compute network gradients After updating the value network, η1 is the learning rate, 0<η1<1

[0115]

[0116] g) Update the strategy network;

[0117] The objective function of the policy network is:

[0118]

[0119] Calculate the advantage function, Indicates that except for the agent The combined action of all agents except Perform statistics to get expected value

[0120]

[0121] Calculate the policy network gradient:

[0122]

[0123] Update the policy network parameters, where η2 is the learning rate, 0<η2<1

[0124]

[0125] h) Update target value network parameters;

[0126]

[0127] i) Repeat step c until k>K, and end the network training.

[0128] During the flywheel control algorithm training phase, the role of this target decomposition network is as follows:

[0129] Instead of using the value network, the strategy network corresponding to each axis can derive the corresponding torque q based on the current satellite attitude q i .

[0130] q i =π i (q),i∈{x,y,z}

[0131] Because each axis has homogeneity, the same state space and the same action space, the policy network can use parameter sharing, greatly reducing the computation of the network.

[0132] The policy network on the actuator of the single-axis flywheel array is trained using a multi-agent reinforcement learning control algorithm. Each flywheel is an agent with an independent policy network. The algorithm uses a centralized training decentralized execution framework and is divided into a training phase and an execution phase. Using hierarchical reinforcement learning, the problem is abstracted into two simple sub-problems, and the goals of the next layer of sub-problems are designed. Each controller of the single-axis flywheel array can output the change in the speed of the corresponding flywheel according to the target posture.

[0133] The training phase steps of the flywheel array control algorithm are as follows:

[0134] Taking the x-axis as an example, assuming that the number of flywheels is n, the flywheels are heterogeneous (have different action spaces), and the same strategy cannot be used. The threshold is ∈, and the experience pool k represents the number of network training iterations, taking values ​​of 0, 1, 2, ..., K. The step length of each round of network iteration is T, and the batch size is B.

[0135] First, design the reward function (that is, the second reward function corresponding to the x-axis):

[0136]

[0137] t′ x It is the axis adjustment torque of the target axis required according to the current posture. is the current combined speed of all flywheels, is the combined output obtained through the strategy network of each flywheel, indicating the amount by which the flywheel speed increases or decreases, and is determined by the current speed Change in flywheel speed The speed of the flywheel at the next moment can be obtained When the combined speed of the flywheel is When , the total torque generated by all flywheels on the x-axis is calculated through the flywheel dynamics model. Represents the error between the torque obtained by the strategy network and the expected torque, and the q at the next moment t+1 , the smaller the gap, the greater the reward value; T x It represents the time difference from the moment when attitude control starts to the current moment. The shorter the time to complete attitude control, the greater the reward value. To smoothly control the attitude of the satellite, the change in speed cannot be too drastic. The smaller the change in speed, the greater the reward value. x ,β x and μ x are hyperparameters, all greater than 0, measuring the impact on the reward.

[0138] a. Randomly initialize the policy network Sum value network {φ0}, or pre-initialize the network using transfer learning;

[0139] b. Initialize the target value network

[0140] c. Use joint strategies Collect a track;

[0141] ω t Input into the joint strategy network to obtain the joint action of all agents By combining the speed change and the current speed, the combined speed ω at the next moment can be obtained. t+1 , will ω t+1 The combined speed at the next moment is passed into the flywheel dynamics model to obtain a new torque Will and t' x Substitute the reward function to get the reward value r, and then use the satellite's dynamic and kinematic models to get the satellite's attitude q at the next moment t+1 ,According to the target decomposition, the target torque at the next moment is obtained, and the process is repeated until t>T, and t starts to take values ​​from 0.

[0142] d. Add to experience pool middle;

[0143] e. From the experience pool B strips are randomly sampled from the middle;

[0144] f. Update V(q according to the TD algorithm t );

[0145] V(q t )=r t +γV(q t+1 )

[0146] Here, γ represents the discount factor γ∈[0,1].

[0147] Target Value Network Enter q t+1 get

[0148]

[0149] Compute network gradients After updating the value network, η1 is the learning rate, 0<η1<1

[0150]

[0151] g. Update strategy network;

[0152] The objective function of the policy network is:

[0153]

[0154] Calculate the advantage function, represents the joint action of all agents except agent i, for all agents in the experience pool Perform statistics to get expected value

[0155]

[0156] Calculate the policy network gradient:

[0157]

[0158] Update the policy network parameters, where η2 is the learning rate, 0<η2<1

[0159]

[0160] h. Update target value network parameters;

[0161]

[0162] i. Repeat step c until k>K, and end network training.

[0163] The final flywheel policy network execution is as follows:

[0164] The value network is no longer used in the execution phase. The strategy network of each agent can derive the corresponding change in flywheel speed a based on q. i (ie target speed adjustment value).

[0165] a i =π i (q),i∈N

[0166] In the above step 5, the random function can be represented by a continuous function. i≠j or i≠j, the interference torque of each agent can be the same or different, but the torque needs to be continuous and the difference between the interference torques of the agents is not large. The final joint torque is

[0167]

[0168] The overall flow chart of this method is as follows Figure 1 As shown in the diagram, the algorithm architecture is as follows Figure 2 shown.

[0169] It should be noted that the satellite attitude in step 1 above can be represented by more than quaternions, and can be represented by other complex forms, such as direction cosine matrix, Euler angle, and the satellite's dynamic model and kinematic model can be based on real satellite attitude control data and use neural networks for supervised learning to model the space environment. The interference torque in step 5 above can be represented by a random function, or an interference torque model can be established, and a neural network can be used to simulate complex space interference.

[0170] Figure 2 is the flow chart of attitude control, Figure 3 and Figure 4 This is an algorithm architecture diagram. The overall process is described in detail. In the above two controller algorithms, the parameters also correspond one to one. The overall process is briefly described as follows:

[0171] like Figure 3 As shown, each controller actor obtains the current satellite attitude from the environment and can obtain the adjustment torque t through its own strategy network. i , i∈{x,y,z}, and transmits the adjustment torque to the value network. The value network returns the value A(t i ,q),i∈{x,y,z}, each controller updates its own policy network according to the value of the advantage function.

[0172] like Figure 4 As shown in the training phase: get the current satellite attitude q from the environment, first decompose the target and pass it into Actori,i∈{X,Y,Z}, and get the adjustment torque t of the x-axis, y-axis, and z-axis x ,t y ,t z , and then pass it to the value network of the corresponding axis of the flywheel to form the reward function of the value network in the training phase. The flywheel controller obtains the current satellite attitude q from the environment and outputs the speed adjustment value through its own strategy network i∈{X,Y,Z},j∈{1,…,n}, the speed adjustment value is passed to the value network, and the value network returns the value of the advantage function corresponding to each controller based on the joint speed adjustment value and the current satellite attitude. i∈{X,Y,Z},j∈{1,…,n}, each controller updates its own strategy network according to the value of the advantage function. Execution phase: Each flywheel controller directly outputs the adjusted speed of the flywheel according to the current satellite attitude i∈{X,Y,Z},j∈{1,…,n}.

[0173] like Figure 5 As shown, Figure 5The device for satellite attitude control based on a multi-flywheel array provided in an embodiment of the present application is shown. It should be understood that the device is similar to the above-mentioned Figure 1 The method embodiment corresponds to the method embodiment, and can execute each step involved in the above method embodiment. The specific functions of the device can refer to the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system of the device. The device for satellite attitude control based on a multi-flywheel array includes: a current attitude attribute value acquisition module 101, a judgment module 102 and a control module 103.

[0174] The current attitude attribute value acquisition module is configured to obtain the current attitude attribute value of the satellite; the judgment module is configured to confirm the adjustment of the satellite attitude according to the current attitude attribute value and the target attitude value; the control module is configured to first obtain the shaft adjustment torque of the shaft belonging to the controller according to the current attitude attribute value, and then obtain the target speed adjustment value of the flywheel set corresponding to the controller according to the shaft adjustment torque.

[0175] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which can implement the following when the program is executed by a processor: Figure 2 The method of any embodiment.

[0176] like Figure 6 As shown, some embodiments of the present application provide an electronic device 500, including a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 reads the program from the memory 520 through a bus 530 and executes the program to achieve the following Figure 1 The method of any embodiment.

[0177] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0178] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code to implement some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the present disclosure embodiment may be used to execute the instructions in the memory 510 to implement Figure 1 The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.

[0179] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0180] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0181] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0182] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0183] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0184] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for satellite attitude control based on a multi-flywheel array, characterized in that: The method comprises: Get the current attitude attribute value of the satellite; Confirming adjustment of the satellite's attitude according to the current attitude attribute value and the target attitude value; Inputting the current attitude attribute value and the target attitude value into a controller, and obtaining a target speed adjustment value corresponding to each flywheel through the controller, wherein the controller is set corresponding to each flywheel in a flywheel array, and the flywheel array is distributed on each axis in a multi-axis; Wherein, the controller is obtained by training the flywheel strategy network by taking the shaft adjustment torque of each shaft obtained as the training target; The axis adjustment torque is obtained by the first agent, wherein, Before obtaining the target speed adjustment value corresponding to each flywheel through the controller, the method includes: The first agent is obtained by training the flywheel strategy network according to a first reward function, wherein the first reward function is positively correlated with the distance between the current posture attribute value and the target posture value and negatively correlated with the time difference from the moment when posture control is started to the current moment; The step of training the strategy network according to the axis adjustment torque to obtain the controller comprises: The controller is obtained by training the flywheel strategy network according to a second reward function, wherein the second reward function is negatively correlated with an error between a torque obtained by the flywheel strategy network and a desired torque, negatively correlated with a time difference from a moment when posture control is started to a current moment, and negatively correlated with a magnitude of an adjusted angular velocity value; The formula of the second reward function is as follows: Among them, t′ j is based on the current satellite attitude q t The total torque required on the j-axis after target decomposition, j∈{X,Y,Z}, It is the combined adjustment speed obtained by combining the outputs of each flywheel strategy network on the j axis. To calculate the total torque generated by all flywheels on the j-axis through the flywheel dynamics model after the flywheel speed is adjusted, t′ j represents the desired torque on the three axes, represents the error between the torque obtained by the flywheel strategy network and the desired torque, T j represents the time difference from the start of attitude control to the current time, α j , β j , and μ j is a hyperparameter.

2. The method according to claim 1, characterized in that The multiple axes include an x-axis, a y-axis, and a z-axis constructed based on a body coordinate system of the satellite, wherein the x-axis, the y-axis, and the z-axis are perpendicular to each other.

3. The method according to claim 2, characterized in that Before obtaining the target speed adjustment value corresponding to each flywheel through the controller, the method further includes: Obtaining the axis adjustment torque of each axis in the multi-axis; The controller is obtained by training the flywheel strategy network according to the shaft adjustment torque.

4. The method according to claim 3, characterized in that The formula of the first reward function is as follows: R1(q t ,t x ,t y ,t z )=-α1D(q t+1 ,q′)-β1T. Among them, q t Used to represent the current posture attribute value obtained during training, q t+1 , represents the next posture attribute value predicted by the strategy network, t x Characterizes the axis adjustment torque of the x-axis, t y Characterizes the axis adjustment torque of the y-axis, t z Characterizes the axis adjustment torque of the z-axis, D(q t+1 ,q') represents the distance between the next posture attribute value and the target posture value, T represents the time difference from the moment of starting posture control to the current moment, and α1 and β1 are hyperparameters.

5. The method according to claim 4, characterized in that The step of training the policy network according to the first reward function to obtain the first agent includes: According to the initial satellite attitude value q0, the axis adjustment torque a0 of each axis in the three axes is obtained through the strategy network. x ,t y ,t z ); Add random interference torque and obtain the first adjusted posture q1 through the satellite dynamics model and the satellite kinematics model. Substitute the first adjusted posture q1 and the target posture value q′ into the first reward function to obtain the global reward value r0. Repeat this step to obtain a trajectory q0, a0, r0, q1, a1, r1, ..., q T-1 ,a T-1 ,r T-1 ,q T ; Put all data corresponding to the trajectory into the experience pool, and complete the training according to the data in the experience pool; The above process is repeated until the training process of the policy network is completed to obtain the first agent.

6. The method according to claim 1, characterized in that The flywheel array includes n flywheels, wherein: The step of training the policy network according to the second reward function to obtain the controller includes: The current satellite attitude q t Input into the flywheel strategy network to obtain the joint action of n control bodies corresponding to any axis Wherein, the combined action is used to characterize the combined speed change; The combined speed at the next moment is obtained by the combined speed change and the current speed. The combined speed The updated torque is passed into the flywheel dynamics model Will and t j Substitute the second reward function to get the reward value r, and then use the satellite dynamics model and satellite kinematics model to get the satellite attitude q at the next moment t+1 , repeat the process until t>T, and t starts from 0.

7. The method according to claim 5 or 6, characterized in that The satellite dynamics model is established through dynamics equations, and the satellite kinematics model is established through kinematics equations. The satellite dynamics model is used to obtain the attitude angular velocity vector of the satellite according to the braking torque, and the satellite kinematics model is used to obtain the updated attitude attribute value according to the attitude angular velocity vector.

8. The method according to claim 1, characterized in that The current posture attribute value and the target posture value are both represented in the form of quaternion, direction cosine matrix or Euler angle.

9. The method according to claim 5, characterized in that The interference torque is represented by a random function; or, The interference torque is obtained through an interference torque model, wherein the interference torque acquisition model uses a neural network to simulate space interference.

10. A device for satellite attitude control based on a multi-flywheel array, characterized in that: The device comprises: A current attitude attribute value acquisition module is configured to acquire the current attitude attribute value of the satellite; A first judgment module is configured to confirm adjustment of the satellite attitude according to the current attitude attribute value and the target attitude value; A controller configured to obtain a target speed adjustment value of a flywheel corresponding to the controller according to the current posture attribute value; Wherein, the controller is obtained by training the flywheel strategy network by taking the shaft adjustment torque of each shaft obtained as the training target; The axis adjustment torque is obtained by the first agent, wherein, Before obtaining the target speed adjustment value corresponding to each flywheel through the controller, the method includes: The first agent is obtained by training the flywheel strategy network according to a first reward function, wherein the first reward function is positively correlated with the distance between the current posture attribute value and the target posture value and negatively correlated with the time difference from the moment when posture control is started to the current moment; The step of training the strategy network according to the axis adjustment torque to obtain the controller comprises: The controller is obtained by training the flywheel strategy network according to a second reward function, wherein the second reward function is negatively correlated with an error between a torque obtained by the flywheel strategy network and a desired torque, negatively correlated with a time difference from a moment when posture control is started to a current moment, and negatively correlated with a magnitude of an adjusted angular velocity value; The formula of the second reward function is as follows: Among them, t′ j is based on the current satellite attitude q t The total torque required on the j-axis after target decomposition, j∈{X,Y,Z}, is the combined adjustment speed obtained by combining the outputs of the flywheel strategy networks of the j-axis, and is the total torque generated by all flywheels of the j-axis calculated by the flywheel dynamics model after the flywheel speed adjustment, t′ j represents the desired torque on the three axes, represents the error between the torque obtained by the flywheel strategy network and the desired torque, T j represents the time difference from the start of attitude control to the current time, α j , β j , and μ j is a hyperparameter.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 9 can be implemented.

12. An information processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method described in any one of claims 1 to 9 can be implemented.

Images