A satellite attitude control method, control device, computer equipment and medium

Abstract

The present application provides a control method, control device, computer equipment and medium for satellite attitude, and the control method includes: obtaining the operation information and target attitude angle of the target satellite at the last moment; inputting the target attitude angle and the operation information at the last moment into a reinforcement learning network with real-time parameter adjustment, and calculating the control parameter value of the momentum wheel in the target satellite at the current moment; adjusting the attitude angle of the target satellite at the current moment according to the calculated control parameter value at the current moment, so that the target satellite runs toward the target attitude angle. In an embodiment of the present application, when the parameters such as the shape and mass of the target satellite can change with time, the present application provides a reinforcement learning network that can adjust parameters in real time, and under the current parameters of the target satellite, uses the reinforcement learning network to learn an optimal control strategy for the current target satellite, so as to adjust the target satellite to the target attitude angle faster and more accurately.

Description

Technical Field

[0001] The present application relates to the field of satellite control, and in particular, to a satellite attitude control method, a control device, a computer device and a medium. Background Art

[0002] With the development of technology, people need more and more information, but the information collected by humans on the surface of the earth can no longer meet human needs, so artificial satellites were born. At present, artificial satellites are operating in their corresponding orbits, but different satellites may have different functions and uses. Generally, satellites can be divided into communication satellites, meteorological satellites, resource satellites, reconnaissance satellites, navigation satellites, etc. The attitude of satellites in orbit is not fixed. Satellites obtain different information by adjusting different attitudes. At this stage, each satellite has a fixed control algorithm to determine the attitude of the satellite. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide a satellite attitude control method, control device, computer equipment and medium, which are used to solve the problem in the prior art that the attitude of the target satellite cannot be accurately adjusted.

[0004] In a first aspect, an embodiment of the present application provides a method for controlling a satellite attitude, including:

[0005] Obtain the last moment operation information and target attitude angle of the target satellite;

[0006] Inputting the target attitude angle and the operation information of the previous moment into a reinforcement learning network for real-time parameter adjustment, and calculating the control parameter value of the momentum wheel in the target satellite at the current moment;

[0007] According to the calculated control parameter value at the current moment, the attitude angle of the target satellite at the current moment is adjusted so that the target satellite moves toward the target attitude angle.

[0008] Optionally, the control method further includes:

[0009] Acquire the current operation information of the target satellite after the momentum wheel adjusts the control parameter value, and if the current attitude angle in the current operation information is not the target attitude angle, adjust the parameters in the reinforcement learning network;

[0010] The operation information in the current operation information is used as the new operation information of the previous moment, and the reinforcement learning network after adjusting the parameters is used as the new reinforcement learning network. The step of re-executing the target attitude angle and the new operation information of the previous moment are input into the new reinforcement learning network, and the control parameter value of the momentum wheel of the target satellite at the new current moment is calculated. The step of adjusting the current attitude angle of the target satellite according to the calculated new control parameter value at the current moment so that the target satellite runs toward the target attitude angle, and obtaining the current operation information of the target satellite after the control parameter value of the momentum wheel is adjusted. If the current attitude angle in the current operation information is not the target attitude angle, the parameters in the reinforcement learning network are adjusted until the adjusted attitude angle of the target satellite is the target attitude angle.

[0011] Optionally, the difference between the control parameter value of the momentum wheel at the current moment and the control parameter value of the momentum wheel in the operation information at the previous moment is less than a preset parameter change.

[0012] Optionally, the control method further includes:

[0013] The operation information at the previous moment, the operation information at the current moment, the target posture angle, the action category and the reward value are stored as a group of experience data in the experience pool of the real-time parameter adjustment reinforcement learning network.

[0014] Optionally, the parameter adjustment process of the reinforcement learning network includes:

[0015] Randomly sample from the experience pool to obtain multiple historical experience data;

[0016] The plurality of historical experience data and the loss function are used to adjust parameters of two action value networks and an actor network in the reinforcement learning network.

[0017] Optionally, the reward value is determined by the following steps:

[0018] The reward value is determined by using the difference between the posture angle of the current running information in the experience data and the target posture angle.

[0019] Optionally, the action category is determined based on a comparison result between the rotation speed in the current operation information and the rotation speed at a previous moment in the previous operation information.

[0020] In a second aspect, an embodiment of the present application provides a satellite attitude control device, including:

[0021] An acquisition module is used to obtain the last moment operation information and target attitude angle of the target satellite;

[0022] A calculation module is used to input the target attitude angle and the operation information of the previous moment into a reinforcement learning network for real-time parameter adjustment, and calculate the control parameter value of the momentum wheel in the target satellite at the current moment;

[0023] The adjustment module is used to adjust the attitude angle of the target satellite at the current moment according to the calculated control parameter value at the current moment, so that the target satellite moves towards the target attitude angle.

[0024] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0026] The target attitude control method proposed in the embodiment of the present application first obtains the target satellite's operating information and target attitude angle at the previous moment; secondly, inputs the target attitude angle and the operating information at the previous moment into a reinforcement learning network with real-time parameter adjustment, and calculates the control parameter value of the momentum wheel in the target satellite at the current moment; finally, according to the calculated control parameter value at the current moment, adjusts the attitude angle of the target satellite at the current moment, so that the target satellite runs toward the target attitude angle.

[0027] In some embodiments, when parameters such as the shape and mass of the target satellite may change over time, the present application provides a reinforcement learning network that can adjust parameters in real time. Under the current parameters of the target satellite, an optimal control strategy for the current target satellite learned by the reinforcement learning network is used to achieve faster and more accurate adjustment of the target satellite to the target attitude angle.

[0028] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1A schematic diagram of a flow chart of a satellite attitude control method provided in an embodiment of the present application;

[0031] Figure 2 A flowchart of a detailed satellite attitude control method provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of the structure of a satellite attitude control device provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0035] At present, the satellite attitude control technology uses the PID control algorithm. For each satellite's PID control algorithm, each parameter in the PID control algorithm is determined according to the shape, mass and parameters of each device of the satellite. That is to say, the PID control algorithm corresponding to each satellite can only be applied to this satellite. When the shape, mass and status of each device of the satellite change with the time in orbit, the control effect of the PID control algorithm will be greatly reduced. If you want to adjust the parameters in the PID control algorithm, you need certain expert experience and time cost to achieve the adjustment, and the adjustment method is relatively cumbersome.

[0036] Based on the above defects, the present application embodiment provides a method for controlling the attitude of a satellite, such as Figure 1 As shown, the following steps are included:

[0037] S101, obtaining the last moment operation information and target attitude angle of the target satellite;

[0038] S102, inputting the target attitude angle and the operation information at the previous moment into a reinforcement learning network for real-time parameter adjustment, and calculating the control parameter value of the momentum wheel in the target satellite at the current moment;

[0039] S103, adjusting the attitude angle of the target satellite at the current moment according to the calculated control parameter value at the current moment, so that the target satellite moves toward the target attitude angle.

[0040] In the above step S101, the target satellite is an artificial satellite that can be used to adjust the attitude angle. The time interval between the last moment and the current moment is a preset value. The operation information of the last moment includes the following information: attitude angle of the last moment, angular velocity of the last moment, rotation speed of the last moment, and PWM (Pulse width modulation wave) value of the last moment. Among them, the attitude angle of the last moment and the angle of the last moment are the self-motion information of the target satellite, and the rotation speed of the last moment and the PWM value of the last moment are the operation information of the momentum wheel in the target satellite. The momentum wheel in the target satellite can adjust the angular velocity of the target satellite by adjusting its own rotation speed and PWM value, and then adjust the attitude angle by adjusting the angular velocity. The target attitude angle is the most suitable attitude angle of the target satellite under the specified working requirements. The target satellite can be controlled by ground instructions, and the internal program of the target satellite will determine the target attitude angle according to the ground instructions, and then the target satellite will gradually adjust from the current attitude angle to the target attitude angle using the control method provided in the present application.

[0041] In the specific implementation, the target satellite will obtain the ground command issued by the control center on the earth. The ground command carries the target attitude angle. The target satellite will analyze the target attitude angle from the ground command, and after receiving the ground command, control the sensors in the target satellite according to the instruction information of the ground command to collect the current operation information of the target satellite.

[0042] In the above step S102, the reinforcement learning network with real-time parameter adjustment will update the parameters once after each output result. The calculated control parameter of the momentum wheel at the current moment is the control parameter value that the momentum wheel is about to adjust. When the control device of the momentum wheel receives the adjustment instruction carrying the control parameter value at the current moment, it will control the momentum wheel to adjust from the operating parameter value at the current moment to the control parameter value at the current moment. The control parameter value includes any one of the following parameter values: the speed of the momentum wheel and the PWM value of the momentum wheel. The control parameter value is a parameter value used to adjust the operation of the momentum wheel.

[0043] The current moment operation information is the information obtained by the target satellite through the sensor during the operation process, and the current moment operation information includes the following information: the current moment attitude angle, the current moment angular velocity, the current moment rotation speed, and the current moment PWM (Pulse width modulation wave) value. Among them, the current moment attitude angle and the current moment angle are the self-motion information of the target satellite, while the current moment rotation speed and the current moment PWM value are the operation information of the momentum wheel in the target satellite. The momentum wheel in the target satellite can adjust the angular velocity of the target satellite by adjusting its own rotation speed or PWM value, and then adjust the attitude angle by adjusting the angular velocity.

[0044] In the specific implementation, the reinforcement learning network with real-time parameter adjustment outputs the change in the control parameters of the momentum wheel (which can be the change in speed or the change in PWM value). By combining the change in the control parameter with the operating parameter value of the momentum wheel at the current moment, the control parameter value of the momentum wheel at the current moment is finally calculated.

[0045] In the above step S103, the control parameter value at the current moment is used to adjust the momentum wheel. After adjusting the operating parameter value of the momentum wheel using the control parameter value at the current moment (the operating parameter value is the parameter value when the momentum wheel is running, which can be a speed or a PWM value, and the operating parameter value matches the control parameter value, for example, when the control parameter value is the speed, the operating parameter value is also the speed; when the control parameter value is the PWM value, the operating parameter value is also the PWM value), the angular velocity of the target satellite is adjusted by changing the operating parameter value of the momentum wheel, and then the attitude angle of the target satellite is adjusted by adjusting the angular velocity. Of course, the adjustment process is not achieved overnight, but a step-by-step process. Therefore, the attitude angle of the target satellite adjusted by the reinforcement learning network with real-time parameter adjustment only gets closer and closer to the target attitude angle. When the attitude angle of the target satellite adjusted reaches the target attitude angle, the adjustment of the target satellite is considered to be completed.

[0046] In the above three steps, when the shape, mass and other parameters of the target satellite can change over time, the present application provides a reinforcement learning network that can adjust parameters in real time. Under the current parameters of the target satellite, an optimal control strategy for the current target satellite learned by the reinforcement learning network is used to achieve faster and more accurate adjustment of the target satellite to the target attitude angle.

[0047] A reinforcement learning network with real-time parameter adjustment means that after the reinforcement learning network is applied to the target satellite, the parameters in the reinforcement learning network will be adjusted according to the operating information generated by the target satellite in real time. In the above step S103, after the control parameters at the current moment are calculated, the momentum wheel is adjusted from the operating parameters at the current moment to the control parameters at the current moment. In the process of adjusting the operating parameter values, the overall attitude of the target satellite will be affected. This change is manifested as the target satellite adjusting from the previous attitude angle to the target attitude angle. In other words, the attitude angle of the target satellite after adjustment is closer to the target attitude angle than the attitude angle in the operating information before adjustment. In order to allow the results output by the reinforcement learning network to enable the target satellite to approach the target attitude angle faster and more accurately, a parameter adjustment will be performed after each result output by the reinforcement learning network, and the control parameter value of the momentum wheel corresponding to the next state of the target satellite will continue to be predicted using the adjusted reinforcement learning network, and then the attitude of the target satellite will continue to be adjusted to achieve a state where the target satellite is at the target attitude angle, such as Figure 2 As shown:

[0048] S104, obtaining the current operation information of the target satellite after the momentum wheel adjusts the control parameter value, and if the current attitude angle in the current operation information is not the target attitude angle, adjusting the parameters in the reinforcement learning network;

[0049] S105, using the operation information in the current operation information as the new operation information of the previous moment, using the reinforcement learning network after adjusting the parameters as the new reinforcement learning network, re-executing the step of inputting the target attitude angle and the new operation information of the previous moment into the new reinforcement learning network, calculating the control parameter value of the momentum wheel of the target satellite at the new current moment, adjusting the current attitude angle of the target satellite according to the calculated new control parameter value at the current moment so that the target satellite runs toward the target attitude angle, and obtaining the current operation information of the target satellite after the momentum wheel adjusts the control parameter value. If the current attitude angle in the current operation information is not the target attitude angle, adjusting the parameters in the reinforcement learning network until the adjusted attitude angle of the target satellite is the target attitude angle.

[0050] In the above step S104, the operation information at the current moment is the operation information obtained by the target satellite during operation after the momentum wheel in the target satellite adjusts the operation parameter value according to the control parameter value at the current moment.

[0051] In the specific implementation, if the attitude angle at the current moment in the operation information obtained at the current moment is not the target attitude angle, it means that the target satellite has not reached the target state and needs to continue to adjust the attitude. In order to ensure more accurate adjustment, the parameters in the reinforcement learning network also need to be updated.

[0052] In the above step S105, only using the reinforcement learning network to predict the control parameter value at the current moment may not be able to directly adjust the attitude of the target satellite to the target attitude. Therefore, this application will repeat steps S102, S103, S104 and S105 until the attitude angle of the target satellite reaches the target attitude angle. During this adjustment process, the reinforcement learning network is constantly updated, so that the control parameter value at the current moment predicted by the updated reinforcement learning network can adjust the target satellite to the target attitude angle more quickly.

[0053] In the process of adjusting the attitude angle of the target satellite, the reason why the attitude angle of the target satellite is adjusted to the target attitude angle through multiple adjustments is to protect the momentum wheel. If the operating parameter value of the momentum wheel is adjusted too fast or too slow (that is, the absolute value of the difference between the operating parameter value of the previous moment and the operating parameter value of the current moment is too large), the loss of the momentum wheel will increase. By controlling the difference between the operating parameter values ​​of two adjacent moments within a certain controllable range, the adjustment span of the operating parameter value of the momentum wheel is reduced, thereby protecting the momentum wheel, reducing the loss, and increasing the service life of the momentum wheel.

[0054] Of course, in the process of adjusting the attitude angle of the target satellite, although the difference between the operating parameter values ​​at two adjacent moments will be controlled within a certain controllable range, the larger the difference within this controllable range, the faster the target satellite adjusts to the target attitude angle, the shorter the time required, and the smaller the difference, the slower the target satellite adjusts to the target attitude angle, and the longer the time required. In some cases, the target satellite needs to adjust its attitude angle to the target attitude angle within a specified time, so combined with the above considerations of protecting the momentum wheel, the control frequency of the momentum wheel will be set during the process of adjusting the attitude angle of the target satellite, and the control frequency will be used to determine the adjustment period of the target satellite's attitude angle each time, that is, the parameter adjustment period of the reinforcement learning network. For example, taking the rotation speed of the momentum wheel as the control parameter value, we want the target satellite to reach the target attitude angle within one second, and the control frequency of the momentum wheel is 50 Hz, which means that the momentum wheel is controlled once every 0.02 seconds. Then every 0.02 seconds, the reinforcement learning network will output the required rotation speed of the momentum wheel, and the control device of the momentum wheel will adjust the momentum wheel according to the rotation speed output by the reinforcement learning network, so that the target satellite will adjust its attitude angle once every 0.02 seconds and finally reach the target attitude angle within one second.

[0055] The parameter adjustment process of the reinforcement learning network with real-time parameter adjustment is a process of continuous trial and error. It takes a lot of trial and error to make the reinforcement learning network more accurate. The trial and error process requires a lot of data. These data are stored in the experience pool of the reinforcement learning network and become the experience data used for reinforcement learning and trial and error. These experience data include the following data: the current moment operation information, the previous moment operation information, target attitude angle, action category and reward value. The above data are all data obtained by the target satellite during operation. Therefore, the control method of this application also includes:

[0056] Step 106: store the operation information at the previous moment, the operation information at the current moment, the target posture angle, the action category and the reward value as a set of experience data in the experience pool of the real-time parameter adjustment reinforcement learning network.

[0057] In the above step 106, the action category includes an acceleration category and a deceleration category. The acceleration category indicates that the current speed is greater than the speed at the previous moment. The deceleration category indicates that the current speed is less than the speed at the previous moment. The reward value represents the closeness between the attitude angle of the target satellite and the target attitude angle in the operation information at the current moment. The larger the reward value, the closer the attitude angle of the target satellite and the target attitude angle in the operation information at the current moment are, and the smaller the reward value, the greater the difference between the attitude angle of the target satellite and the target attitude angle in the operation information at the current moment. The above experience data is stored in the experience pool of the reinforcement learning network, which can be used for the reinforcement learning network to quickly learn and improve the trial and error efficiency. Of course, every time the reinforcement learning network predicts (equivalent to each time the target satellite adjusts the attitude angle once), the operation information of the target satellite after adjusting the operation parameter value (that is, the operation information at the current moment) will be obtained, and then the operation information at the previous moment and the operation information at the current moment will be used to determine the action category and reward value.

[0058] Specifically, in the control method of the present application, the reward value in the experience data is determined by the following steps:

[0059] Step 107, determining a reward value using the difference between the attitude angle of the current running information in the experience data and the target attitude angle.

[0060] In the above step 107, the reward value is calculated using the following formula based on the difference between the posture angle of the current running information and the target posture angle.

[0061] Among them, r represents the reward value, s t Indicates the attitude angle in the current running information, s targetIndicates the target attitude angle. Regardless of the attitude angle in the current operation information or the target attitude angle, they are the attitude angles corresponding to the target satellite in the same dimension (same motion orbit).

[0062] Specifically, in the control method of the present application, the action category is determined based on the comparison result between the speed in the operation information at the previous moment and the speed at the next moment in the operation information at the current moment. If the current moment speed in the operation information at the current moment is greater than the speed in the operation information at the previous moment, it indicates that it is an acceleration category; if the current moment speed in the operation information at the current moment is less than the speed in the operation information at the previous moment, it indicates that it is a deceleration category.

[0063] It is mentioned above that the target satellite will obtain operation information every time it adjusts its attitude angle, and the operation information of the previous moment will be processed to form experience data and stored in the experience pool of the reinforcement learning network. Therefore, there will be experience data corresponding to multiple historical moments in the experience pool of the reinforcement learning network. The experience data in the experience pool are stored in chronological order, so the new sample will overwrite the oldest experience data in time. As for sampling, if the experience data of the latest sequence is taken out every time, the reinforcement learning network is not much different from online learning, and the learning effect is not good. If it is random sampling, that is, randomly obtaining experience data from historical experience data at multiple different moments for reinforcement learning trial and error, the experience data in each learning process usually comes from multiple interaction sequences, so that the fluctuation of a single sequence is reduced a lot, and the training effect is more stable. At the same time, a piece of experience data can also be trained multiple times, which improves the utilization rate of experience data. Specifically, the control method of the present application also includes:

[0064] Step 108, randomly sampling from the experience pool to obtain a plurality of historical experience data;

[0065] Step 109, using the multiple historical experience data and the loss function, adjust the parameters of the two action value networks and the actor network in the reinforcement learning network.

[0066] In the above step 108, the historical experience data is the experience data stored at the historical moment, and the historical experience data includes the operation information at the historical moment, the operation information at the moment before the historical moment, the action category at the historical moment, and the reward value at the historical moment.

[0067] In the above step 109, the result obtained by calculating the loss function is used to update the parameters in the reinforcement learning network. Specifically, the target value is first calculated using the objective function and historical experience data.

[0068] y target =r+γ(1-d)(min(Q target,1 (s′,a′),Q target,2(s′,a′))-αlog(π θ (a′|s′)));

[0069] Among them, y target is the target value, r is the reward value in the historical experience data, γ represents the influence of the reward value in the historical experience data on the reward value corresponding to the subsequent target satellite operation information, d is the label representing whether the target satellite is at the target attitude angle at the historical moment corresponding to the historical experience data, α is the preset coefficient, s′ is the operation information at the historical moment corresponding to the historical experience data, and a ′ is the action category at the historical moment corresponding to the historical experience data, π θ is an actor network with parameters θ, min(Q target,1 (s′,a′),Q target,2 (s′, a′)) is the Q output of the two action-value networks in the case of s′ and a′ respectively. target The minimum value in Q target,1 (s′, a′) is the value output by an action value network in the reinforcement learning network under the conditions of s′ and a′, Q target,2 (s′, a′) is the value output by another action-value network in the reinforcement learning network under the condition of S′ and a′.

[0070] After calculating the target value, the two action value networks in the reinforcement learning network are updated using the first loss function. The first loss function is as follows:

[0071]

[0072] Among them, y target is the target value, r is the reward value, s is the operation information of the previous moment of the historical moment corresponding to the historical experience data, a is the action category of the previous moment of the historical moment corresponding to the historical experience data, d is the label representing whether the target satellite is at the target attitude angle at the historical moment corresponding to the historical experience data, s′ is the operation information of the historical moment corresponding to the historical experience data, Q i (s,a) is the value output by the i-th action value network under the operation information and action category of the previous historical moment, where i∈(1,2), It is used to update the action value function The loss function of the parameters, D is the set of historical experience data in the experience pool.

[0073] The action distribution network in the reinforcement learning network is updated using the second loss function, which is as follows:

[0074] L θ =∑ s∈D (min(Q1(s,a′θ ),Q2(s,a′ θ ))-αlog(π θ (a′|s)));

[0075] Among them, r is the reward value, s is the operation information of the previous moment of the historical moment corresponding to the historical experience data, α is the preset coefficient, π θ is an actor network with parameters θ, a′ θ is the action category predicted by the actor network with parameter θ based on the running information of the previous moment in history, min(Q1(s,a' θ ),Q2(s,a' θ )) is between s and a′ θ In the case of , the two action value networks respectively output the minimum value of the results, and D is the set of historical experience data in the experience pool.

[0076] The reinforcement learning network in this application uses two action value estimation networks for prediction, and these two action value estimation networks are different. By using two different action value networks to predict the same set of historical experience data, two different prediction results can be obtained. Finally, a more accurate result is selected from the two different prediction results, thereby further improving the accuracy of the reinforcement learning network prediction.

[0077] Generally, reinforcement learning networks are written in Python, specifically using the pytorch framework in Python. The parameter update process in the reinforcement learning network is implemented using the optimizer in the pytorch framework. There are many optional optimizers, such as Adam, SGD, RMSprop, etc.

[0078] The present application provides a satellite attitude control device, such as Figure 3 As shown, including:

[0079] The acquisition module 301 is used to acquire the last moment operation information and target attitude angle of the target satellite;

[0080] A calculation module 302 is used to input the target attitude angle and the operation information of the previous moment into a reinforcement learning network for real-time parameter adjustment, and calculate the control parameter value of the momentum wheel in the target satellite at the current moment;

[0081] An adjustment module 303 is used to adjust the attitude angle of the target satellite at the current moment according to the calculated control parameter value at the current moment, so that the target satellite moves toward the target attitude angle;

[0082] Optionally, the control device further includes:

[0083] A judgment module, used for obtaining the current operation information of the target satellite after the momentum wheel adjusts the control parameter value, and adjusting the parameters in the reinforcement learning network if the current attitude angle in the current operation information is not the target attitude angle;

[0084] A loop module is used to use the operation information in the current operation information as the new operation information of the previous moment, use the reinforcement learning network after adjusting the parameters as the new reinforcement learning network, re-execute the step of inputting the target attitude angle and the new operation information of the previous moment into the new reinforcement learning network, calculate the control parameter value of the momentum wheel of the target satellite at the new current moment, adjust the current attitude angle of the target satellite according to the calculated new control parameter value at the current moment so that the target satellite runs toward the target attitude angle, and obtain the current operation information of the target satellite after the momentum wheel adjusts the control parameter value. If the current attitude angle in the current operation information is not the target attitude angle, adjust the parameters in the reinforcement learning network until the adjusted attitude angle of the target satellite is the target attitude angle.

[0085] Optionally, the difference between the control parameter value of the momentum wheel at the current moment and the control parameter value of the momentum wheel in the operation information at the previous moment is less than a preset parameter change.

[0086] Optionally, the control device further includes:

[0087] The storage module is used to store the operation information at the previous moment, the operation information at the current moment, the target posture angle, the action category and the reward value as a group of experience data in the experience pool of the reinforcement learning network with real-time parameter adjustment.

[0088] Optionally, the control device includes:

[0089] The sampling module is used to randomly sample from the experience pool to obtain multiple historical experience data;

[0090] An adjustment module is used to adjust parameters of two action value networks and an actor network in the reinforcement learning network by using the multiple historical experience data and the loss function.

[0091] Optionally, the reward value is determined by the following steps:

[0092] The determination module is used to determine the reward value by using the difference between the posture angle of the current moment running information in the experience data and the target posture angle.

[0093] Optionally, the action category is determined based on a comparison result between the rotation speed in the current operation information and the rotation speed at a previous moment in the previous operation information.

[0094] Corresponds to Figure 1 The satellite attitude control method in the present application embodiment also provides a computer device 400, such as Figure 4 As shown, the device includes a memory 401, a processor 402, and a computer program stored in the memory 401 and executable on the processor 402, wherein the processor 402 implements the satellite attitude control method when executing the computer program.

[0095] Specifically, the above-mentioned memory 401 and processor 402 can be general-purpose memory and processor, which are not specifically limited here. When the processor 402 runs the computer program stored in the memory 401, it can execute the above-mentioned satellite attitude control method, thereby solving the problem in the prior art that the attitude of the target satellite cannot be accurately adjusted.

[0096] Corresponds to Figure 1 The satellite attitude control method in the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned satellite attitude control method are executed.

[0097] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned satellite attitude control method can be executed, which solves the problem in the prior art that the attitude of the target satellite cannot be accurately adjusted. When the shape, mass and other parameters of the target satellite can change over time, the present application provides a reinforcement learning network that can adjust parameters in real time. Under the current parameters of the target satellite, an optimal control strategy for the current target satellite learned by the reinforcement learning network is used to achieve faster and more accurate adjustment of the target satellite to the target attitude angle.

[0098] In the embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0101] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0102] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0103] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for controlling satellite attitude, characterized in that: include: Obtain the last moment operation information and target attitude angle of the target satellite; Inputting the target attitude angle and the operation information of the previous moment into a reinforcement learning network for real-time parameter adjustment, and calculating the control parameter value of the momentum wheel in the target satellite at the current moment; According to the calculated control parameter value at the current moment, adjusting the attitude angle of the target satellite at the current moment, so that the target satellite moves toward the target attitude angle; The control method further comprises: Acquire the current operation information of the target satellite after the momentum wheel adjusts the control parameter value, and if the current attitude angle in the current operation information is not the target attitude angle, adjust the parameters in the reinforcement learning network; The operation information in the operation information at the current moment is used as the new operation information at the previous moment, and the reinforcement learning network after adjusting the parameters is used as the new reinforcement learning network, and the step of re-execution is to input the target attitude angle and the new operation information at the previous moment into the new reinforcement learning network, and calculate the control parameter value of the momentum wheel of the target satellite at the new current moment, and the step of adjusting the current attitude angle of the target satellite according to the calculated new control parameter value at the current moment so that the target satellite runs toward the target attitude angle, and obtaining the current operation information of the target satellite after the control parameter value of the momentum wheel is adjusted, if the current attitude angle in the current operation information is not the target attitude angle, adjusting the parameters in the reinforcement learning network until the adjusted attitude angle of the target satellite is the target attitude angle; The control method further comprises: The operation information at the previous moment, the operation information at the current moment, the target posture angle, the action category and the reward value are stored as a set of experience data in the experience pool of the reinforcement learning network with real-time parameter adjustment; The parameter adjustment process of the reinforcement learning network includes: Randomly sample from the experience pool to obtain multiple historical experience data; Using the plurality of historical experience data and the loss function, adjusting parameters of two action value networks and an actor network in the reinforcement learning network; The parameter adjustment of the two action value networks and the actor network in the reinforcement learning network includes: Calculate the target value using the target function and historical experience data; y target =r+γ(1-d)(min(Q target,1 (s′,a′),Q target,2 (s′,a′))-αlog(π θ (a′|s′))) Among them, y target is the target value, r is the reward value in the historical experience data, γ represents the influence of the reward value in the historical experience data on the reward value corresponding to the subsequent target satellite operation information, d is the label representing whether the target satellite is at the target attitude angle at the historical moment corresponding to the historical experience data, α is the preset coefficient, s′ is the operation information at the historical moment corresponding to the historical experience data, a′ is the action category at the historical moment corresponding to the historical experience data, π θ is an actor network with parameters θ, min(Q target,1 (s′,a′),Q target,2 (s′, a′)) is the Q output of the two action-value networks in the case of s′ and a′ respectively. target The minimum value in Q target,1 (s′, a′) is the value output by an action value network in the reinforcement learning network under the conditions of s′ and s′, Q target,2 (s′, s′) is the value output by another action-value network in the reinforcement learning network under the conditions of S′ and a′; The two action value networks in the reinforcement learning network are updated using the first loss function. The first loss function is as follows: Among them, y target is the target value, r is the reward value, s is the operation information of the previous moment of the historical moment corresponding to the historical experience data, a is the action category of the previous moment of the historical moment corresponding to the historical experience data, d is the label representing whether the target satellite is at the target attitude angle at the historical moment corresponding to the historical experience data, s′ is the operation information of the historical moment corresponding to the historical experience data, Q i (s,a) is the value output by the i-th action value network under the operation information and action category of the previous historical moment, where i∈(1,2), It is used to update the action value function The loss function of the parameter, D is the set of historical experience data in the experience pool; The action distribution network in the reinforcement learning network is updated using the second loss function, which is as follows: L θ =S s∈D (min(Q1(s,a′ θ ),Q2(s,a′ θ ))-αlog(π θ (a′|s))) Among them, r is the reward value, s is the operation information of the previous moment of the historical moment corresponding to the historical experience data, α is the preset coefficient, π θ is an actor network with parameters θ, a′ θ is the action category predicted by the actor network with parameter θ based on the running information of the previous moment in history, min(Q1(s,a'0),Q2(s,a' θ )) is between s and a′ θ In the case of , the two action value networks respectively output the minimum value of the results, and D is the set of historical experience data in the experience pool.

2. The control method according to claim 1, characterized in that: The difference between the control parameter value of the momentum wheel at the current moment and the control parameter value of the momentum wheel in the operation information at the previous moment is less than the preset parameter change.

3. The control method according to claim 1, characterized in that: The reward value is determined by the following steps: The reward value is determined by using the difference between the posture angle of the current running information in the experience data and the target posture angle.

4. The control method according to claim 1, characterized in that: The action category is determined according to a comparison result between the rotation speed in the current operation information and the rotation speed at the previous moment in the previous operation information.

5. A satellite attitude control device, characterized in that: include: An acquisition module is used to obtain the last moment operation information and target attitude angle of the target satellite; A calculation module is used to input the target attitude angle and the operation information of the previous moment into a reinforcement learning network for real-time parameter adjustment, and calculate the control parameter value of the momentum wheel in the target satellite at the current moment; An adjustment module, used for adjusting the attitude angle of the target satellite at the current moment according to the calculated control parameter value at the current moment, so that the target satellite moves towards the target attitude angle; The control device also includes: A judgment module, used for obtaining the current operation information of the target satellite after the momentum wheel adjusts the control parameter value, and adjusting the parameters in the reinforcement learning network if the current attitude angle in the current operation information is not the target attitude angle; A loop module, used to use the operation information in the current operation information as the new operation information at the previous moment, use the reinforcement learning network after adjusting the parameters as the new reinforcement learning network, re-execute the step of inputting the target attitude angle and the new operation information at the previous moment into the new reinforcement learning network, calculate the control parameter value of the momentum wheel of the target satellite at the new current moment, adjust the current attitude angle of the target satellite according to the calculated new control parameter value at the current moment, so that the target satellite runs toward the target attitude angle, and obtain the current operation information of the target satellite after the control parameter value of the momentum wheel is adjusted, if the current attitude angle in the current operation information is not the target attitude angle, adjust the parameters in the reinforcement learning network until the adjusted attitude angle of the target satellite is the target attitude angle; The control device also includes: A storage module is used to store the operation information at the previous moment, the operation information at the current moment, the target posture angle, the action category and the reward value as a set of experience data in the experience pool of the reinforcement learning network with real-time parameter adjustment; The control device comprises: The sampling module is used to randomly sample from the experience pool to obtain multiple historical experience data; An adjustment module, used to adjust parameters of two action-value networks and an actor network in the reinforcement learning network using the multiple historical experience data and the loss function; When the adjustment module is used to adjust the parameters of two action value networks and an actor network in a reinforcement learning network, it is specifically used to: Calculate the target value using the target function and historical experience data; y target =r+γ(1-d)(min(Q target,1 (s′,a′),Q target,2 (s′,a′))-αlog(π θ (a′|s′))) Among them, y target is the target value, r is the reward value in the historical experience data, γ represents the influence of the reward value in the historical experience data on the reward value corresponding to the subsequent target satellite operation information, d is the label representing whether the target satellite is at the target attitude angle at the historical moment corresponding to the historical experience data, α is the preset coefficient, s′ is the operation information at the historical moment corresponding to the historical experience data, a′ is the action category at the historical moment corresponding to the historical experience data, π θ is an actor network with parameters θ, min(Q target,1 (s′,a′),Q target,2 (s′, a′)) is the Q output of the two action-value networks in the case of s′ and a′ respectively. target The minimum value in Q target,1 (s′, a′) is the value output by an action value network in the reinforcement learning network under the conditions of s′ and a′, Q target,2 (s′, a′) is the value output by another action-value network in the reinforcement learning network under the conditions of S′ and a′; The two action value networks in the reinforcement learning network are updated using the first loss function. The first loss function is as follows: Among them, y target is the target value, r is the reward value, s is the operation information of the previous moment of the historical moment corresponding to the historical experience data, a is the action category of the previous moment of the historical moment corresponding to the historical experience data, d is the label representing whether the target satellite is at the target attitude angle at the historical moment corresponding to the historical experience data, s′ is the operation information of the historical moment corresponding to the historical experience data, Q i (s,a) is the value output by the i-th action value network under the operation information and action category of the previous historical moment, where i∈(1,2), It is used to update the action value function The loss function of the parameter, D is the set of historical experience data in the experience pool; The action distribution network in the reinforcement learning network is updated using the second loss function, which is as follows: L θ =∑ s∈D (min(Q1(s,a′ θ ),Q2(s,a′ θ ))-αlog(π θ (a′|s))) Among them, r is the reward value, s is the operation information of the previous moment of the historical moment corresponding to the historical experience data, α is the preset coefficient, π θ is an actor network with parameters θ, a′ θ is the action category predicted by the actor network with parameter θ based on the running information of the previous moment in history, min(Q1(s,a' θ ),Q2(s,a' θ )) is between s and a′ θ In the case of , the two action value networks respectively output the minimum value of the results, and D is the set of historical experience data in the experience pool.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are executed.

Images