A distributed fault-tolerant control method for spacecraft clusters based on reinforcement learning

Abstract

The invention discloses a distributed fault-tolerant control method for a spacecraft cluster based on reinforcement learning, and relates to the field of spacecraft control technology. Through a fixed-time distributed estimator, it is ensured that the followers in the cluster can obtain the position and speed information of the leader within a fixed time. And through a fixed-time distributed estimator and a backstepping control method, an approximate optimal virtual control quantity and an approximate available control quantity are designed. On this basis, radial basis functions and participant-evaluator networks are introduced to process unknown nonlinear terms and Hamilton-Jacobi-Bellman equations in the approximate optimal virtual control quantity and the approximate available control quantity. Finally, the approximate available control quantity is optimized through the actuator fault processing link to ensure that the spacecraft cluster can complete the predetermined formation task under the influence of the actuator fault.

Description

Technical Field

[0001] The present invention relates to the field of spacecraft control technology, and in particular to a distributed fault-tolerant control method of a spacecraft cluster based on reinforcement learning. Background Art

[0002] Under the current development requirements of the aerospace industry, it has become increasingly difficult for a single spacecraft to complete difficult space missions, and there are problems such as long R&D cycles, high construction costs, complex mechanical structures, and high failure rates. Therefore, using multiple small spacecraft with short R&D cycles, low construction costs, and simple mechanical structures to form a spacecraft cluster through remote communications, and then replace traditional large spacecraft to complete space missions has gradually become a research hotspot. In the formation control process of a spacecraft cluster, actuator failures and external interference such as high-energy particles, atmospheric drag, and solar gravity may affect the safety performance of the system. Summary of the invention

[0003] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a distributed fault-tolerant control method of a spacecraft cluster based on reinforcement learning is provided, which aims to solve the current formation control problem of a spacecraft cluster under the influence of actuator failure and external machine interference.

[0004] The technical solution adopted by the present invention to solve the problem is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a distributed fault-tolerant control method for a spacecraft cluster based on reinforcement learning, the method comprising:

[0006] For each follower in the spacecraft cluster, the position estimate and velocity estimate of the leader are obtained through a fixed-time distributed estimator;

[0007] Obtaining an expected formation position corresponding to the follower, and calculating an expected position corresponding to the follower according to the estimated position value of the leader and the expected formation position;

[0008] Inputting the desired position, the estimated position value and the estimated speed value of the leader into the fixed-time formation controller corresponding to the follower to obtain the actual control amount corresponding to the follower;

[0009] Wherein, the fixed time formation controller comprises:

[0010] An actor-evaluator network based on reinforcement learning is used to calculate an approximate optimal virtual control amount according to the desired position and the position estimate of the navigator through a backstepping control method and a radial basis function; and to calculate an approximate available control amount according to the approximate optimal virtual control amount and the speed estimate of the navigator through a backstepping control method and a radial basis function;

[0011] The actuator fault processing module is used to calculate the actual control amount through the upper limit information of the actuator fault and the approximate available control amount.

[0012] In a second aspect, an embodiment of the present invention further provides a spacecraft cluster, comprising a leader and a plurality of followers, each of the followers being controlled by any of the above-mentioned reinforcement learning-based distributed fault-tolerant control methods for spacecraft clusters.

[0013] In a third aspect, an embodiment of the present invention further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the program comprises instructions for executing any of the reinforcement learning-based distributed fault-tolerant control methods for a spacecraft cluster as described above; and the processor is used to execute the program.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of the distributed fault-tolerant control method of a spacecraft cluster based on reinforcement learning as described in any of the above.

[0015] Beneficial effects of the present invention: The embodiments of the present invention use a fixed-time distributed estimator to ensure that the followers in the cluster can obtain the position and velocity information of the leader within a fixed time. The approximate optimal virtual control quantity and the approximate available control quantity are designed by a fixed-time distributed estimator and a backstepping control method. On this basis, radial basis functions and participant-evaluator networks are introduced to handle the unknown nonlinear terms and Hamilton-Jacobi-Bellman equations in the approximate optimal virtual control quantity and the approximate available control quantity. Finally, the approximate available control quantity is optimized through the actuator fault processing link to ensure that the spacecraft cluster can complete the predetermined formation task under the influence of the actuator fault. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a distributed fault-tolerant control method of a spacecraft cluster based on reinforcement learning provided in an embodiment of the present invention.

[0017] Figure 2 This is a structural block diagram of a spacecraft formation control system provided in an embodiment of the present invention.

[0018] Figure 3 A schematic diagram of the spacecraft cluster communication topology provided in an embodiment of the present invention.

[0019] Figure 4 A schematic diagram of position estimation error of a distributed estimator provided in an embodiment of the present invention.

[0020] Figure 5A schematic diagram of speed estimation error of a distributed estimator provided in an embodiment of the present invention.

[0021] Figure 6 A three-dimensional diagram of the formation motion trajectory of a spacecraft cluster provided in an embodiment of the present invention.

[0022] Figure 7 A two-dimensional diagram of the formation motion trajectory of a spacecraft cluster provided in an embodiment of the present invention.

[0023] Figure 8 A schematic diagram of the commentator network weight matrix norm of the spacecraft cluster formation controller provided in an embodiment of the present invention.

[0024] Fig. 9 A schematic diagram of the participant network weight matrix norm of the spacecraft cluster formation controller provided in an embodiment of the present invention.

[0025] Fig.10 A schematic diagram of the position control error of a spacecraft cluster formation controller provided in an embodiment of the present invention.

[0026] Fig.11 A schematic diagram of the speed control error of a spacecraft cluster formation controller provided in an embodiment of the present invention.

[0027] Fig.12 A functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention discloses a distributed fault-tolerant control method for a spacecraft cluster based on reinforcement learning. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0029] In view of the above-mentioned defects of the prior art, the present invention provides a distributed fault-tolerant control method for a spacecraft cluster based on reinforcement learning. Through a fixed-time distributed estimator, it is ensured that the followers in the cluster can obtain the position and velocity information of the leader within a fixed time. And the approximate optimal virtual control quantity and the approximate available control quantity are designed through a fixed-time distributed estimator and a backstepping control method. On this basis, radial basis functions and participant-evaluator networks are introduced to deal with unknown nonlinear terms and Hamilton-Jacobi-Bellman equations in the approximate optimal virtual control quantity and the approximate available control quantity. Finally, the approximate available control quantity is optimized through the actuator fault processing link to ensure that the spacecraft cluster can complete the predetermined formation task under the influence of the actuator fault.

[0030] like Figure 1 As shown, the method specifically includes:

[0031] Step S100: For each follower in the spacecraft cluster, obtain the position estimate and velocity estimate of the leader through a fixed-time distributed estimator.

[0032] Specifically, the spacecraft cluster in this embodiment can use a navigator and The leader is composed of followers. , the semi-major axis of the orbit is The orbital eccentricity is The radial distance vector from the center of the Earth to the navigator is ,in Indicates the distance from the center of the earth to the navigator. This embodiment pre-designs a fixed-time distributed estimator to estimate the position and speed of the navigator.

[0033] For example, a spacecraft cluster consists of a leader and four followers. Figure 3 is the communication topology of the cluster, where 0 is the leader and 1-4 are followers. The leader is at a true anomaly angle of , the semi-major axis of the orbit is The orbital eccentricity is The navigator flies in an ideal elliptical orbit around the earth. The radial distance vector from the center of the earth to the navigator and the distance from the center of the earth to the navigator can be calculated through the semi-major axis and the eccentricity of the orbit.

[0034] In one implementation, the spacecraft orbital dynamics model corresponding to the follower is:

[0035] , formula (1);

[0036] in, For the Followers; is the position vector of the spacecraft; is the velocity vector of the spacecraft; For quality, and The two are reciprocal; is the actual control input; For external interference; It is a multiplicative fault; It is an additive fault; is the gravity vector; For the The distance between each follower and the center of the Earth; is the Earth's gravitational constant; is a skew symmetric matrix; is a time-varying nonlinear matrix; is the actual control quantity; is the available control amount; is the actual equation of spacecraft orbital dynamics, is the ideal equation for spacecraft orbital dynamics.

[0037] Furthermore, the gravity vector , No. The distance between each follower and the center of the Earth , is the distance from the center of the Earth to the navigator, is the radial distance vector from the center of the Earth to the navigator, is the x-axis component of the navigator's position relative to the center of the earth, is the square of the y-axis component of the navigator's position relative to the center of the earth, is the square of the z-axis component of the navigator's position relative to the center of the earth.

[0038] Specifically, the above formula (1) is the spacecraft orbital dynamics model with multiplicative faults and additive faults given by this embodiment according to the spacecraft orbital dynamics. Further, the skew symmetric matrix With the time-varying nonlinear matrix Defined as:

[0039] , formula (2);

[0040] and The definition is

[0041] , formula (3);

[0042] in, is the average orbital angular velocity of the navigator, is the Earth's gravitational constant, a c is the semi-major axis of the orbit.

[0043] For example, based on the spacecraft orbital dynamics, a spacecraft orbital dynamics model is given as shown in the above formula (1), where For the The quality of the followers, is the actual control input, For external interference, For multiplicative faults, For additive faults, and according to the above The gravity vector is obtained by the calculation formula. According to the above The calculation formula is used to obtain the The distance between each follower and the center of the Earth, is the Earth's gravitational constant.

[0044] In one implementation, the fixed-time distributed estimator is constructed based on an adjacency matrix, the adjacency matrix is ​​used to reflect the edge set of a directed graph, the directed graph is used to reflect the communication topology of the spacecraft cluster, and the fixed-time distributed estimator is:

[0045] , formula (4);

[0046] in, Indicates Followers, Indicates Followers; and is a positive constant; , , , is an odd number, satisfying , , Indicates The parameters corresponding to the numerator of the first group of exponents selected by the followers, Indicates The parameters corresponding to the denominator of the first group of exponentials selected by the followers, Indicates The parameters corresponding to the numerator of the second group of exponents selected by the followers, Indicates The parameters corresponding to the denominator of the second group of exponentials selected by the followers; is 1 or 2, Indicates The estimates of the leader's position by the followers, Indicates The followers' estimates of the leader's speed; represents the number of followers; the adjacency matrix is ​​defined as , , if A follower can receive messages from Follower information, then ,otherwise .

[0047] Specifically, this embodiment adopts a directed graph To represent the communication topology in the spacecraft cluster, for The set of nodes corresponding to the followers, is the edge set corresponding to the follower communication topology. The edge set of is represented by the adjacency matrix, where the adjacency matrix is ​​defined as , If A follower can receive messages from Follower information, then ,otherwise , and furthermore define . Then define the Laplacian matrix ,in, is the degree matrix. To further describe the communication topology of the navigator, this embodiment uses a directed graph Based on this, an extended graph containing navigator nodes is introduced. . Define a new diagonal matrix , if Followers can receive information from the leader, then ,otherwise . Then the The estimates of the position and velocity of the leader by the followers are denoted as and , the true values ​​of the pilot's position and velocity are recorded as and Finally, the fixed-time distributed estimator shown in the above formula (4) is constructed.

[0048] It should be noted that in the past, when dealing with the communication problem between individuals within a spacecraft cluster, a topological network based on an undirected graph was often used, which was difficult to handle the situation of unidirectional communication within the cluster. This embodiment uses a fixed-time distributed estimator based on a directed graph to solve the problem of unidirectional information transmission between individuals while ensuring that followers within the cluster can obtain the position and velocity information of the leader within a fixed time.

[0049] For example, a directed graph In , represents the node set corresponding to the 4 followers, and the adjacency matrix is , The Laplace matrix is ,in is the degree of the adjacency matrix The new diagonal matrix is The adjacency matrix describing the communication topology between spacecraft cluster followers is chosen as , the diagonal matrix describing the communication topology between followers and leaders is selected as Finally, the above formula (4) is used to construct a fixed-time distributed estimator, where , , , , , . Figure 4is the position estimation error curve of the fixed time distributed estimator on the three-dimensional coordinates x, y, and z, where position estimation error 1 corresponds to the position estimation error on the x-axis, position estimation error 2 corresponds to the position estimation error on the y-axis, and position estimation error 3 corresponds to the position estimation error on the z-axis; Figure 5 is the speed estimation error curve of the fixed time distributed estimator on the three-dimensional coordinates x, y, and z, where speed estimation error 1 corresponds to the speed estimation error on the x-axis, speed estimation error 2 corresponds to the speed estimation error on the y-axis, and speed estimation error 3 corresponds to the speed estimation error on the z-axis. Figure 4 and Figure 5 It can be seen that the position and velocity estimates obtained by the fixed-time distributed estimator can converge to the true value of the leader within a fixed time, while also ensuring that the leader's state is available to each follower.

[0050] Step S200: Obtain the expected formation position corresponding to the follower, and calculate the expected position corresponding to the follower according to the estimated position value of the leader and the expected formation position.

[0051] Specifically, this embodiment realizes formation by ensuring that the drones are constantly approaching the expected position. The expected formation position of each follower refers to the expected formation position of the follower relative to the leader. The expected position, the expected formation position, and the position of the leader provided by the distributed estimator satisfy a certain numerical relationship. Therefore, the expected position of the follower can be calculated based on the position of the leader and the expected formation position.

[0052] Step S300: input the desired position, the estimated position value and the estimated speed value of the leader into the fixed-time formation controller corresponding to the follower to obtain the actual control amount corresponding to the follower.

[0053] Specifically, this embodiment pre-designs a fixed-time formation controller for each follower to calculate the actual control amount. In actual application, for each follower, the currently calculated expected position of the follower, the position estimation value and the speed estimation value of the leader are input into the fixed-time formation controller, and the actual control amount corresponding to the follower is output through the fixed-time formation controller, and the flight control of the follower is performed through the actual control amount.

[0054] Furthermore, the fixed time formation controller comprises:

[0055] An actor-evaluator network based on reinforcement learning is used to calculate an approximate optimal virtual control amount according to the desired position and the position estimate of the navigator through a backstepping control method and a radial basis function; and to calculate an approximate available control amount according to the approximate optimal virtual control amount and the speed estimate of the navigator through a backstepping control method and a radial basis function;

[0056] The actuator fault processing module is used to calculate the actual control amount through the upper limit information of the actuator fault and the approximate available control amount.

[0057] Specifically, this embodiment designs an estimated value of the optimal virtual control amount, that is, an approximate optimal virtual control amount, by a backstepping control method. In order to obtain an available virtual control amount, the control performance is first estimated using an evaluator network, which is the control performance corresponding to the gradient used when calculating the optimal virtual control amount, and the weight matrix update law of the evaluator network is designed. Then, the approximate optimal virtual control amount is designed through a participant network, and the weight matrix update law of the participant network is designed. In addition, this embodiment also designs an estimated value of the available control amount, that is, an approximate available control amount, by a backstepping control method. In order to obtain an available control amount, the control performance is first estimated using an evaluator network, which is the control performance corresponding to the gradient used when calculating the optimal control amount, and the weight matrix update law of the evaluator network is designed. Then, the approximate available control amount is designed through a participant network. Finally, this embodiment also considers actuator failure to design the actual control amount.

[0058] It should be noted that in the past, when reinforcement learning methods were used to solve the problem of spacecraft cluster formation control, it was difficult to obtain the fixed-time stability of the system under the influence of external interference. This embodiment uses a participant-evaluator network based on reinforcement learning to process the difficult equations and unknown nonlinear terms in the control quantity, which can ensure that the cluster's formation position control error and speed control error can converge within a fixed time. In addition, in the past, when the backstepping control method was used to solve the problem of spacecraft cluster formation control, only external disturbances were processed, and the influence of actuator failures on the control effect of the spacecraft system was mostly ignored. This embodiment uses the upper bound information of actuator failures to construct the actual control quantity, ensuring that the spacecraft cluster can achieve the target formation maneuver while processing actuator failures and external interference, and maintain the predetermined formation queue under the premise of ensuring the safety of the spacecraft system.

[0059] In one implementation, the participant-evaluator network includes an evaluator network and a participant network. When calculating the approximate optimal virtual control amount, the weight matrix update law of the evaluator network is:

[0060] , formula (5);

[0061] in, To calculate the weight matrix of the evaluator network when approximating the optimal virtual control amount, for The transposed matrix of A positive constant designed for calculating the approximate optimal virtual control quantity; The basis function matrix used to calculate the approximate optimal virtual control quantity is: The basis function selected for calculating the approximate optimal virtual control quantity is: is the number of neurons; is transposed;

[0062] The weight matrix update law of the participant network is:

[0063] , formula (6);

[0064] in, To calculate the weight matrix of the participant network when approximating the optimal virtual control amount, for The transposed matrix of , They are all normal numbers designed when calculating the approximate optimal virtual control quantity and satisfy .

[0065] In one implementation, the calculation formula for obtaining the approximate optimal virtual control amount through the participant network is:

[0066] , formula (7);

[0067] in, is the approximate optimal virtual control quantity; For the The desired position of each follower; is the system error corresponding to the position; is the approximate optimal virtual control index term 1, is the approximate optimal virtual control quantity index term 2, , is a positive integer; , Design parameters for calculating approximate optimal virtual control quantities; The weight matrix of the participant network when calculating the approximate optimal virtual control amount; The basis function matrix used to calculate the approximate optimal virtual control quantity is: The basis function selected for calculating the approximate optimal virtual control quantity is: is the number of neurons, is transposed.

[0068] Specifically, Figure 2 As shown, the participant-evaluator network includes an evaluator network (also called a critic network) and a participant network. When designing an approximate available control quantity by the backstepping control method, a calculation formula for the system error is predefined. Then the corresponding performance index function is defined, and the Hamilton-Jacobi-Bellman equation is solved to obtain a calculation formula for representing the optimal virtual control quantity. The gradient term in the calculation formula of the optimal virtual control quantity is decomposed to obtain a decomposition formula of the gradient term. And the unknown nonlinear term in the decomposition formula of the gradient term is fitted using the radial basis function to obtain a fitting formula. According to the decomposition formula of the gradient term and the fitting formula of the unknown nonlinear term, a new calculation formula for representing the optimal virtual control quantity is obtained, wherein, in order to obtain an available virtual control quantity, this embodiment uses a neural network based on a participant-evaluator structure to process the unknown ideal weight matrix in the fitting formula of the unknown nonlinear term to obtain the final approximate optimal virtual control quantity. For the approximate optimal virtual control quantity, the neural network of the participant-evaluator structure is specifically used to: first use the evaluator network to estimate the control performance represented by the decomposition formula of the gradient term, and design the weight matrix update law of the evaluator network. Afterwards, the optimal virtual control amount is approximated by a new calculation formula for expressing the optimal virtual control amount and the participant network design, and the weight matrix update law of the participant network is designed.

[0069] The specific derivation steps of designing the approximate optimal virtual control quantity by backstepping control method and actor-evaluator network include:

[0070] First, the optimal virtual control quantity is introduced: .

[0071] The calculation formula for defining the system error is:

[0072] , formula (8);

[0073] , formula (9);

[0074] in, is the optimal virtual control quantity, is the optimal virtual control quantity The estimated value of , i.e., the approximate optimal virtual control quantity; is the position vector of the spacecraft; is the velocity vector of the spacecraft; , are the system errors corresponding to position and velocity respectively; For the The expected position of the followers, For the The desired formation position of the followers relative to the leader, both of which satisfy the following conditions:

[0075] , formula (10);

[0076] in, By the distributed estimator supply, , The definition of is shown in formula (4).

[0077] The calculation formula for defining the performance indicator function is:

[0078] , formula (11);

[0079] in, It is the performance index function used in analyzing the approximate optimal virtual control quantity; is the utility function, The definition of is shown in formula (8); The definition of is shown in formula (9).

[0080] Further, Designed for .

[0081] According to formula (11), the Hamilton-Jacobi-Bellman equation used to analyze the approximate optimal virtual control quantity can be obtained: .

[0082] At the same time, by solving , the optimal virtual control quantity can be obtained , the specific calculation process is:

[0083] , formula (12);

[0084] , formula (13);

[0085] According to formula (13), the optimal virtual control quantity It is expressed as:

[0086] , formula (14);

[0087] in, is the optimal virtual control quantity The gradient of the calculation formula.

[0088] Since the Hamilton-Jacobi-Bellman equation is difficult to solve, a neural network based on the actor-evaluator structure can be used to obtain the optimal virtual control quantity. .

[0089] First, the gradient in formula (14) To break it down:

[0090] , formula (15);

[0091] in, is the approximate optimal virtual control index term 1, is the approximate optimal virtual control quantity index term 2, , , is a positive integer; , Parameters designed for needs; , The definition of is shown in formulas (8) and (9); It is the unknown nonlinear term used in analyzing the approximate optimal virtual control quantity.

[0092] Defined as:

[0093] , formula (16);

[0094] Due to the nonlinear Unknown, consider fitting it using a radial basis function neural network:

[0095] , formula (17);

[0096] in, To analyze the ideal weight matrix used in approximating the optimal virtual control quantity, is the number of neurons. To analyze the basis function matrix used when approximating the optimal virtual control quantity, It is the basis function used in analyzing the approximate optimal virtual control quantity.

[0097] Substituting formula (15) and formula (17) into formula (14), the optimal virtual control quantity can be rewritten as:

[0098] , formula (18);

[0099] in, , , , , , , , , The definition of is shown in formulas (14), (15), and (17).

[0100] Since the ideal weight matrix used in the analysis of the approximate optimal virtual control quantity Unknown, in order to obtain the available virtual control amount, the evaluator network is first used to estimate the control performance represented by formula (15):

[0101] , formula (19);

[0102] in, is the weight matrix of the evaluator network, is the output of the evaluator network, , , , , , The definition of is shown in formulas (14)-(17).

[0103] The weight matrix update law of the evaluator network is designed as the aforementioned formula (5):

[0104] , formula (5);

[0105] in, A positive constant designed for your needs; , The definition of is shown in formula (5).

[0106] According to formula (18), the participant network is used to design the approximate optimal virtual control quantity, and the aforementioned formula (7) is obtained:

[0107] , formula (7);

[0108] in, is the approximate optimal virtual control quantity; is the weight matrix of the participant network; , , , , The definition of is shown in formula (7).

[0109] The weight matrix update law of the participant network is designed as the aforementioned formula (6):

[0110] , formula (6);

[0111] in, , is a normal number that needs to be designed and satisfies ; , , The definition of is shown in formula (6).

[0112] For example, the specific content of the spacecraft cluster formation mission is that the navigator Start moving to the target position The followers maneuver with the leader as the center, and finally form a formation with a side length of The initial position of the follower is: , , , The initial velocity of the follower is: , The expected trajectory of the navigator is: ,in , , , represents the average orbital angular velocity of the leader. The expected formation position of the follower relative to the leader in formula (3)-formula (12) is ,in, ; =diag(250,500,500-260 ); =diag(500,500,500); ; , , , , For formula (15), set , , , , For formula (17), set is the basis function matrix, , , , is the center of the basis function, is the width of the basis function, For the existence of upper bound The approximate error of For formula (5), set =4, the initial value of the evaluator network weight matrix is For formula (6), set =6, the initial value of the participant network weight matrix is .

[0113] In one implementation, when calculating the approximate available control amount, the weight matrix update law of the evaluator network is:

[0114] , formula (20);

[0115] in, To calculate the weight matrix of the evaluator network when approximating the available control, for The transposed matrix of A positive constant designed for calculating the approximate available control quantity; is the basis function matrix used to calculate the approximate available control quantity. The basis function selected for calculating the approximate available control quantity is: is the number of neurons, is transposed;

[0116] The weight matrix update law of the participant network is:

[0117] , formula (21);

[0118] in, To calculate the weight matrix of the actor network when approximating the available control, for The transposed matrix of , They are all normal numbers designed when calculating the approximate available control quantity and satisfy .

[0119] In one implementation, the calculation formula for obtaining the approximate available control amount through the participant network is:

[0120] , formula (22);

[0121] in, is the estimated value of the available control quantity, that is, the approximate available control quantity; , is the design parameter; is the system error corresponding to the speed; is the approximate available control quantity index 1, is the approximate available control quantity index term 2, , is a positive integer; is the actual equation of spacecraft orbital dynamics; is the basis function matrix used to calculate the approximate available control quantity. The basis function selected for calculating the approximate available control quantity is: is the number of neurons.

[0122] Specifically, Figure 2As shown, the estimated value of the available control amount in this embodiment is the approximate available control amount. When designing the approximate available control amount by the backstepping control method, the corresponding performance index function is first defined, and then the Hamilton-Jacobi-Bellman equation is solved to obtain the calculation formula for representing the optimal control amount. The gradient term in the calculation formula of the optimal control amount is decomposed to obtain the decomposition formula of the gradient term. And the unknown nonlinear term in the decomposition formula of the gradient term is fitted using the radial basis function to obtain the fitting formula. According to the decomposition formula of the gradient term and the fitting formula of the unknown nonlinear term, a new calculation formula for representing the optimal control amount is obtained, wherein, in order to obtain the available control amount, this embodiment uses a neural network based on a participant-evaluator structure to process the unknown ideal weight matrix in the fitting formula of the unknown nonlinear term to obtain the final approximate available control amount. For the approximate available control amount, the neural network of the participant-evaluator structure is specifically used to: first use the evaluator network to estimate the control performance represented by the decomposition formula of the gradient term, and design the weight matrix update law of the evaluator network. Afterwards, the approximate available control quantity is designed through a new calculation formula for expressing the optimal control quantity and a participant network, and the weight matrix update law of the participant network is designed.

[0123] The specific derivation steps for designing the approximate available control quantity by the backstepping control method include:

[0124] Define the performance indicator function:

[0125] , formula (23);

[0126] in, It is the performance index function used in analyzing the approximate available control quantity; In order to analyze the utility function used when approximating the available control quantity, it is designed as ; , The definition of is shown in formula (22). Similar to the design process of the approximate optimal virtual control quantity, the solution process of the Hamilton-Jacobi-Bellman equation used in the analysis of the approximate available control quantity is:

[0127] , formula (24);

[0128] , formula (25);

[0129] in, To calculate the gradient of the optimal control quantity; , , The definition of is shown in formulas (22) and (23); The definition of is shown in formula (1); is the approximate optimal virtual control quantity, The derivative of the approximate optimal virtual control quantity.

[0130] According to formula (25), the optimal control quantity It is expressed as:

[0131] , formula (26);

[0132] After that, the gradient in formula (26) To decompose;

[0133] , formula (27);

[0134] in, , Parameters that need to be designed; To calculate the unknown nonlinear terms when approximating the available control quantity; is the actual equation of spacecraft orbital dynamics; The definition of is shown in formulas (8)-(9); The definition of is shown in formula (25); The definition and value of are shown in formula (15).

[0135] Defined as:

[0136] , formula (28);

[0137] Use radial basis functions to find unknown nonlinear terms Perform the fit:

[0138] , formula (29);

[0139] in, To calculate the ideal weight matrix when approximating the available control volume, is the corresponding number of neurons; To calculate the basis function matrix for approximate available control quantity, is the corresponding selected basis function; For the existence of upper bound The approximate error of .

[0140] Substituting formula (27) and formula (29) into formula (26), the optimal control quantity can be rewritten as:

[0141] , formula (30);

[0142] in, The definition of is shown in formula (1); The definition of is shown in formula (8); , The definition and value of are shown in formula (15); The definition of is shown in formula (25); The definition of is shown in formula (27); , , The definition of is shown in formula (29).

[0143] Similar to designing the approximate optimal virtual control quantity through the backstepping control method, in order to obtain the available control quantity, a neural network based on the actor-evaluator structure can be used to calculate the unknown ideal weight matrix to be processed.

[0144] First, the evaluator network is used to estimate the control performance represented by formula (27):

[0145] , formula (31);

[0146] in, is the weight matrix of the evaluator network; is the output of the evaluator network; The definition of is shown in formula (8); , The definition and value of are shown in formula (15); The definition of is shown in formula (25); , , The definition of is shown in formula (27); , The definition of is shown in formula (29).

[0147] The weight matrix update law of the evaluator network is designed as the aforementioned formula (20):

[0148] , formula (20);

[0149] The specific definition is shown in the above formula (20);

[0150] According to formula (30), the approximate available control quantity is designed using the participant network, and the aforementioned formula (22) is obtained:

[0151] , formula (22);

[0152] The specific definition is shown in the above formula (22);

[0153] The weight matrix update law of the participant network is designed as the aforementioned formula (21):

[0154] , formula (21);

[0155] The specific definition is shown in the above formula (21);

[0156] Finally, considering the actuator failure, according to formula (1), the actual control quantity Designed for:

[0157] , formula (32);

[0158] in, , , , The definition of is shown in formula (1); is a selected normal constant; The definition of is shown in formula (8).

[0159] For example, for the above formula (27), set , , , , , these are the parameters that need to be designed. For the above formula (29), we can set is the basis function matrix; , , , is the center of the basis function, is the width of the basis function. For the existence of upper bound The approximate error of For formula (20), set =4, the initial value of the evaluator network weight matrix is For formula (21), set =6, the initial value of the participant network weight matrix is For formula (32), set =0.0001 is the selected normal number. Figure 6 is a three-dimensional graph of the spacecraft formation motion trajectory, Figure 7 is a two-dimensional diagram of the spacecraft formation motion trajectory. Figure 6 and Figure 7 It can be seen that the spacecraft cluster is able to achieve the required formation maneuvers and maintain the predetermined formation shape. Figure 8 The change of the norm of the weight matrix of the evaluator network over time is shown, where evaluator network 1 is the evaluator network used to calculate the approximate optimal virtual control amount, and evaluator network 2 is the evaluator network used to calculate the approximate available control amount; Fig. 9The figure shows the change of the norm of the weight matrix of the participant network over time, where participant network 1 is the participant network used to calculate the approximate optimal virtual control amount, and participant network 2 is the participant network used to calculate the approximate available control amount. Figure 8 and Fig. 9 It can be seen that the norm of the weight matrices of the evaluator network and the participant network is bounded, and the participant-evaluator network mainly plays a role before the spacecraft cluster reaches a stable state, which improves the dynamic performance of the controller. Fig.10 is the position control error curve of the spacecraft formation in the three-dimensional coordinates x, y, and z, where position control error 1 corresponds to the position control error on the x-axis, position control error 2 corresponds to the position control error on the y-axis, and position control error 3 corresponds to the position control error on the z-axis; Fig.11 is the speed control error curve of the spacecraft formation in the three-dimensional coordinates x, y, and z, where speed control error 1 corresponds to the speed control error on the x-axis, speed control error 2 corresponds to the speed control error on the y-axis, and speed control error 3 corresponds to the speed control error on the z-axis. Fig.10 and Fig.11 It can be seen that the position and speed of the followers in the cluster can converge to the expected value within a fixed time, and can successfully complete the specified formation maneuvering task.

[0160] Summary of beneficial effects:

[0161] 1. The present invention adopts a fixed-time distributed estimator based on a directed graph, which not only solves the problem of one-way information transmission between individuals, but also ensures that followers in the cluster can obtain the position and speed information of the leader within a fixed time.

[0162] 2. The present invention adopts a participant-evaluator network based on reinforcement learning to process the difficult equations and unknown nonlinear terms in the control quantity, which can ensure that the cluster's formation position control error and speed control error can converge within a fixed time.

[0163] 3. The present invention uses the upper limit information of the actuator failure to construct the actual control quantity, ensuring that the spacecraft cluster can achieve the target formation maneuver while dealing with the actuator failure and external interference, and maintain the predetermined formation queue under the premise of ensuring the safety of the spacecraft system. The present invention only requires the upper limit information of the actuator failure, has a simple structure and is easy to implement, and has no requirements on the form of the actuator failure, and has good flexibility and reliability.

[0164] Based on the above embodiments, the present invention further provides a spacecraft cluster, which includes a leader and a number of followers, and each of the followers is controlled by any of the above-mentioned reinforcement learning-based distributed fault-tolerant control methods for spacecraft clusters.

[0165] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Fig.12 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a distributed fault-tolerant control method for a spacecraft cluster based on reinforcement learning is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0166] Those skilled in the art will understand that Fig.12 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0167] In one implementation, the terminal has one or more programs stored in its memory and is configured to be executed by one or more processors. The one or more programs include instructions for performing a distributed fault-tolerant control method for a spacecraft cluster based on reinforcement learning.

[0168] In summary, the present invention discloses a distributed fault-tolerant control method for a spacecraft cluster based on reinforcement learning, which relates to the field of spacecraft control technology. Through a fixed-time distributed estimator, it is ensured that the followers in the cluster can obtain the position and velocity information of the leader within a fixed time. And through a fixed-time distributed estimator and a backstepping control method, an approximate optimal virtual control quantity and an approximate available control quantity are designed. On this basis, radial basis functions and participant-evaluator networks are introduced to deal with unknown nonlinear terms and Hamilton-Jacobi-Bellman equations in the approximate optimal virtual control quantity and the approximate available control quantity. Finally, the approximate available control quantity is optimized through the actuator fault processing link to ensure that the spacecraft cluster can complete the predetermined formation task under the influence of the actuator fault.

Claims

1. A distributed fault-tolerant control method for spacecraft clusters based on reinforcement learning, characterized in that: The method comprises: For each follower in the spacecraft cluster, the position estimate and velocity estimate of the leader are obtained through a fixed-time distributed estimator; Obtaining an expected formation position corresponding to the follower, and calculating an expected position corresponding to the follower according to the estimated position value of the leader and the expected formation position; Inputting the desired position, the estimated position value and the estimated speed value of the leader into the fixed-time formation controller corresponding to the follower to obtain the actual control amount corresponding to the follower; Wherein, the fixed time formation controller comprises: An actor-evaluator network based on reinforcement learning is used to calculate an approximate optimal virtual control amount according to the desired position and the position estimate of the navigator through a backstepping control method and a radial basis function; and to calculate an approximate available control amount according to the approximate optimal virtual control amount and the speed estimate of the navigator through a backstepping control method and a radial basis function; An actuator fault processing module, used for calculating the actual control amount through the upper limit information of the actuator fault and the approximate available control amount; The fixed-time distributed estimator is constructed based on an adjacency matrix, the adjacency matrix is ​​used to reflect the edge set of a directed graph, the directed graph is used to reflect the communication topology of the spacecraft cluster, and the fixed-time distributed estimator is: ； in, Indicates Followers, Indicates Followers; and is a positive constant; , , , is an odd number, satisfying , , Indicates The parameters corresponding to the numerator of the first group of exponents selected by the followers, Indicates The parameters corresponding to the denominator of the first group of exponentials selected by the followers, Indicates The parameters corresponding to the numerator of the second group of exponents selected by the followers, Indicates The parameters corresponding to the denominator of the second group of exponentials selected by the followers; is 1 or 2, Indicates The estimates of the leader's position by the followers, Indicates The followers' estimates of the leader's speed; represents the number of followers; the adjacency matrix is ​​defined as , , if A follower can receive messages from Follower information, then ,otherwise .

2. The distributed fault-tolerant control method for spacecraft clusters based on reinforcement learning according to claim 1 is characterized in that: The spacecraft orbital dynamics model corresponding to the follower is: ； in, For the Followers; is the position vector of the spacecraft; is the velocity vector of the spacecraft; For quality, and The two are reciprocal; is the actual control input; For external interference; It is a multiplicative fault; It is an additive fault; is the gravity vector; For the The distance between each follower and the center of the Earth; is the Earth's gravitational constant; is a skew symmetric matrix; is a time-varying nonlinear matrix; is the actual control quantity; The available control amount.

3. The distributed fault-tolerant control method for spacecraft clusters based on reinforcement learning according to claim 1 is characterized in that: The participant-evaluator network includes an evaluator network and a participant network. When calculating the approximate optimal virtual control amount, the weight matrix update law of the evaluator network is: ； in, To calculate the weight matrix of the evaluator network when approximating the optimal virtual control amount, for The transposed matrix of A positive constant designed for calculating the approximate optimal virtual control quantity; The basis function matrix used to calculate the approximate optimal virtual control quantity is: The basis function selected for calculating the approximate optimal virtual control quantity is: is the number of neurons; The weight matrix update law of the participant network is: ； in, To calculate the weight matrix of the participant network when approximating the optimal virtual control amount, for The transposed matrix of , They are all normal numbers designed when calculating the approximate optimal virtual control quantity and satisfy .

4. The distributed fault-tolerant control method for spacecraft clusters based on reinforcement learning according to claim 3 is characterized in that: The calculation formula for obtaining the approximate optimal virtual control amount through the participant network is: ； in, is the approximate optimal virtual control quantity; For the The desired position of each follower; is the system error corresponding to the position; is the approximate optimal virtual control index term 1, is the approximate optimal virtual control quantity index term 2, , is a positive integer; , Design parameters for calculating approximate optimal virtual control quantities; The weight matrix of the participant network when calculating the approximate optimal virtual control amount; The basis function matrix used to calculate the approximate optimal virtual control quantity is: The basis function selected for calculating the approximate optimal virtual control quantity is: is the number of neurons, is transposed.

5. The distributed fault-tolerant control method for spacecraft clusters based on reinforcement learning according to claim 3 is characterized in that: When calculating the approximate available control amount, the weight matrix update law of the evaluator network is: ； in, To calculate the weight matrix of the evaluator network when approximating the available control, for The transposed matrix of A positive constant designed for calculating the approximate available control quantity; is the basis function matrix used to calculate the approximate available control quantity. The basis function selected for calculating the approximate available control quantity is: is the number of neurons, is transposed; The weight matrix update law of the participant network is: ； in, To calculate the weight matrix of the actor network when approximating the available control, for The transposed matrix of , They are all normal numbers designed when calculating the approximate available control quantity and satisfy .

6. The distributed fault-tolerant control method for spacecraft clusters based on reinforcement learning according to claim 5 is characterized in that: The calculation formula for obtaining the approximate available control amount through the participant network is: ； in, is the approximate available control amount; , is the design parameter; is the system error corresponding to the speed; The weight matrix of the actor network when calculating the approximate available control amount; is the approximate available control quantity index 1, is the approximate available control quantity exponent term 2, , is a positive integer; is the actual equation of spacecraft orbital dynamics; is the basis function matrix used to calculate the approximate available control quantity. The basis function selected for calculating the approximate available control quantity is: is the number of neurons.

7. A spacecraft cluster, characterized in that: The spacecraft cluster includes a leader and a number of followers, and each of the followers is controlled by the distributed fault-tolerant control method of the spacecraft cluster based on reinforcement learning as described in any one of claims 1-6.

8. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the program contains instructions for executing the distributed fault-tolerant control method of a spacecraft cluster based on reinforcement learning as described in any one of claims 1-6; and the processor is used to execute the program.

9. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the distributed fault-tolerant control method of a spacecraft cluster based on reinforcement learning as described in any one of claims 1 to 6 above.

Images