A satellite mission planning evolution method, system and device based on reinforcement learning

Abstract

The present invention provides a satellite mission planning evolution method, system and equipment based on reinforcement learning, the system comprising: a data input module for inputting raw data; an initialization module for initializing algorithm parameters and initializing a Q value table; an optimal solution calculation module for obtaining an optimization solution through a population algorithm and a Q value table update algorithm; a task time window selection module for data interaction with the optimization solution module, and forming a task planning solution based on the transmission data of the optimal solution calculation module. This solution constructs a more refined model, and optimizes the planning method of the optimal solution in combination with QGA design, and can obtain a more practical planning solution for the EDSSP problem, filling the technical gap.

Description

Technical Field

[0001] The present invention relates to the fields of computer data processing and satellite mission planning, and in particular to an evolutionary method, system and device based on reinforcement learning for solving electromagnetic detection satellite mission planning problems. Background Art

[0002] In recent years, the rapid development of aerospace has provided new solutions for various tasks such as information communication, environmental monitoring, and disaster forecasting. The role that satellites can play can be divided into three categories: observation, communication, and navigation satellites according to the different payloads they carry. Observation satellites can be further classified according to whether they carry visible light, infrared, synthetic aperture radar, antennas, and other equipment. Satellites that use signal receivers and antennas as payloads are called electromagnetic detection satellites. Compared with optical imaging satellites, electromagnetic detection satellites have a wider detection range, are not easily affected by weather factors, and can generate more stable and reliable detection benefits. Electromagnetic detection satellites detect and process electromagnetic signals to obtain useful information. On the basis of being able to detect targets with known signal characteristics, electromagnetic detection satellites can also discover unknown signals through a wide range of frequency band searches.

[0003] There are a large number of electromagnetic signals on the surface of the earth where we live, and many electromagnetic signals together constitute the surface electromagnetic environment. The detection of the surface electromagnetic environment can effectively support work in various fields such as environment, agriculture, military, and meteorology. The detection needs are proposed by various types of users, hoping to be effectively met. The electromagnetic detection satellite mission planning problem (EDSSP) is to plan the satellite resources used to execute the needs and determine the specific execution time of the task. The goal of electromagnetic detection satellite mission planning is to generate a reasonable task sequence within a certain time range. This sequence can generate enough detection benefits as much as possible and meet user preferences. However, since the satellite is flying in a fixed orbit, only a part of the tasks are within the satellite detection range, and limited by its own equipment capabilities, the tasks that can be performed during each flight are limited. The phenomenon of oversubscription is very common in satellite mission planning problems, whether it is electromagnetic detection satellite mission planning problems, optical imaging satellite mission planning problems, satellite data downlink planning problems, and satellite measurement and control scheduling problems.

[0004] Electromagnetic signal detection requires an effective balance between detection accuracy and the satellite's own capabilities. From the perspective of signal detection activities, when the number of detection points is increased to improve detection accuracy, the amount of data generated by the detection will also increase accordingly. The capacity of the satellite's onboard storage device is limited. If the detection mission data has been fully used by the satellite, the detection mission cannot be continued before the data is erased. Reasonable setting of detection accuracy based on user needs can maximize detection performance while making full use of satellite resources.

[0005] Existing research on electromagnetic detection satellite mission planning has designed corresponding solutions for multiple scenarios such as multi-region detection and mobile target detection. However, the research adopted a simplified approach to the satellite working mode, which only solved the common problems of various satellite mission planning problems. It can be said that the solutions proposed by existing research are not mature and in-depth enough. Compared with the electromagnetic detection satellite mission planning problem, there are more research results on the optical satellite observation mission planning problem.

[0006] Berger and Barkaoui used quadratic programming to construct a visible light satellite mission planning model. Cho et al. proposed a two-step binary linear programming formulation to develop satellite observation plans for low-orbit satellite constellations. Niu et al. considered using satellites to complete large-area observation tasks after disasters, constructed a multi-objective optimization model and used genetic algorithms to solve this problem. Chen et al. proposed a mixed integer programming model for agile satellite scheduling, which constructed decision variables based on conflicts between time windows and used 5-index variables to obviate the need for a Big-M approach.

[0007] Existing research on reinforcement learning to solve satellite mission planning problems rarely considers combining it with combinatorial optimization methods, and research on using reinforcement learning methods alone to solve this problem is also in its infancy. Summary of the invention

[0008] To solve the above problems, the solution of the present invention combines evolutionary algorithms with reinforcement learning to effectively utilize the advantages of the two methods, so as to improve the search efficiency and search performance. The present invention applies the reinforcement learning method to the evolutionary algorithm, and effectively combines the advantages of the two methods. Control parameters are extremely important for evolutionary algorithms, and these parameters are sensitive to problems. The evolutionary algorithm based on reinforcement learning proposed in this solution allows researchers to invest more energy in algorithm and strategy design. At the same time, in order to verify the effectiveness of the proposed framework, this solution also proposes a genetic algorithm based on Q learning. This solution uses reinforcement learning methods to help evolutionary algorithms improve generalization capabilities and obtain high-quality solutions. This is the first time that reinforcement learning is used as a component embedded in an evolutionary algorithm for satellite mission planning problems.

[0009] Specifically, the present invention provides the following technical solutions:

[0010] On the one hand, the present invention provides a satellite mission planning evolution method based on reinforcement learning, the method comprising:

[0011] Step 1, initialize the algorithm parameters and the Q value table; the algorithm parameters include time t, number of evaluations num_eval, optimal objective function value local_best of the contemporary population, optimal objective function value gobal_best during the search process, optimal individual, optimal individual of the contemporary period, and objective function value control parameter count;

[0012] Step 2: If num_eval < the maximum number of function value evaluations, execute step 3; otherwise, execute step 11.

[0013] Step 3: Calculate the Q value in the table (Q t (s i ,a j ),T) to get the probability of the action being selected; where s i represents the i-th state, T is the control parameter, a j Indicates action j, Q t represents the Q value at time t;

[0014] Step 4: Based on the result of step 3, select an action and an individual, and set the loop variable p from 1 to N. p Increase 1 once to cycle, perform individual crossover mutation, obtain new individuals, form a planning scheme for the new individuals, and calculate the objective function value; where N p Indicates the population size;

[0015] Step 5: Update the Q value after calculating the reward based on the new individual to obtain a new Q value table;

[0016] Step 6: Search the optimal objective function value and its corresponding position number in the population from the contemporary population;

[0017] Step 7: If the optimal objective function value of the contemporary population is greater than the optimal objective function value of the search process, the optimal objective function value of the search process is updated to the optimal objective function value of the contemporary population, and the optimal individual of the search process is updated to the optimal individual of the contemporary population; otherwise, when count is less than the threshold Thre, the optimal individual of the contemporary population is replaced by the optimal individual of the search process, and the optimal function value of the current population is updated to the optimal function value of the search process;

[0018] Step 8: If the current optimal objective function value is worse than the previous optimal objective function value, count is increased by 1;

[0019] Step 9: Update t, set t=t+1;

[0020] Step 10, update the optimal fitness function value of the current population to the optimal fitness function value of the previous generation population, and return to step 2; when the condition that num_eval is equal to the maximum number of function value evaluations is met, execute step 11;

[0021] Step 11: Save the best individual and use the execution plan corresponding to the best individual as the optimization plan.

[0022] Preferably, the initialization in step 1 includes constructing an initial solution, and the individual representations correspond one-to-one to the serial numbers of the task sequences.

[0023] Preferably, in step 4, a method of selecting an individual is:

[0024]

[0025] Among them, fit i represents the fitness function value of individual i, N p represents the population size, p i Represents the probability value of an individual being selected.

[0026] Preferably, the fitness function value uses an objective function value, and the objective function is:

[0027]

[0028] in, Indicates the gain function corresponding to the bandwidth setting, degree j Indicates the degree of importance, represents the bandwidth set based on the importance level, G j represents the detection gain of detection task j, i represents the i-th satellite, S represents the satellite set, T represents the task set, o represents the orbit, O i represents the orbit set of satellite i, k represents a time window, TW represents the time window set, x ijok Indicates whether satellite i is executing the mission in the kth time window on orbit o j .

[0029] Preferably, the detection gain G j The solution is:

[0030]

[0031]

[0032] Among them, u=2.07123sin(θ) / sin(θ 3dB ), J1(u) and J3(u) are the first-order and third-order Bessel functions of the first kind, respectively, θ is the angle between the satellite antenna and the center of the signal source, η is the antenna efficiency, λ is the wavelength, θ 3dB It is the angle of the antenna gain relative to the 3dB attenuation at the beam center, and D represents the antenna aperture.

[0033] Preferably, the The setting method is:

[0034]

[0035] Preferably, the constraints of the objective function include:

[0036] The start and end of the mission must be within the time range required by the mission; the detection angle cannot exceed the maximum angle requirement of the mission; the start and end of the mission must be within the visible time window; the satellite cannot exceed the upper limit of the satellite storage capacity when completing the mission in each orbit; the satellite must meet the conversion time requirements when performing two missions; and meet the mission detection number limit; ijok ∈{0,1}.

[0037] Preferably, the conversion time for executing the two tasks is solved as follows:

[0038] tran ijj′ =max{Γ fre (fre ij ,fre ij′ ),0,Δ

[0039] Γ band (band ij ,band ij′ ),Γ pol (pol ij ,pol ij′ )

[0040] Γ mode (mode ij ,mode ij′ )}

[0041] Among them, Γ pol represents the satellite polarization mode switching time, Γ mode represents the satellite detection mode switching time, Γ band represents the satellite bandwidth setting conversion time, Γ fre represents the satellite frequency band switching time, Δ represents the satellite payload power-on and power-off time, pol ij Indicates the task polarization mode, mode ij Indicates the task detection mode, band ij Indicates the task bandwidth setting, fre ij represents the frequency band setting of the satellite to perform the mission, j and j′ represent task j and task j′.

[0042] Preferably, in step 4, in the crossover mutation, crossover operators of various lengths are used, and the length of the crossover operator is determined by the length of the fragment selected in the task sequence.

[0043] Preferably, in step 4, the crossover variation includes: segment flipping, front position crossover, segment internal order adjustment based on the earliest allowed start time, and segment internal order adjustment based on task required execution time.

[0044] Preferably, in step 5, in the new Q value table, the calculation method for updating the Q value is:

[0045]

[0046] Among them, R t =F t (S t ,A t )-F t-1 (S t-1 ,A t-1 ), α represents the learning rate, γ represents the discount factor, F t represents the objective function value, S t Indicates the state, A t Indicates action.

[0047] Preferably, the Q value table records the performance values ​​of all crossover and mutation combinations during the search process. The Q value in the Q value table is represented by a state-action combination. For an agent in each generation, it is in state s i Take action a j The probability is expressed as:

[0048]

[0049] Among them, Q t (s i ,a j ) / T represents the Q value of the state and action at time t, and T represents the control parameter.

[0050] Preferably, in step 4, a planning scheme is formed using a task time window selection algorithm, and the task time window selection algorithm includes:

[0051] Step 01: Try to arrange the tasks one by one in the order of tasks. If the attempts have been completed, go to step 03, otherwise go to step 02;

[0052] Step 02: Try the time windows in the time window set one by one. If all the attempts are completed, go to step 01 to try to arrange the next task; if all the tasks are arranged, go to step 03;

[0053] Step 03: Output the task planning plan.

[0054] Preferably, the step 02 further comprises:

[0055] Step 021: For a time window, obtain the earliest actual allowed start time (AEVT) of the task. ijok and the latest actual allowed completion time ALVT ijok ;

[0056] Step 022, if (ALVT ijok -AEVT ijok )≥d, then schedule the task in this time window; otherwise, go to step 021 to try the next time window; where d represents the duration of the detection task;

[0057] Step 023: (AEVT ijok +ALVT ijok ) / 2-dur j / 2 as the task start time st ijo , (AEVT ijok +ALVT ijok ) / 2+dur j / 2 as the task end time; if the task start time is earlier than the actual earliest allowed start time, move the task start time backward until the constraint is met; if the task end time is later than the actual latest allowed end time, move the task start time forward until the constraint is met;

[0058] Step 024: Update the time window.

[0059] Preferably, the earliest actual allowed start time AEVT ijok and the latest actual allowed completion time ALVT ijok The calculation method is:

[0060] AEVT ijok =max{est j ,EVT ijok}

[0061] ALVT ijok =in{let j , LVT ijok}

[0062] Among them, est j Indicates the earliest allowed start time of the task, let j Indicates the latest allowed end time of the task; EVT ijok LVT represents the earliest visible time of satellite i to target j in time window k on orbit o. ijok represents the latest visible time of satellite i to target j in time window k on orbit o.

[0063] On the other hand, the present invention also provides a satellite mission planning evolution system based on reinforcement learning, the system comprising:

[0064] A data input module, used for inputting original data, wherein the original data includes task data and original time window data;

[0065] Initialization module, used to initialize algorithm parameters and initialize the Q value table;

[0066] The optimal solution calculation module is used to obtain the optimization solution through the population algorithm and Q value table update algorithm;

[0067] The task time window selection module exchanges data with the optimization scheme module and forms a task planning scheme based on the transmitted data of the optimal scheme calculation module; the transmitted data includes: a task set, a time window set, and a task sequence.

[0068] Preferably, the task time window selection module uses a task time window selection algorithm to form a planning scheme, and the task time window selection algorithm includes:

[0069] Step 01: Try to arrange the tasks one by one in the order of tasks. If the attempts have been completed, go to step 03, otherwise go to step 02;

[0070] Step 02: Try the time windows in the time window set one by one. If all the attempts are completed, go to step 01 to try to arrange the next task; if all the tasks are arranged, go to step 03;

[0071] Step 03: Output the task planning plan.

[0072] Preferably, the step 02 further comprises:

[0073] Step 021: For a time window, obtain the earliest actual allowed start time (AEVT) of the task. ijok and the latest actual allowed completion time ALVT ijok ;

[0074] Step 022, if (ALVT ijok -AEVT ijok )≥d, then schedule the task in this time window; otherwise, go to step 021 to try the next time window; where d represents the duration of the detection task;

[0075] Step 023: (AEVT ijok +ALVT ijok ) / 2-dur j / 2 as the task start time st ijo , (AEVT ijok +ALVT ijok ) / 2+dur j / 2 as the task end time; if the task start time is earlier than the actual earliest allowed start time, move the task start time backward until the constraint is met; if the task end time is later than the actual latest allowed end time, move the task start time forward until the constraint is met;

[0076] Step 024: Update the time window.

[0077] Preferably, the earliest actual allowed start time AEVT ijok and the latest actual allowed completion time ALVT ijok The calculation method is:

[0078] AEVT ijok =max{est j ,EVT ijok}

[0079] ALVT ijok =in{let j , LVT ijok}

[0080] Among them, est j Indicates the earliest allowed start time of the task, let j Indicates the latest allowed end time of the task; EVT ijok LVT represents the earliest visible time of satellite i to target j in time window k on orbit o. ijok represents the latest visible time of satellite i to target j in time window k on orbit o.

[0081] On the other hand, the present invention also provides a satellite mission planning evolution device based on reinforcement learning, which includes a processor and a memory. The processor can call computer instructions in the memory to execute the satellite mission planning evolution method based on reinforcement learning as described above.

[0082] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0083] 1. This scheme constructs a refined model of the EDSSP problem based on mixed integer programming. The model takes into account many practical factors such as detection mode, detection angle, data volume, etc., which is conducive to obtaining a more practical planning scheme.

[0084] 2. The genetic algorithm proposed in this scheme uses reinforcement learning methods to drive the changes of the evolutionary algorithm. According to the algorithm framework, we designed the reward function, Q value update strategy and strategy selection method for QGA, and proposed a two-stage task time window selection algorithm to generate detection plans and calculate the fitness function value.

[0085] 3. This solution provides a new algorithm design and solution to the EDSSP problem, filling the gap in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0087] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0088] Figure 2 The results of comparative tests of multiple algorithms in the embodiments of the present invention are shown;

[0089] Figure 3 This is a comparison diagram of algorithm performance of Example 600-4 of an embodiment of the present invention;

[0090] Figure 4 This is a comparison chart of algorithm performance of Example 800-4 of an embodiment of the present invention. DETAILED DESCRIPTION

[0091] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0093] The present invention is further explained below in conjunction with specific implementation methods and the accompanying drawings.

[0094] The EDSSP problem is to specify a time-ordered task execution sequence for the electromagnetic detection satellite, which maximizes the detection sequence benefits as much as possible while satisfying various satellite constraints. To ensure the execution of the task, it is necessary not only to determine the power-on and power-off time of the receiver, that is, the start and end time of the task, but also to set a series of parameters such as the detection mode, frequency, bandwidth, polarization mode, etc.

[0095] The time range from the start to the end of the signal beam coverage of the electromagnetic satellite is called the visible time window. Since the electromagnetic satellite antenna can effectively detect ground signals in a wide range, in order to reduce noise and improve detection accuracy, the angle between the signal source and the direction of the satellite antenna needs to be within a certain range.

[0096] The detection benefit of electromagnetic detection satellites is affected by two factors. On the one hand, the signal detection benefit is affected by the angular relationship between the satellite antenna and the signal source. The signal gain that can be obtained when the center of the satellite antenna passes directly above the signal source is the largest, and the signal gain is directly linked to the detection effect. The signal gain is inversely correlated with the angle between the antenna and the signal source. In other words, the detection effect is most ideal when the center line of the signal source beam coincides with the extension line of the satellite antenna pointing direction. On the other hand, the detection accuracy will also affect the effect of signal detection. The detection accuracy is limited by the inherent capabilities of the receiver, and the number of sampling points will also have a direct impact on the detection accuracy. There is a one-to-one correspondence between the bandwidth and the number of sampling points, and there are obvious differences in the amount of data obtained by detections with different sampling point settings. The satellite has limited fixed storage capacity, which means that only some tasks can be set to the highest bandwidth, while many tasks need to be set to a smaller bandwidth.

[0097] Since the Earth observation single satellite mission planning problem has been proven to be an NP-hard problem, and the more complex EDSSP problem obviously cannot be solved by a polynomial algorithm, in order to solve the EDSSP problem, the present invention proposes a genetic algorithm based on reinforcement learning.

[0098] For the EDSSP problem, based on a large number of experiments, in order to improve the local search ability of the genetic algorithm, this scheme adds a neighborhood search function to the genetic algorithm. The genetic algorithm based on reinforcement learning (QGA algorithm for short) proposed in this scheme uses the Q learning method to complete the variation of the population. The variation includes two genetic operations, crossover and mutation, and each operation has multiple operation operators. The population obtained by the QGA algorithm uses a task time window selection algorithm to complete the decoding and generate a task execution plan. The change in the benefit of the task execution plan obtained by the task time window selection algorithm updates the Q value, providing a basis for the learning method to select new actions.

[0099] Below, the meanings of various parameters involved in the present invention are first explained as follows:

[0100] T: Task set, total |T| tasks. j , the following properties are defined: j : The earliest allowed start time of the task; let j : The latest allowed end time of the task; dur j : Task required duration; The maximum allowed detection angle of the task; degree j : Task importance; m j : The amount of data for the task; G j :The signal gain that can be obtained by the mission.

[0101] S: satellite set, total |S| satellites. For satellite s i , the following properties are defined: D: satellite antenna aperture; η: antenna efficiency; O i : orbital set, a total of satellites s i |O i | orbits; β: satellite detection unit data volume; M: satellite storage capacity; Γ pol : Satellite polarization mode switching time; Γ mode : Satellite detection mode switching time; Γ band : Satellite bandwidth setting conversion time; Γ fre : Satellite frequency band switching time; Δ: Satellite payload power on / off time; pol ij :Task polarization mode; mode ij :Task detection mode; band ij : Task bandwidth setting; fre ij : Frequency band setting for satellite to perform missions.

[0102] TW: time window set, total |TW| time windows. For time window tw ijok , the following properties are defined: EVT ijok : The earliest visible time of satellite i to target j in time window k on orbit o; LVT ijok : The latest visible time of satellite i to target j in time window k on orbit o; The detection angle of satellite i to target j at time t in time window k on orbit o; The minimum detection angle of satellite i on orbit o to detect target j in the kth time window; The time corresponding to the minimum detection angle of satellite i to target j in the kth time window on orbit o.

[0103] Decision variables: x ijok, indicating whether satellite i performs mission j in the kth time window on orbit o. If it does, then x ijok =1; otherwise, x ijok =0. st ijo , represents the start time of satellite i’s mission j on orbit o.

[0104] Regarding satellite mission planning, in a specific embodiment, we make general assumptions about satellite missions and the operation of electromagnetic detection satellites as follows: all electromagnetic detection satellites have the same structure and function; the detection process will not be affected by external factors; the detection mission is determined and there will be no temporary changes or cancellations; the satellite has sufficient energy during its in-orbit operation; each mission can be completed after one detection and no repeated detection is required.

[0105] Based on the above, the detection gain that a single detection task may obtain is calculated as follows:

[0106]

[0107]

[0108] Among them, u=2.07123sin(θ) / sin(θ 3dB ), J1(u) and J3(u) are the first-order and third-order Bessel functions of the first kind, respectively, θ is the angle between the satellite antenna and the center of the signal source, η is the antenna efficiency, λ is the wavelength, θ 3dB It is the angle of the antenna gain relative to the 3dB attenuation at the center of the beam. The calculation formula is as follows:

[0109] θ 3dB =70λ / D

[0110] Among them, λ represents the wavelength and D represents the aperture of the antenna.

[0111] The bandwidth setting also affects the detection gain. The resulting gain is expressed as the gain function To indicate that the gain function here can be further defined in detail by using the conventional bandwidth setting gain solution method in the art, for example, the gain function can be calculated using the gain-bandwidth product, or the gain function can be calculated using the Shannon formula, etc. Of course, other radar gain calculation formulas can also be used as the gain function here, and those skilled in the art can also make adjustments based on the commonly used gain function categories in the art and according to the specific accuracy requirements, specific reference conditions, etc. in the satellite mission planning. The specific expression of the gain function is not limited here.

[0112] In this scheme, the bandwidth used by the satellite to perform the detection task is dynamically matched according to the priority of the detection task. The higher the mission importance, the larger the bandwidth used, so the detection effect will be better. However, due to the limitation of satellite storage, it is necessary to scientifically set the detection bandwidth. In a more specific implementation, according to the degree of importance, j The formula for setting the bandwidth is as follows:

[0113]

[0114] When the bandwidth of detection task j is based on After setting according to the above formula, the amount of data generated per second is Combined with the task detection time dur j , we can get the total data volume of task j as:

[0115]

[0116] Among them, β is the unit data volume of satellite detection, dur j Probe time for the task.

[0117] Electromagnetic detection satellites need to adjust the parameters of onboard equipment when performing different tasks. The conversion time between tasks is composed of four parts. The first part is the time required for the change of polarization mode, the second part is the time required for the change of detection mode, the third part is the frequency band conversion time, and the fourth part is the bandwidth conversion time. In addition, each time the onboard equipment is turned on and off, it also takes a certain amount of time, and this time interval must be met between two tasks. In order to simplify the constraint judgment, we introduce the variable tran ijj′ , represents the total conversion time. The conversion time of tasks j and j′ is:

[0118] tran ijj′ =max{Γ fre (fre ij ,fre ij′ ),0,Δ

[0119] Γ band (band ij ,band ij′ ),Γ pol (pol ij ,pol ij′ )

[0120] Γ mode (mode ij ,mode ij′ )}

[0121] The planning goal of the EDSSP problem is to obtain the highest detection benefit possible. The objective function is expressed as follows:

[0122] Objective function:

[0123]

[0124] The above objective function can be used as the objective function of the subsequent QGA algorithm for subsequent solution.

[0125] Constraints:

[0126] Condition 1:

[0127] Condition 2:

[0128] Condition 3:

[0129] Condition 4:

[0130] Condition 5:

[0131] Condition 6:

[0132] Condition 7:

[0133] Condition 8:

[0134] Condition 9: x ijok ∈{0,1}.

[0135] The above constraints 1-2 indicate that the start and end of the mission must be within the time range required by the mission, constraint 3 indicates that the detection angle cannot exceed the maximum angle requirement of the mission, constraints 4-5 indicate that the start and end of the mission must be within the visible time window, constraint 6 indicates that the satellite cannot exceed the upper limit of the satellite storage capacity when completing the mission in each orbit, constraint 7 indicates that the satellite must meet various conversion time requirements when performing two tasks, constraint 8 indicates the limit on the number of detections of the mission, and constraint 9 indicates the value range of the decision variable.

[0136] In a specific embodiment, in combination Figure 1 , the genetic algorithm based on reinforcement learning proposed in this scheme can be implemented in the following ways:

[0137] The input of the algorithm includes: population, population size, learning rate, discount factor, Q table, crossover operator, mutation operator, maximum function value evaluation times, and threshold. The specific execution process is as follows:

[0138] Step 1: Initialize algorithm parameters, initialize Q table, set t = 1, record evaluation times num_eval = 0, optimal objective function value of contemporary population local_best = 0, optimal objective function value during search gobal_best = 0, optimize optimal individual gobal_best_individual = [], optimal contemporary individual local_best_individual = [], objective function value control parameter count = 0;

[0139] Step 2: If num_eval < the maximum number of function value evaluations, execute step 3; otherwise, execute step 11;

[0140] Step 3: Use the softmax strategy to calculate (Q t (s i ,a j ),T) to get the probability of the action being selected; where s i represents the i-th state, T represents a control parameter, a j Indicates action j, Q t represents the Q value at time t;

[0141] Step 4: Let p range from 1 to N p The number is increased by 1 each time to perform a cycle to control the crossover variation within the population; where p represents the cycle variable, N p Indicates the population size; in a specific implementation, this step can be implemented in the following ways:

[0142] Step 4.1: Select an action based on the probability of action selection calculated by softmax, for example, by roulette;

[0143] Step 4.2: Select an individual from the population by, for example, roulette;

[0144] Step 4.3: Take the selected action for the individuals in the population and increase p from 1 to N p The number is increased by 1 each time and the cycle is repeated. Starting from the selected individual, the crossover mutation calculation is performed within the population to obtain a new individual.

[0145] Step 4.4: For the generated new individuals, use the task time window selection algorithm to generate a planning scheme;

[0146] Step 4.5: Calculate the objective function value for the generated planning scheme, which can also be used as the fitness function value of the population;

[0147] Step 4.6: num_eval value increases by 1;

[0148] Step 5: Based on the new individual, calculate the reward and update the Q value to obtain a new Q table;

[0149] Step 6: Find the optimal objective function value and its position number in the population from the contemporary population;

[0150] Step 7: If the optimal objective function value of the contemporary population is greater than the optimal objective function value of the search process, the optimal objective function value of the search process is updated to the optimal objective function value of the contemporary population, and the optimal individual of the search process is updated to the optimal individual of the contemporary population; otherwise, when count is less than the threshold Thre, the optimal individual of the contemporary population is replaced by the optimal individual of the search process, and the optimal function value of the current population is updated to the optimal function value of the search process;

[0151] Step 8: If the current optimal objective function value is worse than the previous optimal objective function value, count is increased by 1;

[0152] Step 9: Update the t value and add 1 to it;

[0153] Step 10: Save the optimal fitness function value of the current population as the optimal function value of the previous generation population, and go to step 2 to perform the next generation population solution process; when the condition that num_eval is equal to the maximum number of function value evaluations is met, execute step 11.

[0154] Step 11: Save the individual corresponding to the final population optimal function value as the optimal individual, and use the task execution plan of the optimal individual as the final optimization plan.

[0155] At this point, the basic QGA algorithm process is completed.

[0156] Furthermore, for step 1, initialization is the first step in the population search process of the evolutionary algorithm. It is necessary to construct an initial solution in a certain way, and further search to find a higher quality solution based on this solution.

[0157] Individual representations can be encoded using real numbers, which correspond one-to-one to the serial numbers in the task sequence. Decoding uses the task time window selection algorithm to determine whether to execute the task according to the task sequence of the individual code. If executed, a specific execution time is specified, or the task is abandoned. The final task execution plan is obtained by arranging them in sequence. The advantage of real number encoding is that it ensures the unique correspondence between the elements at each position in the sequence and the task, and the decoding process is simple. Another advantage is that it effectively guarantees the legitimacy of the solution structure without the need to use any repair method.

[0158] Furthermore, in this scheme, fitness evaluation is introduced. Fitness evaluation has two functions in the QGA algorithm. One is to select individuals from the population, and the other is to evaluate the reward update results in the Q-Learning method. Fitness evaluation is obtained by calculating the objective function value of the individual corresponding to the planning scheme. The planning scheme is obtained from the individual by the task time window selection algorithm.

[0159] Furthermore, in the QGA algorithm, attention should be paid to the setting of individual selection. Individual selection is the prerequisite for population evolution. After selecting an individual, the corresponding operation can be performed on it to obtain a new individual. Individual selection should, on the one hand, allow individuals with good performance to have more chances of being successfully selected, and on the other hand, allow other individuals to have a certain chance of being selected to ensure the diversity of algorithm optimization. Therefore, this scheme determines the individuals used for the evolutionary algorithm in the following way:

[0160]

[0161] Among them, fit i represents the fitness function value of individual i, N p represents the population size, p i Represents the probability value of the individual being selected. The fitness function value is calculated using the objective function calculation formula, and the result of the objective function formula is used as the fitness function value of the corresponding individual i. The objective function formula has been given in the previous article and will not be repeated here. For the objective function formula, the task time window selection algorithm is used to solve it.

[0162] Furthermore, in step 4, crossover is a key step in the evolutionary algorithm. The purpose of evolution is to improve the performance of individuals in the population. We use crossover and mutation to achieve the evolution of individuals in the population. The two are collectively referred to as variation. For the crossover operation, this scheme sets crossover operators of various lengths, which are respectively called short-segment crossover operators (denoted as C1), medium-segment crossover operators (denoted as C2), and long-segment crossover operators (denoted as C3).

[0163] Short segment crossover operator (C1): Short segment crossover refers to selecting two task sequence segments of length L from an individual and exchanging the positions of the two segments, where L represents the length control parameter.

[0164] Medium fragment crossover operator (C2): Medium fragment crossover refers to selecting two task sequence fragments of length 2L from an individual and exchanging the positions of the two fragments.

[0165] Long segment crossover operator (C3): Long segment crossover refers to selecting two task sequence segments of length 3L from an individual and exchanging the positions of the two segments.

[0166] Furthermore, the following operations are performed in the crossover:

[0167] Fragment reversal (C4): Select a task sequence fragment of length L from an individual and rearrange each task in the task fragment in reverse order.

[0168] Front position crossover (C5): Select a task sequence segment of length L from an individual and exchange it with a task sequence segment of the same length at the front position of the individual.

[0169] Adjustment of the internal order of segments based on the earliest allowed start time (C6): A task segment of length L is selected from an individual, and the internal order of the segment is adjusted from front to back according to the earliest allowed start time of the task.

[0170] Adjustment of the internal order of segments based on the required execution time of the task (C7): A task segment of length L is selected from an individual, and the internal order of the segment is adjusted from short to long according to the required execution time of the task.

[0171] Furthermore, in step 4, in the QGA algorithm, the variation of mutation is smaller than that of crossover, which is also helpful for improving the algorithm search performance. Mutation is performed by exchanging two positions in the sequence. We combine the crossover operator and the mutation operator to complete the evolution of the population individuals. After the change is completed, the fitness is evaluated. In a more preferred embodiment, the fitness value is calculated using the objective function formula and the task time window selection algorithm for the individual arrangement scheme, that is, the fitness value is calculated by directly using the objective function formula, and the objective function formula is solved by the task time window selection algorithm, and the obtained value is used as the individual fitness value.

[0172] In the RL-EA algorithm, the crossover mutation method used for population evolution is evaluated using the Q-learning method. The operations used for individual evolution within the population are selected based on the Q value, and the reward is calculated and the Q value is updated based on the effect of the selected action.

[0173] Furthermore, in a specific implementation, the calculation formula of Reward is as follows:

[0174] R t =F t (S t ,A t )-F t-1 (S t-1 ,A t-1 )

[0175] Among them, F t represents the objective function value, S t Indicates the state, A t Indicates action.

[0176] The update formula of Q value is as follows:

[0177]

[0178] Among them, α represents the learning rate and γ represents the discount factor.

[0179] Furthermore, this scheme uses the Q-learning method to guide the search of the evolutionary algorithm. In this scheme, the Q-learning method selects the population update strategy by updating the Q value, replacing the traditional operator selection method in the evolutionary algorithm. We regard each individual in the population as an agent, and select and evaluate the search strategy for the agent's performance through the Q value. The interaction between Q-learning and the environment is evaluated by the fitness function, and the calculation of the fitness function value requires the selection of the algorithm according to the task time window.

[0180] Furthermore, an important component of Q learning is the Q value table, which records the performance values ​​of all crossover and mutation combinations used in the search process. c crossover operator, n m When there are mutation operators, the total number of actions contained in the Q value table is n c ×n m +n c +n m . In the framework proposed in this scheme, there are a total of 7 crossover operators and 1 mutation operator. Therefore, the number of actions that each agent (i.e., individual or intelligent body) can use is 15. Further preferably, in a more preferred embodiment, this scheme sets the state to two levels according to the change of the fitness function value, namely higher (denoted as I) and lower (denoted as II), and divides them based on the calculation result of the fitness value. Of course, those skilled in the art can also set the fitness function value to more levels according to the requirements of calculation complexity and accuracy. In this embodiment, in the algorithm proposed in this scheme, the state-action combination, i.e.<state,action> There are 30 combinations in total.

[0181] For each agent (i.e. individual), in state s i Take action a j The probability of is adjusted by the Softmax strategy, and the specific formula is:

[0182]

[0183] Among them, Q t (s i ,a j ) / T is the Q value of the state and action at time t, T is the control parameter, and the control parameter T is generally set to a constant and can be set by, for example, experimental testing.

[0184] Since the objective function value in the EDSSP problem is relatively large, this solution effectively prevents some individuals from being difficult to be selected by introducing the control parameter T. At the same time, Q-learning requires the exploration and mining of a balanced algorithm. In a more preferred embodiment, in order to prevent the Q-learning algorithm from converging too early and falling into a local search and being unable to jump out, we use the ε method. When the randomly generated probability value is less than ε, a random selection is made from the crossover mutation strategy combination as the method adopted by the individual evolution in this population.

[0185] Furthermore, this scheme uses the elite individual retention strategy in the QGA algorithm to speed up the convergence of the algorithm. When the elite individual retention retains the best individual obtained by the search to the next generation of the population, since the use of elite individuals will affect the global search ability of the algorithm after a certain stage of search, we introduce a parameter count and a threshold Thre. When the current population search has not improved compared to the previous generation of the population, that is, the maximum fitness value of the population has not changed, the parameter count will increase by 1. If the value of the parameter count is equal to the threshold Thre, the elite individual will no longer be retained.

[0186] Furthermore, the above QGA algorithm needs to end its operation after a certain number of iterative searches, and output the optimal result as the final task execution plan. Since this solution adopts the crossover and mutation method based on reinforcement learning, one or two evolution operations can be corresponded when Q learning selects an action. Therefore, it is reasonable to use the number of evolutions MFE as an evaluation indicator to record the iterative search and algorithm termination of the genetic algorithm. When the number of algorithm evolutions reaches the set maximum number of evolutions MFE, the algorithm search is stopped.

[0187] In a more preferred embodiment, the task time window selection algorithm can be implemented in the following way:

[0188] After deciding to execute a task through algorithm optimization, it is necessary to specify the start time and end time of the task. The setting of the task start time and end time will not only affect the detection effect of the task, but also the satisfaction of various constraints of the problem. In this scheme, a two-stage task time window selection algorithm is proposed, which is a preliminary screening and then a time specification.

[0189] The purpose of preliminary screening is to ensure that the task execution can meet the requirements of various constraints of the detection task, such as angle, conversion time and other constraints. Therefore, preliminary screening needs to trim some time windows that do not meet the constraints, or abandon the current time window and try to schedule the task in the next time window. The basis for preliminary screening is the earliest actual allowed start time AEVT obtained after trimming. ijok and the latest actual allowed completion time ALVT ijok The earliest actual allowed start time is AEVT. ijok and the latest actual allowed completion time ALVT ijok The calculation method is shown in the following formula:

[0190] AEVT ijok =max{est j ,EVT ijok}

[0191] ALVT ijok = min{let j ,LVT ijok}

[0192] Among them, est j Indicates the earliest allowed start time of the task, let j Indicates the latest allowed end time of the task; EVT ijok LVT represents the earliest visible time of satellite i to target j in time window k on orbit o. ijok represents the latest visible time of satellite i to target j in time window k on orbit o.

[0193] After the time window is trimmed, the start and end time of the detection task need to be specified. Before determining the task execution time, the new time window after trimming needs to be checked and the task is scheduled after the check.

[0194] The signal gain that a satellite can obtain from a signal source generally follows the rule of increasing first and then gradually decreasing. In order to obtain more signal gain, the detection process should ensure that the angle between the antenna and the signal source is as small as possible. The time range selected for the mission should be such that the detection can obtain more benefits as much as possible. Generally speaking, the change in the angle between the satellite and the target within the time window can be regarded as a symmetrical process. Therefore, the middle position of the clipped time window (AEVT ijok +ALVT ijok ) / 2 is taken as the middle moment of the detection task. This moment is usually also the moment when the detection angle is the smallest. The start execution time and the end time of the task are determined based on this position. The calculation formula for the start execution time is as follows:

[0195] st ijo =(AEVT ijok+ALVT ijok ) / 2-dur j / 2

[0196] Among them, st ijo represents the start time of satellite i's mission j on orbit o, dur j Indicates the required duration of the task.

[0197] If this moment can satisfy other constraints as the start time of the task, it will be used as the final start time. ijo If it is earlier than the actual earliest allowed start time, it moves backward (i.e., to st ijo The task starts when all constraints are met, and if the task ends when (st ijo +dur j ) is later than the actual latest allowed end time, then move forward (i.e., to st ijo The start time is moved in the direction of decreasing until all constraints are met. After all tasks are arranged in sequence, the execution plan of the detection task is obtained.

[0198] Specifically, the task time window selection algorithm steps provided by this solution are:

[0199] Input: Genetic algorithm individual task ordering (i.e. task sequence), task set, time window set

[0200] Output: Planning results

[0201] Step 1: Try to arrange the tasks one by one according to the order of tasks in the individual. If you have tried them all, go to step 3;

[0202] Step 2: Try each time window in the time window set. If all attempts are completed, go to step 1 to try to schedule the next task.

[0203] Step 2.1: Calculate the earliest actual allowed start time (AEVT) ijok and the latest actual allowed completion time ALVT ijok ;

[0204] Step 2.2: If (ALVT ijok -AEVT ijok )≥d, then schedule the task, otherwise, go to step 2 to try the next time window; where d represents the duration of the detection task;

[0205] Step 2.3: (AEVT ijok +ALVT ijok ) / 2-dur j / 2 as the task start time st ijo , (AEVTijok +ALVT ijok ) / 2+dur j / 2 is used as the task end time. If it is earlier than the actual earliest allowed start time, the start time is moved backward until all constraints are met. If the task end time is later than the actual latest allowed end time, the start time is moved forward until all constraints are met.

[0206] Step 2.4: Update the time window to [est j ,st ijo ] and [st ijo +dur j ,let j ].

[0207] Step 3: Output the task scheduling results, that is, output the task planning plan.

[0208] In a more specific embodiment, the actual comparison effect of this solution can be evaluated in combination with the performance evaluation of RL-EA algorithm. The comparison test environment is set as follows: a desktop computer with a configuration system Core I7-7700, a 3.6GHz CPU, 8GB of memory, and a Windows 10 operating system, and the coding is done using Matlab 2020a.

[0209] Comparison algorithm: In order to verify the effect of the algorithm proposed in this scheme, genetic algorithm (GA), adaptive large neighborhood search algorithm (ALNS) and heuristic algorithm (HA) are selected as experimental comparison algorithms. The above three algorithms have a large number of applications in satellite mission planning and other planning and scheduling problems, and have good planning performance.

[0210] Examples and evaluation benchmark metrics: In this example, 6 examples of tasks ranging from 400 to 800 are designed, and each algorithm is run 20 times on each example. The detection benefit of the task sequence is the basis for algorithm evaluation. We calculate the maximum value (denoted as Best), average value (denoted as Mean) and standard deviation (denoted as Std.Dev.) of the obtained results. For the comparison results of the algorithms, see Figure 2 shown.

[0211] The experimental results are as follows Figure 2As shown. In the vast majority of examples, the RL-EA algorithm we proposed achieved the maximum detection benefit. In the 500-1 example, the GA algorithm searched for the maximum benefit. It can be said that the RL-EA algorithm has a good performance in global search. According to the mean and standard deviation, it can be seen that the RL-EA algorithm has a good average performance in multiple runs, and the algorithm has good stability. It is worth mentioning that the HA algorithm with a simple structure can also search for high-quality detection task execution plans. In order to more intuitively reflect the algorithm operation results, we show the algorithm results of running the 600-4 and 800-4 examples 10 times. The results are shown as follows Figure 3 , 4 shown.

[0212] from Figure 3 , 4 It can be seen that RL-EA can always obtain higher detection benefits in the 10 running results. In the example of 800-4, the overall performance of the ALNS algorithm is not ideal. In summary, the RL-EA algorithm proposed in this paper can solve the EDSSP problem well and has good application prospects.

[0213] In this scheme, the changes in the evolutionary algorithm are guided by reinforcement learning, which allows each individual to select the appropriate change strategy based on Reward and Q value. We construct the <state, action> combination based on the search performance and change strategy of the evolutionary algorithm, and update the corresponding value after each search. It can be seen from experimental verification that the algorithm proposed in this scheme has obvious advantages in solving the EDSSP problem. The evolutionary algorithm based on reinforcement learning is also very versatile and can be used to solve a variety of planning problems.

[0214] In another specific embodiment, the solution proposed by the present invention can also be implemented in a systematic manner, that is, through a satellite mission planning evolution system based on reinforcement learning, the system comprising:

[0215] A data input module, used for inputting original data, wherein the original data includes task data and original time window data;

[0216] Initialization module, used to initialize algorithm parameters and initialize the Q value table;

[0217] The optimal solution calculation module is used to obtain the optimization solution through the population algorithm and Q value table update algorithm;

[0218] The task time window selection module exchanges data with the optimization scheme module and forms a task planning scheme based on the transmitted data of the optimal scheme calculation module; the transmitted data includes: a task set, a time window set, and a task sequence.

[0219] In the execution of the system, more preferably, the task time window selection module uses a task time window selection algorithm to form a planning scheme, and the task time window selection algorithm includes:

[0220] Step 01: Try to arrange the tasks one by one in the order of tasks. If the attempts have been completed, go to step 03, otherwise go to step 02;

[0221] Step 02: Try the time windows in the time window set one by one. If all the attempts are completed, go to step 01 to try to arrange the next task; if all the tasks are arranged, go to step 03;

[0222] Step 03: Output the task planning plan.

[0223] Preferably, the step 02 further comprises:

[0224] Step 021: For a time window, obtain the earliest actual allowed start time (AEVT) of the task. ijok and the latest actual allowed completion time ALVT ijok ;

[0225] Step 022, if (ALVT ijok -AEVT ijok )≥d, then schedule the task in this time window; otherwise, go to step 021 to try the next time window; where d represents the duration of the detection task;

[0226] Step 023: (AEVT ijok +ALVT ijok ) / 2-dur j / 2 as the task start time st ijo , (AEVT ijok +ALVT ijok ) / 2+dur j / 2 as the task end time; if the task start time is earlier than the actual earliest allowed start time, move the task start time backward until the constraint is met; if the task end time is later than the actual latest allowed end time, move the task start time forward until the constraint is met;

[0227] Step 024: Update the time window.

[0228] Preferably, the earliest actual allowed start time AEVT ijok and the latest actual allowed completion time ALVT ijok The calculation method is:

[0229] AEVT ijok =max{est j ,EVT ijok}

[0230] ALVT ijok = min{let j ,LVT ijok}

[0231] Among them, est j Indicates the earliest allowed start time of the task, let j Indicates the latest allowed end time of the task; EVT ijok LVT represents the earliest visible time of satellite i to target j in time window k on orbit o. ijok represents the latest visible time of satellite i to target j in time window k on orbit o.

[0232] In another embodiment, the present solution can be implemented by means of a device, which may include a corresponding module that performs each or several functions in the above-mentioned various embodiments. Therefore, each step or several steps of the above-mentioned various embodiments can be performed by a corresponding module, and the device may include one or more of these modules. The module may be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0233] The device can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus connects various circuits including one or more processors, memories, and / or hardware modules together. The bus can also connect various other circuits such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0234] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiment of the present solution includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present solution. The processor performs the various methods and processes described above. For example, the method implementation in the present solution can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0235] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0236] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0237] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A satellite mission planning evolution method based on reinforcement learning, characterized in that: The method comprises: Step 1: Initialize algorithm parameters and Q value table; Step 2: If the number of evaluations num_eval < the maximum number of function value evaluations, execute step 3; otherwise, execute step 11; Step 3: Calculate the Q value in the table Get the probability of the action being selected; where, represents the i-th state, T is the control parameter, a j Indicates action j, Q t represents the Q value at time t; Step 4: Based on the result of step 3, select an action and an individual, and set the loop variable p from 1 to Increase 1 once for a cycle, perform individual crossover mutation, obtain new individuals, form a planning scheme for the new individuals, and calculate the objective function value; where, Indicates the population size; Step 5: Update the Q value after calculating the reward based on the new individual to obtain a new Q value table; Step 6: Search the optimal objective function value and its corresponding position number in the population from the contemporary population; Step 7: If the optimal objective function value of the contemporary population is greater than the optimal objective function value in the search process, then the optimal objective function value in the search process is updated to the optimal objective function value of the contemporary population, and the optimal individual in the search process is updated to the optimal individual in the contemporary population; otherwise, Less than threshold When , the optimal individual of the contemporary population is replaced by the optimal individual of the search process, and the optimal function value of the current population is updated to the optimal function value of the search process; Step 8: If the current optimal objective function value is worse than the previous optimal objective function value, the objective function value control parameter count is increased by 1; Step 9: Update time t, set t=t+1; Step 10: Update the optimal fitness function value of the current population to the optimal function value of the previous generation population, and return to step 2; when the evaluation times num_eval is equal to the maximum function value evaluation times, execute step 11; Step 11: Save the optimal individual corresponding to the final population optimal function value, and use the execution plan corresponding to the optimal individual as the optimization plan; The fitness function value is represented by the objective function value, and the objective function is: in, represents the gain function corresponding to the bandwidth setting, Indicates the degree of importance, Indicates the bandwidth set based on the importance level, represents the detection gain of detection task j, i represents the i-th satellite, S represents the satellite set, T represents the task set, o represents the orbit, represents the orbit set of satellite i, k represents a time window, TW represents the time window set, Indicates satellite Is it on track? On Whether to execute the task in the time window ; The detection gain The solution is: in, , and are the 1st and 3rd order Bessel functions of the first kind, respectively. is the angle between the satellite antenna and the center of the signal source, is the antenna efficiency, λ is the wavelength, It is the angle of the antenna gain relative to the 3dB attenuation at the beam center, and D represents the antenna aperture.

2. The method according to claim 1, characterized in that The initialization in step 1 includes constructing an initial solution, and the individual representations correspond one-to-one to the serial numbers of the task sequences.

3. The method according to claim 1, characterized in that In step 4, in the crossover mutation, crossover operators of various lengths are used, and the length of the crossover operator is determined by the length of the segment selected in the task sequence.

4. The method according to claim 1, characterized in that In step 5, in the new Q value table, the calculation method for updating the Q value is: in, , represents the learning rate, represents the discount factor, F t represents the objective function value, S t Indicates the state, A t Indicates action.

5. The method according to claim 1, characterized in that The Q value table records the performance values ​​of all crossover and mutation combinations during the search process. The Q value in the Q value table is represented by a state-action combination. For an agent in each generation, Take action The probability is expressed as: in, represents the Q value of the state and action at time t, and T represents the control parameter.

6. The method according to claim 1, characterized in that In step 4, a planning scheme is formed using a task time window selection algorithm, and the task time window selection algorithm includes: Step 01: Try to arrange the tasks one by one in the order of tasks. If the attempts have been completed, go to step 03, otherwise go to step 02; Step 02: Try the time windows in the time window set one by one. If all the attempts are completed, go to step 01 to try to arrange the next task; if all the tasks are arranged, go to step 03; Step 03: Output the task planning plan.

7. A satellite mission planning evolution system based on reinforcement learning, characterized in that: The system is applied to the satellite mission planning evolution method based on reinforcement learning described in any one of claims 1 to 6, and the system comprises: A data input module, used for inputting original data, wherein the original data includes task data and original time window data; Initialization module, used to initialize algorithm parameters and initialize the Q value table; The optimal solution calculation module is used to obtain the optimization solution through the population algorithm and Q value table update algorithm; The task time window selection module exchanges data with the optimization scheme module and forms a task planning scheme based on the transmitted data of the optimal scheme calculation module; the transmitted data includes: a task set, a time window set, and a task sequence.

8. A satellite mission planning evolution device based on reinforcement learning, characterized in that: The device includes a processor and a memory, and the processor can call computer instructions in the memory to execute the satellite mission planning evolution method based on reinforcement learning as described in claims 1-6.

Images