Intelligent evolution method and device for remote sensing satellite application system

Abstract

The application discloses a kind of remote sensing satellite application system intelligent evolution method and device, the method includes: remote sensing satellite application system is modeled, and the remote sensing satellite application system system twin is obtained;Information collection is carried out to the remote sensing satellite application system system twin, and the twin state information is obtained;The twin state information includes satellite system condition information, ground control condition information, satellite data storage calculation condition information and service task completion degree information to detection object;The twin state information is processed, and intelligent evolution strategy is obtained.The application is deduced running by constructing remote sensing satellite application system system twin based on scene, and is integrated with the interaction of depth reinforcement learning DQN algorithm, overcome the problem that actual system running data is difficult to obtain and reuse, realizes the intelligent evolution of remote sensing satellite application system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of simulation deduction, and in particular to an intelligent evolution method and device for a remote sensing satellite application system. BACKGROUND

[0002] In the current field of remote sensing satellite applications, with the expansion of remote sensing technology in various industries, the requirements for intensive application of remote sensing satellite resources, high-quality services, and efficient performance are increasing. In order to improve the service efficiency and quality of remote sensing satellite applications, artificial intelligence technology is applied to remote sensing satellites and application systems, and a deep reinforcement learning algorithm is used to associate the state space, action space, service quality, and satisfaction of the remote sensing satellite application system, so as to realize the optimization and improvement of the strategy.

[0003] The DQN (Deep Q-Network) algorithm is a Q-Learning algorithm and reinforcement learning algorithm based on deep neural networks. It is an algorithm that applies deep learning to reinforcement learning. It replaces the traditional Q-value function with the expression capability of a neural network to evaluate the advantages of using an action strategy in a given state and guide the system to select the optimal strategy. In the intelligent evolution of the remote sensing satellite application system, the state, action, and task completion of the satellite system can be mapped to the state space and action space of the DQN algorithm, and the task completion and resource consumption indicators can be used as reward values to solve the intelligent perception and decision-making problems of the remote sensing satellite application system.

[0004] Unlike other actual systems, due to the complexity of the composition of the remote sensing satellite application system, the deployment domain, the diversity of needs, and the long running time, it is particularly difficult to obtain data such as the actual working state, working parameters, and service quality of the remote sensing satellite application system through actual system operation. Therefore, modeling and simulation technology can be used to build a system twin model of the remote sensing satellite application system. Through the operation of the twin, with the help of a deep reinforcement learning model, the interaction between the remote sensing satellite application system and the environment is simulated, and the optimal strategy is obtained through iterative learning training to realize the intelligent evolution of the remote sensing satellite system. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an intelligent evolution method and device for a remote sensing satellite application system, which combines simulation deduction technology with deep reinforcement learning artificial intelligence technology to realize the intelligent evolution of the remote sensing satellite application system, thereby providing a technical method for improving the service quality and application efficiency of the remote sensing satellite application system.

[0006] To solve the above technical problems, the first aspect of the embodiment of the present application discloses an intelligent evolution method for a remote sensing satellite application system, which comprises:

[0007] S1, modeling the remote sensing satellite application system to obtain a system twin of the remote sensing satellite application system;

[0008] S2, information collection is performed on the system-in-silico of the remote sensing satellite application system to obtain silico state information; the silico state information comprises satellite system condition information, ground control condition information, satellite data storage and calculation condition information, and service task completion degree information of a detection object;

[0009] S3, the silico state information is processed to obtain an intelligent evolution strategy.

[0010] As an optional implementation, in the first aspect of the embodiment of the present application, the modeling of the remote sensing satellite application system to obtain the system-in-silico of the remote sensing satellite application system comprises:

[0011] S11, the entities and behaviors of the remote sensing satellite application system are modeled to obtain a behavior model;

[0012] S12, resource allocation, task allocation, storage compression, detection and identification, data query, data processing, and data analysis are modeled to obtain an abstract model;

[0013] S13, the detection requirements and detection objects of the remote sensing satellite application system are modeled to obtain an object model;

[0014] S14, the behavior model, the abstract model, and the object model are fused to obtain the system-in-silico of the remote sensing satellite application system.

[0015] As an optional implementation, in the first aspect of the embodiment of the present application, the processing of the silico state information to obtain an intelligent evolution strategy comprises:

[0016] S31, the DNQ model is improved to obtain an optimized DNQ model;

[0017] S32, the silico state information is processed by using the optimized DNQ model to obtain intelligent evolution parameter information;

[0018] S33, the intelligent evolution parameter information is processed to obtain an intelligent evolution strategy.

[0019] As an optional implementation, in the first aspect of the embodiment of the present application, the processing of the silico state information by using the optimized DNQ model to obtain intelligent evolution parameter information comprises:

[0020] S321, the remote sensing satellite application system is processed by using the optimized DNQ model to obtain a state space, an action space, and neuron output;

[0021] S322, processing the remote sensing satellite application system to obtain a reward value; the reward value includes a task completion degree reward, a task completion timeliness reward, and an additional reward;

[0022] S323, processing the remote sensing satellite application system to obtain a training hyperparameter; the training hyperparameter includes a learning rate, a discount factor, a greedy growth factor, an experience replay pool capacity, and a batch size;

[0023] S234, integrating the state space, the action space, the neuron output, the reward value, and the training hyperparameter to obtain intelligent evolution parameter information.

[0024] As an optional implementation, in the first aspect of the embodiment of the present application, the processing of the remote sensing satellite application system by using the optimized DNQ model to obtain the state space, the action space, and the neuron output includes:

[0025] S3211, performing feature extraction and coding on satellite resources, computing resources, storage resources, and the importance level of the current processing task to obtain a state space;

[0026] S3212, processing the remote sensing satellite application system to obtain an action space; the action space includes satellite resource adjustment, storage capacity adjustment, and computing resource adjustment;

[0027] S3213, processing the remote sensing satellite application system to obtain a neuron output; the neuron output includes the number of observation areas, observation time for the area, and result providing time.

[0028] As an optional implementation, in the first aspect of the embodiment of the present application, the processing of the remote sensing satellite application system to obtain a reward value; the reward value includes a task completion degree reward, a task completion timeliness reward, and an additional reward;

[0029] S3221, processing the action of the remote sensing satellite application system to obtain a task completion degree reward and a task completion timeliness reward;

[0030] S3222, processing the satellite resource action to obtain a satellite resource reward; the satellite resource reward expression is:

[0031] re=-nsatellite*ke

[0032] Wherein, if an action is to increase satellite resources, a negative reward value is given; if an action is to reduce satellite resources, a positive reward value is given, re is a reward value, nsatellite is the number of satellites increased or reduced, increase is positive and decrease is negative; ke is a reward coefficient, which is set according to the importance of the increased or reduced satellite in the system, and * represents multiplication;

[0033] S3223, the storage resource action is processed to obtain a storage resource reward;

[0034] S3224, the computing resource action is processed to obtain a computing resource reward;

[0035] S3225, the action fault information is processed to obtain a system reliability reward;

[0036] S3226, the satellite resource reward, the storage resource reward, the computing resource reward and the system reliability reward are integrated to obtain an additional reward.

[0037] As an optional implementation, in the first aspect of the embodiment of the application, the action of the remote sensing satellite application system is processed to obtain a task completion degree reward and a task completion timeliness reward, comprising:

[0038] S32211, the action of the remote sensing satellite application system is processed to obtain a task completion degree reward; the expression of the task completion degree reward rtc is:

[0039]

[0040] Wherein, Aa i is the actual observation area of the remote sensing satellite application system based on the research results of intelligent evolution for the i-th required area detection, which is derived from the actual task demand; Ar i is the area of the i-th required detection area; ta i is the required observation time length of the i-th required detection area; tr i is the actual observation time length of the i-th required detection area, w i is a preset weight coefficient, ktc is a reward coefficient, which is set by experiment, n is the number of required detection areas, and * represents multiplication;

[0041] S32212, the action of the remote sensing satellite application system is processed to obtain a task completion timeliness reward; the expression of the task completion timeliness reward is:

[0042]

[0043] Wherein kt i is the importance coefficient of the i-th task, treqi is the completion time of the i-th task, tactual i is the required completion time of the i-th task.

[0044] The second aspect of the embodiment of the application discloses an intelligent evolution device of a remote sensing satellite application system, and the device comprises:

[0045] a system modeling module, configured to model the remote sensing satellite application system to obtain a system twin of the remote sensing satellite application system;

[0046] a state information collection module, configured to collect information of the system twin of the remote sensing satellite application system to obtain twin state information; the twin state information comprises satellite system information, ground control information, satellite data storage and calculation information, and service task completion degree information of a detection object;

[0047] an intelligent evolution strategy generation module, configured to process the twin state information to obtain an intelligent evolution strategy.

[0048] As an optional implementation, in the second aspect of the embodiment of the application, the modeling of the remote sensing satellite application system to obtain the system twin of the remote sensing satellite application system comprises:

[0049] S11, modeling entities and behaviors of the remote sensing satellite application system to obtain a behavior model;

[0050] S12, modeling resource allocation, task allocation, storage compression, detection and identification, data query, data processing, and data analysis to obtain an abstract model;

[0051] S13, modeling detection requirements and detection objects of the remote sensing satellite application system to obtain an object model;

[0052] S14, fusing the behavior model, the abstract model, and the object model to obtain the system twin of the remote sensing satellite application system.

[0053] As an optional implementation, in the second aspect of the embodiment of the application, the processing of the twin state information to obtain the intelligent evolution strategy comprises:

[0054] S31, improving a DNQ model to obtain an optimized DNQ model;

[0055] S32, processing the twin state information by using the optimized DNQ model to obtain intelligent evolution parameter information;

[0056] S33, processing the intelligent evolution parameter information to obtain the intelligent evolution strategy.

[0057] As an optional implementation, in the second aspect of the embodiment of the present application, the processing of the remote sensing satellite application system by using the optimized DNQ model to obtain the state space, the action space and the neuron output comprises:

[0058] S321, processing the remote sensing satellite application system by using the optimized DNQ model to obtain the state space, the action space and the neuron output;

[0059] S322, processing the remote sensing satellite application system to obtain the reward value; the reward value comprises the task completion degree reward, the task completion timeliness reward and the additional reward;

[0060] S323, processing the remote sensing satellite application system to obtain the training hyperparameter; the training hyperparameter comprises the learning rate, the discount factor, the greediness growth factor, the experience replay pool capacity and the batch size;

[0061] S234, integrating the state space, the action space, the neuron output, the reward value and the training hyperparameter to obtain the intelligent evolution parameter information.

[0062] As an optional implementation, in the second aspect of the embodiment of the present application, the processing of the remote sensing satellite application system by using the optimized DNQ model to obtain the state space, the action space and the neuron output comprises:

[0063] S3211, performing feature extraction and coding on the satellite resources, the computing resources, the storage resources and the importance level of the current processing task of the satellite resources performing the current remote sensing satellite application service task to obtain the state space;

[0064] S3212, processing the remote sensing satellite application system to obtain the action space; the action space comprises the satellite resource adjustment, the storage capacity adjustment and the computing resource adjustment;

[0065] S3213, processing the remote sensing satellite application system to obtain the neuron output; the neuron output comprises the number of observation areas, the observation time for the area and the result providing time.

[0066] As an optional implementation, in the second aspect of the embodiment of the present application, the processing of the remote sensing satellite application system to obtain the reward value; the reward value comprises the task completion degree reward, the task completion timeliness reward and the additional reward;

[0067] S3221, processing the action of the remote sensing satellite application system to obtain the task completion degree reward and the task completion timeliness reward;

[0068] S3222, processing the satellite resource action to obtain the satellite resource reward; the satellite resource reward expression is:

[0069] re = -nsatellite * ke

[0070] Wherein, if an action is to increase satellite resources, a negative reward value is given; if an action is to reduce satellite resources, a positive reward value is given, re is a reward value, nsatellite is the number of satellites increased or reduced, positive for increase and negative for reduction; ke is a reward coefficient, which is set according to the importance of the increased or reduced satellites in the system, and * indicates multiplication;

[0071] S3223, the storage resource action is processed to obtain a storage resource reward;

[0072] S3224, the computing resource action is processed to obtain a computing resource reward;

[0073] S3225, the action fault information is processed to obtain a system reliability reward;

[0074] S3226, the satellite resource reward, the storage resource reward, the computing resource reward and the system reliability reward are integrated to obtain an additional reward.

[0075] As an optional implementation, in the second aspect of the embodiment, the action of the remote sensing satellite application system is processed to obtain a task completion degree reward and a task completion timeliness reward, which comprises:

[0076] S32211, the action of the remote sensing satellite application system is processed to obtain a task completion degree reward; the expression of the task completion degree reward rtc is:

[0077]

[0078] Wherein, Aa i is the actual observation area of the remote sensing satellite application system based on the research results of intelligent evolution for the i-th area to be detected, which is derived from the actual task demand; Ar i is the area of the i-th area to be detected; ta i is the time length of observation required for the i-th area to be detected; tr i is the actual observation time length of the i-th area to be detected, w i is a preset weight coefficient, ktc is a reward coefficient, which is set by experiment, and n is the number of areas to be detected, and * indicates multiplication;

[0079] S32212, the action of the remote sensing satellite application system is processed to obtain a task completion timeliness reward; the expression of the task completion timeliness reward is:

[0080]

[0081] wherein kt i is the importance coefficient of the ith task, treq i is the completion time of the ith task, tactual i is the required completion time of the ith task.

[0082] The third aspect of the present application discloses another intelligent evolution device of a remote sensing satellite application system, which comprises:

[0083] a memory in which executable program codes are stored;

[0084] a processor coupled with the memory;

[0085] The processor invokes the executable program codes stored in the memory to execute part or all of the steps of the intelligent evolution method of the remote sensing satellite application system disclosed in the first aspect of the present application.

[0086] The fourth aspect of the present application discloses a computer storage medium, which stores computer instructions, and when the computer instructions are invoked, part or all of the steps of the intelligent evolution method of the remote sensing satellite application system disclosed in the first aspect of the present application are executed.

[0087] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0088] 1. The present application proposes a framework of intelligent evolution of a remote sensing satellite application system by integrating simulation deduction technology and deep reinforcement learning technology, which overcomes the problem that actual system operation data is difficult to obtain and reuse by constructing a remote sensing satellite application system system twin body deduction operation based on a scene and interacting with a deep reinforcement learning DQN algorithm, and realizes intelligent evolution of the remote sensing satellite application system.

[0089] 2. A dense reward method for intelligent evolution of a remote sensing satellite application system using a DQN algorithm is proposed, which overcomes the problem of slow convergence, low training efficiency, high training cost and even difficulty in convergence caused by reward sparsity and multi-dimensional complex factors in traditional deep reinforcement learning, effectively improves the convergence efficiency of intelligent evolution of the remote sensing satellite application system, and significantly speeds up the evolution research process.

[0090] 3. An intelligent evolution method of a remote sensing satellite application system based on a dense reward designed DNQ algorithm is proposed, which provides a dense reward value, deep reinforcement learning hyperparameters and experimental parameter design in combination with the actual work of the remote sensing satellite application system, adopts double neural network to update Q values, clearly defines the algorithm steps, and clearly defines the method steps for realizing intelligent evolution of the remote sensing satellite application system. BRIEF DESCRIPTION OF DRAWINGS

[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort.

[0092] Figure 1 is a flow diagram of an intelligent evolution method of a remote sensing satellite application system disclosed by an embodiment of the present application;

[0093] Figure 2 is a flow diagram of another intelligent evolution method of a remote sensing satellite application system disclosed by an embodiment of the present application;

[0094] Figure 3 is a structural diagram of an intelligent evolution device of a remote sensing satellite application system disclosed by an embodiment of the present application;

[0095] Figure 4 is a structural diagram of another intelligent evolution device of a remote sensing satellite application system disclosed by an embodiment of the present application. DETAILED DESCRIPTION

[0096] In order to make the technical personnel in the art better understand the present application scheme, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0097] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, not to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or equipment.

[0098] In this paper, the term "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it independent or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0099] The application discloses a kind of remote sensing satellite application system intelligent evolution method and device, which comprises: remote sensing satellite application system is modeled, and remote sensing satellite application system system twin is obtained;The information collection is carried out to the remote sensing satellite application system system twin, and twin state information is obtained;The twin state information includes satellite system condition information, ground control condition information, satellite data storage calculation condition information and service task completion degree information to detection object;The twin state information is handled, and intelligent evolution strategy is obtained.The application is deduced running by constructing remote sensing satellite application system system twin based on scene, and is integrated with the interaction of depth reinforcement learning DQN algorithm, overcomes the problem that actual system running data is difficult to obtain and reuse, realizes the intelligent evolution of remote sensing satellite application system.The following are described in detail.

[0100] Embodiment one

[0101] Please refer to Figure 1 , Figure 1 It is a kind of remote sensing satellite application system intelligent evolution method flow diagram disclosed in the embodiment of the application. Among them, Figure 1 The remote sensing satellite application system intelligent evolution method described is applied to simulation deduction technical field, and the embodiment of the application is not limited.As shown in Figure 1 The remote sensing satellite application system intelligent evolution method can include the following operations:

[0102] S1, remote sensing satellite application system is modeled, and remote sensing satellite application system system twin is obtained;

[0103] S2, the information collection is carried out to the remote sensing satellite application system system twin, and twin state information is obtained;The twin state information includes satellite system condition information, ground control condition information, satellite data storage calculation condition information and service task completion degree information to detection object;

[0104] S3, the twin state information is handled, and intelligent evolution strategy is obtained.

[0105] Optionally, the remote sensing satellite application system is modeled, and remote sensing satellite application system system twin is obtained, including:

[0106] S11, the entity and behavior of remote sensing satellite application system are modeled, and behavior model is obtained;

[0107] S12, resource deployment, task allocation, storage compression, detection identification, data query, data processing, data analysis are modeled, and abstract model is obtained;

[0108] S13, model the detection requirements and detection objects of the remote sensing satellite application system to obtain an object model;

[0109] S14, fuse the behavior model, the abstract model and the object model to obtain a remote sensing satellite application system system twin.

[0110] Optionally, the processing of the twin state information to obtain an intelligent evolution strategy comprises:

[0111] S31, improving the DNQ model to obtain an optimized DNQ model;

[0112] A double DDNQ (Double DNQ) method is adopted to overcome the limitation of overestimation of Q value in the DNQ method. Two deep neural networks are introduced, one is an online network for selecting and estimating the current Q value, and the other is a target network for estimating the target Q value according to the next state and action. The double neural network mechanism avoids the overestimation problem caused by using the same network for action selection and target estimation in the traditional DQN algorithm, so that the remote sensing satellite application system can more accurately learn the optimal strategy in the intelligent evolution process, and the efficiency of evolution calculation is improved.

[0113] S32, processing the twin state information by using the optimized DNQ model to obtain intelligent evolution parameter information;

[0114] S33, processing the intelligent evolution parameter information to obtain an intelligent evolution strategy, comprising:

[0115] The maximum training task number is set as Nepi, the decision time interval of the remote sensing satellite application system is set as tinterval in a task, the maximum decision step number is set as Nstep, and the state, action, reward and next state are collected and recorded for each decision step, and are provided to the DQN network for training. The method flow is as follows:

[0116] (1) initialization setting

[0117] The data that need to be initially set include: experience replay pool memory, maximum training task number, training data collection number, current Q value network parameter and target Q value network parameter, and DQN method hyperparameter.

[0118] (2) single round loop deduction

[0119] The system twin of the remote sensing satellite application system includes a task scene, a task demand, a satellite resource, a storage resource, a computing resource, a behavior rule, and a detected object entity. The system twin of the remote sensing satellite application system is deduced to run, and the intelligent evolution strategy generation module obtains running state information of the system twin of the remote sensing satellite application system. An action to be executed is selected under a greedy strategy. Specifically, a random number is compared with a greedy growth factor. If the random number is greater than the greedy growth factor, an action with the maximum Q value is selected as an output. Otherwise, an action is randomly selected from an action space and output.

[0120] Next state information and reward information are obtained. The reward information is a reward value designed for a dense reward. The reward value is generated by comprehensively generating a task completion degree reward, a task completion timeliness reward, a satellite resource reward, a storage resource reward, a computing resource reward, and a system reliability reward.

[0121] (3) Collecting experience values

[0122] If a set of experience values (sj, aj, rj, sj+1) reaches a set experience collection number, the set of experience values is stored in a replay experience pool. If the experience pool buffer is exceeded, the original experience values are overwritten.

[0123] (4) Updating DQN network parameters

[0124] Experience values with a number of groups equal to a batch_size number are extracted from the experience pool. An estimated value Q(sj, aj) is calculated using a current Q network. A gradient descent is performed using a loss function {[rj+γ*Qtarget(sj+1, aj+1)]-Qcurrent(sj, aj)}2 to update current network parameters θ. Target network parameters θ' are updated using θ at a specific interval number.

[0125] When a task is completed, the next task is trained.

[0126] (5) Storing an intelligent evolution strategy generation model

[0127] During the process of deducing and running the system twin of the remote sensing satellite application system in any task, if a gradient descent reaches a convergence target, an intelligent evolution strategy generation model is saved. When a maximum training task number Nepi is completed, the intelligent evolution strategy generation model is also saved.

[0128] Optionally, the optimized DNQ model is used to process the twin body state information to obtain intelligent evolution parameter information, including:

[0129] S321, the optimized DNQ model is used to process the remote sensing satellite application system to obtain a state space, an action space, and a neuron output.

[0130] S322, processing the remote sensing satellite application system to obtain a reward value; the reward value includes a task completion degree reward, a task completion timeliness reward, and an additional reward;

[0131] S323, processing the remote sensing satellite application system to obtain a training hyperparameter; the training hyperparameter includes a learning rate, a discount factor, a greedy growth factor, an experience replay pool capacity, and a batch size;

[0132] S234, integrating the state space, the action space, the neuron output, the reward value, and the training hyperparameter to obtain intelligent evolution parameter information.

[0133] Optionally, the processing the remote sensing satellite application system to obtain the state space, the action space, and the neuron output using the optimized DNQ model includes:

[0134] S3211, performing feature extraction and coding on satellite resources, computing resources, storage resources, and a current processing task importance level of a satellite resource performing a current remote sensing satellite application service task to obtain a state space;

[0135] S3212, processing the remote sensing satellite application system to obtain an action space; the action space includes satellite resource adjustment, storage capacity adjustment, and computing resource adjustment;

[0136] S3213, processing the remote sensing satellite application system to obtain a neuron output; the neuron output includes an observation region number, an observation time for a region, and a result providing time.

[0137] Optionally, the processing the remote sensing satellite application system to obtain a reward value; the reward value includes a task completion degree reward, a task completion timeliness reward, and an additional reward;

[0138] S3221, processing an action of the remote sensing satellite application system to obtain a task completion degree reward and a task completion timeliness reward;

[0139] S3222, processing a satellite resource action to obtain a satellite resource reward; the satellite resource reward expression is:

[0140] re = -nsatellite * ke

[0141] wherein, if an action is to increase satellite resources, a negative reward value is given; if an action is to reduce satellite resources, a positive reward value is given, re is a reward value, nsatellite is the number of satellites increased or reduced, positive for increase and negative for decrease; ke is a reward coefficient, which is set according to the importance of the type of satellite increased or reduced in the system, and * indicates multiplication;

[0142] S3223, the storage resource action is processed to obtain a storage resource reward; the storage resource reward rs expression is:

[0143] rs=-nstore*ks

[0144] In the task process, if the action increases the storage resource, a negative reward value is given; if an action reduces the satellite storage resource, a positive reward value is given, nstore is the storage resource adjustment amount, positive for increase and negative for decrease; ks is the reward coefficient, which is set according to the importance of the increased or decreased storage resource in the system.

[0145] S3224, the computing resource action is processed to obtain a computing resource reward; the computing resource reward rc expression is:

[0146] rc=-ncalculate*kc,

[0147] In the task process, if the action increases the computing resource, a negative reward value is given; if an action reduces the computing resource, a positive reward value is given, ncalculate is the storage resource adjustment amount, positive for increase and negative for decrease; kc is the reward coefficient, which is set according to the importance of the increased or decreased computing resource in the system.

[0148] S3225, the action failure information is processed to obtain a system reliability reward; the system reliability reward expression is:

[0149] rs-stable=-kstable

[0150] In the task process, if the remote sensing satellite application system fails after an action, such as data transmission interruption, satellite attitude anomaly and other problems, a corresponding punishment rs-stable=-kstable is given according to the severity of the failure, and the reward is negative.

[0151] S3226, the satellite resource reward, the storage resource reward, the computing resource reward and the system reliability reward are integrated to obtain an additional reward.

[0152] Optionally, the action of the remote sensing satellite application system is processed to obtain a task completion degree reward and a task completion timeliness reward, comprising:

[0153] S32211, the action of the remote sensing satellite application system is processed to obtain a task completion degree reward; the task completion degree reward rtc expression is:

[0154]

[0155] Wherein, Aai is the actual observation area of the remote sensing satellite application system based on the intelligent evolution research result for the i-th required area detection, which is derived from the actual task requirement; Ar i is the area of the i-th required detection area; ta i is the time length of observation required for the i-th required detection area; tr i is the actual observation time length for the i-th required detection area, w i is a preset weight coefficient, ktc is a reward coefficient, which is set by experiment, n is the number of required detection areas, and * represents multiplication;

[0156] S32212, the action of the remote sensing satellite application system is processed to obtain a task completion timeliness reward; the task completion timeliness reward expression is:

[0157]

[0158] where kt i is the importance coefficient of the i-th task, treq i is the completion time of the i-th task, tactual i is the required completion time of the i-th task.

[0159] Embodiment Two

[0160] The embodiment is an intelligent evolution method of a remote sensing satellite application system, which fuses modeling simulation technology and deep reinforcement learning technology. Firstly, the modeling simulation technology is used to construct a system twin of the remote sensing satellite application system, the twin interacts with an intelligent evolution strategy generation calculation module, provides state space input for the intelligent evolution strategy module, calculates decision actions, guides the twin to run, obtains service efficiency of the remote sensing satellite application system, and solves the problem that the actual experience value is difficult to set in the traditional DNQ algorithm. Secondly, by customizing the method of running process data of the twin, the state space, action space and reward value in the training process of the intelligent evolution strategy of the remote sensing satellite application system obtained by the DNQ algorithm are reasonably designed in combination with the data acquisition feasibility of the system twin of the remote sensing satellite application system, to provide a basis for evolution strategy generation. Thirdly, the method of dense reward design is used to design rewards for key and critical actions in the running process of the system twin of the remote sensing satellite application system, to change the problem of sparse rewards in the traditional DNQ learning algorithm, and to accelerate the efficiency of intelligent evolution of the remote sensing satellite application system.

[0161] The application mainly proposes an intelligent evolution learning framework of a remote sensing satellite application system by fusing simulation deduction technology and deep reinforcement learning technology, designs intelligent evolution parameters of the remote sensing satellite application system based on a DNQ algorithm designed based on dense rewards, and clearly defines specific method steps of the intelligent evolution of the remote sensing satellite application system based on the DNQ algorithm designed based on dense rewards. The application can provide technical support for efficiently realizing the intelligent evolution of the remote sensing satellite application system.

[0162] 1. An intelligent evolution learning framework of a remote sensing satellite application system by fusing simulation deduction technology and deep reinforcement learning technology

[0163] In combination with a working scene of the remote sensing satellite application system, a modeling simulation and a digital twin body technology are adopted to construct an intelligent evolution learning framework of the remote sensing satellite application system.

[0164] (1) System twin body construction of the remote sensing satellite application system

[0165] Satellite equipment, sensor load equipment, ground stations, communication equipment and the like entities of the remote sensing satellite application system, and corresponding behaviors of orbit maneuver, information communication, ground control, data acquisition, information distribution and the like are modeled; resource allocation, task allocation, storage compression, detection identification, data query, data processing, data analysis and the like links are abstractly modeled; detection requirements and detection objects of the remote sensing satellite application system are modeled; and the above models are integrated and fused to construct and generate a system twin body of the remote sensing satellite application system.

[0166] (2) Twin body running state information acquisition

[0167] The twin body running state acquisition is responsible for acquiring key information of dynamic running of the system twin body of the remote sensing satellite application system in running, including satellite system condition information, ground control condition information, satellite data storage and calculation condition information, and service task completion degree information of detection objects and the like.

[0168] (3) Intelligent evolution strategy generation

[0169] The intelligent evolution strategy generation is responsible for calculating and generating specific execution actions according to the system twin body state information of the remote sensing satellite application system and according to the predetermined action selection strategy. The intelligent evolution strategy generation can cyclically output the next action according to the interval state specified by the system twin body of the remote sensing satellite application system. The intelligent evolution strategy generation can also output the whole action sequence to guide the working running of the actual remote sensing satellite application system.

[0170] An intelligent evolution learning framework of the remote sensing satellite application system based on simulation deduction technology is shown in Figure 2 .

[0171] 2. Intelligent evolution parameter design of the remote sensing satellite application system based on a DNQ algorithm designed based on dense rewards

[0172] The application adopts a double DDNQ (Double DNQ) method in intelligent evolution strategy calculation, and overcomes the limitation of overestimation of Q value in the DNQ method. Two deep neural networks are introduced, one is an online network for selecting and estimating the current Q value, and the other is a target network for estimating the target Q value according to the next state and action. The double neural network mechanism avoids the overestimation problem caused by the use of the same network for action selection and target estimation in the traditional DQN algorithm, so that the remote sensing satellite application system can more accurately learn the optimal strategy in the intelligent evolution process, and improve the efficiency of evolution calculation.

[0173] In the evolution learning process, the intelligent evolution strategy generates an action a according to the current system state s through the online network and executes it. The remote sensing satellite application system twin body responds to the action and returns a new state s' and a reward r. The setting of the reward is closely related to the optimization goal of the system. If the action corresponding to the strategy reduces resource use, improves resource utilization, improves task completion quality or meets specific task requirements, a higher reward is given; on the contrary, if the action leads to resource increase, resource waste, task delay or error, a lower reward is given. According to the reward signal and the new state, the parameters of the online network are updated, and this process is repeated to gradually learn the optimal strategy that can maximize the long-term cumulative reward.

[0174] (1) State space and action space design

[0175] For the intelligent evolution of the remote sensing satellite application system, the related state space, action space and neuron output of the DNQ algorithm are designed based on dense rewards as follows:

[0176] State space: satellite resources, computing resources, storage resources for executing current remote sensing satellite application service tasks, and the importance level of the current processing task. After feature extraction and encoding, a vector containing multiple feature values is formed as input. If the final encoded feature vector dimension is n, the number of input layer neurons is n.

[0177] Design action space: satellite resource adjustment, storage capacity adjustment, and computing resource adjustment.

[0178] Output neuron number: number of observation areas, observation time for each area, and result provision time.

[0179] (2) Reward value design

[0180] In the intelligent evolution process of remote sensing satellite application system, if the task completion degree and task completion timeliness after one task is directly taken as the reward, the reward and punishment will be delayed due to the influence of sparse reward and multi-dimensional complex factors, it will be difficult to determine the influence of behavior and final reward, resulting in slow convergence, low training efficiency, high training cost and even difficult to converge and other problems. The invention adopts the method of dense reward value, sets a reward for each action of the remote sensing satellite application system, in addition to the final task completion degree and task completion quality reward, an additional reward is designed, including satellite resource reward, storage resource reward, computing resource reward and system reliability reward.

[0181] Satellite resource reward: during the task process, if an action is to increase satellite resources, a negative reward value is given; if an action is to reduce satellite resources, a positive reward value is given. The reward value is re=-nsatellite*ke, nsatellite is the number of satellites increased or decreased, positive for increase and negative for decrease; ke is the reward coefficient, which is set according to the importance of the increased or decreased satellites in the system.

[0182] Storage resource reward: during the task process, if the action increases the storage resource, a negative reward value is given; if an action is to reduce satellite resources, a positive reward value is given. The storage resource adjustment amount is nstore, the reward rs=-nstore*ks, nstore is the number of storage resources increased or decreased, positive for increase and negative for decrease; ks is the reward coefficient, which is set according to the importance of the increased or decreased storage resources in the system.

[0183] Computing resource reward: during the task process, if the action increases the computing resource, a negative reward value is given; if an action is to reduce satellite resources, a positive reward value is given. The storage resource adjustment amount is ncalculate, the reward rc=-ncalculate*kc, wherein ncalculate is the number of computing resources increased or decreased, positive for increase and negative for decrease; kc is the reward coefficient, which is set according to the importance of the increased or decreased computing resources in the system.

[0184] System reliability reward: during the task process, if the remote sensing satellite application system fails after an action, such as data transmission interruption, satellite attitude abnormality and other problems, a corresponding punishment rs-stable=-kstable is given according to the severity of the failure, and the reward is negative.

[0185] Task completion degree reward: after the completion of a remote sensing satellite application system task, a task completion reward is given according to the task completion percentage, the reward value Wherein Aa_i is the actual observation area of the remote sensing satellite application system based on the research results of intelligent evolution for the i-th required area detection, which comes from the actual task demand; Ar_i is the area of the i-th required detection area; ta_i is the required observation time length of the i-th required detection area; tr_i is the actual observation time length of the i-th required detection area.

[0186] Task timeliness reward: if the task result providing time is later than the required providing time, the reward is negative; if the result providing time is earlier than the required providing time, the reward is positive. rt= Wherein kt is the task importance coefficient, treq is the required task completion time, and tactual is the required task completion time.

[0187] According to the dense reward design method, the total reward of the intelligent evolution strategy generated by the above various reward designs is:

[0188] R=re+rs+rc+rs-stable+rtc+rt

[0189] The intelligent evolution strategy module will continuously adjust the strategy according to the total reward signal to realize the intelligent evolution of the system.

[0190] (3) Training hyperparameter design

[0191] Learning rate: during the training process, learning rate decay strategy can be used, and the initial learning rate can be set to a small value. As the training step increases, the learning rate is gradually reduced to balance the convergence speed and stability of the algorithm. After every N training steps, the learning rate is multiplied by the decay factor kr, that is, the learning rate a=kr* a, so that the network can quickly explore different strategies in the early stage of training, and then more finely adjust the parameters to approach the optimal solution.

[0192] Discount factor: the discount factor γ is used to measure the importance of future rewards. A larger discount factor means that the agent pays more attention to long-term rewards and tends to choose actions that can bring long-term benefits; a smaller discount factor makes the agent pay more attention to short-term rewards. In the remote sensing satellite application system, due to the complexity and long-term nature of the task, a discount factor close to 1 is selected, which tends to meet the final task goal and make decisions that are beneficial to the long-term stable operation and performance optimization of the system.

[0193] Greediness growth factor: with a probability of 1-ε, the action with the maximum Q value is selected, and with a probability of ε, other actions are randomly selected. In this method, a small ε is set to allow the intelligent evolution strategy to use the known best action most of the time, while also allowing some opportunities to explore new actions to discover better strategies.

[0194] Experience replay pool capacity: the experience replay pool is used to store the experience samples (sj, aj, rj, sj+1) generated by the interaction of the agent and the environment, wherein s is the current state, a is the action taken, r is the reward obtained, and sj+1 is the next state. The experience replay pool capacity M is set to a large value, which ensures that the network can learn from rich experience during training, avoiding overfitting caused by too high correlation of consecutive samples. When the experience replay pool is full, the new experience sample will overwrite the earliest stored sample.

[0195] Batch size: during each training, a batch of samples is randomly selected from the experience replay pool for network parameter update, and the batch size batch_size can ensure that sufficient information is contained in one update to stabilize the gradient, and the batch size is not too large to cause excessive consumption of computing resources and slow training speed.

[0196] Relying on the operation of the system twin of the remote sensing satellite application system, the intelligent evolution strategy generation module gradually realizes intelligent evolution of the remote sensing satellite application system through the change of the twin operation state and strategy calculation training, can better adapt to various complex and variable task requirements and environmental condition changes, and the overall performance is optimized.

[0197] 3. The intelligent evolution method of the remote sensing satellite application system based on the DNQ algorithm designed based on dense rewards

[0198] The method provided by the application designs a DNQ algorithm based on dense rewards, combines the operation deduction of the system twin of the remote sensing satellite application system, sets the maximum training task number to Nepi, sets the decision time interval of the remote sensing satellite application system to tinterval in one task, sets the maximum decision step number to Nstep, collects and records the state, action, reward and next state for each decision step, and provides the DQN network for training. The method flow is as follows:

[0199] (1) initialization setting

[0200] The data that need to be initially set include: experience replay pool memory, maximum training task number, training data collection number, current Q value network parameter and target Q value network parameter, and DQN method hyperparameter.

[0201] (2) single round loop deduction

[0202] The system twin of the remote sensing satellite application system is constructed, including a task scene, a task demand, a satellite resource, a storage resource, a computing resource, a behavior rule, and a detected object entity. The system twin of the remote sensing satellite application system is deduced to run, and the intelligent evolution strategy generation module obtains the running state information of the system twin of the remote sensing satellite application system, selects an output action to be executed under a greedy strategy, and specifically, compares a random number with a greedy growth factor, if the random number is greater than the greedy growth factor, selects an action with the maximum Q value as the output, or otherwise, randomly selects an action in an action space as the output.

[0203] Next state information and reward information are obtained, the reward information is a reward value designed according to a dense reward, and the reward value is generated by comprehensively generating a task completion degree reward, a task completion timeliness reward, a satellite resource reward, a storage resource reward, a computing resource reward and a system reliability reward.

[0204] (3) Collecting experience values

[0205] If a set of experience values (sj, aj, rj, sj+1) reach a set experience collection number, the experience values are stored in a replay experience pool, and if the experience pool buffer is exceeded, the original experience values are overwritten.

[0206] (4) Updating DQN network parameters

[0207] Experience values with a number of groups of batch_size are extracted from the experience pool, an estimated value Q(sj, aj) is calculated using a current Q network, a loss function {[rj+γ*Qtarget(sj+1, aj+1)]-Qcurrent(sj, aj)}2 is used to perform gradient descent to update current network parameters θ, and θ target network parameters θ' are updated using θ at a specific interval number.

[0208] When a task is completed, the next task training is continued.

[0209] (5) Storing an intelligent evolution strategy generation model

[0210] In any process of deducing and running a task of the system twin of the remote sensing satellite application system, if gradient descent reaches a convergence target, an intelligent evolution strategy generation model is saved. When a maximum training task number Nepi is completed, the intelligent evolution strategy generation model is also saved.

[0211] Embodiment three

[0212] Please refer to Figure 3 , Figure 3 is a structural schematic diagram of an intelligent evolution device of a remote sensing satellite application system disclosed in an embodiment of the present application. In the diagram, Figure 3 The remote sensing satellite application system intelligent evolution device described is applied to the technical field of simulation deduction, and embodiments of the present application are not limited. For example Figure 3As shown in the figure, the intelligent evolution device of the remote sensing satellite application system can include the following operations:

[0213] S301, a system modeling module, configured to model the remote sensing satellite application system to obtain a system twin of the remote sensing satellite application system;

[0214] S302, a state information acquisition module, configured to acquire information of the system twin of the remote sensing satellite application system to obtain twin state information; the twin state information includes satellite system information, ground control information, satellite data storage and calculation information, and service task completion degree information of a detection object;

[0215] S303, an intelligent evolution strategy generation module, configured to process the twin state information to obtain an intelligent evolution strategy.

[0216] Embodiment Four

[0217] Please refer to Figure 4 , Figure 4 is another structure schematic diagram of the intelligent evolution device of the remote sensing satellite application system disclosed by the embodiment of the present application. Among them, Figure 4 The described intelligent evolution device of the remote sensing satellite application system is applied to the technical field of simulation deduction, and the embodiment of the present application is not limited. As Figure 4 As shown in the figure, the intelligent evolution device of the remote sensing satellite application system can include the following operations:

[0218] The memory 401 stores executable program codes;

[0219] The processor 402 is coupled with the memory 401;

[0220] The processor 402 calls the executable program codes stored in the memory 401, and is configured to execute the steps in the intelligent evolution method of the remote sensing satellite application system described in Embodiment One and Embodiment Two.

[0221] Embodiment Five

[0222] The embodiment of the present application discloses a computer readable storage medium, which stores a computer program for electronic data exchange, wherein the computer program makes the computer execute the steps in the intelligent evolution method of the remote sensing satellite application system described in Embodiment One and Embodiment Two.

[0223] The apparatus embodiments described above are only illustrative, wherein the modules illustrated as separate components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules, i.e., can be located in one place or distributed to multiple network modules. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0224] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, and the computer software product can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer readable medium capable of carrying or storing data.

[0225] Finally, it should be noted that: the remote sensing satellite application system intelligent evolution method and device disclosed by the embodiments of the application are only the preferred embodiments of the application, and are used to illustrate the technical solutions of the application, but not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that; it can still modify the technical solutions recorded in the foregoing embodiments, or replace some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A method for intelligent evolution of remote sensing satellite application systems, characterized in that, The method comprises: S1, modeling a remote sensing satellite application system to obtain a remote sensing satellite application system system twin; S2, collecting information of the remote sensing satellite application system system twin to obtain twin state information; the twin state information comprises satellite system condition information, ground control condition information, satellite data storage and calculation condition information, and service task completion degree information of a detection object; S3, processing the twin state information to obtain an intelligent evolution strategy, comprising: S31, improving a DQN model to obtain an optimized DQN model; S32, processing the twin state information by using the optimized DQN model to obtain intelligent evolution parameter information, comprising: S321, processing the remote sensing satellite application system by using the optimized DQN model to obtain a state space, an action space, and neuron output; S322, processing the remote sensing satellite application system to obtain a reward value; the reward value comprises a task completion degree reward, a task completion timeliness reward, and an additional reward; S323, processing the remote sensing satellite application system to obtain training hyperparameters; the training hyperparameters comprise a learning rate, a discount factor, a greediness growth factor, an experience replay pool capacity, and a batch size; S324, integrating the state space, the action space, the neuron output, the reward value, and the training hyperparameters to obtain the intelligent evolution parameter information; S33, processing the intelligent evolution parameter information to obtain the intelligent evolution strategy.

2. The intelligent evolution method for remote sensing satellite application systems according to claim 1, characterized in that, The modeling of the remote sensing satellite application system to obtain the remote sensing satellite application system system twin comprises: S11, modeling entities and behaviors of the remote sensing satellite application system to obtain a behavior model; S12, modeling resource allocation, task allocation, storage compression, detection and identification, data query, data processing, and data analysis to obtain an abstract model; S13, modeling detection requirements and detection objects of the remote sensing satellite application system to obtain an object model; S14, fusing the behavior model, the abstract model, and the object model to obtain the remote sensing satellite application system system twin.

3. The intelligent evolution method for remote sensing satellite application systems according to claim 1, characterized in that, The processing of the remote sensing satellite application system by using the optimized DQN model to obtain the state space, the action space, and the neuron output comprises: S3211, feature extraction and coding of satellite resources, computing resources, storage resources, and an importance level of a current processing task of a satellite resource performing a current remote sensing satellite application service task to obtain a state space; S3212, processing the remote sensing satellite application system to obtain an action space; the action space comprises satellite resource adjustment, storage capacity adjustment, and computing resource adjustment; S3213, processing the remote sensing satellite application system to obtain neuron output; the neuron output comprises an observation region quantity, an observation time for a region, and a result providing time.

4. The intelligent evolution method for remote sensing satellite application systems according to claim 1, characterized in that, The processing of the remote sensing satellite application system to obtain a reward value; the reward value comprises a task completion degree reward, a task completion timeliness reward, and an additional reward; S3221, processing the action of the remote sensing satellite application system to obtain a task completion degree reward and a task completion timeliness reward; S3222, processing the satellite resource action to obtain a satellite resource reward; the satellite resource reward expression is: re=-nastellite*ke Wherein, if an action is to increase satellite resources, a negative reward value is given; if an action is to reduce satellite resources, a positive reward value is given, re is the reward value, nsatellite is the number of satellites increased or decreased, positive for increase and negative for decrease; ke is the reward coefficient, which is set according to the importance of the increased or decreased satellite in the system, and * indicates multiplication; S3223, processing the storage resource action to obtain a storage resource reward; r s = -n store *ks Wherein, r s is the storage resource reward, n store is the number of increased or decreased storage resources, with positive value for increase and negative value for decrease; ks is the reward coefficient, set according to the importance of the increased or decreased storage resources in the system; S3224, processing the computing resource action to obtain a computing resource reward; rc = -n calculate *kc Wherein, rc is the computing resource reward, n calculate is the number of computing resources added or reduced, the addition being positive and the reduction being negative; and kc is the reward coefficient, which is set according to the importance of the added or reduced computing resources in the system; S3225, processing the action failure information to obtain a system reliability reward; r s-stable = -k stable wherein, r s-stable is the system reliability reward, in the task process, if the remote sensing satellite application system appears failure after a certain action, according to the severity k stable of the failure, the corresponding punishment r s-stable is given, and the reward is negative. S3226, integrating the satellite resource reward, the storage resource reward, the computing resource reward and the system reliability reward to obtain an additional reward.

5. The intelligent evolution method for remote sensing satellite application systems according to claim 4, characterized in that, The processing of the action of the remote sensing satellite application system to obtain a task completion degree reward and a task completion timeliness reward comprises: S32211, processing the action of the remote sensing satellite application system to obtain a task completion degree reward; the task completion degree reward rtc expression is: wherein, Aa i is the actual observation area of the i-th required area detection based on the remote sensing satellite application system of intelligent evolution research results, derived from actual task requirements; Ar i is the area of the i-th required detection area; ta i is the required observation time length of the i-th required detection area; tr i is the actual observation time length of the i-th required detection area, w i is a preset weight coefficient, ktc is a reward coefficient, set by experiment, n is the number of required detection areas, and * indicates multiplication. S32212, processing the action of the remote sensing satellite application system to obtain a task completion timeliness reward; the task completion timeliness reward expression is: where kt i is the importance coefficient of the ith task, treq i is the completion time of the ith task, tactual i is the required completion time of the ith task.

6. The intelligent evolution method for remote sensing satellite application systems according to claim 1, characterized in that, Comprise: A system modeling module for modeling the remote sensing satellite application system to obtain a remote sensing satellite application system system twin; A state information acquisition module for acquiring information of the remote sensing satellite application system system twin to obtain twin state information; the twin state information includes satellite system information, ground control information, satellite data storage and computing information, and service task completion degree information of the detected object; An intelligent evolution strategy generation module for processing the twin state information to obtain an intelligent evolution strategy.

7. An apparatus for intelligent evolution of remote sensing satellite application systems, characterized in that, The device comprises: A memory storing executable program code; A processor coupled with the memory; The processor invokes the executable program code stored in the memory to execute the remote sensing satellite application system intelligent evolution method of any one of claims 1-5.

8. A computer storable medium, characterized by The computer storage medium stores computer instructions, which when invoked, are used to execute the remote sensing satellite application system intelligent evolution method of any one of claims 1-5.

Images