A track pursuit game method, system, device and medium

Abstract

The present application relates to the field of space orbit control, and discloses an orbit pursuit and escape game method, system, device and medium. The method comprises the following steps: converting the relative motion of a pursuer and an escaper into a sequence decision model; training a decision neural network of the pursuer based on a double-value evaluation network according to the sequence decision model; obtaining the position, speed, acceleration and time of the pursuer and the escaper and inputting them into the decision neural network to obtain the overload of the pursuer. The present application can cope with the problem of counter-adaptability of strategy mutation of the escaper in real time in a high-dynamic counter-environment.

Description

Technical Field

[0001] This invention relates to the field of aerospace orbit control, and in particular to an orbital pursuit and escape game method, system, device and medium. Background Technology

[0002] In the field of spacecraft orbital control, orbital pursuit-escape game is a typical model for studying non-cooperative target adversarial situations in space. The tracker aims to approach the escape vehicle, while the escape vehicle attempts to escape the track or prolong its capture time. Because the strategies of both sides are difficult to predict accurately and fixed trajectories cannot be pre-planned, control strategies need to be adjusted in real-time during the interactive adversarial process. This makes the solution process extremely challenging. Existing classical game theory methods cannot fully meet these requirements in terms of real-time performance and disturbance resistance, and cannot address the adversarial adaptation problem of sudden changes in escape vehicle strategy in highly dynamic adversarial environments. Summary of the Invention

[0003] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes an orbital pursuit-escape game theory method that can address the adversarial adaptability problem of sudden changes in escaper strategy in a highly dynamic adversarial environment in real time.

[0004] The present invention also proposes a system, device and medium having the above-mentioned orbital pursuit game method.

[0005] A method for orbital pursuit and escape game according to a first aspect of the present invention includes: The relative motion of the tracker and escape device is converted into a sequential decision model; The decision neural network of the tracker, based on the dual-value evaluation network, is trained according to the sequence decision model. The position, velocity, acceleration, and time of the tracker and escape device are obtained and input into the decision neural network to obtain the tracker's overload.

[0006] According to an embodiment of the present invention, an orbital pursuit and escape game method has at least the following beneficial effects: The present invention addresses the geometric game relationship characteristics of orbital pursuit and escape games by designing a pursuit decision method based on prediction terminal error guidance using a dual-value evaluation network. It establishes a spacecraft orbital game training interaction environment based on deep reinforcement learning by constructing a training network architecture and a reward function based on prediction terminal error. This establishes a pursuit and escape game that can adapt in real time to dynamically changing spacecraft orbits, addressing the adversarial adaptability problem of sudden escaper strategy changes in a highly dynamic adversarial environment.

[0007] According to some embodiments of the present invention, the step of converting the relative motion of the tracker and the escaper into a sequential decision model includes: Get the position, velocity, and acceleration of the tracker and escaper at the same moment, and integrate them into the state space corresponding to that moment. The overload in the three axes of the tracker is used as the motion space.

[0008] According to some embodiments of the present invention, training the decision neural network of the tracker based on the dual-value evaluation network according to the sequence decision model includes: According to the sequence decision model, the position, velocity, acceleration, and time of the tracker and the escape device are respectively input into the decision neural network of the tracker to obtain the overload of the tracker; The overload of the tracker is input into the sequence decision model to obtain the state at the next time step, and the state at the next time step is added to the experience pool. If the amount of the experience pool is greater than a preset threshold, several states are randomly sampled from the experience pool; based on the sampled states, the reward value of the decision neural network is determined, and the decision neural network is updated accordingly.

[0009] According to some embodiments of the present invention, the reward value of the decision neural network includes: whether the tracker has reached the safe area of ​​the escaper, the terminal error corresponding to the tracker and the escaper respectively, the relative position of the tracker and the escaper, and the overload of the tracker.

[0010] According to some embodiments of the present invention, the decision neural network includes an online network and a target network; wherein the online network and the target network respectively include a policy network and a dual-value evaluation network, the policy network is used to obtain the overload of the tracker based on the position, velocity, acceleration and time of the tracker and the escaper respectively; the dual-value evaluation network is used to evaluate the output of the policy network.

[0011] According to some embodiments of the present invention, training the decision neural network of the tracker based on the dual-value evaluation network according to the sequence decision model includes: The network parameters of the online network are updated in a first cycle, with the goal of minimizing the difference between the evaluation value output by the value evaluation network in the online network and the learning objective corresponding to the target network. The network parameters of the target network are updated with a second period as the cycle; wherein the first period is shorter than the second period.

[0012] According to some embodiments of the present invention, it further includes: The product of the minimum value output by the value evaluation network in the target network and a preset discount factor is used as the action evaluation output by the strategy network in the target network. The learning objective is obtained by adding the action evaluation output by the policy network in the target network to the reward value of the decision neural network.

[0013] A trajectory pursuit game system according to a second aspect of the present invention includes: The system comprises a tracker, an escape device, and an agent; wherein the agent is used to convert the relative motion of the tracker and the escape device into a sequential decision model; the decision neural network of the tracker, based on a dual-value evaluation network, is trained according to the sequential decision model; the position, velocity, acceleration, and time of the tracker and the escape device are obtained and input into the decision neural network to obtain the overload of the tracker.

[0014] An electronic device according to a third aspect of the present invention includes: Memory, used to store programs; A processor for executing a program stored in the memory, wherein when the processor executes the program stored in the memory, the processor is configured to perform the method as described in any one of the first aspects.

[0015] According to a fourth aspect of the present invention, a storage medium stores computer-executable instructions for performing the method as described in any one of the first aspects.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0017] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0018] Figure 1 This is a training framework diagram of a decision neural network in an orbital pursuit game method provided by an embodiment of the present invention; Figure 2 This is a three-dimensional flight trajectory diagram obtained by training an orbital pursuit and escape game method according to another embodiment of the present invention; Figure 3 This is a schematic diagram showing the relative distance between trackers and escapers at different initial positions in an orbital pursuit and escape game method provided by another embodiment of the present invention; Figure 4 This is a tracker control quantity change curve diagram of an orbital pursuit and escape game method provided in another embodiment of the present invention; Figure 5 This is a diagram showing the results of 1000 target tests of an orbital pursuit and escape game method provided in another embodiment of the present invention; Figure 6 This is a schematic diagram of a scenario for an orbital pursuit and escape game method provided in another embodiment of the present invention; Figure 7 This is a training flowchart of an orbital pursuit game method provided in another embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] It should be understood that in the description of the embodiments of the present invention, "multiple" (or "amounts") means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. If "first," "second," etc., are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0021] like Figure 6 As shown, consider the following near-Earth orbit continuous thrust pursuit-escape game scenario. Assume a probe satellite is performing an escape mission in orbit, and a tracking satellite approaches the escape satellite's safe zone from close range via orbital maneuvers, intending to interfere with or capture it. The escape satellite possesses orbital maneuvering capabilities and, upon sensing the approaching tracker, can actively evade it based on the tracker's position and velocity, thus forming a pursuit-escape game confrontation. Therefore, an orbital pursuit-escape game is necessary.

[0022] This invention provides a method for orbital pursuit and escape game theory, including: Step S100: Convert the relative motion of the tracker and the escaper into a sequence decision model; Step S200: Train the decision neural network of the tracker based on the dual-value evaluation network according to the sequence decision model; Step S300: Obtain the position, velocity, acceleration, and time of the tracker and escape device and input them into the decision neural network to obtain the tracker's overload.

[0023] This invention addresses the geometric game relationship characteristics of orbital pursuit and escape games. Based on a dual-value evaluation network, it designs a pursuit decision-making method for pursuit and escape games guided by prediction terminal error. By constructing a training network architecture and a reward function based on prediction terminal error, a spacecraft orbital game training interaction environment based on deep reinforcement learning is established. This creates a pursuit and escape game that can adapt in real time to dynamically changing spacecraft orbits, and addresses the adversarial adaptability problem of sudden changes in escaper strategy in a highly dynamic adversarial environment.

[0024] In one embodiment, in step S100, the relative motion of the tracker and the escaper is converted into a sequence decision model, including: Get the position, velocity, and acceleration of the tracker and escaper at the same moment, and integrate them into the state space corresponding to that moment. The overload in the three axes of the tracker is used as the motion space.

[0025] In the model construction phase of the orbital pursuit-escape game, the first step is to construct the relative motion model and state-space equations of the spacecraft, and set appropriate thrust constraints and mission completion conditions. Then, the problem is modeled as a Markov decision process to adapt to the reinforcement learning training framework. Since the relative distance between the tracker and the escaper in this scenario is much smaller than the orbital radius of the reference near-Earth orbit, and the entire pursuit-escape game is relatively short, the influence of orbital perturbations can be ignored. The CW (Clohessy Wiltshire) equations are used to approximate the relative motion of the two spacecraft, as shown below: In the formula, Let P represent the tracker and E represent the escape device. The state variables of the tracker and escape device relative to the reference origin can be represented as follows: , , and These are the position vector, velocity vector, and acceleration vector in the coordinate system, respectively. Indicates the angular velocity of the reference orbit; The control variable represents the acceleration of the spacecraft along each axis.

[0026] The above equation can be rewritten in the form of a state-space equation: in: Linear systems can be based on matrices The state transition matrix of the system is obtained by solving: in, ; ; .

[0027] The above state-space equations can be expressed by the following equation based on the state transition matrix: Based on the above relative motion model, a game theory model for spacecraft orbital pursuit and escape is established. Considering the following constraints that the tracking spacecraft must satisfy during the game: In the formula, This represents the safe distance of the escape vehicle. In this game scenario, the tracker only needs to enter the safe area of ​​the escape vehicle via orbital maneuver to be considered to have successfully completed the tracking mission.

[0028] The tracker and escape device need to be realistic when applying continuous thrust, and the thrust overload has upper and lower saturation limits: in, The overload limit for the tracker and escape device.

[0029] Furthermore, the above content is modeled as a Markov Decision Process (MDP). An MDP consists of a state space... Action space State transition function Reward function and discount factor It is composed primarily of components used to solve sequential decision-making problems with uncertainty and dynamism. Its specific definition is as follows: At every decision moment The state observable by the tracker Includes time and relative motion information of the tracker and escaper in the LVLH coordinate system, state space Defined as: The tracker's control output is a continuous thrust overload in three axes, therefore the motion space Defined as: The state transition function is determined by the relative motion equations of the continuous thrust trajectory, given the current state. and The state at the next moment is obtained through state transition. The state transition function can be directly derived from the state transition matrix described above. express.

[0030] To balance immediate and future returns, a discount factor is introduced. The cumulative expected return is: The closer the value is to 1, the more the agent tends to consider maximizing long-term returns.

[0031] Through the above MDP modeling, the orbital pursuit game problem is transformed into a sequential decision problem, which is suitable for the reinforcement learning framework, and the decision model can be obtained through reinforcement learning training.

[0032] In one embodiment, in step S200, training the decision neural network of the tracker based on the dual-value evaluation network according to the sequence decision model includes: Based on the sequence decision model, the position, velocity, acceleration, and time of the tracker and escape device are input into the tracker's decision neural network to obtain the tracker's overload. The overload of the tracker is input into the sequence decision model to obtain the state at the next time step, and the state at the next time step is added to the experience pool. If the amount of data in the experience pool exceeds a preset threshold, several states are randomly sampled from the experience pool. Based on the sampled states, the reward value of the decision neural network is determined, and the decision neural network is updated accordingly.

[0033] It should be noted that decision neural networks actually include value assessment and decision-making.

[0034] In one embodiment, the reward value of the decision neural network includes: whether the tracker has reached the safe area of ​​the escaper, the terminal error corresponding to the tracker and the escaper respectively, the relative position of the tracker and the escaper, and the overload of the tracker.

[0035] In the training architecture and algorithm design phase, this invention addresses the geometric game relationship characteristics of orbital pursuit-escape game. Based on a dual-value evaluation network, target policy smoothing regularization, and a delayed policy update mechanism, it designs a pursuit decision-making method for pursuit-escape game guided by predicted terminal error. A spacecraft orbital game training interactive environment based on deep reinforcement learning is established by constructing a training network architecture and a reward function based on predicted terminal error. This invention trains only the tracker's decision neural network; the escaper's control law is represented by a differential game method, and the tracker's decision network is updated with parameters and weights using an intelligent orbital game algorithm. The training architecture mainly consists of an environment, an experience replay pool, a policy network, and a dual-value function network module. The training framework is as follows: Figure 1 As shown. The training process is as follows. Figure 7 As shown.

[0036] The underlying parameter optimization mechanism of the reinforcement learning decision-making algorithm guided by terminal error prediction is based on the idea of ​​the dual-delay deep deterministic policy gradient algorithm. It directly generates actions by learning a deterministic policy, making it applicable to continuous action spaces. This algorithm references the deep Q-learning algorithm, training the network by sampling data from an experience replay pool to break data correlation and improve learning efficiency. Firstly, the algorithm uses a policy-value evaluation network architecture as its foundation. The policy network is responsible for generating actions and aims to maximize cumulative reward. It learns the optimal policy by updating parameters through the gradient of expected reward. in, Indicates the expected return; For the policy function of the policy network; The action-value function estimated by the valuation network; It is the probability distribution of the state generated by the policy; These are the parameters of the current policy network.

[0037] It should be noted that, for a general value evaluation network, the value evaluation network is responsible for evaluating the performance score of the current policy, and learns the state-action value function by minimizing the temporal difference (TD) error. This enables the network to learn to reasonably predict future benefits: In the formula, For learning objectives; The reward value is sampled from the experience pool; The network parameters of the current value assessment network, and These are the network parameters of the target value assessment network and the target policy network, respectively. This indicates that a batch of data is sampled from the experience replay pool; Discount factor; For the state and action in the next moment; Used to evaluate the difference between the value of the current action and the learning objective.

[0038] It's important to clarify that the y-value is the learning objective for the value network to score state-action pairs (s, a). Without a reference value, how would the network know whether a state-action pair is good or bad? In simple terms, the y-value represents the average total reward that can be obtained if every subsequent step follows the optimal path from the current state. Essentially, it indicates whether adopting this strategy will lead to good or bad future rewards. The value network learns how to evaluate whether a strategy is good or bad, and a higher Q-value is better.

[0039] In one embodiment, to address the limitations of overestimation of Q-value and unstable policy training in the Deep Deterministic Policy Gradient (DDPG) algorithm, the algorithm of this invention first introduces two value evaluation networks. and Estimate the value for the same state-action separately, and reduce the overestimation bias of the Q-value by taking the smaller of the two values, thus preventing the policy network from being misled by excessively high values ​​during training. Secondly, to avoid instability caused by frequent policy network updates, this invention chooses to update parameters only after every N rounds. Simultaneously, the value estimation network in the algorithm updates its parameters n times before updating the policy network once, effectively improving the stability of the algorithm's training. Finally, a target action smoothing mechanism is introduced when calculating the target value. At that time, Gaussian noise is added to the actions generated by the policy network and then pruned: in, It is Gaussian noise. This is the upper limit for pruning. This mechanism can make the target Q-value smoother, improving the robustness of the training strategy.

[0040] Unlike the parameter updates of the current network (i.e., the online network), the algorithm updates the parameters of the target network through soft updates. The current network parameters are obtained through hard updates during initialization, and then updated using gradient descent, as shown in the following equation: The soft update method for the target network is shown in the following formula: In the formula, As a soft update factor, This refers to the collective parameters of the weight matrix and the bias matrix in a network. Indicates the current moment. Let the expected return function be... For loss function, and These are the learning rates for the policy network and the value network, respectively. Soft updates use a convex combination of the current network and the target network for parameter updates, ensuring greater stability of the target network with each iteration and providing a lagging but stable reference target network for the parameter updates of the current network. It should be noted that the tracker's decision neural network includes the current network (online network) and the target network. The current network and the target network each consist of one policy network (Actor) and two value networks (Critic), respectively. The current network directly interacts with the environment and continuously adjusts its parameters through backpropagation; it is the object that the algorithm truly trains and optimizes. The target network, on the other hand, is a "historical copy" of the current network, slowly tracking it through soft updates; its role is to provide stable y-values.

[0041] The maximum number of rounds in the entire training process is Each round contains There are 10 steps, and the step size of each step is 1. In the scenario described in this invention, the initial position of the tracker in each round is randomly initialized within a certain range, and then the current state is... Input is fed into the policy network to generate actions. The state at the next moment is obtained by applying the spacecraft's relative motion equations and the fourth-order Runge-Kutta integral. and reward value , will get The data is stored in the experience pool, and when the conditions are met, the network parameters are updated, repeating the above process.

[0042] In the reward function design phase, the quality of the reward function directly affects the convergence and effectiveness of the reinforcement learning algorithm, making it a crucial part of reinforcement learning. The strategy space of the orbital game problem is large, with numerous constraints. In complex environments, trained strategies are prone to getting trapped in local optima or even failing to converge. To address this issue and guide the tracker to gradually approach the safe region of the escaper with minimal energy consumption, this invention proposes a reward function design method based on prediction terminal error guidance. The reward function consists of the following components.

[0043] When the tracker successfully reaches the escape vehicle's safe zone, it receives a positive reward, and the training round ends. Because the reward function is sparsity and deceptive, only Since it is difficult to achieve convergence of the training curve, continuous guided rewards are needed. This invention utilizes the predicted terminal error and velocity direction to provide continuous rewards to the tracker, thereby guiding the tracker to quickly learn an effective pursuit strategy in the early stages. in, and All of these are hyperparameters. To predict the terminal error vector, Given the relative position vectors of the tracker and the escaper, the terminal moment of the tracker under uncontrolled conditions can be predicted based on the state transition matrix and state equations. Relative to the position of the escape vehicle: In the formula, , It is an identity matrix. ; ; . This represents the angular velocity of the reference orbit. This is the state transition matrix at the terminal time. This represents the relative state of the tracker and the escaper; It represents the first-order differential of a relative state quantity.

[0044] In the continuous maneuver model, the spacecraft's fuel consumption can be represented by the overload during the maneuver: In the formula, For hyperparameters, This is due to overload of the tracker.

[0045] In summary, the tracker's total reward function is: In the final online application phase of the model, after the tracker's network training is completed, the network is saved and its parameters are maintained. The position, velocity, acceleration, and time of each character are input into the policy network, which will generate the overload magnitude of the tracker on each axis. Figure 2-5 The results of applying this method in an orbital pursuit game scenario are shown.

[0046] like Figure 3 , Figure 5 As shown, validation is required after training; to verify the robustness of the model trained by this method, Figure 5 The study shows the relative distance between the tracker and the escape vehicle at each point of intersection during 1000 target practice tests, with most points within 3 meters. Figure 3 The diagram shows the relative distance changes of the tracker during 1000 target practice tests with random perturbations added to its initial position. It can be seen that the tracker was able to approach the safe zone of the escape vehicle quite well. The blue part contains 1000 curves showing the relative distance changes.

[0047] This invention also provides an orbital pursuit and escape game system, including a tracker, an escapee, and an agent, wherein the agent is used to execute the above-described orbital pursuit and escape game method.

[0048] This invention also provides an electronic device, which includes, but is not limited to: Memory, used to store programs; The processor is used to execute programs stored in memory. When the processor executes programs stored in memory, it is used to execute the aforementioned orbital chase game method.

[0049] The processor and memory can be connected via a bus or other means.

[0050] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the method described in the embodiments of the present invention. The processor implements the above method by running the non-transitory software program and instructions stored in the memory.

[0051] The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function; the data storage area may store data for executing the methods described above. Furthermore, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0052] The non-transitory software program and instructions required to implement the above terminal selection method are stored in memory and are executed by one or more processors.

[0053] This invention also provides a storage medium storing computer-executable instructions for performing the above-described methods.

[0054] In one embodiment, the storage medium stores computer-executable instructions that are executed by one or more control processors.

[0055] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0056] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0057] This document describes embodiments of the invention, including preferred embodiments known to the inventors for carrying out the invention. Variations of these embodiments will become apparent to those skilled in the art upon reading the foregoing description. The inventors encourage those skilled in the art to adopt such variations as appropriate, and the inventors intend to practice embodiments of the invention in ways other than those specifically described herein. Therefore, the scope of the invention includes all modifications and equivalents of the subject matter set forth in the appended claims, as permitted by applicable law. Furthermore, the scope of the invention covers any combination of the foregoing elements in all possible variations thereof, unless otherwise indicated herein or otherwise clearly contradicted by the context.

Claims

1. A method for orbital pursuit and escape game theory, characterized in that, include: The relative motion of the tracker and escape device is converted into a sequential decision model; The decision neural network of the tracker, based on the dual-value evaluation network, is trained according to the sequence decision model. The position, velocity, acceleration, and time of the tracker and escape device are obtained and input into the decision neural network to obtain the tracker's overload.

2. The orbital pursuit game method according to claim 1, characterized in that, The process of converting the relative motion of the tracker and the escaper into a sequential decision model includes: Get the position, velocity, and acceleration of the tracker and escaper at the same moment, and integrate them into the state space corresponding to that moment. The overload in the three axes of the tracker is used as the motion space.

3. The orbital pursuit game method according to claim 1, characterized in that, The step of training the decision neural network of the tracker based on the dual-value evaluation network according to the sequence decision model includes: According to the sequence decision model, the position, velocity, acceleration, and time of the tracker and the escape device are respectively input into the decision neural network of the tracker to obtain the overload of the tracker; The overload of the tracker is input into the sequence decision model to obtain the state at the next time step, and the state at the next time step is added to the experience pool. If the amount of the experience pool is greater than a preset threshold, several states are randomly sampled from the experience pool; based on the sampled states, the reward value of the decision neural network is determined, and the decision neural network is updated accordingly.

4. The orbital pursuit game method according to claim 1, characterized in that, The decision neural network includes an online network and a target network; wherein the online network and the target network each include a policy network and a dual-value evaluation network, the policy network is used to obtain the overload of the tracker based on the position, velocity, acceleration and time of the tracker and the escape device respectively; the dual-value evaluation network is used to evaluate the output of the policy network.

5. The orbital pursuit game method according to claim 4, characterized in that, The step of training the decision neural network of the tracker based on the dual-value evaluation network according to the sequence decision model includes: The network parameters of the online network are updated in a first cycle, with the goal of minimizing the difference between the evaluation value output by the value evaluation network in the online network and the learning objective corresponding to the target network. The network parameters of the target network are updated with a second period as the cycle; wherein the first period is shorter than the second period.

6. The orbital pursuit game method according to claim 5, characterized in that, Also includes: The product of the minimum value output by the value evaluation network in the target network and a preset discount factor is used as the action evaluation output by the strategy network in the target network. The learning objective is obtained by adding the action evaluation output by the policy network in the target network to the reward value of the decision neural network.

7. The orbital pursuit game method according to claim 1, characterized in that, The reward value of the decision neural network includes: whether the tracker has reached the safe zone of the escaper, the terminal error corresponding to the tracker and the escaper respectively, the relative position of the tracker and the escaper, and the overload of the tracker.

8. A trajectory-based pursuit and escape game system, characterized in that, include: The system comprises a tracker, an escape device, and an agent; wherein the agent is used to convert the relative motion of the tracker and the escape device into a sequential decision model; the decision neural network of the tracker, based on a dual-value evaluation network, is trained according to the sequential decision model; the position, velocity, acceleration, and time of the tracker and the escape device are obtained and input into the decision neural network to obtain the overload of the tracker.

9. An electronic device, characterized in that, include: Memory, used to store programs; A processor for executing a program stored in the memory, wherein when the processor executes the program stored in the memory, the processor is configured to perform the method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The device stores computer-executable instructions for performing the method as described in any one of claims 1 to 7.

Images