A group target tracking reinforcement learning control method, system and electronic device
By constructing common coverage relationships and priority prior attention among targets, generating velocity and heading sequences for future control steps, and applying kinematic constraints, the problems of lack of geometric constraints in target selection and unstable attention in multi-target pursuit are solved, achieving efficient and stable pursuit control.
Patent Information
- Application Number
- CN202611460384.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multi-target pursuit reinforcement learning methods suffer from problems such as lack of circular coverage scale constraints in target selection, unstable attention allocation, insufficient utilization of opportunities for multi-target common coverage, and insufficient executableness of output actions.
By constructing the common coverage relationship between targets based on the circular radius of action, target priority prior attention is generated, reinforcement learning observation vectors are constructed, and velocity and heading control sequences for multiple future control steps are output. Control commands are then decoded in combination with strict kinematic constraints.
It significantly improves coverage efficiency in dense scenarios, stabilizes attention allocation, increases sample utilization and policy convergence speed, solves control jitter and non-executability issues, and ensures the physical feasibility of control commands.
Smart Images

Figure CN123113376A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of group target pursuit reinforcement learning control technology, specifically to a group target pursuit reinforcement learning control method, system, and electronic device. Background Technology
[0002] In multi-target pursuit missions, the pursuing agent often does not only need to approach a single target, but also needs to cover, intercept, or deal with multiple moving targets within a limited time. When the pursuing agent has a circular scope with radius R, a target can be considered covered once it enters the circular scope. Therefore, the key to this type of task is not only to shorten the distance to the nearest target, but also to how to select the next pursuit target, how to use the circular scope to cover multiple adjacent targets at once, and how to quickly clear boundary targets and isolated targets after the target group is scattered.
[0003] Existing reinforcement learning pursuit methods typically involve directly concatenating all target states into the policy network, or designing reward functions based solely on nearest distance, single-target closure, and the number of hits. These methods tend to learn local policies that chase the nearest target, making it difficult to explicitly express the shared coverage relationship between targets induced by the circular radius of action R. When the target group exhibits clustering, chaining, splitting, boundary residuals, or sparse residuals, the policy may suffer from problems such as unreasonable sweep paths, insufficient shared coverage of dense areas, no effective hits after crossing the center of the target group, and long-term residuals of isolated targets.
[0004] There are some existing reinforcement learning pursuit schemes. For example, the patent CN115952729A published by Northwestern Polytechnical University realizes the generation of strategies for multi-agent pursuit and escape games through self-games. It uses fuzzy learning and Q-learning to explore the environment and autonomously generate game strategies. However, the game strategies are mainly designed for adversarial scenarios between the pursuer and the escaper. Its target selection logic lacks explicit modeling of the physical domain scale of the pursuing agent, making it difficult to express the common coverage relationship between targets induced by the circular radius of action. In scenarios with dense target groups, it is easy to have unreasonable sweep paths.
[0005] Xi'an University of Technology's patent CN116225065A studies the multi-agent reinforcement learning algorithm for multi-UAV pursuit problems and establishes a multi-UAV encirclement strategy inference method based on reinforcement learning. However, this method mainly focuses on single-target or small-target pursuit under a multi-UAV cooperative framework. For group target scenarios with a large number of targets distributed in clusters or chains, its policy network is difficult to explicitly utilize the spatial topological relationships and coverage overlap opportunities between targets, and is prone to duplicate coverage or coverage omissions in dense areas. Similarly, Tsinghua University's patent CN117408159A evaluates the performance of each pursuit task through a task evaluator and dynamic pursuit strategy, and selects target pursuit tasks based on the performance evaluation results to train a multi-agent reinforcement learning algorithm. However, its training framework focuses on optimizing task selection and sampling strategies, and does not introduce target priority prior information based on physical domain into the observation space or attention mechanism of the policy network, making it difficult to stably allocate attention in multi-target coverage scenarios.
[0006] On the other hand, single-step speed or heading maneuvers are insufficient to express short-term intentions during continuous pursuit, easily leading to control command jitter or near-misses near the target. If only a large number of empirical terms are added to the reward function without providing the policy network with interpretable target selection priors, the training process will suffer from low sample efficiency, unstable attention allocation, and policy degradation in the residual phase. Therefore, a reinforcement learning control method is needed that unifies the circular radius of action R, the spatial morphology of the target group, the cost of the coverage route, target priority priors, and multi-step executable control. This method should enable the policy to utilize the circular scope for dense target coverage and to stably transition to high-value target closure and sweeping after target sparseness.
[0007] Therefore, existing technologies still need further development. Summary of the Invention
[0008] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a group target pursuit reinforcement learning control method to solve the technical problems of existing multi-target pursuit reinforcement learning methods, such as the lack of circular coverage scale constraints in target selection, unstable attention allocation, insufficient utilization of opportunities for multi-target common coverage, and insufficient executableness of output actions.
[0009] To achieve the above-mentioned technical objectives, according to a first aspect of the present invention, the present invention provides a group target pursuit reinforcement learning control method, comprising: S100: Obtain the state of the pursuing agent, the states of multiple targets, and the circular radius of action of the pursuing agent; S200. Based on the circular radius of action, determine the common coverage relationship between each target, and generate target priority prior attention based on the common coverage relationship; S300. Construct a reinforcement learning observation vector based on the state of the pursuing agent, the state of the target, and the prior attention of the target priority; S400. Input the observation vector into the masked attention policy network and output the velocity and heading control sequence for multiple future control steps; S500: Decode the speed and heading control sequence to obtain executable control instructions that satisfy the motion constraints of the pursuing agent, and execute the executable control instructions.
[0010] Specifically, determining the common coverage relationship between targets based on the circular radius of action, and generating target priority prior attention based on the common coverage relationship, includes: A coverage disk for each target is formed with the target location as the center and the radius of the circle as the radius. It is determined whether the coverage disks of any two targets have an intersection. If there is an intersection, it is determined that the two have a common coverage relationship. Based on the common coverage relationship, the motion state of the target relative to the pursuing agent, and the spatial distribution between targets, the target priority score of each target is calculated, and the target priority score is normalized to obtain the target priority prior attention.
[0011] Specifically, determining whether the coverage disks of any two targets intersect, and if they do intersect, determining that they have a common coverage relationship, includes: The common coverage relationship between two targets is constructed based on whether their coverage disks intersect, as specifically represented below: ; in, Indicates the first objective Second objective The shared coverage relationship between them , They represent the first objective. Second objective The position vector, Indicates the radius of the circle. Represents the Euclidean distance between two target locations, where the first target... With the second objective If the distance between them is less than or equal to 2R, then they are considered to have an intersection, that is, they have a common coverage relationship.
[0012] Specifically, the method for calculating the target priority score of each target based on the shared coverage relationship, the motion state of the target relative to the tracking agent, and the spatial distribution between targets includes: Based on the shared coverage relationship, calculate the coverage route item and the shared coverage order item; Based on the target's relative motion state to the pursuing agent, calculate the proximity term, heading alignment term, and closing velocity term; Based on the spatial distribution of targets, calculate the coverage route cost, endpoint threat, edge isolation, and target locking terms; Then calculate the target priority score as follows: ; in, For the primary goal Target priority score To cover route items, For jointly covering sequential items, For near-range items, To cover the route cost item, As the endpoint threat item, For heading alignment, For closed velocity terms, For marginal isolated terms, For target locking items, , , , , , , , and These are the corresponding weights.
[0013] Specifically, the calculation method for the coverage route item is as follows: Construct a common coverage mask for candidate targets based on the common coverage relationship. According to the common coverage mask Calculate with the first objective The coverage route cost of the first target is used to obtain the coverage route item, which is then used to create a common coverage mask. The specific calculation method is as follows: ; Among them, common coverage mask To select a target When serving as the entry point for a coverage path, it represents a set of targets with shared coverage opportunities.
[0014] Specifically, the method for constructing reinforcement learning observation vectors includes: Based on the state of the pursuing agent, construct the state features of the pursuing agent itself; Based on the target state, construct the target node features and target survival mask for each target; The target node features include at least: the target's position relative to the pursuing agent, the target's speed relative to the pursuing agent, the distance between the target and the pursuing agent, the azimuth angle of the target's heading relative to the pursuing agent, the closing speed of the pursuing agent towards the target, the distance from the target to the mission endpoint, and the target's survival marker. The target liveness mask is used to identify whether the corresponding target is alive or has been covered; The state features of the pursuing agent, the target node features of each target, the target survival mask of each target, and the prior attention of the target priority are combined to form the reinforcement learning observation vector.
[0015] Specifically, the step of inputting the observation vector into the masked attention policy network and outputting the velocity and heading control sequences for multiple future control steps includes: The state features of the tracking agent in the observation vector are input into a state encoding network to obtain its own embedding. The features of each target node in the observation vector are input into the target encoding network to obtain the embedding of each target; The target priority prior attention in the observation vector is input into the prior embedding network to obtain the prior embedding; The self-embedding, each target embedding, and the prior embedding are input into the attention prediction network, and the learned attention of each surviving target is calculated in combination with the target survival mask. Invalid targets are then masked according to the target survival mask. Based on the learned attention and the target survival mask, the target group context features are aggregated. After fusing the self-embedding with the target group context features, the input is given to the action output network to obtain the velocity and heading control sequences for the future multiple control steps.
[0016] Specifically, the method for decoding the speed and heading control sequences includes: Set upper and lower speed limits, acceleration constraints, and maximum turning angular velocity constraints. Decode the speed and heading control sequences based on these constraints. The specific calculation method is as follows: future The velocity and heading control sequence for each control step is represented as follows: ; in, For the first A speed command that controls the step size. For the first A heading command or heading increment of one control step. It is a positive integer; The speed and heading control sequences are decoded step by step according to the following constraints: ; ; ; in, and These are the minimum speed and the maximum speed, respectively. For maximum acceleration, To control the step size, For speed The maximum turning angular velocity.
[0017] According to a second aspect of the present invention, a group target tracking reinforcement learning control system is provided, comprising: The state acquisition module is used to acquire the state of the pursuing agent, the states of multiple targets, and the circular radius of action of the pursuing agent. The prior generation module is used to determine the common coverage relationship between each target based on the circular radius of action, and to generate target priority prior attention based on the common coverage relationship; An observation construction module is used to construct reinforcement learning observation vectors based on the state of the pursuing agent, the state of the target, and the prior attention of the target priority. The strategy reasoning module is used to input the observation vector into the masked attention policy network and output the velocity and heading control sequences for multiple future control steps; The decoding and execution module is used to decode the speed and heading control sequence to obtain executable control instructions that satisfy the motion constraints of the pursuing agent, and to execute the executable control instructions.
[0018] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory; and a processor, wherein the memory stores computer-readable instructions, which, when executed by the processor, implement the above-described group target pursuit reinforcement learning control method.
[0019] Beneficial effects: This invention provides a reinforcement learning control method and system for group target tracking. By constructing common coverage relationships between targets based on the radius of a circle and transforming this into target priority prior attention, it solves the problem of lack of geometric constraints in target selection in existing technologies. This prior attention explicitly encodes the spatial topology of the target group, guiding the policy network to prioritize target combinations with common coverage potential, rather than simply greedily chasing the nearest target, significantly improving coverage efficiency in dense scenes. By constructing a reinforcement learning observation vector containing target priority prior attention and target survival masks, and cooperating with a mask attention policy network, it achieves control over a variable number of targets. Adaptive processing and the introduction of prior attention provide a stable anchor point for the attention allocation of the policy network, effectively alleviating the problems of attention distraction in the early stage of training and policy degradation in the later stage, and improving sample utilization and policy convergence speed. By outputting the velocity and heading control sequences of multiple future control steps and applying strict velocity, acceleration and turning angular velocity constraints in the decoding stage, the control jitter and non-executability problems caused by single-step action output are solved. This multi-step prediction and constrained decoding mechanism ensures the physical feasibility of control commands, enabling the agent to adjust its posture in advance during the pursuit process, reducing near-range errors and invalid following, and improving the overall task completion quality. Attached Figure Description
[0020] Figure 1 This is a flowchart of the group target pursuit reinforcement learning control method provided in a specific embodiment of the present invention; Figure 2 This is a schematic diagram of the composition of the group target pursuit reinforcement learning control system provided in a specific embodiment of the present invention. Detailed Implementation
[0021] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments in this application, other similar embodiments obtained by those skilled in the art without creative effort should all fall within the scope of protection of this application. Furthermore, directional terms mentioned in the following embodiments, such as "up," "down," "left," and "right," are only for reference to the directions in the accompanying drawings; therefore, the directional terms used are for illustrative purposes and not for limiting the invention.
[0022] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments.
[0023] Example 1 Please see Figure 1This embodiment provides a reinforcement learning control method for group target pursuit. This method achieves efficient and stable pursuit of multiple targets by fusing geometric priors and reinforcement learning strategies. Specifically, the group target pursuit reinforcement learning control method includes the following steps: Step S100: Obtain the state of the pursuing agent, the states of multiple targets, and the circular radius of action of the pursuing agent.
[0024] Understandably, the state of the pursuing agent typically includes its current position coordinates, velocity vector, heading angle, and remaining task time; the states of multiple targets encompass dynamic data such as the relative positions, relative velocities, and type identifiers of all surviving targets within the field of view; the circular radius of action R defines the geometric boundary of the range within which the pursuing agent can effectively cover or intercept targets. In this embodiment, the circular radius of action is not only used to determine whether a single target is covered, but more importantly, it serves as a benchmark for calculating the topological relationships between targets in subsequent steps, thereby directly introducing the operational capability constraints of the physical world into the decision logic of the algorithm.
[0025] Step S200: Based on the circular radius of action, determine the common coverage relationship between each target, and generate target priority prior attention based on the common coverage relationship.
[0026] Understandably, the radius R of a circular action can be used to assess the geometric possibility that any two targets may be simultaneously covered by the same domain; this possibility is known as a co-coverage relationship. For example, when the distance between two targets is less than or equal to twice the radius of the circular action, theoretically there exists a center position where the action circle can simultaneously contain both targets. The target priority prior attention generated based on this geometric relationship is a guiding signal. Unlike attention weights obtained through autonomous learning by neural networks, it explicitly encodes the spatial clustering degree and coverage efficiency of the target group. In this way, complex geometric planning problems can be transformed into prior knowledge that is easy for policy networks to understand, effectively alleviating the problem that reinforcement learning struggles to autonomously discover multi-target cooperative coverage strategies under sparse rewards.
[0027] Step S300: Construct reinforcement learning observation vectors based on the state of the pursuing agent, the state of the target, and the prior attention of the target priority.
[0028] Specifically, in this embodiment, the observation vector not only includes basic state features describing the current situation but also incorporates target priority prior attention. This allows the policy network to receive explicit cues about which target combinations are more valuable when receiving input, thereby reducing the difficulty for the network to extract complex geometric features from the original high-dimensional state. Furthermore, to adapt to scenarios where the number of targets changes dynamically, a masking mechanism is used in the construction process of the observation vector to ensure that invalid or non-existent targets do not interfere with policy inference.
[0029] Step S400: Input the observation vector into the masked attention policy network and output the velocity and heading control sequences for multiple future control steps.
[0030] Specifically, the masked attention policy network is a deep learning model specifically designed for variable-length sequence inputs. Internally, it dynamically aggregates target information through an attention mechanism. Unlike existing technologies that only output the single-step action at the current moment, the policy network in this embodiment outputs the velocity and heading control sequence for the next K control steps. On the one hand, this allows the agent to express its pursuit intention in a continuous time dimension, rather than just an instantaneous reaction. On the other hand, it provides sufficient optimization space for subsequent kinematic constraint decoding, making the generated trajectory smoother and more in line with dynamic characteristics, avoiding control jitter and near-target misses that are easily caused by single-step decision-making.
[0031] In step S500, the speed and heading control sequences are decoded to obtain executable control instructions that satisfy the motion constraints of the pursuing agent, and the executable control instructions are executed.
[0032] Specifically, the original control sequences output by neural networks are usually normalized or dimensionless abstract values that cannot directly drive physical entities. The decoding process is the process of mapping these abstract values back to the physical feasible domain. This process strictly enforces kinematic constraints such as upper and lower limits of speed, maximum acceleration, and maximum turning angular velocity. For example, even if the network outputs a command for a sharp turn, the decoder will correct it to a safe and feasible turning rate based on the current actual speed and the platform's maneuverability. The decoded executable control commands are sent to the underlying controller of the pursuing intelligent agent to drive it to complete the actual maneuver. This ensures that the decision results of the artificial intelligence algorithm can be safely and reliably implemented in the physical system, forming a complete closed loop from perception, decision-making to execution.
[0033] As one implementation method, based on the circular radius of action, the common coverage relationship between each target is determined, and the target priority prior attention is generated based on the common coverage relationship. This includes: forming a coverage disk for each target with the target position as the center and the circular radius of action as the radius; determining whether the coverage disks of any two targets have an intersection; if there is an intersection, it is determined that the two have a common coverage relationship; and calculating the target priority score for each target based on the common coverage relationship, the motion state of the target relative to the tracking agent, and the spatial distribution between targets, and normalizing the target priority score to obtain the target priority prior attention.
[0034] It should be noted that this mechanism is not simply based on distance ranking, but rather uses the physical operational capability of the pursuing agent, i.e. the circular radius of action R, as the core metric to construct a comprehensive evaluation system that integrates geometric topology, motion status, and task value. The prior attention generated in this way not only reflects the relative probability of the target being selected, but also implies how to efficiently utilize the physical strategy of covering more targets within the scope, thus providing a clear and interpretable supervisory signal for subsequent reinforcement learning networks.
[0035] Specifically, the aforementioned circular scope Represented as: ; in, For a moment The circular scope, In order to track the location of the intelligent agent, The radius of the circular action is when the first target Location satisfy belong At that time, determine the first target At any moment It was covered.
[0036] At the geometric level, it determines whether the coverage disks of any two targets intersect. If they intersect, it determines that they have a common coverage relationship. This includes constructing the common coverage relationship between the two targets based on whether their coverage disks intersect, as specifically represented below: ; in, Indicates the first objective Second objective The shared coverage relationship between them , They represent the first objective. Second objective The position vector, Indicates the radius of the circle. Represents the Euclidean distance between two target locations, where the first target... With the second objective If the distance between two targets is less than or equal to 2R, then they are considered to have an intersection, i.e., a common coverage relationship. The physical meaning of this criterion is that only when the distance between two targets does not exceed twice the radius of action can there be a legal position for the pursuing agent, so that its circular scope can simultaneously contain both targets. The above steps transform the continuous spatial positional relationship into a discrete graph structure connection relationship, enabling the algorithm to geometrically identify which targets can be packaged and processed, avoiding path repetition and reduced coverage efficiency caused by only using the nearest target as the basis for pursuit.
[0037] At the topology construction and path evaluation level, the calculation method for coverage route items is as follows: A common coverage mask for candidate targets is constructed based on common coverage relationships. According to the common coverage mask Calculate with the first objective The cost of the coverage route for the first objective is used to obtain the coverage route item, which is then used to create a common coverage mask. The specific calculation method is as follows: ; Among them, common coverage mask To select a target When serving as the entry point for a coverage path, it represents a set of targets with shared coverage opportunities.
[0038] It should be noted that the above calculation method essentially defines a locally connected subgraph starting from the target. Based on this, through dynamic programming or an equivalent path search algorithm, the minimum path cost required to traverse all uncovered targets in the subgraph from the target is estimated. The covered route item is negatively correlated with this cost, that is, the shorter the path and the more targets that can be connected, the higher the score of this item. This design enables prior attention to identify those targets that are in the hub position of the target group, guiding the agent to prioritize going to these key nodes that can trigger a chain of covered effects, rather than just chasing the nearest isolated target at the current moment.
[0039] Furthermore, in this embodiment, at the multi-dimensional value assessment level, the method for calculating the target priority score of each target based on the common coverage relationship, the target's motion state relative to the pursuing agent, and the spatial distribution between targets includes: calculating the coverage route item and the common coverage order item according to the common coverage relationship; calculating the proximity item, the heading alignment item, and the closing velocity item according to the target's motion state relative to the pursuing agent; calculating the coverage route cost item, the endpoint threat item, the edge isolation item, and the target locking item according to the spatial distribution between targets; and then calculating the target priority score in the following manner: ; in, For the primary goal Target priority score To cover route items, For jointly covering sequential items, For near-range items, To cover the route cost item, As the endpoint threat item, For heading alignment, For closed velocity terms, For marginal isolated terms, For target locking items, , , , , , , , and The formula constructs a nine-dimensional evaluation space, with corresponding weights, covering multiple aspects such as efficiency, safety, task urgency, and situational stability. For example, the edge isolation term is usually determined by the radial distance of the target relative to the centroid of the target group and the number of local neighbors. Its role is to increase the priority of targets located at the boundaries of the cluster or the endpoints of the chain structure, preventing these targets from remaining for a long time after the main body is covered. The heading alignment term measures the degree of alignment between the current heading of the pursuing agent and the target's orientation. The higher the alignment, the lower the cost of maneuvering adjustments. The endpoint threat term reflects the urgency of the target approaching the task endpoint, preventing the agent from allowing high-threat targets to escape due to excessive pursuit of coverage efficiency. Each weight coefficient can be configured according to the specific task scenario. For example, it can be increased in the sweeping phase to increase attention to residual targets, while it can be increased in the interception phase to strengthen the response to endpoint threats. This explicit multi-factor weighting structure makes the prior generation process highly adjustable and physically interpretable, which is different from the pure data-driven black box model.
[0040] Finally, the calculated target priority scores are... Temperature normalization is performed to obtain the target priority prior attention. The normalization process can be performed using the Softmax function, i.e. ,in, For temperature coefficient, For the set of surviving targets, this operation transforms the absolute scores into a probability distribution, allowing them to be directly input into the neural network as attention weights. It's important to emphasize that this prior attention is fully differentiable. This means that during training, although the prior itself is based on rule-based computation, its guiding effect on the policy network can indirectly optimize other network parameters through gradient backpropagation. Furthermore, since each score component corresponds to a clear physical meaning, when the policy network exhibits abnormal behavior, engineers can analyze the distribution of the prior attention to trace the cause, such as determining whether edge targets are being ignored due to excessively low weights for isolated edge terms. This achieves white-box debugging and verification of the reinforcement learning decision-making process.
[0041] In a preferred embodiment, based on a common coverage mask, the route cost required to cover the target set when each candidate target is the first target is calculated through dynamic programming or equivalent route cost search. Select a target from the already covered target set. The newly added common coverage set for: ; when The more targets there are, and the more targets... The lower the cost of subsequent coverage routes for the first objective, the better the objective. The higher the coverage route item and coverage order item, the more the mechanism prioritizes the path entry that can bring the benefit of multiple targets being covered together, rather than simply choosing the nearest target; Simultaneously, calculate the target edge isolation term, the target edge isolation term It is determined by the radial distance of the target relative to the centroid of the target group and the number of local neighbors: ; in, For the survival target group's centroid, For the goal The number of neighbors within the preset neighborhood. To achieve the target number of survivors, This is a weighting coefficient used to increase the priority of boundary targets, isolated targets, and chain endpoint targets; the system further calculates the heading alignment item, closing velocity item, endpoint threat item, and target lock item, and obtains the target priority scoring formula according to the invention. Then, the target priority prior attention is obtained through softmax normalization. .
[0042] As one implementation method, the approach to constructing reinforcement learning observation vectors includes: constructing the state features of the pursuing agent based on its state; constructing target node features and target survival masks for each target based on its state; and combining the state features of the pursuing agent, the target node features of each target, the target survival masks of each target, and the prior attention based on target priority to form reinforcement learning observation vectors. This structured design enables the network not only to perceive the motion state of individual targets but also to understand the overall topology and dynamic changes of a group of targets through prior attention and masking mechanisms, thereby effectively supporting adaptive processing of a variable number of targets.
[0043] In constructing the state characteristics of the pursuing agent, the system typically acquires the raw information of the pursuing agent at the current moment, such as its position coordinates, velocity vector, heading angle, and remaining task time, and transforms it into a local coordinate system with the pursuing agent itself as the origin. For example, the global position is converted into relative displacement, and the absolute heading is converted into the deflection angle relative to the current direction of motion. This localization process eliminates the influence of the selection of the global coordinate origin on the policy generalization ability, so that the control law learned by the agent has translation invariance and can focus on its own motion trend and the relative relationship with the surrounding environment.
[0044] Regarding the construction of target node features, for each observable target, the system extracts multiple key features to comprehensively describe its interaction with the pursuing agent. Specifically, target node features include at least: the target's position relative to the pursuing agent, the target's velocity relative to the pursuing agent, the distance between the target and the pursuing agent, the azimuth angle of the target relative to the pursuing agent's heading, the closing velocity of the pursuing agent towards the target, the distance from the target to the mission endpoint, and the target's survival marker. Among these, the target's position and velocity relative to the pursuing agent provide the most basic relative kinematic information, which is the basis for judging collision risk and interception trajectory. The distance between the target and the pursuing agent intuitively reflects the current proximity and is often used to trigger close-range maneuvers or determine coverage results. The target's position and velocity relative to the pursuing agent's heading provide the most basic relative kinematic information, which is the basis for judging collision risk and interception trajectory. The distance between the target and the pursuing agent directly reflects the current proximity and is often used to trigger close-range maneuvers or determine coverage results. The azimuth angle describes whether the target is to the left, right, or directly in front of the agent's direction of travel, which is crucial for deciding the turning direction. The closing speed of the pursuing agent towards the target characterizes how quickly the distance between the two parties decreases. A high closing speed means a shorter interception window, requiring more precise lead control. The distance from the target to the task endpoint introduces a task-level threat assessment to prevent the agent from allowing high-threat targets at a distance to escape due to excessive focus on nearby targets. The target survival label, as a binary or probabilistic feature, clearly indicates whether the target is still in a pending state, providing a direct basis for subsequent masking mechanisms. These features together constitute a multi-dimensional target descriptor, enabling the policy network to comprehensively evaluate the value and urgency of each target from multiple perspectives such as geometry, motion, and task.
[0045] In a preferred embodiment, the pursuing agent moves within a two-dimensional plane or a local tangent plane. Multiple targets move along their respective velocities and headings, and have a mission tendency to reach the target endpoint. In each control cycle, the system acquires the pursuing agent's position, velocity, heading angle, relative distance to the mission boundary, remaining control time, relative position to the target endpoint, and circular radius of action. .
[0046] For each surviving target The position and velocity are converted into relative states in the local coordinates of the pursuing agent to obtain the target node features: ; in, For the goal Relative to the position of the pursuing agent, For the goal Relative to the speed of the pursuing agent, The distance between the target and the pursuing agent. The azimuth angle relative to the target's trajectory of the intelligent agent. To track the closing velocity of the intelligent agent toward the target, The distance from the target to the destination.
[0047] In terms of constructing the target survival mask, the target survival mask is used to identify whether the corresponding target is alive or has been covered. Specifically, since the number of targets in the actual scene is dynamically changing, while the input dimension of the neural network is usually fixed, it is necessary to set an upper limit M of the maximum observable target number. When the actual number of surviving targets is less than M, the extra target positions will be filled with zero vectors or specific placeholders. At this time, the corresponding position in the target survival mask is set to 0 (indicating invalidity), while the corresponding position of the real surviving target is set to 1 (indicating validity). This allows the policy network to automatically ignore targets that do not exist or have been successfully covered during training and inference, avoiding interference from invalid data on attention allocation. The existence of the mask allows the same network model to seamlessly adapt to any scene from a single target to a full-load target group without retraining or adjusting the network structure for different target numbers.
[0048] Finally, the generated state features of the pursuing agent, the target node features of each target, the target survival mask of each target, and the target priority prior attention are combined to form the final reinforcement learning observation vector. The target priority prior attention is explicitly input as part of the observation vector, which is fundamentally different from traditional methods that only use it as an external reward signal or post-processing logic. By directly embedding the geometric prior with clear physical meaning into the observation space, the policy network can see which target combinations have common coverage potential and which targets are in an isolated state at the first layer of encoding. This explicit guidance signal complements the implicit state information in the target node features, effectively avoiding the problem that pure data-driven methods cannot autonomously discover efficient coverage strategies under sparse rewards. At the same time, the target survival mask ensures that only the component corresponding to the surviving target participates in the calculation in the prior attention, ensuring the logical consistency and completeness of the entire observation vector.
[0049] As one implementation method, the process of inputting the observation vector into a masked attention policy network and outputting velocity and heading control sequences for multiple future control steps specifically includes the following data processing flow: First, the state features of the pursuing agent in the observation vector are input into a state encoding network to obtain a self-embedding. The state encoding network is usually implemented using a multilayer perceptron or a one-dimensional convolutional network. Its role is to map the high-dimensional and physically different self-states, such as position, velocity, and heading, into a compact implicit feature space. This self-embedding represents the current movement capability and task urgency of the pursuing agent, providing a primary reference benchmark for subsequent decision-making. Simultaneously, the features of each target node in the observation vector are input into a target encoding network to obtain target embeddings. The target encoding network extracts features independently for each target, generating an embedding sequence corresponding to the number of targets. This independent encoding method preserves the individual differences of each target, enabling the network to distinguish the threat level and movement trend of different targets. In addition, the target priority prior attention in the observation vector is input into a prior embedding network to obtain a prior embedding. The role of the prior embedding network is to transform the scalar form of the prior attention value into a feature vector that matches the dimensions of the self-embedding and target embedding, transforming explicit knowledge calculated based on geometric rules into an implicit representation that can be processed by the neural network, providing physically meaningful guiding signals for subsequent attention mechanisms.
[0050] Secondly, the self-embedding, target embeddings, and prior embeddings are input into the attention prediction network. The learned attention of each surviving target is calculated by combining the target survival mask, and invalid targets are masked according to the target survival mask. The attention prediction network is the core component for realizing prior-guided learning. In this module, the three types of embeddings are fused through concatenation, addition, or gating modulation. For example, the prior embedding can be added to the target embedding as a bias term, or the amplitude of the target embedding can be modulated by multiplication, so that the prior information directly affects the calculation of the attention score. When calculating the attention score, the system strictly applies the target survival mask. For invalid target slots with a mask value of 0, their attention scores are set to a minimum value (such as negative infinity) to ensure that the weights of these positions are exactly zero after Softmax normalization. This mechanism ensures that the network only focuses on real targets when dealing with a variable number of targets, avoiding interference from padding data on the policy. It is important to emphasize that this masking mechanism remains completely consistent between the training and inference phases. This not only prevents overfitting during training but also ensures the generalization stability of the model under different target quantities. The final output of the learned attention includes features that the network autonomously mines from the data and is subject to reasonable constraints from geometric priors, thus avoiding the oscillation and drift problems that are prone to occur in pure learning attention under sparse rewards.
[0051] Furthermore, based on the learned attention and target survival mask, the target group context features are aggregated. The aggregation process usually adopts a weighted summation method, that is, the learned attention is used as the weight to linearly combine the embeddings of each target. The generated target group context features are a fixed-dimensional global representation, including the situational information of the entire target group, including the distribution center of the targets, the main direction of movement, and the degree of aggregation of high-value targets. Since invalid targets have been removed during the aggregation process, the context features can accurately reflect the real battlefield situation at the current moment and provide a reliable basis for decision-making.
[0052] Finally, the embedded information is fused with the contextual features of the target group and input into the action output network to obtain a speed and heading control sequence with multiple control steps in the future. The action output network deeply fuses the subject's state with global environmental information and maps it to the action space. Unlike traditional networks that only output single-step actions, the action output network in this embodiment is designed to directly output K sets of speed and heading parameters. This multi-step output structure can be implemented in various ways, such as using fully connected layers to directly regress the K-step sequence or using recurrent neural networks to gradually generate the sequence. Regardless of the specific network form used, its output represents the agent's continuous pursuit intention over a period of time in the future. By unifying prior guidance, variable length processing, and multi-step prediction in an end-to-end network architecture, efficient collaboration from perception to decision-making is achieved, ensuring the rationality of target selection and improving the temporal coherence and executability of control commands.
[0053] Preferably, the masked attention strategy network includes a tracking agent state encoding module, a target node encoding module, a prior attention embedding module, a target attention prediction module, a cross-target attention aggregation module, and an action output module; The agent state encoding module will track the agent's own state. Mapping to its own embedding; the target node encoding module maps the features of each target node Mapped to target embedding; the prior attention embedding module will The prior embedding is mapped and input into the target attention prediction module along with its own embedding and the target embedding. The target attention prediction module outputs the learned attention. : ; in, For target attention scoring network, The learned attention output by the policy network; The cross-target attention aggregation module uses learned attention and target survival masks to aggregate target group context, forming policy features. The action output module outputs based on these policy features. Step speed and heading control sequence; During training, the policy network is updated using the proximal policy optimization algorithm, while prioritizing the target with prior attention. With learning attention Between Divergence is added to the training loss, and this auxiliary supervision enables the policy network to maintain a reasonable target selection direction even when there are insufficient samples, changes in the number of targets, or sudden changes in the morphology of the target group.
[0054] As one implementation method, the method for decoding the speed and heading control sequences includes: setting upper and lower speed limits, acceleration constraints, and maximum turning angular velocity constraints; and decoding the speed and heading control sequences based on these constraints. The original action sequences output by the masked attention strategy network are typically normalized or dimensionless values, which cannot directly drive the physical actuators of the pursuing agent. The decoding process is the key step in mapping these abstract values back to the physically feasible domain. By applying strict kinematic constraints, it ensures that the generated control commands conform to the dynamic performance boundaries of the pursuing agent while guaranteeing the continuity and smoothness of the trajectory, thereby solving the jitter, overshoot, or inexecutability problems commonly encountered when reinforcement learning algorithms directly output actions.
[0055] In terms of sequence representation, the velocity and heading control sequence for the next K control steps is represented as follows: ; in, For the first A speed command that controls the step size. For the first A heading command or heading increment of one control step. As a positive integer, this sequence structure indicates that the policy network does not only decide the instantaneous state at the current moment, but plans the continuous motion intentions over a period of time in the future. During each decoding step, the speed and heading are updated using the previous decoding state, thereby forming a short-term control sequence that satisfies the motion constraints of the pursuing agent. This multi-step prediction mechanism enables the pursuing agent to adjust its attitude in advance to cope with the target's maneuvering changes, which can reduce single-step action jitter and near-target misses. In actual decoding, the system first maps the normalized components of the network output to a preset physical dimension range as the initial input for subsequent constraint processing.
[0056] In terms of the specific calculations for constraint decoding, the speed and heading control sequences are decoded step by step according to the following constraints: ; ; ; in, and These are the minimum speed and the maximum speed, respectively. For maximum acceleration, To control the step size, For speed The maximum turning angular velocity.
[0057] It should be noted that the above three constraints ensure the physical feasibility of the control commands from different dimensions, including the upper and lower speed limits. The basic motion envelope of the pursuing agent is defined to prevent the generation of velocity commands that exceed the limits of the propulsion system. For example, for a fixed-wing UAV, this typically corresponds to the stall velocity; below this value, lift cannot be maintained. Acceleration constraints are also defined. Limiting the rate of speed change between adjacent control steps directly corresponds to the physical response capability of the engine thrust or braking system, preventing abrupt changes in speed commands, thereby protecting the power unit and reducing mechanical wear; maximum turning angular velocity constraint. This illustrates the coupling relationship between speed and time-varying turning capability. In the physical world, the turning capability of most mobile platforms is not constant but varies with speed. For example, high-speed flying drones are limited by aerodynamic loads and structural strength, and their maximum allowable turning angular velocity is usually much lower than that in low-speed hovering or cruising states. Therefore, in this embodiment... Designed for speed The function is a function of the aircraft, rather than a fixed constant. This ensures that no violent turning commands that would cause loss of control or structural damage are generated during high-speed pursuits, while the platform’s agility can be fully utilized for rapid adjustments during low-speed maneuvers. In contrast, if a fixed upper limit for turning rate is used, there will be safety hazards at high speeds and a waste of maneuverability at low speeds.
[0058] In terms of execution logic, the above decoding process adopts a step-by-step decoding method, which means that the first... The decoding result of step 1 strictly depends on the first step. Instead of relying solely on the network's original predictions, the system calculates the actual output value of each step. Specifically, starting from the beginning, the system sequentially prunes the constraints on velocity and heading for each step, using the pruned legal values as the baseline state for the next constraint calculation. This mechanism forms a cumulative error correction loop. Even if the network outputs an aggressive instruction that violates the constraints at a certain step, the decoder will pull it back within the safe boundary, and subsequent decoding steps will continue based on this corrected state. This ensures the temporal continuity and dynamic consistency of the entire trajectory. In this way, high-frequency noise and non-physical jumps that may exist in the neural network output are avoided, so that the final executable control instructions retain the intent of the policy network and fully meet the safe operation requirements of the pursuing agent.
[0059] Example 2 This embodiment provides an optimal implementation of a group target pursuit reinforcement learning control method. This embodiment introduces a two-stage training mechanism based on attention supervision and a stage adaptive reward system, thereby significantly improving the convergence speed and final performance of the strategy.
[0060] In the model training phase, to overcome the difficulties of cold start and low sample efficiency in complex multi-objective scenarios of reinforcement learning, this embodiment adopts a two-stage training strategy that combines behavior cloning pre-training with reinforcement learning fine-tuning: In the first stage of behavior cloning pre-training, the system utilizes an expert planner built based on rules or traditional planning algorithms (such as A*, RRT, or artificial potential field method) to generate high-quality speed and heading control sequences as supervision signals under various preset scenarios. In this stage, the policy network imitates the expert behavior by minimizing the mean square error between its own output and the expert's actions, enabling the network to quickly learn basic obstacle avoidance, target approach, and kinematic constraint compliance capabilities. It is worth noting that even in the behavior cloning stage, the system still introduces target priority prior attention to assist in the supervision of learning attention, ensuring that while the network imitates the actions, it can also initially establish a target selection logic that conforms to the geometric prior, avoiding simply imitating the trajectory while ignoring the alignment of decision intentions.
[0061] In the second stage of reinforcement learning fine-tuning, the system uses a proximal policy optimization algorithm to update the masked attention policy network end-to-end. Preferably, the total loss function of the training process consists of four weighted terms, as shown below: ; in, The loss is optimized for near-end strategies to maximize expected cumulative returns while ensuring the stability of strategy updates. The value function loss is used to train the state value network to accurately assess the long-term rewards of the current situation. This is a policy entropy regularization term, which encourages the policy to maintain a certain degree of exploratory nature and prevents it from getting trapped in local optima too early. For attention monitoring loss, , and These are the corresponding weight coefficients. In particular, the attention supervision loss is a key design feature that distinguishes this embodiment from conventional reinforcement learning methods, and its calculation formula is as follows: ; in, Prioritize the target with prior attention. The learned attention output by the policy network. For the set of survival targets, To prevent the small constant of division by zero, this loss function essentially calculates the prior distribution. With learning distribution Minimizing the KL divergence between the two factors, by minimizing this loss term, is equivalent to distilling explicit prior knowledge based on the circular radius of action and geometric topology into the neural network. This mechanism ensures that the policy network is always guided by physical rules during the exploration process, even under conditions of sparse rewards or drastic changes in the shape of the target group, thus improving the learning of attention. It will not deviate from a reasonable selection range, thus solving the problems of unstable attention allocation and susceptibility to noise interference in pure data-driven methods.
[0062] Regarding the reward function design, this embodiment proposes a phase-adaptive reward mechanism to address the varying tactical requirements at different stages of the pursuit mission. The system monitors the number of surviving targets and the potential coverage opportunities within a preset multiple R around the pursuing agent in real time, thereby determining the current mission phase. When there are many surviving targets and multiple targets that can be simultaneously covered by a circular scope, the system identifies it as a target-dense phase. In this phase, the reward function focuses on improving coverage efficiency, specifically including additional coverage rewards. Continuous coverage reward and joint coverage opportunity rewards Among them, the shared coverage opportunity reward Based on the priority of targets in the prior knowledge and their corresponding common coverage masks, the agent is explicitly incentivized to move to pivotal locations that can trigger a chain reaction of coverage effects. Conversely, when the number of surviving targets falls below a preset threshold or local coverage opportunities are scarce, the system automatically switches to a target sparsity phase. In this phase, the reward function reduces the weight of common coverage and instead increases the target closure reward in the sparsity phase. Near-field miss penalty In addition, there is an invalid follow penalty, which guides the agent to quickly switch to precise sweeping of isolated targets on the boundary, avoiding the waste of time and the retention of residual targets caused by blindly searching the cluster center.
[0063] Specifically, at the end of each control cycle, the system calculates the number of newly covered targets, the number of currently surviving targets, target priority consistency, and the preset multiplier for pursuing nearby agents. Target coverage opportunity within the range, selected target closure, line-of-sight angular velocity, whether close-range miss occurs, and smoothness of the action sequence; The revenue from circular coverage can be expressed as: ; in, To add coverage rewards, For continuous coverage rewards, To share the opportunity for rewards, Rewards for consistency in target priority. The goal is to close the reward loop during the sparse phase. For the punishment of the missed opportunity, Penalty for smooth movement As a time penalty.
[0064] Among them, the shared coverage opportunity reward Based on the shared coverage relationship diagram Calculate the set of targets around the current pursuing agent's location.
[0065] In one implementation, the system statistically analyzes the prior information of target priorities. The number of uncovered targets corresponding to a common coverage mask for each target is weighted according to their reachability distance from the pursuing agent: ; in, Prioritize the targets by ranking them first. One goal, For the goal Prioritize target attention For the goal Common overlay mask, For the already covered target set, To track down the intelligent agent to the target distance, The distance normalization scale; When there are a large number of surviving targets and there are still multiple targets around the pursuing agent that can be covered by a circular scope, the system determines that the current stage is a target-dense stage. In the target-dense stage, the reward function increases the weight of the common coverage opportunity reward and the continuous coverage reward, guiding the pursuing agent to choose the path entry that can bring multiple target coverage benefits. When the number of surviving targets decreases, or when pursuing agents around a preset multiplier... When the coverage opportunities within the range are below the threshold, the system determines that it has entered the target sparsity stage. In the target sparsity stage, the reward function reduces the weight of simple common coverage opportunities and increases the target closure reward, the missed penalty and the invalid follow penalty, guiding the pursuing agent to quickly switch to isolated target or boundary target sweeping.
[0066] During the online deployment phase, this embodiment employs a rolling time-domain control strategy. Although the masked attention strategy network outputs the velocity and heading control sequences for the next K control steps, in actual execution, the system typically only issues the first control step command or the first few control step commands in the sequence. When the next control cycle arrives, the system re-acquires the latest environmental state, recalculates the target priority prior attention, and infers and generates a new K-step control sequence again. This rolling optimization mechanism endows the system with extremely strong dynamic response capabilities. On the one hand, it preserves the trajectory smoothness and intent coherence brought about by multi-step prediction. On the other hand, through high-frequency feedback correction, it can promptly compensate for deviations caused by environmental disturbances, target maneuvers, or perception errors, ensuring the safety and robustness of the pursuit process.
[0067] Example 3 Please see Figure 2 This embodiment provides a group target pursuit reinforcement learning control system, including a state acquisition module 100, a priori generation module 200, an observation construction module 300, a policy reasoning module 400, and a decoding execution module 500.
[0068] The state acquisition module 100 is used to acquire the state of the pursuing agent, the states of multiple targets, and the circular radius of action of the pursuing agent. This module constitutes the system's perception input interface, responsible for real-time acquisition of raw data from airborne sensors, external positioning systems, or the simulation environment. This module not only provides basic pose and velocity information but also explicitly acquires the circular radius of action parameter characterizing the pursuing agent's operational capabilities. The introduction of this parameter allows subsequent processing to move beyond purely geometric distance calculations and incorporate decisions based on the actual coverage capabilities of physical entities. In practical deployment, the state acquisition module 100 may include a data preprocessing unit for coordinate transformation, filtering and noise reduction, and time alignment of the raw observation data, ensuring that the data input to downstream modules has a unified spatiotemporal reference.
[0069] The prior generation module 200 is used to determine the common coverage relationship between targets based on the circular radius of action, and to generate target priority prior attention based on the common coverage relationship. Specifically, the prior generation module 200 receives target position and radius of action information from the state acquisition module 100, constructs a common coverage topology map by calculating the relationship between target spacing and twice the radius of action, and further calculates the target priority prior attention by combining motion situation and spatial distribution characteristics. This module transforms discrete geometric constraints into continuous attention weight signals, providing explicit guidance for subsequent neural network inference.
[0070] The observation construction module 300 is used to construct reinforcement learning observation vectors based on the state of the pursuing agent, the state of the target, and the prior attention of the target priority. Specifically, the observation construction module 300 acts as a fusion of multi-source heterogeneous information, structurally concatenating the self-state provided by the state acquisition module 100, the target node features, and the prior attention output by the prior generation module. At the same time, it generates a target survival mask to identify valid targets. This structured observation vector design enables the system to adaptively handle dynamically changing target groups and transmit geometric prior information to the decision layer. The output of this module directly determines the situational richness that the policy network can perceive, and it is a key bridge connecting environmental perception and intelligent decision-making.
[0071] The strategy inference module 400 is used to input observation vectors into the masked attention policy network and output velocity and heading control sequences for multiple future control steps. Internally, the strategy inference module 400 runs a trained masked attention policy network. This module receives standardized observation vectors generated by the observation construction module 300, dynamically aggregates the target group context using an attention mechanism, and directly regresses the velocity and heading sequences for the next K time steps. Compared to traditional architectures that only output single-step actions, the temporal control sequences output by this module implicitly contain the planning intent for future short-term motion trends, effectively suppressing control jitter and improving the coherence of the pursuit process. In engineering implementation, the strategy inference module is typically deployed on a GPU or a dedicated AI acceleration chip to meet the computational power requirements of real-time inference.
[0072] The decoding and execution module 500 decodes the speed and heading control sequences to obtain executable control commands that satisfy the motion constraints of the pursuing agent, and then executes these executable control commands. Specifically, the decoding and execution module 500 receives the abstract control sequence output by the policy inference module 400, and performs step-by-step decoding and pruning based on preset kinematic constraints such as upper and lower speed limits, acceleration limits, and maximum turning angular velocity. It maps the nonlinear output of the neural network back to the physical feasible domain. The decoded commands are sent to the underlying flight control or drive system to drive the pursuing agent to complete the actual maneuver. The existence of this module ensures that even if the policy network produces aggressive or unreasonable outputs during the exploration process, the commands ultimately applied to the physical entity are still safe and executable.
[0073] In one specific embodiment, a training update module is further included. This module updates the masked attention policy network based on the coverage gain, attention supervision loss, and action smoothing cost. The action smoothing cost is determined based on the second-order difference of the continuous control sequence. ; in, , For the cost of smooth motion, For smoothing weights.
[0074] Example 4 This embodiment provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the group target pursuit reinforcement learning control method as described in any of the foregoing method embodiments.
[0075] Specifically, the memory, as a non-volatile or volatile storage medium, can take the form of read-only memory, random access memory, disk, optical disk, or flash memory. In this embodiment, the memory not only stores the underlying code constituting the operating system, but also specifically stores computer program instructions for implementing the group target pursuit reinforcement learning control method. These instructions cover the complete logical flow from state acquisition, target priority prior generation, observation vector construction, masked attention policy network inference to action sequence decoding and execution. In addition, the memory also stores the model weight parameters of the masked attention policy network, various weight coefficients required for target priority score calculation, temperature normalization parameters, and configuration data such as the upper and lower limits of velocity, maximum acceleration, and maximum turning angular velocity on which kinematic constraint decoding depends. It should be understood that although this embodiment uniformly describes the program and data as being stored in memory, in actual engineering implementation, a distributed storage architecture can also be adopted. For example, the model weights can be stored in high-speed video memory to accelerate inference, while historical trajectory data can be stored in a large-capacity disk, as long as the processor can access it on demand.
[0076] As the core of computation and control in electronic devices, the processor can be one or more combinations of a central processing unit, a graphics processing unit, a neural network processor, or a field-programmable gate array. The processor communicates with memory via a bus and is configured to read and execute computer programs stored in memory. When the processor executes the program, its internal logic circuits or arithmetic units sequentially trigger the steps in the aforementioned method embodiments: first, driving the sensor interface or communication module to acquire the state of the pursuing agent and multiple target states; then, calling the arithmetic logic unit to calculate the common coverage relationship between targets based on the circular radius of action and generating target priority prior attention; next, using the tensor computation unit to input the observation vector into the masked attention policy network for forward propagation, outputting a speed and heading control sequence for multiple future control steps; finally, applying kinematic constraints to the control sequence through the numerical computation module to decode and generate executable control instructions. This hardware-level execution mechanism ensures that abstract reinforcement learning algorithms can be transformed into real-time physical control signals.
[0077] In one implementation, the electronic device can be integrated into the body of the pursuing intelligent agent, serving as an airborne computing platform to directly complete the closed loop of perception, decision-making, and control; alternatively, the electronic device can be deployed in a ground control station, cloud server, or edge computing node, remotely receiving the target status and issuing control commands to the pursuing intelligent agent via a wireless link. Regardless of the deployment form, as long as it has a memory and processor and can run the method program described in this application, it falls within the protection scope of this application.
[0078] Example 5 This embodiment provides an extended application scenario for a group target tracking reinforcement learning control method. Regarding the semantic equivalence of the circular action domain, the circular action radius R physically represents the effective range of the tracking agent on the target. In this extended application, the specific physical meaning of this effective range can be equivalently replaced according to the task type. Specifically, when the task is area reconnaissance or environmental monitoring, the circular action radius R can be defined as the effective detection radius or resolution-compliant radius of the airborne sensor. In this case, coverage indicates that the target has entered the sensor's effective field of view and has been successfully identified. When the task is electronic warfare or communication relay, the circular action radius R can be defined as the effective signal suppression radius or stable communication link radius. In this case, coverage indicates that the target is within the effective electromagnetic range. When the task is fire strike or physical interception, the circular action radius R corresponds to the damage radius or the range of the acquisition device. The geometric logic of determining the common coverage relationship between targets based on twice the circular action radius in step S200 remains unchanged. This design allows the same algorithm framework to adapt to various heterogeneous tasks such as perception, communication, and strike by only modifying the physical definition of parameter R, avoiding the cost of developing separate decision algorithms for each task.
[0079] Regarding heterogeneous adaptation of target objects, the target state and motion constraints in the aforementioned embodiments are mainly designed for aerial vehicles. However, in practical applications, targets can also be ground vehicles, surface ships, mobile robots, or other moving entities. Specifically, when the target is a ground vehicle, the target node features in the observation vector construction module need to be adjusted accordingly. For example, the three-dimensional relative position can be replaced with two-dimensional planar relative coordinates, and features such as terrain elevation difference or road accessibility can be added. At the same time, the calculation of edge isolated terms in the target priority prior generation module needs to consider road topology constraints, rather than simply Euclidean distance. In the action decoding stage, the kinematic constraints of the pursuing agent also need to be replaced according to the platform characteristics. For example, for wheeled robots, the maximum turning angular velocity constraint should be replaced with the minimum turning radius constraint or nonholonomic kinematic constraints; for surface ships, water drift and inertial braking distance need to be additionally considered.
[0080] As an extended implementation, when multiple tracking agents cooperate in the system, cooperative constraints can be introduced to optimize global performance while retaining the single-machine geometric prior generation mechanism. Specifically, each tracking agent can independently calculate its own target priority prior attention, but a cooperative division of labor factor or overlap penalty term can be added during the calculation process. For example, when calculating the target... When prioritizing the target, a penalty value proportional to the current attention of other friendly agents to the target can be subtracted. This guides agents to automatically differentiate targets at the geometric prior level, avoiding resource waste caused by multiple agents repeatedly chasing the same target. Alternatively, the joint coverage capability of multiple chasing agents can be regarded as a dynamically changing composite circular scope. Based on this, the common coverage relationship graph can be recalculated so that the prior attention can reflect the coverage gain brought by the multi-agent formation.
[0081] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0082] The technical features described above can be combined arbitrarily. Although not all possible combinations of these technical features are described, any combination of these technical features should be considered to be covered by this specification, provided that such combination does not contain contradictions.
[0083] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A multi-target pursuit reinforcement learning control method, characterized in that, include: S100: Obtain the state of the pursuing agent, the states of multiple targets, and the circular radius of action of the pursuing agent; S200. Based on the circular radius of action, determine the common coverage relationship between each target, and generate target priority prior attention based on the common coverage relationship; S300. Construct a reinforcement learning observation vector based on the state of the pursuing agent, the state of the target, and the prior attention of the target priority; S400. Input the observation vector into the masked attention policy network and output the velocity and heading control sequence for multiple future control steps; S500: Decode the speed and heading control sequence to obtain executable control instructions that satisfy the motion constraints of the pursuing agent, and execute the executable control instructions. 2.The group target pursuit reinforcement learning control method according to claim 1, characterized in that, The step of determining the common coverage relationship between targets based on the circular radius of action, and generating target priority prior attention based on the common coverage relationship, includes: A coverage disk for each target is formed with the target location as the center and the radius of the circle as the radius. It is determined whether the coverage disks of any two targets have an intersection. If there is an intersection, it is determined that the two have a common coverage relationship. Based on the common coverage relationship, the motion state of the target relative to the pursuing agent, and the spatial distribution between targets, the target priority score of each target is calculated, and the target priority score is normalized to obtain the target priority prior attention. 3.The group target pursuit reinforcement learning control method according to claim 2, characterized in that, The step of determining whether the coverage disks of any two targets intersect, and if they do intersect, determining that they have a common coverage relationship, includes: The common coverage relationship between two targets is constructed based on whether their coverage disks intersect, as specifically represented below: ; wherein, represents a common coverage relationship between a first target and a second target , , represents a position vector of a first target and a second target , represents a circular action radius, represents a Euclidean distance between two target positions, when the distance between a first target and a second target is less than or equal to 2R, it is determined that there is an intersection between the two, i.e. a common coverage relationship. 4.The group target pursuit reinforcement learning control method according to claim 2, characterized in that, The method for calculating the target priority score of each target based on the shared coverage relationship, the motion state of the target relative to the tracking agent, and the spatial distribution between targets includes: Based on the shared coverage relationship, calculate the coverage route item and the shared coverage order item; Based on the target's relative motion state to the pursuing agent, calculate the proximity term, heading alignment term, and closing velocity term; Based on the spatial distribution of targets, calculate the coverage route cost, endpoint threat, edge isolation, and target locking terms; Then calculate the target priority score as follows: ; wherein, is a first target has a target priority score, is a cover route item, is a common cover order item, is a close distance item, is a cover route cost item, is an end threat item, is a heading alignment item, is a closure speed item, is an edge isolation item, is a target lock item, , , , , , , , and are respective weights.
5. The group target pursuit reinforcement learning control method according to claim 4, characterized in that, The calculation method for the coverage route item is as follows: constructing a common coverage mask of the candidate target based on the common coverage relationship , calculating a coverage route cost of the first target as the first target according to the common coverage mask , obtaining the coverage route item, and a specific calculation method of the common coverage mask is as follows: ; wherein the common coverage mask to select targets a set of targets having a common coverage opportunity as an entry to a coverage path.
6. The group target pursuit reinforcement learning control method according to claim 1, characterized in that, The method for constructing reinforcement learning observation vectors includes: Based on the state of the pursuing agent, construct the state features of the pursuing agent itself; Based on the target state, construct the target node features and target survival mask for each target; The target node features include at least: the target's position relative to the pursuing agent, the target's speed relative to the pursuing agent, the distance between the target and the pursuing agent, the azimuth angle of the target's heading relative to the pursuing agent, the closing speed of the pursuing agent towards the target, the distance from the target to the mission endpoint, and the target's survival marker. The target liveness mask is used to identify whether the corresponding target is alive or has been covered; The state features of the pursuing agent, the target node features of each target, the target survival mask of each target, and the prior attention of the target priority are combined to form the reinforcement learning observation vector.
7. The group target pursuit reinforcement learning control method according to claim 6, characterized in that, The process of inputting the observation vector into a masked attention policy network and outputting a velocity and heading control sequence for multiple future control steps includes: The state features of the tracking agent in the observation vector are input into a state encoding network to obtain its own embedding. The features of each target node in the observation vector are input into the target encoding network to obtain the embedding of each target; The target priority prior attention in the observation vector is input into the prior embedding network to obtain the prior embedding; The self-embedding, each target embedding, and the prior embedding are input into the attention prediction network, and the learned attention of each surviving target is calculated in combination with the target survival mask. Invalid targets are then masked according to the target survival mask. Based on the learned attention and the target survival mask, the target group context features are aggregated. After fusing the self-embedding with the target group context features, the input is given to the action output network to obtain the velocity and heading control sequences for the future multiple control steps. 8.The group target pursuit reinforcement learning control method of claim 1, wherein, The method for decoding the speed and heading control sequences includes: Set upper and lower speed limits, acceleration constraints, and maximum turning angular velocity constraints. Decode the speed and heading control sequences based on these constraints. The specific calculation method is as follows: Future The speed and heading control sequence for one control step is represented as: ; wherein, is the speed command for the control step, is the heading command or heading increment for the control step, is a positive integer; The speed and heading control sequences are decoded step by step according to the following constraints: ; ; ; wherein, and Vmin and Vmax are minimum and maximum speed, a is maximum acceleration, s is control step size, v is speed ωmax is maximum turning angular velocity at v.
9. A swarm target pursuit reinforcement learning control system, characterized in that, include: The state acquisition module is used to acquire the state of the pursuing agent, the states of multiple targets, and the circular radius of action of the pursuing agent. The prior generation module is used to determine the common coverage relationship between each target based on the circular radius of action, and to generate target priority prior attention based on the common coverage relationship; An observation construction module is used to construct reinforcement learning observation vectors based on the state of the pursuing agent, the state of the target, and the prior attention of the target priority. The strategy reasoning module is used to input the observation vector into the masked attention policy network and output the velocity and heading control sequences for multiple future control steps; The decoding and execution module is used to decode the speed and heading control sequence to obtain executable control instructions that satisfy the motion constraints of the pursuing agent, and to execute the executable control instructions.
10. An electronic device, comprising: It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the group target pursuit reinforcement learning control method as described in any one of claims 1 to 8.
