Intelligent target distribution method and system based on deep reinforcement learning

Abstract

The invention discloses an intelligent target allocation method and system based on deep reinforcement learning, particularly relates to the field of multi-agent systems, and is used for solving the real-time decision problem under multi-dimensional constraints in a dynamic confrontation simulation scene. Time, space and resource constraints are decomposed into independent models, and the decision complexity of a multi-agent system in a dynamic confrontation scene is reduced; by means of adaptive Monte Carlo sampling, prediction perception path planning and dynamic threshold value resource allocation, diversified high-quality candidate schemes are quickly generated; when the environment changes, the multi-layer influence propagation network and the spatial topological relation graph are used for positioning an affected agent, and scheme fragments are generated through local adjustment, so that global re-calculation is avoided, the overhead is reduced, and real-time response is ensured; meanwhile, an efficient and reliable optimal solution is screened out through resource delay coupling and path efficiency comprehensive evaluation in combination with multi-objective optimization and task consistency verification.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent systems, and more specifically, to an intelligent target allocation method and system based on deep reinforcement learning. Background Art

[0002] In dynamic confrontation simulation scenarios, multi-agent systems need to perform collaborative tasks in complex environments. The core challenge is to simultaneously meet the multi-dimensional dynamic constraints of time, space, and resources. In a typical scenario, the agent needs to plan a path in a dynamically changing obstacle area and ensure the timing logic and resource consumption limits of task execution. For example, the agent's actions must strictly match the preset time window to avoid conflicts, avoid no-go areas that are updated in real time, and complete target allocation under limited energy supply conditions. These constraints are intertwined in a dynamic environment, forming a complex decision space with strong coupling of time and space resources, which places stringent requirements on the system's real-time response capabilities.

[0003] Existing target allocation methods have inherent defects when dealing with multi-constraint coupling problems. Traditional solutions use a rigid joint optimization strategy to serially process time, space, and resource constraints as an indivisible whole. When a constraint in a certain dimension changes dynamically (such as adding a no-go zone or adjusting task priorities), the system is unable to effectively decouple the correlation between time and space resources and is forced to re-traverse the global constraint relationship and perform full calculations. This strongly coupled processing mode causes local adjustments to trigger a chain reaction, significantly increasing decision delays. Especially in the scenario of sudden constraint changes, the system lacks dynamic decoupling and incremental optimization capabilities, making it difficult to quickly generate feasible solutions that meet multi-dimensional constraints, causing some tasks to fail due to response delays, seriously restricting the system's real-time decision-making efficiency in a dynamic environment.

[0004] In order to solve the above problems, a technical solution is now provided. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an intelligent target allocation method and system based on deep reinforcement learning. By constructing a timing dependency graph, a gridded traffic matrix and a capacity balance table, the time, space and resource constraints are decomposed into independent models, thereby reducing the decision-making complexity of the multi-agent system in a dynamic confrontation scenario; with the help of adaptive Monte Carlo sampling, predictive perception path planning and dynamic threshold resource allocation, diversified and high-quality candidate solutions are quickly generated; when the environment changes, the affected agents are located by using a multi-layer influence propagation network and a spatial topological relationship map, and solution fragments are generated through local adjustments to avoid global recalculation, reduce overhead, and ensure real-time response; at the same time, through resource delay coupling and path efficiency comprehensive evaluation, combined with multi-objective optimization and task consistency verification, an efficient and reliable optimal solution is screened out to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions: The intelligent target allocation method based on deep reinforcement learning includes the following steps: S1. Decompose the time constraint into a time-dependent graph with relaxation factors, where nodes represent task time windows and edges represent task precedence relationships; discretize the spatial constraint into a gridded binary access matrix, with each grid annotated with the state of dynamic obstacles; transform the resource constraint into a capacity balance table, and record the resource margin of each agent in real time; S2. Generate task sequences through Monte Carlo sampling in the time domain, use the improved A* algorithm to generate path fragments in the space domain, and execute the greedy strategy to allocate resource packages in the resource domain; combine the candidate fragments of the three domains through the graph matching algorithm and output a set of candidate solutions; S3. Establish a change impact propagation model to locate the set of agents affected by the constraint change based on the spatial topological relationship map; trigger the local constraint satisfaction solver to generate a new set of candidate solutions based on the task sequence, path segment and resource allocation data of the affected agents; S4. Obtain resource delay information and path efficiency information from the new set of candidate solutions, and perform Pareto frontier screening on the candidate solution set; use the sequential logic verification tool to check the consistency of task dependencies in the solution, and output a set of conflict-free feasible solutions.

[0007] In a preferred embodiment, step S1 includes the following contents: Time constraint processing: Based on the task time window and sequential dependency, a relaxation factor is introduced to quantify the degree of relaxation of the timing dependency between tasks, and a timing dependency graph with relaxation factors is constructed in real time. The relaxation factor is dynamically calculated based on the time difference between the task end time and the subsequent task start time to reflect the time flexibility of task execution; Spatial constraint processing: Using the dynamic obstacle prediction model, the environment is rasterized in real time to generate a binary traffic matrix. The traffic status of each grid is dynamically updated based on historical data and the trend predicted by the exponential smoothing method to characterize the real-time traffic capacity of the spatial environment. Resource constraint processing: A nonlinear attenuation model is used to calculate the real-time consumption of agent resources by tasks, and a capacity balance table is constructed. The resource margin is adjusted in real time with the progress of task execution and load changes to track the resource status of each agent.

[0008] In a preferred embodiment, step S2 includes the following contents: Based on the timing dependency graph, adaptive Monte Carlo sampling is used to generate time domain task sequences. The sampling weight is dynamically adjusted according to the task dependency strength. The dependency strength is calculated by the weighted sum of the incoming edge relaxation factors, so that tasks with high dependency strength are sampled first. Based on the dynamic access matrix, the predictive-aware A* algorithm is used to generate spatial domain path fragments. The predictive-aware heuristic function is introduced. The predictive-aware heuristic function combines the traditional heuristic function with the cumulative probability of inaccessibility at future moments to optimize the adaptability of the path to the dynamic environment. Based on the capacity balance table, a dynamic threshold greedy strategy is used to allocate resource domain resources. The dynamic threshold is adjusted with the changes in the resource surplus and load of the agent. Resources are allocated only when the task resource demand is lower than the threshold. Agents with large resource surplus and low load are given priority. Through the multi-domain graph matching algorithm, the time, space and resource solution fragments are mapped to the multi-domain graph, and the compatibility score is calculated. The compatibility score is the weighted sum of the matching degree of time and space, space and resources, and resources and time. The combination scheme with the highest compatibility is screened out through maximum weight matching, and a set of candidate solutions is output.

[0009] In a preferred embodiment, step S3 includes the following contents: The agents, tasks and resources are mapped into a multi-layer network, in which the nodes are agents, tasks and resources, and the edges represent allocation relationships, dependency relationships and timing relationships. By calculating the impact intensity of the change source on other nodes in the network when the constraints change and iteratively propagating it, the agents with impact intensity exceeding the threshold are located to determine the set of affected agents; a spatial topological relationship map is constructed based on the spatial position of the agent in the path segment, the affected agents are clustered, the center of the cluster space is calculated to determine the affected area, and the agents in the area are included in the affected set; the task sequence, path segment and resource allocation of the affected agents are extracted from the candidate solutions, the local time, space and resource constraints are updated, and the constraint satisfaction solver is used to generate new solution fragments in the local range, which are combined with the solution fragments of the unaffected agents to form a new candidate solution; an incremental update strategy is adopted to replace only the solution fragments of the affected agents and retain the unaffected parts, and the cross-domain constraint compatibility of the updated solution is verified to output a new set of candidate solutions.

[0010] In a preferred embodiment, step S4 includes the following contents: Resource delay information and path efficiency information are obtained from the new candidate solution set, wherein the resource delay information includes a resource delay coupling index, and the path efficiency information includes a path efficiency skewness index.

[0011] In a preferred implementation, the logic for obtaining the resource delay coupling index is: For each solution in the new candidate solution set , calculate its total resource consumption and the total time delay ;in ,in Representing an Agent On Task Resource consumption in represents the set of all agents, Representing an Agent The set of all tasks assigned to be executed; ,in For the task The actual completion time, is the ideal completion time of the task time window, Represents the set of all tasks; defines the resource delay coupling index : ,in A small positive number.

[0012] In a preferred embodiment, the logic for obtaining the path efficiency skewness index is: For each solution in the new candidate solution set , calculate its total path length and the ideal path length ;in ,in For intelligent agents ; where ,in Representing an Agent The straight-line distance from the starting point to the end point, where It is an intelligent agent The starting point, It is an intelligent agent The end point of ; calculate the statistics of the path corners: is the corner sequence of all path segments in the solution, calculated from the spatial coordinates of the path segments in step S2; Indicates the extreme value of the turning angle, reflecting the degree of path tortuosity; represents the average turning angle; defines the path efficiency skewness index : ,in A small positive number.

[0013] In a preferred embodiment, the optimal solution is screened out from the set of candidate solutions by adopting a multi-objective optimization method; the specific processing process is as follows: first, each candidate solution is mapped to a two-dimensional target space, and the corresponding two-dimensional target space is composed of a resource delay coupling index and a path efficiency skewness index; then, the Pareto frontier algorithm is applied to screen out a set of Pareto optimal solutions, that is, solutions that cannot be improved by other solutions at the same time in terms of resource delay coupling index and path efficiency skewness index; finally, the Pareto optimal solution set is output.

[0014] In a preferred embodiment, in order to ensure that the task execution timing of the Pareto optimal solution satisfies the dependency, a linear temporal logic model checker is used for verification; the specific processing process is as follows: first, a timing trajectory of task execution is generated for each Pareto optimal solution, including the start time and end time of each task; then, a linear temporal logic formula is defined to express the timing dependency between tasks; then, a linear temporal logic model checker is used to verify whether the timing trajectory of each solution satisfies all dependency constraints; finally, a final set of feasible solutions that have passed verification is output.

[0015] Intelligent target allocation system based on deep reinforcement learning, including: constraint decomposition modeling module, solution generation and verification module, local replanning module and solution selection and verification module; Constraint decomposition modeling module: decomposes time constraints into a time-dependent graph with relaxation factors, where nodes represent task time windows and edges represent task precedence relationships; discretizes spatial constraints into a gridded binary access matrix, with each grid annotated with the state of dynamic obstacles; converts resource constraints into a capacity balance table, and records the resource margin of each agent in real time; Solution generation and verification module: Generate task sequences through Monte Carlo sampling in the time domain, use the improved A* algorithm to generate path fragments in the space domain, and execute the greedy strategy to allocate resource packages in the resource domain; then combine the candidate fragments in the time, space and resource domains into a complete set of candidate solutions through the graph matching algorithm, verify their compatibility, and output the candidate solution set; Local replanning module: In response to environmental changes, a change impact propagation model is established to locate the set of agents affected by the constraint change based on the spatial topological relationship map; then, based on the task sequence, path segment and resource allocation data of the affected agents, the local constraint satisfaction solver is triggered to generate a new set of candidate solutions; Solution selection and verification module: extract resource delay information and path efficiency information from the new candidate solution set, and perform multi-objective optimization through Pareto front screening; then use the sequential logic verification tool to check the consistency of task dependencies in the solution, and finally output a set of conflict-free feasible solutions.

[0016] The technical effects and advantages of the intelligent target allocation method and system based on deep reinforcement learning of the present invention are as follows: By constructing a timing dependency graph, a gridded traffic matrix, and a capacity balance table, the time, space, and resource constraints are decomposed into independent models, significantly reducing the decision-making complexity of the multi-agent system in dynamic confrontation simulation scenarios. With the help of adaptive Monte Carlo sampling, predictive-aware path planning, and dynamic threshold resource allocation, diverse and high-quality candidate solutions are quickly generated to improve the system's adaptability to environmental changes. When the environment changes, the multi-layer impact propagation network and spatial topological relationship map locate the affected agents, locally adjust the solution fragments, avoid global recalculation, and ensure real-time response. Through the comprehensive evaluation of resource delay coupling and path efficiency, combined with multi-objective optimization and task consistency verification, an efficient and reliable optimal solution is screened out. This significantly improves the decision-making speed, flexibility, and stability of the multi-agent system in a dynamic environment, providing reliable support for the efficient execution of real-time collaborative tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the process of the intelligent target allocation method based on deep reinforcement learning of the present invention.

[0018] Figure 2 It is a structural schematic diagram of the intelligent target allocation system based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Embodiment 1: Figure 1 The present invention provides an intelligent target allocation method based on deep reinforcement learning, including: S1. Decompose the time constraint into a timing dependency graph with relaxation factors, where nodes represent task time windows and edges represent task precedence relationships; discretize the spatial constraint into a gridded binary access matrix, with each grid annotated with the state of dynamic obstacles; transform the resource constraint into a capacity balance table, and record the resource margin of each agent in real time.

[0021] S2. Generate task sequences through Monte Carlo sampling in the time domain, use the improved A* algorithm to generate path fragments in the space domain, and execute the greedy strategy to allocate resource packages in the resource domain; combine the candidate fragments of the three domains through the graph matching algorithm and output a set of candidate solutions.

[0022] S3. Establish a change impact propagation model, locate the set of agents affected by the constraint change based on the spatial topological relationship map; trigger the local constraint satisfaction solver to generate a new set of candidate solutions based on the task sequence, path fragments and resource allocation data of the affected agents.

[0023] S4. Obtain resource delay information and path efficiency information from the new set of candidate solutions, and perform Pareto frontier screening on the candidate solution set; use the sequential logic verification tool to check the consistency of task dependencies in the solution, and output a set of conflict-free feasible solutions.

[0024] In dynamic confrontation simulation scenarios, the time, space, and resource constraints faced by multi-agent systems are intertwined and highly coupled. If these constraints are handled directly, the decision-making process will be too complicated and difficult to quickly adapt to environmental changes. Therefore, it is necessary to model the time constraints (such as task time windows and precedence relationships), space constraints (such as obstacles and traffic conditions), and resource constraints (such as energy supply and consumption) separately. The complexity of the problem is reduced through hierarchical decoupling, thereby improving the system's real-time decision-making ability. This method can decompose complex joint optimization problems into relatively independent sub-problems, avoid the calculation from falling into local optimality or bottlenecks, and maintain global consistency at the same time, so that the system can not only flexibly respond to dynamic changes, but also quickly adapt to the environment through local adjustments, enhancing robustness and efficiency.

[0025] Step S1 includes the following contents: In order to achieve the decoupling of time, space and resource constraints, step S1 adopts the following specific technical logic and is completed step by step in three parts.

[0026] A1. Time Constraint Modeling: Timing Dependency Graph with Relaxation Factor: In order to reflect the temporal dependency between tasks and quantify their flexibility, a temporal dependency graph is constructed. The specific method is as follows: first, the time window of each task is defined as a node, and the node is represented by the start time and end time range of the task; then, the order relationship between tasks is defined as a directed edge, indicating that one task must be completed before another task; then, a relaxation factor is introduced to measure the degree of relaxation of the temporal dependency between tasks. The relaxation factor is calculated as follows: the difference between the end time of task i and the start time of task j is calculated. If the difference is negative (that is, the time windows overlap), the relaxation factor is the absolute value of the difference divided by the absolute time difference between the end time of task i and the start time of task j; if the difference is positive (that is, the time windows do not overlap), the relaxation factor is zero. To avoid division by zero, a very small positive value is added as an adjustment item in the denominator during the calculation. Finally, a temporal dependency graph is output, which contains all task nodes, directed edges representing the order relationship, and the corresponding relaxation factor set.

[0027] A2. Spatial Constraint Modeling: Rasterized Binary Pass Matrix: In order to describe the spatial constraints of the environment and reflect the real-time traffic status, a gridded binary traffic matrix is ​​constructed. The specific method is as follows: first, the entire environment is discretized into grid-like grids, and the traffic status of each grid at a certain moment is represented by a binary value, where passable is recorded as 1 and impassable is recorded as 0; then, a dynamic obstacle prediction model is introduced to predict the traffic status at future moments based on the historical traffic status and change trend. The exponential smoothing method is used for prediction, and the calculation method is as follows: the predicted traffic status at the current moment is equal to the weighted average of the actual traffic status at the previous moment and the predicted traffic status at the previous moment, where the weight is determined by a smoothing coefficient, which is usually between 0 and 1 and is used to adjust the relative importance of historical data and new data; finally, a dynamic traffic matrix is ​​output, which updates the traffic status of each grid in real time to reflect the spatial constraints of the environment.

[0028] A3. Resource Constraint Modeling: Dynamic Capacity-Consumption Balance In order to quantify the change of resources over time and task load and track the resource status of the agent, a capacity balance table is constructed. The specific method is: first, the resource surplus of each agent at a certain moment is equal to its initial resource amount minus the cumulative sum of resources consumed by executing tasks; then, the resource consumption of the agent by the task is calculated using a nonlinear decay model, that is, the basic resource consumption increases nonlinearly with the increase of the task execution progress, where the task execution progress is defined as the difference between the current time and the task start time divided by the total duration of the task, and the specific growth mode of resource consumption can be set to a nonlinear form such as a quadratic function or an exponential function according to the actual scenario; then, the capacity balance table is updated in the following way: the resource surplus at each moment is equal to the resource surplus at the previous moment minus the sum of resources consumed by executing all tasks at the current moment; finally, a capacity balance table is output to record the resource surplus of each agent in real time and reflect its resource constraint status.

[0029] Step S1 successfully decouples time, space, and resource constraints into three independent models through the above method: the time dependency graph with relaxation factors clearly describes the time dependency of tasks, the rasterized binary access matrix reflects the spatial access status of the environment in real time, and the capacity balance table accurately tracks the resource surplus of the agent. These models provide a reliable data basis for subsequent steps (such as sampling based on the time dependency graph, planning paths using the access matrix, and allocating resources based on the capacity balance table), meeting the multi-dimensional constraint requirements of dynamic confrontation simulation scenarios.

[0030] Step S1 successfully decouples time, space, and resource constraints into three independent models by constructing a timing dependency graph with relaxation factors, a dynamically updated gridded traffic matrix, and a capacity balance table that reflects nonlinear resource consumption. These models describe the time dependency of tasks, the traffic status of space, and the resource margin of agents, respectively, providing basic data support for step S2. Step S2 uses these outputs to generate and verify a set of candidate solutions in a dynamic confrontation simulation scenario, ensuring the compatibility of time, space, and resource domain solution fragments, reducing invalid calculations in subsequent local replanning, and providing the system with a rich selection space, enhancing its ability to respond quickly to sudden constraint changes, and laying a solid foundation for real-time collaborative task execution of multi-agent systems in complex confrontation environments.

[0031] Step S2 includes the following contents: B1. Time domain: Task sequence generation: In order to generate diverse task execution sequences, an adaptive Monte Carlo sampling method is used. The specific calculation and processing process is as follows: First, the dependency strength of each task is calculated. The dependency strength is defined as the weighted sum of the relaxation factors of the task input edges, where the weight is determined by the priority or importance of the task; then, the adaptive sampling weight is calculated. The adaptive sampling weight is dynamically adjusted according to the change of the task dependency strength. The specific calculation method is to map the difference between the dependency strength and the average dependency strength of all tasks through the Sigmoid function, so that tasks with high dependency strength have a higher sampling probability; finally, Monte Carlo sampling is performed based on these adaptive sampling weights to generate multiple task sequences, thereby forming a set of candidate task sequences in the time domain.

[0032] B2. Spatial domain: path segment generation: In order to generate a path that adapts to the dynamic environment, an improved A* algorithm based on predictive perception is adopted. The specific calculation and processing process is as follows: First, the traffic status at the future moment is predicted. The prediction method is calculated by the linear combination of the current traffic status and the historical change trend; then, the predictive perception heuristic function is introduced into the A* algorithm. On the basis of the traditional heuristic function, this heuristic function adds the cumulative probability of impassability at the future moment to guide the path selection to avoid potential obstacles; finally, the A* algorithm is run based on this predictive perception heuristic function to generate multiple path segments, thereby forming a set of candidate path segments in the spatial domain.

[0033] B3. Resource domain: Resource allocation: In order to optimize resource allocation, a greedy strategy based on dynamic threshold is adopted. The specific calculation and processing process is as follows: First, the dynamic resource threshold of each agent is calculated. The dynamic resource threshold is dynamically adjusted according to the resource margin and current load of the agent. Specifically, the higher the load, the lower the threshold; then, resource allocation is performed in the greedy strategy. Only when the resource demand of the task is less than or equal to the dynamic resource threshold of the agent, the resource package is allocated to the agent; finally, according to the principle of giving priority to agents with large resource margin and low load, a resource allocation plan is generated, thereby forming a candidate resource allocation set for the resource domain.

[0034] B4. Cross-domain combination and verification: In order to ensure the compatibility of time, space and resource solution fragments, a multi-domain graph matching algorithm is used for verification. The specific calculation and processing process is as follows: First, the task sequence in the time domain, the path fragment in the space domain and the resource allocation in the resource domain are mapped into a multi-domain graph, where the nodes of the graph represent the combination of the three and the edges represent the constraint compatibility between the three; then, the compatibility score is calculated, which is defined as the weighted sum of the matching degrees of time and space, space and resources, and resources and time, where the weights are adjusted according to the specific needs of the dynamic confrontation simulation scenario; finally, the combination with the highest compatibility score is screened out through the maximum weight matching algorithm, thereby outputting a set of candidate solutions that have passed the verification.

[0035] Step S2 successfully generates the task sequence in the time domain, the path fragments in the space domain, and the resource allocation plan in the resource domain through the above method, and ensures the compatibility of these plan fragments through cross-domain verification, and finally outputs a high-quality set of candidate plans, which provides a solid foundation for decision-making in dynamic confrontation simulation scenarios.

[0036] In dynamic confrontation simulation scenarios, environmental changes (such as obstacle position adjustment, task priority change) may cause some of the candidate solution sets generated in step S2 to fail. To cope with such changes, step S3 updates the candidate solution set by locating the affected agents and regenerating their time, space, and resource solution fragments to ensure solution adaptability and system response speed. Step S3 directly uses the output of step S2, the candidate solution set, as input data to achieve local replanning, providing a high-quality candidate solution selection basis for step S4.

[0037] Step S3 includes the following contents: C1. Establishment of change impact propagation model: In order to quickly locate the affected agents, a multi-layer influence propagation network is constructed. The specific calculation and processing process is as follows: First, the agents, tasks and resources are mapped into a multi-layer network, in which the nodes of the network include agents, tasks and resources, and the edges represent the distribution relationship between agents and tasks, the dependency relationship between agents and resources, and the temporal relationship between tasks; then, when a constraint change occurs, the change source is regarded as a node in the network, and its influence on other nodes is calculated. The calculation of the influence intensity adopts an iterative propagation method, that is, the influence intensity of each node is equal to the sum of the influence intensity of all its adjacent nodes multiplied by the weight of the corresponding edge, and then divided by the number of adjacent nodes of the node plus one to normalize the influence diffusion; finally, an influence threshold is set for each agent node. If the influence intensity of an agent node exceeds the threshold, the agent is determined to be an affected agent.

[0038] C2. Construction and positioning of spatial topological relationship maps: In order to accurately locate the spatial distribution of the affected agents, a spatial topological map of the agents is constructed. The specific calculation and processing process is as follows: First, based on the spatial position of each agent path segment in step S2, a topological map is constructed, in which nodes represent agents and edges represent spatial adjacency between agents; then, the affected agents are marked in the topological map, and these affected agents are divided into several clusters through a clustering algorithm, and the spatial center of each cluster is calculated. The specific method is to select a position so that the sum of the distances from all agents in the cluster to the position is minimized; finally, the affected area is determined based on the spatial center of each cluster, and all agents in the area are included in the set of affected agents.

[0039] C3. Triggering of the local constraint satisfaction solver and regeneration of solution fragments: In order to regenerate the solution fragments of the affected agents, a local constraint satisfaction solver is used. The specific calculation and processing process is as follows: first, the task sequence, path fragment and resource allocation of the affected agents are extracted from the candidate solutions of step S2; then, the local constraints of these affected agents are updated according to the changed time constraints, space constraints and resource constraints; then, the constraint satisfaction solver is used to solve in the local range of the affected agents to generate new task sequences, path fragments and resource allocation solution fragments; finally, the newly generated solution fragments are combined with the original solution fragments of the unaffected agents to form a complete candidate solution.

[0040] C4. Update the candidate solution set: In order to quickly generate a new set of candidate solutions, an incremental update strategy is adopted. The specific calculation and processing process is as follows: for each original candidate solution in step S2, only the solution fragment of the affected agent is replaced with the newly generated solution fragment, while the solution fragment of the unaffected agent remains unchanged; then, the replaced candidate solution is quickly verified to ensure that it meets the cross-domain compatibility of time constraints, space constraints and resource constraints; finally, the updated new candidate solution set is output for use in subsequent steps.

[0041] Step S3 constructs a multi-layer influence propagation network and a spatial topological relationship map to efficiently locate the set of agents affected by the constraint change, and uses a local constraint satisfaction solver to regenerate the time, space, and resource solution fragments of the affected agents, thereby realizing the incremental update of the candidate solution set. This method avoids the computational overhead of global replanning, ensures the adaptability and computational efficiency of the solution in a dynamic environment, and provides a reliable set of candidate solutions for the optimal solution selection and task consistency verification in the subsequent step S4, significantly improving the real-time decision-making ability of the multi-agent system in a complex confrontation environment.

[0042] In the dynamic confrontation simulation scenario, step S3 establishes a change impact propagation model, locates the set of agents affected by the constraint change based on the spatial topological relationship map, triggers the local constraint satisfaction solver for the task sequence, path fragment and resource allocation data of the affected agents, and updates and outputs a new set of candidate solutions. Step S4 directly inherits the output of the new candidate solution set in step S3, aiming to extract key indicators of resource delay and path efficiency from the set, selects the optimal solution through indicator calculation, multi-objective optimization and timing consistency verification, and outputs the final feasible solution that meets all constraints. The technical logic of step S4 relies on the task sequence, path fragment and resource allocation data of each solution in the new candidate solution set to ensure clear connection with the previous steps and consistent parameter definition.

[0043] Step S4 includes the following contents: Step S4 obtains resource delay information and path efficiency information from the new candidate solution set output by step S3, wherein the resource delay information includes the resource delay coupling index, and the path efficiency information includes the path efficiency skewness index. Through the following specific technical logic, the optimal solution is selected and the task dependency consistency is verified, and the final feasible solution is output.

[0044] The logic for obtaining the resource delay coupling index is: The resource-delay coupling index quantifies the coupling effect of resource consumption and time delay, highlighting the interactive impact of the two in a dynamic environment.

[0045] For each solution in the new candidate solution set , calculate its total resource consumption and the total time delay : ,in Representing an Agent On Task Resource consumption in represents the set of all agents, Representing an Agent The collection of all tasks that are assigned for execution.

[0046] ,in For the task The actual completion time, is the ideal completion time of the task time window, Represents the collection of all tasks.

[0047] Defining resource delay coupling index : in A small positive number used to avoid division by zero.

[0048] This formula amplifies the ratio effect of resource consumption and time delay through the square term and introduces nonlinearity through the logarithmic term, which makes the exponent increase significantly when both resources and time delay are large, reflecting the strong coupling characteristics.

[0049] The logic for obtaining the path efficiency skewness index is: The path efficiency skewness index is used to assess the deviation of the actual efficiency of a path from the ideal efficiency and to quantify the tortuosity of the path.

[0050] For each solution in the new candidate solution set , calculate its total path length and the ideal path length : ,in For intelligent agents The path segment.

[0051] ,in Representing an Agent The straight-line distance from the starting point to the end point, where It is an intelligent agent The starting point, It is an intelligent agent The end point.

[0052] Compute statistics for path corners: is the corner sequence of all path segments in the solution, calculated from the spatial coordinates of the path segments in step S2.

[0053] Indicates the extreme value of the turning angle, reflecting the degree of path tortuosity.

[0054] Represents the average value of the rotation angle.

[0055] Defining the path efficiency skewness index : Be a small positive number to avoid division by zero.

[0056] This formula evaluates efficiency through the ratio of path lengths and quantifies path skewness through the ratio of the turning range to the average, comprehensively reflecting the actual performance of the path.

[0057] In order to select the best solution from the candidate solution set, a multi-objective optimization method is adopted. The specific processing process is as follows: first, each candidate solution is mapped to a two-dimensional target space, which is composed of a resource delay coupling index and a path efficiency skewness index; then, the Pareto frontier algorithm is applied to select the Pareto optimal solution set, that is, the solution that cannot be improved by other solutions at the same time in terms of resource delay coupling index and path efficiency skewness index; finally, the Pareto optimal solution set is output as the input for the next step of verification.

[0058] In order to ensure that the task execution timing of the Pareto optimal solution satisfies the dependency, a linear temporal logic (LTL) model checker is used for verification. The specific processing process is as follows: First, the timing trajectory of task execution is generated for each Pareto optimal solution, including the start time and end time of each task; then, a linear temporal logic formula is defined to express the timing dependency between tasks, for example, "task A must be completed before task B" can be expressed as LTL: G(A->FB); then, the linear temporal logic model checker is used to verify whether the timing trajectory of each solution satisfies all dependency constraints; finally, the final set of feasible solutions that have passed verification is output.

[0059] Linear Temporal Logic (LTL) is a formal logic used to describe and verify time-varying properties in a system, and is particularly suitable for expressing timing constraints on system behavior. LTL accurately describes the relationship between events or states on the timeline by introducing timing operators such as Globally (indicating that a property holds at all future moments), Finally (indicating that a property holds at a certain moment in the future), and Next (indicating that a property holds at the next moment). This logic can describe the conditions that a system must meet at a certain moment in the future or for a period of time. For example, LTL can be used to express timing dependencies such as Task A must be completed before Task B or a state must be maintained until another state appears. In practical applications, such as dynamic adversarial simulation scenarios, LTL is used to verify whether the task execution timing of candidate solutions meets the preset dependencies and time constraints, thereby ensuring the correctness and reliability of the solution. In a formal way, LTL provides powerful tool support for system design and verification.

[0060] Step S4 is based on the candidate solution set output by step S3. By calculating the resource-delay coupling index and path efficiency skewness index, combined with Pareto frontier screening and LTL model checking, it successfully screens out the solution with the lowest resource-delay coupling and the highest path efficiency from the perspective of multi-objective optimization, and ensures its task dependency consistency. In the dynamic confrontation simulation scenario, this method effectively responds to environmental changes, ensures that the final feasible solution set output meets the time, space and resource constraints, and improves the execution efficiency and reliability of multi-agent collaborative tasks.

[0061] By constructing a timing dependency graph, a gridded traffic matrix, and a capacity balance table, the time, space, and resource constraints are decomposed into independent models, significantly reducing the decision-making complexity of the multi-agent system in dynamic confrontation simulation scenarios. With the help of adaptive Monte Carlo sampling, predictive-aware path planning, and dynamic threshold resource allocation, diverse and high-quality candidate solutions are quickly generated to improve the system's adaptability to environmental changes. When the environment changes, the multi-layer impact propagation network and spatial topological relationship map locate the affected agents, locally adjust the solution fragments, avoid global recalculation, and ensure real-time response. Through the comprehensive evaluation of resource delay coupling and path efficiency, combined with multi-objective optimization and task consistency verification, an efficient and reliable optimal solution is screened out. This significantly improves the decision-making speed, flexibility, and stability of the multi-agent system in a dynamic environment, providing reliable support for the efficient execution of real-time collaborative tasks.

[0062] Embodiment 2: Figure 2 The present invention provides an intelligent target allocation system based on deep reinforcement learning, including: a constraint decomposition modeling module, a solution generation and verification module, a local replanning module and a solution optimization verification module; Constraint decomposition modeling module: decomposes time constraints into a timing dependency graph with relaxation factors, where nodes represent task time windows and edges represent task precedence relationships; discretizes spatial constraints into a gridded binary access matrix, with each grid annotated with the state of dynamic obstacles; and converts resource constraints into a capacity balance table to record the resource margin of each agent in real time.

[0063] Solution generation and verification module: Generate task sequences through Monte Carlo sampling in the time domain, use the improved A* algorithm to generate path fragments in the space domain, and execute the greedy strategy to allocate resource packages in the resource domain; then use the graph matching algorithm to combine the candidate fragments in the time, space and resource domains into a complete set of candidate solutions, verify their compatibility, and output the set of candidate solutions.

[0064] Local replanning module: In response to environmental changes, a change impact propagation model is established, and the set of agents affected by the constraint changes is located based on the spatial topological relationship map; then, based on the task sequence, path fragments and resource allocation data of the affected agents, the local constraint satisfaction solver is triggered to generate a new set of candidate solutions.

[0065] Solution selection and verification module: extract resource delay information and path efficiency information from the new candidate solution set, and perform multi-objective optimization through Pareto front screening; then use the sequential logic verification tool to check the consistency of task dependencies in the solution, and finally output a set of conflict-free feasible solutions.

[0066] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0067] It should be noted that the system of the present invention can be deployed on the device itself to realize embedded applications, and can also be run on a PC or other terminal with a user interface, thereby meeting a variety of hardware environments and usage requirements.

[0068] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0069] It should be noted that, in this article, if there are relational terms such as first and second, etc., they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0070] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An intelligent target allocation method based on deep reinforcement learning, characterized in that: Includes steps: S1. Decompose the time constraint into a time-dependent graph with relaxation factors, where nodes represent task time windows and edges represent task precedence relationships; discretize the spatial constraint into a gridded binary access matrix, with each grid annotated with the state of dynamic obstacles; transform the resource constraint into a capacity balance table, and record the resource margin of each agent in real time; S2. Generate task sequences through Monte Carlo sampling in the time domain, use the improved A* algorithm to generate path fragments in the space domain, and execute the greedy strategy to allocate resource packages in the resource domain; combine the candidate fragments of the three domains through the graph matching algorithm and output a set of candidate solutions; S3. Establish a change impact propagation model to locate the set of agents affected by the constraint change based on the spatial topological relationship map; trigger the local constraint satisfaction solver to generate a new set of candidate solutions based on the task sequence, path segment and resource allocation data of the affected agents; S4. Obtain resource delay information and path efficiency information from the new set of candidate solutions, and perform Pareto frontier screening on the candidate solution set; use the sequential logic verification tool to check the consistency of task dependencies in the solution, and output a set of conflict-free feasible solutions.

2. The intelligent target allocation method based on deep reinforcement learning according to claim 1 is characterized in that: Step S1 includes the following contents: Time constraint processing: Based on the task time window and sequential dependency, a relaxation factor is introduced to quantify the degree of relaxation of the timing dependency between tasks, and a timing dependency graph with relaxation factors is constructed in real time. The relaxation factor is dynamically calculated based on the time difference between the task end time and the subsequent task start time to reflect the time flexibility of task execution; Spatial constraint processing: Using the dynamic obstacle prediction model, the environment is rasterized in real time to generate a binary traffic matrix. The traffic status of each grid is dynamically updated based on historical data and the trend predicted by the exponential smoothing method to characterize the real-time traffic capacity of the spatial environment. Resource constraint processing: A nonlinear attenuation model is used to calculate the real-time consumption of agent resources by tasks, and a capacity balance table is constructed. The resource margin is adjusted in real time with the progress of task execution and load changes to track the resource status of each agent.

3. The intelligent target allocation method based on deep reinforcement learning according to claim 2 is characterized in that: Step S2 includes the following contents: Based on the timing dependency graph, adaptive Monte Carlo sampling is used to generate time domain task sequences. The sampling weight is dynamically adjusted according to the task dependency strength. The dependency strength is calculated by the weighted sum of the incoming edge relaxation factors, so that tasks with high dependency strength are sampled first. Based on the dynamic access matrix, the predictive-aware A* algorithm is used to generate spatial domain path fragments. The predictive-aware heuristic function is introduced. The predictive-aware heuristic function combines the traditional heuristic function with the cumulative probability of inaccessibility at future moments to optimize the adaptability of the path to the dynamic environment. Based on the capacity balance table, a dynamic threshold greedy strategy is used to allocate resource domain resources. The dynamic threshold is adjusted with the changes in the resource surplus and load of the agent. Resources are allocated only when the task resource demand is lower than the threshold. Agents with large resource surplus and low load are given priority. Through the multi-domain graph matching algorithm, the time, space and resource solution fragments are mapped to the multi-domain graph, and the compatibility score is calculated. The compatibility score is the weighted sum of the matching degree of time and space, space and resources, and resources and time. The combination scheme with the highest compatibility is screened out through maximum weight matching, and a set of candidate solutions is output.

4. The intelligent target allocation method based on deep reinforcement learning according to claim 3 is characterized in that: Step S3 includes the following contents: Mapping agents, tasks and resources into a multi-layer network, where nodes are agents, tasks and resources, and edges represent allocation relationships, dependency relationships and timing relationships. By calculating the impact strength of the change source on other nodes in the network when the constraint changes and iteratively propagating it, locate the agents whose impact strength exceeds the threshold to determine the set of affected agents; Based on the spatial position of the agent in the path fragment, a spatial topological relationship map is constructed, the affected agents are clustered, the center of the cluster space is calculated to determine the affected area, and the agents in the area are included in the affected set; the task sequence, path fragments and resource allocation of the affected agents are extracted from the candidate solutions, the local time, space and resource constraints are updated, and the constraint satisfaction solver is used to generate new solution fragments in the local range, which are combined with the solution fragments of the unaffected agents to form a new candidate solution; an incremental update strategy is adopted to replace only the solution fragments of the affected agents and retain the unaffected parts, and the cross-domain constraint compatibility of the updated solution is verified to output a new candidate solution set.

5. The intelligent target allocation method based on deep reinforcement learning according to claim 4 is characterized in that: Step S4 includes the following contents: Resource delay information and path efficiency information are obtained from the new candidate solution set, wherein the resource delay information includes a resource delay coupling index, and the path efficiency information includes a path efficiency skewness index.

6. The intelligent target allocation method based on deep reinforcement learning according to claim 4 is characterized in that: The logic for obtaining the resource delay coupling index is: For each solution in the new candidate solution set , calculate its total resource consumption and the total time delay ;in ,in Representing an Agent On Task Resource consumption in represents the set of all agents, Representing an Agent The set of all tasks that are assigned to be executed; in ,in For the task The actual completion time, is the ideal completion time of the task time window, Represents the set of all tasks; Defining resource delay coupling index : ,in A small positive number.

7. The intelligent target allocation method based on deep reinforcement learning according to claim 4 is characterized in that: The logic for obtaining the path efficiency skewness index is: For each solution in the new candidate solution set , calculate its total path length and the ideal path length ;in ,in For intelligent agents ; where ,in Representing an Agent The straight-line distance from the starting point to the end point, where It is an intelligent agent The starting point, It is an intelligent agent The end point of ; calculate the statistics of the path corners: is the corner sequence of all path segments in the solution, calculated from the spatial coordinates of the path segments in step S2; Indicates the extreme value of the turning angle, reflecting the degree of path tortuosity; represents the average turning angle; defines the path efficiency skewness index : ,in A small positive number.

8. The intelligent target allocation method based on deep reinforcement learning according to claim 4 is characterized in that: The optimal solution is selected from the set of candidate solutions using a multi-objective optimization method. The specific processing process is as follows: first, each candidate solution is mapped to a two-dimensional target space, and the corresponding two-dimensional target space is composed of a resource delay coupling index and a path efficiency skewness index; then, the Pareto frontier algorithm is applied to select the Pareto optimal solution set, that is, the solution that cannot be improved by other solutions at the same time in terms of the resource delay coupling index and the path efficiency skewness index; finally, the Pareto optimal solution set is output.

9. The intelligent target allocation method based on deep reinforcement learning according to claim 8, characterized in that: In order to ensure that the task execution timing of the Pareto optimal solution satisfies the dependency, a linear temporal logic model checker is used for verification. The specific processing process is as follows: first, a timing trajectory of task execution is generated for each Pareto optimal solution, including the start time and end time of each task; then, a linear temporal logic formula is defined to express the timing dependency between tasks; then, a linear temporal logic model checker is used to verify whether the timing trajectory of each solution satisfies all dependency constraints; finally, a final set of feasible solutions that have passed verification is output.

10. An intelligent target allocation system based on deep reinforcement learning, used to implement the intelligent target allocation method based on deep reinforcement learning according to any one of claims 1 to 9, characterized in that: include: Constraint decomposition modeling module, solution generation and verification module, local replanning module and solution optimization verification module; Constraint decomposition modeling module: decomposes the time constraints into a timing dependency graph with relaxation factors, where nodes represent task time windows and edges represent the order of tasks; Discretize the spatial constraints into a gridded binary access matrix, with each grid annotated with the dynamic obstacle status; transform the resource constraints into a capacity balance table, and record the resource surplus of each agent in real time; Solution generation and verification module: Generate task sequences through Monte Carlo sampling in the time domain, use the improved A* algorithm to generate path fragments in the space domain, and execute the greedy strategy to allocate resource packages in the resource domain; then combine the candidate fragments in the time, space and resource domains into a complete set of candidate solutions through the graph matching algorithm, verify their compatibility, and output the candidate solution set; Local replanning module: In response to environmental changes, a change impact propagation model is established to locate the set of agents affected by the constraint change based on the spatial topological relationship map; then, based on the task sequence, path segment and resource allocation data of the affected agents, the local constraint satisfaction solver is triggered to generate a new set of candidate solutions; Solution selection and verification module: extract resource delay information and path efficiency information from the new candidate solution set, and perform multi-objective optimization through Pareto front screening; then use the sequential logic verification tool to check the consistency of task dependencies in the solution, and finally output a set of conflict-free feasible solutions.

Images