Multi-target resource allocation method based on deep reinforcement learning

Abstract

The invention discloses a multi-target resource allocation method based on deep reinforcement learning. The method comprises the following steps: acquiring and processing real-time scene information; construction of a resource allocation multi-objective optimization mathematical model, including decision variable representation, objective function construction and constraint condition design; performing multi-objective optimization design based on deep reinforcement learning; in combination with an optimal resource allocation scheme obtained by a multi-target optimization algorithm based on deep reinforcement learning, radar and launch vehicle combinations in different time periods are allocated to incoming targets as many as possible for target interception, so that the purposes of maximizing the successful interception probability, minimizing the interception cost and achieving the highest resource utilization rate are achieved. Optimized targets cover the aspects of increasing the interception success rate, reducing resource consumption and the like, constraint conditions are considered at the same time, and a mixed integer linear model is formed to achieve optimal balance among multiple targets; the reinforcement learning method is modified and improved, and the stopping condition is met more accurately.

Description

Technical Field

[0001] The present invention belongs to the technical field of resource allocation, and in particular relates to a multi-objective resource allocation method based on deep reinforcement learning. Background Art

[0002] The resource allocation problem is essentially a constrained combinatorial optimization problem. Based on constraints such as attack time, confrontation resources, threat targets, and tactical requirements, the strike plan is optimized according to the environmental situation and the real-time status of the target to obtain the best strike result. The current solutions to this type of problem can be roughly divided into two categories:

[0003] The first category is improved versions of traditional solving algorithms, such as enhanced large-scale neighborhood search algorithms, rule-based methods, geometric methods, approximate dynamic programming, Lagrangian relaxation methods, game theory frameworks, etc. These algorithms can effectively solve problems of smaller dimensions, but it is difficult to solve high-dimensional problems.

[0004] The second category is heuristic optimization algorithms, such as efficient heuristic algorithms, precise heuristic algorithms, virtual mutation and taboo search heuristic algorithms, evolutionary heuristic algorithms, and efficient rule-based heuristic algorithms. These algorithms can improve optimization capabilities by using heuristic methods, but their global optimization capabilities are still weak. Intelligent optimization algorithms such as immune-based ant colony optimization algorithms, improved ant colony optimization methods based on dominance relationships, multi-objective optimization algorithms with adaptive allocation of decomposition vectors, and discrete particle swarm optimization belong to meta-heuristic algorithms, which can simulate the foraging and reproduction process of biological populations. These algorithms have good global optimization capabilities, but their convergence speed is slow.

[0005] In addition, with the development of artificial intelligence technology, deep learning technology has broken the barriers of traditional methods in many fields and achieved remarkable breakthroughs. In the field of computer vision, more than ten years ago, scholars mainly used manually designed algorithms for feature extraction and image processing, but now deep learning has become the current core method. Deep Neural Networks (DNN) can automatically learn the features of images, replacing human manual algorithm design. As another important branch of deep learning, deep reinforcement learning (DRL) is mainly used to make sequential decisions, that is, to make action choices based on the current environmental state, and continuously adjust its own strategy based on the feedback of the action, so as to achieve the set goal. In recent years, the performance of deep reinforcement learning on AlphaGo Zero, Atari and other problems has demonstrated its powerful learning ability and optimization decision-making ability. Traditional sequential decision optimization problems involve the optimal selection of decision variables in a discrete or continuous decision space, which is naturally similar to the "action selection" of reinforcement learning. The "offline training, online decision-making" characteristics of deep reinforcement learning make it possible to solve traditional sequential decision optimization problems in real time online. Compared with traditional iterative optimization algorithms, the DRL-based combinatorial optimization algorithm has a series of advantages such as fast solution speed and strong generalization ability. Once the model is trained, it can solve all problem instances with the same distribution characteristics without retraining, thus achieving "offline training, online decision-making". Using deep reinforcement learning methods to solve traditional optimization problems has broad prospects.

[0006] Compared with single-objective optimization problems, using deep reinforcement learning methods to solve multi-objective optimization problems has obvious challenges. It is necessary to design an effective mechanism that can output the Pareto optimal solution set of the problem in an end-to-end manner, rather than just outputting a single solution. In order to solve this problem, a decomposition strategy and a domain-based parameter migration strategy are used to model the multi-objective combinatorial optimization problem. First, the decomposition strategy is used to decompose the multi-objective optimization problem into multiple sub-problems. Each sub-problem is modeled as a neural network based on the pointer network model. Then, the neural network parameters of all sub-problems are collaboratively optimized according to the neighborhood-based parameter migration strategy and reinforcement learning training algorithm, thereby achieving the solution of the multi-objective combinatorial optimization problem.

[0007] Although the above-mentioned solution methods can improve the optimization capability, considering the strict requirements on the allocation time in actual scenarios, in order to shorten the solution time and improve the solution quality, it is necessary to further improve the optimization capability and timeliness of these algorithms. In addition, how to build a multi-objective optimization model that is closer to real problems is also an important basis for solving such optimization problems. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a new reinforcement learning solution method for complex multi-objective resource allocation problems, to give full play to the self-training and evolution advantages of the intelligent agent of the reinforcement learning method, to achieve autonomous learning of the optimal resource allocation plan without the intervention of preset empirical rules, and to solve the problems of slow combination selection decision-making and poor real-time performance under time-sensitive tasks.

[0009] The technical solution is to first obtain real-time environmental information and perform data preprocessing on various types of information in the environment. Then, a multi-objective resource allocation optimization model is constructed in combination with actual confrontation requirements. Then, for the multi-objective optimization scenario generated by the resource allocation solution, a decomposition strategy and a field-based parameter migration strategy are used to model the multi-objective combination optimization problem. Finally, the optimal matching solution of "target-radar-launch vehicle" is output.

[0010] To achieve the above object, the multi-objective resource allocation method based on deep reinforcement learning disclosed in the present invention comprises the following steps:

[0011] Step 1: Real-time scene information acquisition and processing: First, the scene information is acquired through the final control radar; second, the target trajectory and flight time are calculated based on the scene information and the trajectory is discretized according to the preset control time interval; then, the positional relationship between the trajectory and the radar detection area and the launch vehicle air defense area is determined, so as to calculate the radar and launch vehicle combination that can be used for each target and the corresponding hit probability;

[0012] Step 2: Construction of mathematical model for multi-objective optimization of resource allocation: including: representation of decision variables, construction of objective function, and design of constraint conditions;

[0013] Step 3: Multi-objective optimization design based on deep reinforcement learning: Construct a basic confrontation management model consisting of three types of nodes: "target-radar-launcher", and mathematically model the resource allocation problem; according to the specific characteristics of the resource allocation problem, construct a deep neural network model based on the multi-head attention mechanism; adopt the decomposition strategy and the field-based parameter migration strategy to model the multi-objective combinatorial optimization problem. First, the decomposition strategy is used to decompose the multi-objective optimization problem into multiple sub-problems. Each sub-problem is modeled as a neural network based on the pointer network model. Then, the neural network parameters of all sub-problems are collaboratively optimized according to the neighborhood-based parameter migration strategy and reinforcement learning training algorithm, so as to realize the generation of multi-objective resource allocation solutions;

[0014] Step 4: Combine the optimal resource allocation plan obtained by the multi-objective optimization algorithm based on deep reinforcement learning to allocate radar and launch vehicle combinations at different times to intercept as many incoming targets as possible, so as to maximize the probability of successful interception, minimize the interception cost and achieve the highest resource utilization.

[0015] Furthermore, the scenario information includes the number, location information, type and launch angle of the target; the hit probability, interception resources and radar combination costs are normalized to obtain standardized data with consistent dimensions.

[0016] Furthermore, the decision variable representation includes:

[0017] Define the task set T and resource set R, where T contains all the enemy targets to be intercepted and their attributes, and R includes all available interception resources and their characteristics; introduce a binary decision variable x for each pair of targets and resources ij , and organized into a decision matrix to express the configuration relationship between tasks and resources; the optimization objectives include improving the interception success rate and reducing resource consumption, while considering the constraints, and finally forming a mixed integer linear program to achieve the optimal balance among multiple objectives.

[0018] Furthermore, the objective function construction includes:

[0019] Maximizing interception success rate: Each successful interception mission directly contributes to the security of the defense system;

[0020] Maximizing the number of interception targets: In some cases, the threat level or importance between different targets may be difficult to quantify and compare;

[0021] Maximizing resource utilization efficiency: Resource utilization efficiency reflects the ratio of the benefits obtained to the input resources, and the input resources include interception resources and radar usage time.

[0022] Furthermore, the constraint design includes:

[0023] Cumulative constraints: This type of constraint is used to ensure the effective operation and protection of interception resources within the planning cycle and avoid failures caused by overload operation, including radar channel occupation resource constraints and launch vehicle use resource constraints; the radar channel occupation resource constraints involve limiting the number of targets that a single radar or radar network can handle within a specific time; the launch vehicle use resource constraints involve limiting the number of interception resources loaded and launched by the launch vehicle;

[0024] Rolling constraints: Rolling constraints focus on intercepting the load intensity of the system in the short term to prevent the system from being overloaded;

[0025] Mission attribute constraints: Mission attribute constraints ensure that the operation of the interception mission complies with the physical limitations of the actual environment;

[0026] Task dependency constraints: Dependency constraints between tasks ensure that the planned interception tasks do not conflict in time and logic.

[0027] Furthermore, the mathematical formulation of the multi-objective optimization problem is as follows:

[0028] Maximize the total interception probability:

[0029]

[0030] P i is the comprehensive interception probability of target i, is the probability that target i is intercepted by launch vehicle l at time t, is a decision variable. It is 1, indicating that target i is intercepted by the interception resource launched by launch vehicle l at time t, otherwise it is 0. I is the set of targets to be attacked, I = {1, 2, 3, ..., n1}, n1 is the total number of targets to be attacked, n2 is the total number of launch vehicles, is the amount of resources transmitted to target i at time t, is the maximum flight time of target i;

[0031] Maximize interception value:

[0032]

[0033] L is the set of launching vehicles, L = {1, 2, 3, ..., n2}, ζ i is the interception value of target i;

[0034] Minimize interception cost:

[0035]

[0036] c l The value of interception resources equipped for launch vehicle l;

[0037] A typical constraint is expressed as:

[0038] The number of interception resources launched at target i at time t is

[0039] T i is the set of interceptable time nodes of target i;

[0040] The number of interception resources launched to target i does not exceed the upper limit of the constraint

[0041] q max The maximum number of interception resources launched to the target;

[0042] The number of interception resources launched to launch vehicle l cannot exceed the upper limit

[0043] θ l The number of interception resources equipped for launch vehicle l;

[0044] The time interval constraint between multiple interception resources launched at target i is

[0045]

[0046] is the moment when the pth interception resource hits target i, is the moment when the p+1th interception resource hits the target i, and η is the maximum interval between the moments when multiple interception resources are launched towards the target;

[0047] The number of channels used by radar r at any time t does not exceed the upper limit:

[0048]

[0049] is the moment when the pth interception resource is launched towards target i, R is the radar set, R = {1,2,3,…,n3}, μ is the time discretization standard, λ i,r is the number of channels occupied by target i on radar r, is the maximum number of channels of radar r;

[0050] The radar is not switched before the target is intercepted to avoid interference with the terminal interception:

[0051]

[0052] is a decision variable, which is 1 if target i is tracked by radar r at time t, otherwise it is 0.

[0053] Furthermore, a basic confrontation management model consisting of three types of nodes, namely “target-radar-launcher”, is constructed, and on this basis, a mathematical modeling of the resource allocation problem is carried out, including:

[0054] For the resource allocation solution generation problem s, the parameterization of the model is as follows: In the resource allocation problem, the feature x of node i is i Defined by its geographical location, i.e. its x- and y-coordinates, we can solve for π = (π1,...,π k , k≤n) represents the resource allocation plan; in order to solve the resource allocation problem, a deep neural network model with parameter θ is designed. The model takes the node position of the resource allocation problem s as input and outputs the plan π. The model strategy is defined as p(π|s) to achieve the mapping from s to π:

[0055]

[0056] Specifically, given a resource allocation optimization problem instance s, a deep neural network model parameterized by parameter θ is defined, which produces a conditional probability distribution p θ (π|s), solving π|s is done by starting from p θ (π|s) is sampled from the conditional probability distribution;

[0057] According to the specific characteristics of the resource allocation problem, a deep neural network model based on the multi-head attention mechanism is constructed, including:

[0058] To maximize the interception probability L(π), the above deep neural network model is trained using the turn-based REINFORCE algorithm, which uses a turn-based Monte Carlo method to calculate the reward value, that is, the strategy is executed until a complete solution is constructed, at which time the total objective function is calculated as the reward L(π). Based on this definition, the strategy parameters are continuously updated through state-action-reward;

[0059] Specifically, given the action π|s and the corresponding reward value / loss function L(π), based on the REINFORCE algorithm, the model parameters θ are updated by gradient descent:

[0060]

[0061] The construction of the resource allocation scheme samples a large number of actions from the probability distribution. Due to the uncertainty of sampling and the sampling process in each round, the rewards obtained in different rounds have variances. This variance will cause conflicts in the gradient descent direction of the policy parameters and affect the convergence speed. In order to reduce the variance of the policy parameter update, the baseline value b(s) is introduced to update the policy gradient formula:

[0062]

[0063] b(s) is used to evaluate the average performance of the strategy. If the performance of the current strategy is better than the average level b(s), L(π)-b(s) is a negative number, and the strategy is positively incentivized. If the performance of the current strategy is worse than the average level b(s), L(π)-b(s) is a positive number, and the strategy is negatively incentivized.

[0064] Given a resource allocation problem instance s, b(s) is defined as using the baseline policy p θ The objective function obtained by solving the problem instance s is a greedy strategy for sampling probability distribution during the solution process: the baseline strategy p θ*At the end of each training cycle, the strategy is evaluated on the validation set. Once the current strategy is better than the current baseline strategy, the baseline strategy is replaced by the current model, so that the baseline strategy is the best model trained so far. b(s) is the current best strategy performance, and the strategy gradient update is optimized through this benchmark performance.

[0065] The decomposition strategy is to decompose the multi-objective optimization problem using the Weighted Sum method, the Chebyshev method or the penalty-based boundary crossing method.

[0066] Furthermore, the neighborhood-based parameter migration strategy includes:

[0067] Since the subproblems are modeled as deep neural networks, the neural network model parameters of the (i-1)th subproblem are expressed as [ω * ,b * ] represents the optimal neural network model parameters, and [ω,b] represents the neural network model parameters that have not yet been optimized; assuming that the (i-1)th subproblem is solved, that is, its neural network parameters have been optimized to near the optimal level, then the model parameters of the (i-1)th subproblem are Set as the initial parameters of the neural network model of the i-th sub-problem, and start training from the initial parameters, that is, migrate the neural network model parameters of the neighborhood sub-problem to the next adjacent sub-problem, thereby reducing the model training time.

[0068] The present invention provides a multi-objective resource allocation method based on deep reinforcement learning, which can achieve the following beneficial effects:

[0069] 1. This paper proposes a multi-objective optimization modeling method for resource allocation schemes. This model expresses the configuration relationship between tasks and resources. The optimization objectives include improving the interception success rate, reducing resource consumption, etc., while considering constraints such as resource capacity, state and time window, and finally forming a mixed integer linear model to achieve the optimal balance between multiple objectives.

[0070] 2. The present invention solves the problems of traditional optimization methods, such as slow response speed, easy to fall into local optimum, difficulty in handling a large number of constraints, and difficulty in meeting the stopping conditions. The reinforcement learning method is modified and improved by optimizing the algorithm structure, enhancing the constraint processing capability, improving the stopping conditions, and improving the robustness. These improvements enable the algorithm to handle complex constraints more efficiently, meet the stopping conditions more accurately, and show better adaptability and flexibility in a variety of tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is the overall structural diagram of the present invention;

[0072] Figure 2 A framework diagram of the algorithm for generating a multi-objective resource allocation solution based on deep reinforcement learning designed for the present invention;

[0073] Figure 3 Schematic diagram of the decomposition strategy in the reinforcement learning training designed for the present invention;

[0074] Figure 4 Schematic diagram of parameter migration strategy in reinforcement learning training designed for the present invention;

[0075] Figure 5 Flowchart of the multi-objective optimization algorithm based on deep reinforcement learning designed for the present invention;

[0076] Figure 6 This is a schematic diagram of a resource allocation solution obtained by using the reinforcement learning method designed by the present invention. DETAILED DESCRIPTION

[0077] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0078] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0079] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0080] The technical solutions provided in the embodiments of the present application involve technologies such as machine learning and natural language processing of artificial intelligence, which are specifically introduced and explained through the following embodiments.

[0081] This paper proposes a multi-objective resource allocation method based on deep reinforcement learning. The overall structure is as follows Figure 1 As shown, the specific steps are as follows:

[0082] Step 1: Real-time scene information acquisition and processing. First, the final control radar is used to obtain scene information such as the number, location information, type, and launch angle of the incoming targets. Secondly, the target trajectory and flight time are calculated based on the summarized environmental information, and the trajectory is discretized according to the preset control time interval. Subsequently, the positional relationship between the trajectory and the radar detection area and the launch vehicle's air defense area is determined to calculate the radar and launch vehicle combination that can be used for each target and the corresponding hit probability. Finally, in order to facilitate the solution of the optimization problem, the present invention also normalizes the hit probability, interception resources, and the cost of the radar combination to obtain standardized data with consistent dimensions.

[0083] Step 2: Construction of a mathematical model for multi-objective optimization of resource allocation, specifically:

[0084] (1) Decision variable representation

[0085] When constructing a multi-objective optimization model for resource allocation, we first define the task set T and resource set R, where T contains all the enemy targets to be intercepted and their attributes, and R includes all available interception resources and their characteristics. By introducing a binary decision variable x for each pair of (target, resource) combinations ij , and organized into a decision matrix. This model expresses the configuration relationship between tasks and resources. The optimization objectives include improving the interception success rate and reducing resource consumption, while considering constraints such as resource capacity, state and time window, and finally forming a mixed integer linear program to achieve the optimal balance between multiple objectives.

[0086] (2) Objective function construction

[0087] When building a multi-objective optimization model for resource allocation, the objective function is a key indicator to measure the quality of the solution. Based on certain assumptions, the objective function is regarded as a deterministic function of the decision variables, that is, for a fixed set of decision variables, the value of the objective function is uniquely determined. The following are several typical objective functions and their basic forms applied to the field:

[0088] Maximizing the interception success rate: This means that each successfully intercepted mission directly contributes to the security of the defense system. Therefore, maximizing the total benefit of all successful interception missions is the most intuitive and commonly used objective function in confrontation.

[0089] Maximizing the number of interception targets: In some cases, the threat level or importance between different targets may be difficult to quantify and compare. In this case, maximizing the number of interception targets can be selected as an indicator to evaluate the effectiveness of the scheduling scheme, where each target is considered equally important.

[0090] Maximizing resource utilization efficiency: Resource utilization efficiency reflects the ratio of benefits gained to resources invested (such as interception resources, radar usage time, etc.). In confrontation, this usually involves how to achieve the greatest effect with the least resource consumption.

[0091] The above objective function provides a multi-dimensional evaluation framework for resource allocation, enabling decision makers to develop the most appropriate resource allocation plan based on different environmental requirements and resource constraints.

[0092] (3) Constraint design

[0093] In the multi-objective optimization modeling of resource allocation problems, constraints are one of the key factors to ensure effective, practical and safe countermeasure solutions. Specifically, the constraints of resource allocation solutions can be divided into the following four categories:

[0094] Cumulative constraints: This type of constraint is mainly used to ensure the effective operation and protection of interception resources within the planning cycle and avoid failures caused by overload operation. The two main typical types of constraints are radar channel occupancy resource constraints and launch vehicle usage resource constraints. Specifically, it can be described as: Radar channel occupancy resource constraints: Radar channel occupancy resource constraints involve limiting the number of targets that a single radar or radar network can handle within a specific time. This helps prevent radar system overload and ensures that all targets can be tracked with sufficient accuracy and continuity for effective target identification and interception. Launch vehicle usage resource constraints: Launch vehicle usage resource constraints involve restrictions on the number of launch vehicle loading and launch interception resources. This helps ensure that in continuous defensive confrontation, the launch vehicle can effectively replenish resources and maintain confrontation capabilities.

[0095] Rolling constraints: Rolling constraints focus on the load intensity of the interception system in the short term to prevent the system from being overloaded. For example, you can set "a single interception system can only conduct a certain number of interceptions in any consecutive minutes" or "the total launch time of the system cannot exceed a certain number of seconds in any consecutive seconds" to protect the equipment from overheating or excessive wear.

[0096] Mission attribute constraints: Mission attribute constraints ensure that the operation of the interception mission complies with the physical limitations of the actual environment. For example, "the minimum warning time for any interception mission must not be less than a few seconds", or "under certain electronic jamming conditions, only certain types of interception resources are allowed to be used for interception".

[0097] Task relevance constraints: Task relevance constraints ensure that the planned interception tasks do not conflict in time and logic. For example, "at least a few minutes must pass between two interception tasks to re-prepare the interception system", or "the same target cannot be attacked by two different interception systems at the same time", and "tasks to ensure key areas such as command centers or important facilities are executed first".

[0098] These constraints ensure that the resource allocation plan is not only optimal in theory, but also feasible and effective in the actual environment, and protects interception resources from overuse and potential damage. These constraints help decision makers make fast and reliable decisions in changing scenarios and maximize the overall defense effectiveness.

[0099] Taking the minimization problem as an example, the mathematical expression of the multi-objective optimization problem is as follows:

[0100]

[0101] in, is the objective function to be optimized, which can be any of the typical objective functions mentioned above. is the optimization variable, g i (·) and h j (·) is the constraint function that the objective optimization variable needs to satisfy. When m=1, it is a single-objective optimization problem, and when m≥2, it is a multi-objective optimization problem. In particular, multi-objective optimization problems with at least 4 objectives are called multi-objective optimization problems.

[0102] Combining the above definitions of objective functions and constraints, a generalized multi-objective mathematical optimization model for solving resource allocation problems is constructed.

[0103]

[0104]

[0105] In actual confrontation, due to the influence of factors such as the different flight times of incoming targets and the different flight destinations of the targets, the constructed mathematical model needs to convert the above factors into the interception priority of the target for optimization. Therefore, the three typical optimization goals of this application can be expressed as:

[0106] Maximize the total interception probability:

[0107] Maximize interception value:

[0108] Minimize interception cost:

[0109] A typical constraint can be expressed as:

[0110] The number of interception resources launched at target i at time t is

[0111] The number of interception resources launched to target i does not exceed the upper limit of the constraint

[0112] The number of interception resources launched to launch vehicle l cannot exceed the upper limit

[0113] The time interval constraint between multiple interception resources launched at target i is

[0115] The number of channels used by radar r at any time t does not exceed the upper limit:

[0116]

[0117] The radar is not switched before the target is intercepted to avoid interference with the terminal interception:

[0118]

[0119] Step 3: Multi-objective optimization design based on deep reinforcement learning.

[0120] Different from traditional combinatorial optimization methods, the use of artificial intelligence methods to solve combinatorial optimization methods involves a large number of new technologies, new concepts, and new ideas. Therefore, we first study the scientific principles of solving combinatorial optimization problems based on deep reinforcement learning, and propose a deep reinforcement learning method for multi-objective resource allocation problems.

[0121] First, a basic model of confrontation management consisting of three types of nodes, "target-radar-launcher", is constructed, and on this basis, the resource allocation problem is mathematically modeled; secondly, according to the specific characteristics of the resource allocation problem, a deep neural network model based on the multi-head attention mechanism is constructed, and a deep reinforcement learning algorithm for training the model parameters is proposed, and the "offline training, online optimization" method is adopted to achieve rapid generation of resource allocation plans. For the multi-objective optimization scenario of resource allocation plan generation, the decomposition strategy and the field-based parameter migration strategy are used to model the multi-objective combinatorial optimization problem. First, the decomposition strategy is used to decompose the multi-objective optimization problem into multiple sub-problems. Each sub-problem is modeled as a neural network based on the pointer network model. Then, the neural network parameters of all sub-problems are collaboratively optimized according to the neighborhood-based parameter migration strategy and reinforcement learning training algorithm, so as to achieve multi-objective resource allocation plan generation. The algorithm framework for generating multi-objective resource allocation plans based on deep reinforcement learning is shown in the figure below.

[0122] (1) Parametric modeling

[0123] For the resource allocation solution generation problem s, the parameterization of the model is as follows: In the resource allocation problem, the feature x of node i is i It is defined by its geographical location and other features. For the convenience of expression, the geographical location is used as the feature, that is, its x-coordinate and y-coordinate. Solve π=(π1,...,π k ,k≤n) represents the resource allocation plan. In order to solve the resource allocation problem, it is necessary to design a deep neural network model with parameter θ. The model takes the node position of the resource allocation problem s as input and outputs the plan π. The model strategy is defined as p(π|s), which can realize the mapping from s to π:

[0124]

[0125] Specifically, given a resource allocation optimization problem instance s, a deep neural network model parameterized by parameter θ is defined, which produces a conditional probability distribution p θ (π|s). Solving π|s can be done from p θ (π|s) conditional probability distribution. Since the optimization goal of the resource allocation problem is to maximize the interception probability, maximize the interception value, and minimize the interception cost, etc. Taking the maximization of the interception probability L(π) as an example, it is suitable to use the turn-based REINFORCE algorithm to train the above deep neural network model. The algorithm uses the turn-based Monte Carlo method to calculate the reward value, that is, the strategy is executed until a complete solution is constructed, at which time the total objective function is calculated as the reward L(π). Based on this definition, the strategy parameters can be continuously updated through state-action-reward.

[0126] Specifically, given an action π|s and the corresponding reward value / loss function L(π), the model parameters θ can be updated by gradient descent based on the REINFORCE algorithm:

[0127]

[0128] The construction of the resource allocation scheme requires sampling a large number of actions from the probability distribution. Due to the uncertainty of sampling and the large number of sampling processes in each round, the rewards obtained in different rounds have a large variance, which will cause conflicts in the gradient descent direction of the policy parameters and affect the convergence speed. In order to reduce the variance of the policy parameter update, the baseline value b(s) is introduced to update the policy gradient formula:

[0129]

[0130] b(s) can be regarded as the average performance used to evaluate the strategy. If the performance of the current strategy is better than the average level b(s), L(π)-b(s) is a negative number, and the strategy is positively motivated (minimizing the loss function). If the performance of the current strategy is worse than the average level b(s), L(π)-b(s) is a positive number, then the strategy is negatively motivated. The current mainstream method is to train an additional neural network to approximate b(s), namely the Critic network. Its parameters are trained using (s,L(π)) collected during the training process. However, simultaneously training the strategy parameterized by the neural network and the Critic neural network will result in slow convergence. Therefore, this patent adopts the greedy rollout baseline method to calculate b(s), which has been proven in experiments to be superior to the training method based on the Critic neural network.

[0131] Specifically, there is no need to design an additional neural network to calculate b(s). In the patent, given a resource allocation problem instance s, b(s) is defined as using the benchmark strategy p θ The objective function obtained by solving the problem instance s is a greedy strategy for sampling probability distribution during the solution process. θ* is the best model so far in the training process, that is, the strategy is evaluated on the validation set at the end of each training cycle. Once the current strategy is better than the current baseline strategy, the baseline strategy is replaced by the current model, so that the baseline strategy is the best model trained so far. b(s) is the current optimal strategy performance, and the policy gradient update is optimized through the benchmark performance.

[0132] (2) Reinforcement Learning Training

[0133] Decomposition strategy: Decomposition is an effective method for solving multi-objective optimization problems. There are already a large number of traditional multi-objective optimization algorithms designed using decomposition strategies, such as MOGA, MOEA / D, MOEA / DD, and NSGA-III. The decomposition strategy is used to model the resource allocation problem, decomposing the resource allocation problem into a set of scalar optimization sub-problems, solving the sub-problems in a collaborative optimization manner, and obtaining the optimal solution for each sub-problem. The solutions of all sub-problems constitute the Pareto Front (PF) of the original problem.

[0134] Specifically, the widely used Weighted Sum method is used to decompose the original problem. Other methods, such as the Chebyshev method and the penalty-based boundary intersection (PBI) method, can also be used to decompose the multi-objective optimization problem. The decomposition method used is as follows: First, a set of uniformly distributed weight vectors λ is given. 1 ,...,λ N For example, for a resource allocation problem with two optimization objectives, the weight vector is (1,0), (0.9, 0.1), ..., (0,1), where Where M represents the number of objectives. Therefore, the original resource allocation problem can be converted into N scalar optimization sub-problems through the Weighted Sum method, and the objective function of the j-th sub-problem is as follows:

[0135]

[0136] Therefore, the Pareto front can be composed of N solutions obtained by solving N sub-problems. The process is shown in the figure below. The dotted line represents the direction of sub-problem optimization. This method decomposes the original resource allocation problem into N sub-problems, and further uses deep neural networks to model the sub-problems, thereby achieving the solution of multi-objective combinatorial optimization problems.

[0137] Neighborhood-based parameter migration strategy: In order to solve each subproblem through deep reinforcement learning methods, it is necessary to decompose the complex resource allocation optimization problem into N subproblems, and then model the N subproblems as deep neural networks. Finally, the N scalar optimization subproblems are solved in a collaborative optimization manner through a neighborhood-based parameter migration strategy. The domain-based parameter migration strategy is introduced as follows.

[0138] By observing the Weighted Sum formula, it can be seen that for two adjacent sub-problems, since their weight vectors are similar, their optimal solutions are similar. Based on this fact, each sub-problem can be solved by drawing on the information of its adjacent sub-problems.

[0139] Since the subproblems are modeled as deep neural networks, the neural network model parameters of the (i-1)th subproblem can be expressed as [ω * ,b * ] represents the optimal neural network model parameters, and [ω,b] represents the neural network model parameters that have not yet been optimized. Assuming that the (i-1)th subproblem has been solved, that is, its neural network parameters have been optimized to near the optimal level, then the model parameters of the (i-1)th subproblem can be Set as the initial parameters of the neural network model of the ith subproblem, and start training from the initial parameters, that is, migrate the neural network model parameters of the neighborhood subproblem to the next adjacent subproblem, thereby greatly reducing the model training time. There are many migration methods. The most basic parameter migration method is used in the technology, that is, the neural network parameters are migrated from the previous subproblem to the next subproblem in sequence. The process is as follows Figure 4 shown.

[0140] If the neighborhood-based parameter migration strategy is not adopted, training N sub-problems modeled as neural network models will take a lot of time and cannot be used in practice. The neighborhood-based parameter migration strategy makes the multi-objective combinatorial optimization method practically applicable.

[0141] (3) Resource allocation plan generation

[0142] In summary, the resource allocation problem is decomposed into multiple sub-problems, each of which is modeled by a deep neural network. The sub-problems are solved by reinforcement learning methods. All sub-problems can be collaboratively optimized through parameter migration strategies. The Pareto frontier can be output end-to-end through the obtained neural network model set. Combining the decomposition strategy and the neighborhood-based parameter migration strategy, the multi-objective combinatorial optimization method based on deep reinforcement learning has the following process: Figure 5 shown.

[0143] Step 4: Generate resource allocation plan.

[0144] Combined with the optimal resource allocation plan obtained by the multi-objective optimization algorithm based on deep reinforcement learning, radar and launch vehicle combinations at different times are allocated to intercept as many incoming targets as possible, thereby achieving the goals of maximizing the probability of successful interception, minimizing the interception cost and maximizing resource utilization.

[0145] The present invention provides a multi-objective resource allocation method based on deep reinforcement learning, which can achieve the following beneficial effects:

[0146] 1. This paper proposes a multi-objective optimization modeling method for resource allocation schemes. This model expresses the configuration relationship between tasks and resources. The optimization objectives include improving the interception success rate, reducing resource consumption, etc., while considering constraints such as resource capacity, state and time window, and finally forming a mixed integer linear model to achieve the optimal balance between multiple objectives.

[0147] 2. The present invention solves the problems of traditional optimization methods, such as slow response speed, easy to fall into local optimum, difficulty in handling a large number of constraints, and difficulty in meeting the stopping conditions. The reinforcement learning method is modified and improved by optimizing the algorithm structure, enhancing the constraint processing capability, improving the stopping conditions, and improving the robustness. These improvements enable the algorithm to handle complex constraints more efficiently, meet the stopping conditions more accurately, and show better adaptability and flexibility in a variety of tasks.

[0148] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0149] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0150] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0151] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A multi-objective resource allocation method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Real-time scene information acquisition and processing: First, the scene information is acquired through the final control radar; second, the target trajectory and flight time are calculated based on the scene information and the trajectory is discretized according to the preset control time interval; then, the positional relationship between the trajectory and the radar detection area and the launch vehicle air defense area is determined, so as to calculate the radar and launch vehicle combination that can be used for each target and the corresponding hit probability; Step 2: Construction of mathematical model for multi-objective optimization of resource allocation: including: representation of decision variables, construction of objective function, and design of constraint conditions; Step 3: Multi-objective optimization design based on deep reinforcement learning: Construct a basic confrontation management model consisting of three types of nodes: "target-radar-launcher", and mathematically model the resource allocation problem; according to the specific characteristics of the resource allocation problem, construct a deep neural network model based on the multi-head attention mechanism; adopt the decomposition strategy and the field-based parameter migration strategy to model the multi-objective combination optimization problem. First, the decomposition strategy is used to decompose the multi-objective optimization problem into multiple sub-problems. Each sub-problem is modeled as a neural network based on the pointer network model. Then, the neural network parameters of all sub-problems are collaboratively optimized according to the neighborhood-based parameter migration strategy and reinforcement learning training algorithm, so as to realize the generation of multi-objective resource allocation solutions; Step 4: Combine the optimal resource allocation plan obtained by the multi-objective optimization algorithm based on deep reinforcement learning to allocate radar and launch vehicle combinations at different times to intercept as many incoming targets as possible, so as to maximize the probability of successful interception, minimize the interception cost and achieve the highest resource utilization.

2. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The scenario information includes the number, location information, type and launch angle of the target; the hit probability, interception resources and radar combination costs are normalized to obtain standardized data with consistent dimensions.

3. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The decision variables are represented as follows: Define the task set T and resource set R, where T contains all the enemy targets to be intercepted and their attributes, and R includes all available interception resources and their characteristics; introduce a binary decision variable x for each pair of targets and resources ij , and organized into a decision matrix to express the configuration relationship between tasks and resources; the optimization objectives include improving the interception success rate and reducing resource consumption, while considering the constraints, and finally forming a mixed integer linear program to achieve the optimal balance among multiple objectives.

4. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The objective function construction includes: Maximizing interception success rate: Each successful interception mission directly contributes to the security of the defense system; Maximizing the number of interception targets: In some cases, the threat level or importance between different targets may be difficult to quantify and compare; Maximizing resource utilization efficiency: Resource utilization efficiency reflects the ratio of the benefits obtained to the input resources, and the input resources include interception resources and radar usage time.

5. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The constraint design includes: Cumulative constraints: This type of constraint is used to ensure the effective operation and protection of interception resources within the planning cycle and avoid failures caused by overload operation, including radar channel occupation resource constraints and launch vehicle use resource constraints; the radar channel occupation resource constraints involve limiting the number of targets that a single radar or radar network can handle within a specific time; the launch vehicle use resource constraints involve limiting the number of interception resources loaded and launched by the launch vehicle; Rolling constraints: Rolling constraints focus on intercepting the load intensity of the system in the short term to prevent the system from being overloaded; Mission attribute constraints: Mission attribute constraints ensure that the operation of the interception mission complies with the physical limitations of the actual environment; Task dependency constraints: Dependency constraints between tasks ensure that the planned interception tasks do not conflict in time and logic.

6. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The mathematical expression of the multi-objective optimization problem is as follows: Maximize the total interception probability: P i is the comprehensive interception probability of target i, is the probability that target i is intercepted by launch vehicle l at time t, is a decision variable. It is 1, indicating that target i is intercepted by the interception resource launched by launch vehicle l at time t, otherwise it is 0. I is the set of targets to be attacked, I = {1, 2, 3, ..., n1}, n1 is the total number of targets to be attacked, n2 is the total number of launch vehicles, is the amount of resources transmitted to target i at time t, is the maximum flight time of target i; Maximize interception value: L is the set of launching vehicles, L = {1, 2, 3, ..., n2}, ζ i is the interception value of target i; Minimize interception cost: c l The value of interception resources equipped for launch vehicle l; A typical constraint is expressed as: The number of interception resources launched at target i at time t is T i is the set of interceptable time nodes of target i; The number of interception resources launched to target i does not exceed the upper limit of the constraint q max The maximum number of interception resources launched to the target; The number of interception resources launched to launch vehicle l cannot exceed the upper limit θ l The number of interception resources equipped for launch vehicle l; The time interval constraint between multiple interception resources launched at target i is is the moment when the pth interception resource hits target i, is the moment when the p+1th interception resource hits the target i, and η is the maximum interval between the moments when multiple interception resources are launched towards the target; The number of channels used by radar r at any time t does not exceed the upper limit: is the moment when the pth interception resource is launched towards target i, R is the radar set, R = {1,2,3,…,n3}, μ is the time discretization standard, λ i,r is the number of channels occupied by target i on radar r, is the maximum number of channels of radar r; The radar is not switched before the target is intercepted to avoid interference with the terminal interception: is a decision variable, which is 1 if target i is tracked by radar r at time t, otherwise it is 0.

7. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: A basic confrontation management model consisting of three types of nodes, namely "target-radar-launcher", is constructed, and mathematical modeling of resource allocation problems is carried out on this basis, including: For the resource allocation solution generation problem s, the parameterization of the model is as follows: In the resource allocation problem, the feature x of node i is i Defined by its geographical location, i.e. its x- and y-coordinates, we can solve for π = (π1,…,π k ,k≤n) represents the resource allocation scheme, and n is the maximum number of allocation schemes. In order to solve the resource allocation problem, a deep neural network model with parameter θ is designed. The model takes the node position of the resource allocation problem s as input and outputs the scheme π. The model strategy is defined as p(π|s) to achieve the mapping from s to π: Specifically, given a resource allocation optimization problem instance s, a deep neural network model parameterized by parameter θ is defined, which produces a conditional probability distribution p θ (π|s), solving π|s is done by starting from p θ (π|s) is sampled from the conditional probability distribution; According to the specific characteristics of the resource allocation problem, a deep neural network model based on the multi-head attention mechanism is constructed, including: To maximize the interception probability L(π), the above deep neural network model is trained using the turn-based REINFORCE algorithm, which uses a turn-based Monte Carlo method to calculate the reward value, that is, the strategy is executed until a complete solution is constructed, at which time the total objective function is calculated as the reward L(π). Based on this definition, the strategy parameters are continuously updated through state-action-reward; Specifically, given the action π|s and the corresponding reward value / loss function L(π), based on the REINFORCE algorithm, the model parameters θ are updated by gradient descent: The construction of the resource allocation scheme samples a large number of actions from the probability distribution. Due to the uncertainty of sampling and the sampling process in each round, the rewards obtained in different rounds have variances, which causes conflicts in the gradient descent direction of the policy parameters and affects the convergence speed. In order to reduce the variance of the policy parameter update, the baseline value b(s) is introduced to update the policy gradient formula: b(s) is used to evaluate the average performance of the strategy. If the performance of the current strategy is better than the average level b(s), L(π)-b(s) is a negative number, and the strategy is positively incentivized. If the performance of the current strategy is worse than the average level b(s), L(π)-b(s) is a positive number, and the strategy is negatively incentivized. Given a resource allocation problem instance s, b(s) is defined as using the baseline policy p θ The objective function obtained by solving the problem instance s is a greedy strategy for sampling probability distribution during the solution process: the baseline strategy p θ* At the end of each training cycle, the strategy is evaluated on the validation set. Once the current strategy is better than the current baseline strategy, the baseline strategy is replaced by the current model, so that the baseline strategy is the best model trained so far. b(s) is the current best strategy performance, and the strategy gradient update is optimized through this benchmark performance. The decomposition strategy is to decompose the multi-objective optimization problem using the Weighted Sum method, the Chebyshev method or the penalty-based boundary crossing method.

8. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: Neighborhood-based parameter migration strategies include: Since the subproblems are modeled as deep neural networks, the neural network model parameters of the (i-1)th subproblem are expressed as [ω * ,b * ] represents the optimal neural network model parameters, and [ω,b] represents the neural network model parameters that have not yet been optimized; assuming that the (i-1)th subproblem is solved, that is, its neural network parameters have been optimized to near the optimal level, then the model parameters of the (i-1)th subproblem are Set as the initial parameters of the neural network model of the i-th sub-problem, and start training from the initial parameters, that is, migrate the neural network model parameters of the neighborhood sub-problem to the next adjacent sub-problem, thereby reducing the model training time.

Images