A multi-objective resource allocation method based on deep reinforcement learning

Abstract

The application discloses a kind of multi-objective resource allocation methods based on deep reinforcement learning, including steps: real-time scene information acquisition and processing;Resource allocation multi-objective optimization mathematical model construction: including decision variable representation, objective function construction, constraint condition design;Multi-objective optimization design based on deep reinforcement learning;The optimal resource allocation scheme obtained by combining the multi-objective optimization algorithm based on deep reinforcement learning is allocated to as many incoming targets as possible in different periods of time to intercept targets with radar and launch vehicle combination, so as to realize the purpose of maximizing the success interception probability, minimizing the interception cost and the highest resource utilization rate.The optimization target of the application covers improving the interception success rate, reducing resource consumption and the like, and simultaneously considers the constraint condition to form a mixed integer linear model to achieve the optimal balance among multiple objectives;The reinforcement learning method is modified and improved to more accurately meet the stopping condition.

Description

Technical Field

[0001] The present invention belongs to the technical field of resource allocation, and in particular relates to a multi-objective resource allocation method based on deep reinforcement learning. Background Art

[0002] The resource allocation problem is essentially a constrained combinatorial optimization problem. Based on constraints such as attack time, countermeasure resources, threat targets, and tactical requirements, it optimizes the strike plan according to the environmental situation and the target's real-time status to achieve the best strike results. Current solutions to this type of problem can be roughly divided into two categories:

[0003] The first category is improved versions of traditional problem-solving algorithms, such as enhanced large-scale neighborhood search algorithms, rule-based methods, geometric methods, approximate dynamic programming, Lagrangian relaxation methods, game theory frameworks, etc. These algorithms can effectively solve problems of smaller dimensions, but have difficulty solving high-dimensional problems.

[0004] The second category is heuristic optimization algorithms, such as efficient heuristic algorithms, precise heuristic algorithms, virtual mutation and taboo search heuristic algorithms, evolutionary heuristic algorithms, and efficient rule-based heuristic algorithms. These algorithms utilize heuristic methods to improve optimization capabilities, but their global optimization capabilities are still relatively weak. Intelligent optimization algorithms such as the immune-based ant colony optimization algorithm, the improved dominance-based ant colony optimization method, the multi-objective optimization algorithm with adaptive allocation of decomposed vectors, and discrete particle swarm optimization are metaheuristic algorithms that can simulate the foraging and reproduction processes of biological populations. These algorithms have good global optimization capabilities, but their convergence speed is slow.

[0005] Furthermore, with the development of artificial intelligence (AI), deep learning has broken through the barriers of traditional methods in many fields, achieving remarkable breakthroughs. In the field of computer vision, over a decade ago, scholars primarily used manually designed algorithms for feature extraction and image processing. However, deep learning has now become the core method. Deep Neural Networks (DNNs) can automatically learn image features, replacing manual algorithm design. Deep reinforcement learning (DRL), another important branch of deep learning, is primarily used for sequential decision-making: selecting actions based on the current state of the environment and continuously adjusting strategies based on feedback from these actions to achieve a set goal. In recent years, deep reinforcement learning's performance on problems like AlphaGo Zero and Atari has demonstrated its powerful learning and decision-making capabilities. Traditional sequential decision optimization problems involve optimally selecting decision variables within a discrete or continuous decision space, sharing similarities with the "action selection" problem of reinforcement learning. Furthermore, the "offline training, online decision-making" nature of deep reinforcement learning makes it possible to solve these problems online in real time. Compared to traditional iterative optimization algorithms, DRL-based combinatorial optimization algorithms offer advantages such as fast solution speed and strong generalization. Once trained, the model can solve all problem instances with the same distribution characteristics without the need for retraining, thus achieving "offline training, online decision-making." Using deep reinforcement learning methods to solve traditional optimization problems holds great promise.

[0006] Compared with single-objective optimization problems, using deep reinforcement learning methods to solve multi-objective optimization problems has obvious challenges. It is necessary to design an effective mechanism that can output the Pareto optimal solution set of the problem in an end-to-end manner, rather than just outputting a single solution. In order to solve this problem, a decomposition strategy and a domain-based parameter migration strategy are adopted to model the multi-objective combination optimization problem. First, the decomposition strategy is used to decompose the multi-objective optimization problem into multiple sub-problems. Each sub-problem is modeled as a neural network based on the pointer network model. Then, the neural network parameters of all sub-problems are collaboratively optimized according to the neighborhood-based parameter migration strategy and reinforcement learning training algorithm, thereby achieving the solution of the multi-objective combination optimization problem.

[0007] While the aforementioned methods can improve optimization capabilities, given the stringent time constraints imposed by real-world scenarios, further improvements in the optimization capabilities and timeliness of these algorithms are needed to shorten solution times and improve solution quality. Furthermore, building multi-objective optimization models that are more realistic is crucial for solving these types of optimization problems. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a new reinforcement learning solution method for complex multi-objective resource allocation problems, to give full play to the self-training and evolution advantages of the intelligent agent of the reinforcement learning method, to achieve autonomous learning of the optimal resource allocation plan without the intervention of preset empirical rules, and to solve the problems of slow combination selection decision-making and poor real-time performance in time-sensitive tasks.

[0009] The technical solution involves first acquiring real-time environmental information and preprocessing it. Then, based on actual combat requirements, a multi-objective resource allocation optimization model is constructed. Based on the multi-objective optimization scenarios generated by the resource allocation solution, a decomposition strategy and domain-based parameter transfer strategies are used to model the multi-objective combinatorial optimization problem. Finally, the optimal matching solution for the "target-radar-launch vehicle" is output.

[0010] To achieve the above objectives, the present invention discloses a multi-objective resource allocation method based on deep reinforcement learning, comprising the following steps:

[0011] Step 1: Real-time scene information acquisition and processing: First, scene information is acquired through the final control radar. Second, the target trajectory and flight time are calculated based on the scene information and the trajectory is discretized according to the pre-set control time interval. Then, the positional relationship between the trajectory and the radar detection area and the launch vehicle's air defense area is determined, thereby calculating the applicable radar and launch vehicle combination for each target and the corresponding hit probability.

[0012] Step 2: Construction of mathematical model for multi-objective optimization of resource allocation: including: representation of decision variables, construction of objective function, and design of constraints;

[0013] Step 3: Multi-objective optimization design based on deep reinforcement learning: A basic adversarial management model consisting of three types of nodes, namely “target-radar-launcher,” is constructed to mathematically model the resource allocation problem. Based on the specific characteristics of the resource allocation problem, a deep neural network model based on a multi-head attention mechanism is constructed. A decomposition strategy and a domain-based parameter transfer strategy are used to model the multi-objective combinatorial optimization problem. First, the decomposition strategy is used to decompose the multi-objective optimization problem into multiple sub-problems. Each sub-problem is modeled as a neural network based on a pointer network model. Then, the neural network parameters of all sub-problems are collaboratively optimized using a neighborhood-based parameter transfer strategy and a reinforcement learning training algorithm, thereby achieving the generation of a multi-objective resource allocation solution.

[0014] Step 4: Combine the optimal resource allocation plan obtained by the multi-objective optimization algorithm based on deep reinforcement learning to allocate radar and launch vehicle combinations at different time periods to intercept as many incoming targets as possible, thereby maximizing the probability of successful interception, minimizing the interception cost, and achieving the highest resource utilization.

[0015] Further, the scene information includes the number of targets, location information, types, and launch angles; the hit probability, interception resources, and radar combination costs are normalized to obtain standardized data with consistent dimensions.

[0016] Further, the decision variable representation includes:

[0017] Define the task set T and the resource set R, where T contains all the targets to be intercepted and their attributes, and R includes all available interception resources and their characteristics; introduce a binary decision variable x for each pair formed by a target and a resource ij , and organize it into a decision matrix to express the configuration relationship between tasks and resources; the optimization objective covers improving interception success rate and reducing resource consumption, while considering constraint conditions, and finally forms a mixed integer linear program to achieve optimal balance among multiple objectives.

[0018] Further, the objective function construction includes:

[0019] Maximize interception success rate: each successful interception directly contributes to the security of the defense system;

[0020] Maximize the number of intercepted targets: in some cases, the threat level or importance between different targets may be difficult to quantify and compare;

[0021] Maximize resource utilization efficiency: resource utilization efficiency reflects the ratio of income to input resources, including interception resources and radar usage time.

[0022] Further, the constraint condition design includes:

[0023] Cumulative constraints: these constraints are used to ensure the effective operation and protection of interception resources within the planning period, avoiding failures caused by overloading, including radar channel resource constraints and launch vehicle usage resource constraints; the radar channel resource constraints involve limiting the number of targets that a single radar or radar network can handle within a certain time; the launch vehicle usage resource constraints involve limiting the number of interception resources loaded and launched by the launch vehicle;

[0024] Rolling constraints: rolling constraints focus on the load intensity of the interception system in the short term to prevent system overload;

[0025] Task attribute constraints: task attribute constraints ensure that the operation of interception tasks complies with the physical limitations of the actual environment;

[0026] Task relevance constraints: task relevance constraints ensure that the planned interception tasks do not conflict in time and logic.

[0027] Furthermore, the mathematical formulation of the multi-objective optimization problem is as follows:

[0028] Maximize the total interception probability:

[0029]

[0030] P i is the comprehensive interception probability of target i, is the probability that target i is intercepted by launch vehicle l at time t, is a decision variable, 1 means that target i is intercepted by the interception resource launched by launch vehicle l at time t, otherwise it is 0. I is the set of targets to be attacked, I = {1, 2, 3, ..., n1}, n1 is the total number of targets to be attacked, n2 is the total number of launch vehicles, is the amount of resources launched to target i at time t, is the maximum flight time of target i;

[0031] Maximize interception value:

[0032]

[0033] L is the set of launching vehicles, L = {1, 2, 3, ..., n2}, ζ i is the interception value of target i;

[0034] Minimize interception costs:

[0035]

[0036] c l The interception resource value equipped for launch vehicle l;

[0037] A typical constraint is expressed as:

[0038] The number of interception resources launched at target i at time t is

[0039] T i is the set of interceptable time nodes of target i;

[0040] The number of interception resources launched to target i does not exceed the upper limit of the constraint

[0041] q max The maximum number of interception resources launched against the target;

[0042] The number of interception resources launched to launch vehicle 1 cannot exceed the upper limit

[0043] θ l The number of interception resources equipped for launch vehicle l;

[0044] The time interval constraint between multiple interception resources launched at target i is

[0045]

[0046] is the moment when the pth interception resource hits target i, is the moment when the p+1th interceptor resource hits the target i, and η is the maximum distance between the moments when multiple interceptor resources are launched towards the target;

[0047] The number of channels used by radar r at any time t does not exceed the upper limit:

[0048]

[0049] is the moment when the pth interceptor resource is launched towards target i, R is the radar set, R = {1, 2, 3, ..., n3}, μ is the time discretization standard, λ i,r is the number of channels occupied by target i on radar r, is the maximum number of channels of radar r;

[0050] The radar does not switch before the target is intercepted to avoid interference with the terminal interception:

[0051]

[0052] is a decision variable, which is 1 if target i is tracked by radar r at time t, and 0 otherwise.

[0053] Furthermore, a basic confrontation management model consisting of three types of nodes, namely “target-radar-launcher”, is constructed. On this basis, a mathematical model of resource allocation is conducted, including:

[0054] For the resource allocation solution generation problem s, the parameterization of the model is as follows: In the resource allocation problem, the feature x of node i is i Defined by its geographical location characteristics, that is, its x-coordinate and y-coordinate, solve π=(π1,...,π k ,k≤n) represents the resource allocation plan; in order to solve the resource allocation problem, a deep neural network model with parameter θ is designed. The model takes the node position of the resource allocation problem s as input and outputs the plan π. The model strategy is defined as p(π|s), which realizes the mapping from s to π:

[0055]

[0056] Specifically, given a resource allocation optimization problem instance s, a deep neural network model parameterized by parameters θ is defined, which produces a conditional probability distribution p θ (π|s), solve π|s by starting from p θ (π|s) is sampled from the conditional probability distribution;

[0057] According to the specific characteristics of the resource allocation problem, a deep neural network model based on the multi-head attention mechanism is constructed, including:

[0058] To maximize the interception probability L(π), the above deep neural network model is trained using the turn-based REINFORCE algorithm. This algorithm uses a turn-based Monte Carlo method to calculate the reward value. That is, the strategy is executed until a complete solution is constructed. At this time, the total objective function is calculated as the reward L(π). Based on this definition, the strategy parameters are continuously updated through the state-action-reward method.

[0059] Specifically, given the action π|s and the corresponding reward value / loss function L(π), the model parameters θ are updated by gradient descent based on the REINFORCE algorithm:

[0060]

[0061] The resource allocation scheme is constructed by sampling a large number of actions from a probability distribution. Due to the uncertainty of sampling and the sampling process in each round, the rewards obtained in different rounds have variance. This variance can cause conflicts in the gradient descent direction of the policy parameters and affect the convergence speed. To reduce the variance of the policy parameter update, a baseline value b(s) is introduced to update the policy gradient formula:

[0062]

[0063] b(s) is used to evaluate the average performance of the strategy. If the performance of the current strategy is better than the average level b(s), L(π)-b(s) is a negative number, and the strategy is positively incentivized. If the performance of the current strategy is worse than the average level b(s), L(π)-b(s) is a positive number, and the strategy is negatively incentivized.

[0064] Given a resource allocation problem instance s, b(s) is defined as using the baseline policy p θ The objective function obtained by solving the problem instance s is a greedy strategy for sampling the probability distribution during the solution process: the baseline strategy p θ*At the end of each training cycle, the policy is evaluated on the validation set. Once the current policy is better than the current baseline policy, the baseline policy is replaced by the current model, making the baseline policy the best model trained so far. b(s) is the current best policy performance, and the policy gradient update is optimized based on this baseline performance.

[0065] The decomposition strategy is to decompose the multi-objective optimization problem using the Weighted Sum method, the Chebyshev method or the penalty-based boundary crossing method.

[0066] Furthermore, neighborhood-based parameter migration strategies include:

[0067] Since the subproblem is modeled as a deep neural network, the neural network model parameters of the (i-1)th subproblem are expressed as [ω * ,b * ] represents the optimal neural network model parameters, and [ω,b] represents the unoptimized neural network model parameters; assuming that the (i-1)th subproblem is solved, that is, its neural network parameters have been optimized to near optimality, then the model parameters of the (i-1)th subproblem are Set as the initial parameters of the neural network model of the i-th sub-problem, and start training from the initial parameters, that is, migrate the neural network model parameters of the neighborhood sub-problem to the next adjacent sub-problem, thereby reducing the model training time.

[0068] The present invention provides a multi-objective resource allocation method based on deep reinforcement learning, which can achieve the following beneficial effects:

[0069] 1. This paper proposes a multi-objective optimization modeling method for resource allocation. This model expresses the configuration relationship between tasks and resources. Optimization objectives range from improving interception success rate to reducing resource consumption, while also considering constraints such as resource capacity, state, and time window. Ultimately, a mixed integer linear model is formed to achieve an optimal balance among these multiple objectives.

[0070] 2. This invention addresses the problems of traditional optimization methods, such as slow response, susceptibility to local optima, difficulty handling a large number of constraints, and difficulty meeting stopping conditions. By optimizing the algorithm structure, enhancing constraint handling capabilities, refining stopping conditions, and improving robustness, the reinforcement learning method is modified and improved. These improvements enable the algorithm to more efficiently handle complex constraints, more accurately meet stopping conditions, and demonstrate greater adaptability and flexibility in a variety of tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is the overall structural diagram of the present invention;

[0072] Figure 2 Generate an algorithm framework diagram for the multi-objective resource allocation solution based on deep reinforcement learning designed by the present invention;

[0073] Figure 3 Schematic diagram of the decomposition strategy in reinforcement learning training designed for the present invention;

[0074] Figure 4 Schematic diagram of the parameter migration strategy in reinforcement learning training designed for the present invention;

[0075] Figure 5 Flowchart of the multi-objective optimization algorithm based on deep reinforcement learning designed for this invention;

[0076] Figure 6 This is a schematic diagram of a resource allocation solution obtained by using the reinforcement learning method designed by the present invention. DETAILED DESCRIPTION

[0077] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0078] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0079] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0080] The technical solutions provided in the embodiments of this application involve technologies such as machine learning and natural language processing of artificial intelligence, and are specifically introduced and explained through the following embodiments.

[0081] This paper proposes a multi-objective resource allocation method based on deep reinforcement learning. The overall structure is as follows Figure 1 The specific steps are as follows:

[0082] Step 1: Real-time scene information acquisition and processing. First, the final control radar is used to obtain scene information such as the number, location information, type, and launch angle of the incoming targets. Secondly, the target trajectory and flight time are calculated based on the summarized environmental information, and the trajectory is discretized according to the preset control time interval. Subsequently, the positional relationship between the trajectory and the radar detection area and the launch vehicle air defense area is determined to calculate the radar and launch vehicle combination that can be used for each target and the corresponding hit probability. Finally, in order to facilitate the solution of the optimization problem, the present invention also normalizes the hit probability, interception resources, and the cost of the radar combination to obtain standardized data with consistent dimensions.

[0083] Step 2: Constructing a mathematical model for multi-objective optimization of resource allocation, specifically:

[0084] (1) Decision variable representation

[0085] When constructing a multi-objective optimization model for resource allocation, we first define the task set T and resource set R, where T contains all the enemy targets to be intercepted and their attributes, and R includes all available interception resources and their characteristics. By introducing a binary decision variable x for each pair of (target, resource) combinations ij , and organized into a decision matrix. This model expresses the configuration relationship between tasks and resources. Optimization objectives range from improving interception success rate to reducing resource consumption, while also considering constraints such as resource capacity, state, and time window. Ultimately, a mixed integer linear program is formed to achieve the optimal balance between these multiple objectives.

[0086] (2) Objective function construction

[0087] When building a multi-objective optimization model for resource allocation, the objective function is a key indicator of solution quality. Based on specific assumptions, the objective function is considered a deterministic function of the decision variables. That is, for a fixed set of decision variables, the value of the objective function is uniquely determined. The following lists several typical objective functions and their basic forms used in various fields:

[0088] Maximizing the interception success rate: This means that every successful interception mission directly contributes to the security of the defense system. Therefore, maximizing the total benefit of all successful interception missions is the most intuitive and commonly used objective function in confrontation.

[0089] Maximizing the number of interception targets: In some cases, the threat level or importance of different targets may be difficult to quantify and compare. In this case, maximizing the number of interception targets can be selected as an indicator to evaluate the effectiveness of the scheduling scheme, where each target is considered equally important.

[0090] Maximizing resource utilization efficiency: Resource utilization efficiency reflects the ratio of benefits gained to resources invested (such as interception resources, radar usage time, etc.). In confrontation, this usually involves how to achieve maximum effect with minimal resource consumption.

[0091] The above objective function provides a multi-dimensional evaluation framework for resource allocation, enabling decision makers to formulate the most appropriate resource allocation plan based on different environmental requirements and resource constraints.

[0092] (3) Constraint design

[0093] In multi-objective optimization modeling of resource allocation problems, constraints are one of the key factors to ensure effective, practical, and safe countermeasure solutions. Specifically, the constraints of resource allocation solutions can be divided into the following four categories:

[0094] Cumulative constraints: This type of constraint is mainly used to ensure the effective operation and protection of interception resources within the planning cycle, and to avoid failures caused by overload operation. The two main typical types of constraints are radar channel occupancy resource constraints and launch vehicle usage resource constraints. Specifically, they can be described as: Radar channel occupancy resource constraints: Radar channel occupancy resource constraints involve limiting the number of targets that a single radar or radar network can handle within a specific time. This helps prevent radar system overload and ensures that all targets can be tracked with sufficient accuracy and continuity for effective target identification and interception. Launch vehicle usage resource constraints: Launch vehicle usage resource constraints involve restrictions on the number of launch vehicle loading and launch interception resources. This helps ensure that in continuous defensive confrontation, the launch vehicle can effectively replenish resources and maintain confrontation capabilities.

[0095] Rolling constraints: These focus on the short-term load on the interception system to prevent system overload. For example, you can set constraints such as "a single interception system can only conduct a certain number of intercepts within any consecutive number of minutes" or "the system's total launch time cannot exceed a certain number of seconds within any consecutive number of seconds" to protect equipment from overheating or excessive wear.

[0096] Mission attribute constraints: Mission attribute constraints ensure that interception missions operate within the physical limitations of the actual environment. For example, "the minimum warning time for any interception mission must not be less than a certain number of seconds," or "under certain electronic jamming conditions, only certain types of interception assets are allowed to be used for interception."

[0097] Task-related constraints: The correlation constraints between tasks ensure that the planned interception tasks do not conflict in time and logic. For example, "there needs to be at least several minutes of interval between two interception tasks to re-preparation of the interception system", or "the same target cannot be attacked by two different interception systems at the same time", and "tasks that protect critical areas such as command centers or important facilities around them are given priority" and so on.

[0098] Through these constraints, it can be ensured that the formulated resource allocation scheme is not only optimal in theory, but also feasible and effective in actual environment, and protects the interception resources from overuse and potential damage. These constraints help decision-makers make quick and reliable decisions in a variable scenario, maximizing overall defense effectiveness.

[0099] Taking the minimization problem as an example, the mathematical expression of the multi-objective optimization problem is as follows:

[0100]

[0101] wherein, is the objective function to be optimized, which can be several of the above typical objective functions, is the optimization variable, g i and h j (·) are constraint functions that the objective optimization variable needs to satisfy. When m = 1, it is a single-objective optimization problem, and when m ≥ 2, it is a multi-objective optimization problem (Multi-Objective Optimization Problems). In particular, a multi-objective optimization problem with at least 4 objectives is called a many-objective optimization problem (Many Objective Optimization Problems).

[0102] Based on the above definitions of objective functions and constraints, a general multi-objective mathematical optimization model for solving resource allocation problems is constructed.

[0103]

[0104]

[0105] In actual confrontation, due to the influence of factors such as different flight times of incoming targets and different target flight destinations, the mathematical model needs to convert the above factors into the interception priority of the target for optimization. Therefore, the three typical optimization objectives of the present application can be represented as:

[0106] Maximize the total interception probability:

[0107] Maximize the interception value:

[0108] Minimize interception costs:

[0109] A typical constraint can be expressed as:

[0110] The number of interception resources launched at target i at time t is

[0111] The number of interception resources launched to target i does not exceed the upper limit of the constraint

[0112] The number of interception resources launched to launch vehicle 1 cannot exceed the upper limit

[0113] The time interval constraint between multiple interception resources launched at target i is

[0114] The number of channels used by radar r at any time t does not exceed the upper limit:

[0115]

[0116] The radar does not switch before the target is intercepted to avoid interference with the terminal interception:

[0117]

[0118] Step 3: Multi-objective optimization design based on deep reinforcement learning.

[0119] Unlike traditional combinatorial optimization methods, the use of artificial intelligence methods to solve combinatorial optimization methods involves a large number of new technologies, new concepts, and new ideas. Therefore, we first study the scientific principles of solving combinatorial optimization problems based on deep reinforcement learning, and propose a deep reinforcement learning method for multi-objective resource allocation problems.

[0120] First, a basic adversarial management model consisting of three types of nodes (target, radar, and launch vehicle) is constructed. Based on this model, the resource allocation problem is mathematically modeled. Second, based on the specific characteristics of the resource allocation problem, a deep neural network model based on a multi-head attention mechanism is constructed. A deep reinforcement learning algorithm is proposed to train the model's parameters. This approach uses an "offline training, online optimization" approach to rapidly generate resource allocation solutions. For the multi-objective optimization scenario of resource allocation solution generation, a decomposition strategy and a domain-based parameter transfer strategy are used to model the multi-objective combinatorial optimization problem. The decomposition strategy is first used to decompose the multi-objective optimization problem into multiple sub-problems. Each sub-problem is modeled as a neural network using a pointer network model. Then, using a neighborhood-based parameter transfer strategy and a reinforcement learning training algorithm, the neural network parameters of all sub-problems are collaboratively optimized to achieve multi-objective resource allocation solution generation. The framework of the deep reinforcement learning-based multi-objective resource allocation solution generation algorithm is shown in the figure below.

[0121] (1) Parametric modeling

[0122] For the resource allocation solution generation problem s, the parameterization of the model is as follows: In the resource allocation problem, the feature x of node i is i Defined by its geographical location and other features, for the convenience of expression, the geographical location is used as the feature, that is, its x-coordinate and y-coordinate, and the solution is π=(π1,...,π k ,k≤n) represents the resource allocation plan. In order to solve the resource allocation problem, it is necessary to design a deep neural network model with parameter θ. This model takes the node position of the resource allocation problem s as input and outputs a plan π. The model strategy is defined as p(π|s), which can achieve the mapping from s to π:

[0123]

[0124] Specifically, given a resource allocation optimization problem instance s, a deep neural network model parameterized by parameters θ is defined, which produces a conditional probability distribution p θ (π|s). Solving π|s can be done from p θ The optimization goal of the resource allocation problem is to maximize the interception probability, maximize the interception value, and minimize the interception cost. Taking maximizing the interception probability L(π) as an example, the turn-based REINFORCE algorithm is suitable for training the above deep neural network model. This algorithm uses a turn-based Monte Carlo method to calculate the reward value. Specifically, the policy is executed until a complete solution is constructed, at which point the total objective function is calculated as the reward L(π). Based on this definition, the policy parameters can be continuously updated through the state-action-reward method.

[0125] Specifically, given an action π|s and the corresponding reward value / loss function L(π), the model parameters θ can be updated by gradient descent based on the REINFORCE algorithm:

[0126]

[0127] The construction of a resource allocation plan requires sampling a large number of actions from a probability distribution. Due to the uncertainty of sampling and the large number of sampling processes in each round, the rewards obtained in different rounds have a large variance. This variance can cause conflicts in the gradient descent direction of the policy parameters and affect the convergence speed. To reduce the variance of the policy parameter update, a baseline value b(s) is introduced to update the policy gradient formula:

[0128]

[0129] b(s) can be thought of as the average performance of a policy. If the current policy outperforms the average b(s), L(π)-b(s) is negative, and the policy is positively incentivized (minimizing the loss function). If the current policy underperforms the average b(s), L(π)-b(s) is positive, and the policy is negatively incentivized. The current mainstream approach is to train an additional neural network to approximate b(s), namely the Critic network. Its parameters are trained using the (s, L(π)) collected during training. However, simultaneously training a policy parameterized by a neural network and a Critic neural network results in slow convergence. Therefore, this patent adopts a greedy rollout baseline method to calculate b(s), which has been shown in experiments to outperform training methods based on Critic neural networks.

[0130] Specifically, there is no need to design additional neural networks to calculate b(s). In the patent, given a resource allocation problem instance s, b(s) is defined as using the benchmark strategy p θ The objective function obtained by solving the problem instance s is obtained by sampling the probability distribution in the solution process using a greedy strategy. The baseline strategy p θ* It is the best model so far in the training process, that is, the policy evaluation is performed on the validation set at the end of each training cycle. Once the current policy is better than the current baseline policy, the baseline policy is replaced by the current model, so that the baseline policy is the best model trained so far. b(s) is the current best policy performance, and the policy gradient update is optimized by the benchmark performance.

[0131] (2) Reinforcement learning training

[0132] Decomposition strategy: Decomposition is an effective method for solving multi-objective optimization problems. Numerous traditional multi-objective optimization algorithms, such as MOGA, MOEA / D, MOEA / DD, and NSGA-III, have been designed using this strategy. This strategy is used to model the resource allocation problem, breaking it down into a set of scalar optimization subproblems. These subproblems are then solved in a collaborative optimization approach to obtain the optimal solution for each subproblem. The solutions to all these subproblems constitute the Pareto Front (PF) of the original problem.

[0133] Specifically, the widely used Weighted Sum method is used to decompose the original problem. Other methods, such as the Chebyshev method and the penalty-based boundary intersection (PBI) method, can also be applied to decompose the multi-objective optimization problem. The decomposition method used is as follows: First, a set of uniformly distributed weight vectors λ is given. 1 ,...,λ N For example, for a resource allocation problem with two optimization objectives, the weight vector is (1,0), (0.9, 0.1), ..., (0,1), where Where M represents the number of objectives. Therefore, the original resource allocation problem can be converted into N scalar optimization sub-problems using the Weighted Sum method. The objective function of the j-th sub-problem is as follows:

[0134]

[0135] Therefore, the Pareto front can be composed of N solutions obtained by solving N subproblems. This process is shown in the figure below. The dotted line represents the direction of subproblem optimization. This method decomposes the original resource allocation problem into N subproblems, and further uses deep neural networks to model the subproblems, thereby achieving the solution of multi-objective combinatorial optimization problems.

[0136] Neighborhood-based parameter transfer strategy: To solve each subproblem using deep reinforcement learning, the complex resource allocation optimization problem needs to be decomposed into N subproblems. These N subproblems are then modeled as deep neural networks. Finally, the N scalar optimization subproblems are solved in a collaborative optimization manner using a neighborhood-based parameter transfer strategy. The domain-based parameter transfer strategy is introduced below.

[0137] By observing the Weighted Sum formula, it can be seen that for two adjacent subproblems, since their weight vectors are similar, their optimal solutions are similar. Based on this fact, each subproblem can be solved by drawing on the information of its adjacent subproblems.

[0138] Since the subproblems are modeled as deep neural networks, the neural network model parameters of the (i-1)th subproblem can be expressed as [ω * ,b * ] represents the optimal neural network model parameters, and [ω,b] represents the unoptimized neural network model parameters. Assuming that the (i-1)th subproblem is solved, that is, its neural network parameters have been optimized to near-optimal, then the model parameters of the (i-1)th subproblem can be Set as the initial parameters of the neural network model of the i-th sub-problem, and start training from the initial parameters, that is, migrate the neural network model parameters of the neighboring sub-problem to the next adjacent sub-problem, thereby greatly reducing the model training time. There are many migration methods. The most basic parameter migration method is used in this technology, that is, the neural network parameters are migrated from the previous sub-problem to the next sub-problem in sequence. The process is as follows Figure 4 shown.

[0139] If the neighborhood-based parameter migration strategy is not adopted, training N sub-problems modeled as neural network models will take a lot of time and cannot be used in practice. The neighborhood-based parameter migration strategy adopted enables the practical application of this multi-objective combinatorial optimization method.

[0140] (3) Resource allocation plan generation

[0141] In summary, the resource allocation problem is decomposed into multiple sub-problems, each of which is modeled by a deep neural network. The sub-problems are solved by reinforcement learning methods. All sub-problems can be collaboratively optimized through parameter migration strategies. The resulting neural network model set can be used to output the Pareto front end-to-end. Combining the decomposition strategy and the neighborhood-based parameter migration strategy, the multi-objective combination optimization method based on deep reinforcement learning has the following process: Figure 5 shown.

[0142] Step 4: Generate resource allocation plan.

[0143] Combined with the optimal resource allocation plan obtained by the multi-objective optimization algorithm based on deep reinforcement learning, radar and launch vehicle combinations at different time periods are allocated to intercept as many incoming targets as possible, thereby achieving the goals of maximizing the probability of successful interception, minimizing the interception cost, and maximizing resource utilization.

[0144] The present invention provides a multi-objective resource allocation method based on deep reinforcement learning, which can achieve the following beneficial effects:

[0145] 1. This paper proposes a multi-objective optimization modeling method for resource allocation. This model expresses the configuration relationship between tasks and resources. Optimization objectives range from improving interception success rate to reducing resource consumption, while also considering constraints such as resource capacity, state, and time window. Ultimately, a mixed integer linear model is formed to achieve an optimal balance among these multiple objectives.

[0146] 2. This invention addresses the problems of traditional optimization methods, such as slow response, susceptibility to local optima, difficulty handling a large number of constraints, and difficulty meeting stopping conditions. By optimizing the algorithm structure, enhancing constraint handling capabilities, refining stopping conditions, and improving robustness, the reinforcement learning method is modified and improved. These improvements enable the algorithm to more efficiently handle complex constraints, more accurately meet stopping conditions, and demonstrate greater adaptability and flexibility in a variety of tasks.

[0147] As used herein, the word "preferred" is intended to serve as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word "preferred" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any of the naturally inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing examples.

[0148] Moreover, although the present disclosure has been shown and described with respect to one or implementation, those skilled in the art will think of equivalent variations and modifications based on reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if structurally different from the disclosed structure that performs the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that can be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0149] The functional units in the embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or multiple or more units may be integrated into a single module. The aforementioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc. The aforementioned devices or systems may execute the storage method in the corresponding method embodiment.

[0150] In summary, the above embodiment is one implementation method of the present invention, but the implementation method of the present invention is not limited to the described embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A multi-objective resource allocation method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Real-time scene information acquisition and processing: First, scene information is acquired through the final control radar. Second, the target trajectory and flight time are calculated based on the scene information and the trajectory is discretized according to the pre-set control time interval. Then, the positional relationship between the trajectory and the radar detection area and the launch vehicle's air defense area is determined, thereby calculating the applicable radar and launch vehicle combination for each target and the corresponding hit probability. Step 2: Construction of mathematical model for multi-objective optimization of resource allocation: including: representation of decision variables, construction of objective function, and design of constraints; Step 3: Multi-objective optimization design based on deep reinforcement learning: A basic adversarial management model consisting of three types of nodes: "target-radar-launcher" is constructed to mathematically model the resource allocation problem. Based on the specific characteristics of the resource allocation problem, a deep neural network model based on the multi-head attention mechanism is constructed. A decomposition strategy and a domain-based parameter transfer strategy are used to model the multi-objective combinatorial optimization problem. First, the decomposition strategy is used to decompose the multi-objective optimization problem into multiple sub-problems. Each sub-problem is modeled as a neural network based on the pointer network model. Then, the neural network parameters of all sub-problems are collaboratively optimized using a neighborhood-based parameter transfer strategy and a reinforcement learning training algorithm to achieve the generation of a multi-objective resource allocation solution. Step 4: Combine the optimal resource allocation solution obtained using a multi-objective optimization algorithm based on deep reinforcement learning to assign radar and launch vehicle combinations at different times to intercept as many incoming targets as possible, thereby maximizing the probability of successful interception, minimizing interception costs, and maximizing resource utilization. Among them, a basic confrontation management model consisting of three types of nodes, "target-radar-launcher", is constructed. On this basis, a mathematical model of the resource allocation problem is carried out, including: Generate questions about resource allocation solutions , the parameterization of the model is as follows. In the resource allocation problem, the node Features Defined by its geographical location characteristics, namely its x-coordinate and y-coordinate, solution Represents the resource allocation scheme, n is the maximum number of allocation schemes; in order to solve the resource allocation problem, the design parameters are A deep neural network model based on the resource allocation problem The node position is taken as input and the output is the solution , the model strategy is defined as ,accomplish arrive The mapping: ； Specifically, given a resource allocation optimization problem instance s, define the parameters A parameterized deep neural network model that produces a conditional probability distribution ,untie By Sampling from the conditional probability distribution; According to the specific characteristics of the resource allocation problem, a deep neural network model based on the multi-head attention mechanism is constructed, including: To maximize the probability of interception , the above deep neural network model is trained using the turn-based REINFORCE algorithm, which uses a turn-based Monte Carlo method to calculate the reward value, that is, the strategy is executed until a complete solution is constructed, at which time the total objective function is calculated as the reward ,Based on this definition, the policy parameters are continuously updated through,state-action-reward; Specifically, given an action and the corresponding reward / loss function , based on the REINFORCE algorithm, updates the model parameters by gradient descent : ； The construction of the resource allocation scheme samples a large number of actions from the probability distribution. Due to the uncertainty of sampling and the sampling process in each round, the rewards obtained in different rounds have variance, which causes conflicts in the gradient descent direction of the policy parameters and affects the convergence speed. In order to reduce the variance of the policy parameter update, the baseline value is introduced. To update the policy gradient formula: ； Used to evaluate the average performance of the strategy. If the current strategy performs better than the average , If the current strategy performs worse than the average, the strategy is positively incentivized. , If is a positive number, the strategy is negatively incentivized; Given a resource allocation problem instance s, Defined as utilizing a baseline strategy The objective function obtained by solving the problem instance s is the greedy strategy used to sample the probability distribution during the solution process: the baseline strategy At the end of each training cycle, the strategy is evaluated on the validation set. Once the current strategy is better than the current baseline strategy, the baseline strategy is replaced by the current model, making the baseline strategy the best model trained so far. That is, the current optimal policy performance, through which the policy gradient update is optimized; The decomposition strategy is to decompose the multi-objective optimization problem using the Weighted Sum method, the Chebyshev method or the penalty-based boundary crossing method.

2. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The scenario information includes the number, location information, type and launch angle of the targets; the hit probability, interception resources and radar combination costs are normalized to obtain standardized data with consistent dimensions.

3. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The decision variables are represented as follows: Define the task set T and resource set R, where T contains all the enemy targets to be intercepted and their attributes, and R includes all available interception resources and their characteristics; introduce a binary decision variable x for each pair of targets and resources. ij , and organized into a decision matrix to express the configuration relationship between tasks and resources; the optimization objectives include improving the interception success rate and reducing resource consumption, while considering the constraints, and finally forming a mixed integer linear program to achieve the optimal balance among multiple objectives.

4. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The objective function construction includes: Maximizing interception success rate: This means that every successful interception mission directly contributes to the security of the defense system; Maximizing the number of interception targets: This means that in some cases, the threat level or importance between different targets may be difficult to quantify and compare; Maximizing resource utilization efficiency: Resource utilization efficiency reflects the ratio of benefits gained to input resources, including interception resources and radar usage time.

5. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The constraint design includes: Cumulative constraints: These constraints are used to ensure the effective operation and protection of interception resources within the planning cycle and to avoid failures caused by overload. They include radar channel occupancy constraints and launch vehicle usage constraints. The radar channel occupancy constraints limit the number of targets that a single radar or radar network can handle within a specific timeframe. The launch vehicle usage constraints limit the number of interception resources that can be loaded and launched by the launch vehicle. Rolling constraints: Rolling constraints focus on intercepting the load intensity of the system in the short term to prevent the system from being overloaded; Mission attribute constraints: Mission attribute constraints ensure that the operation of the interception mission complies with the physical limitations of the actual environment; Task dependency constraints: Dependency constraints between tasks ensure that the planned interception tasks do not conflict in time and logic.

6. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: The mathematical expression of the multi-objective optimization problem is as follows: Maximize the total interception probability: ， ； is the comprehensive interception probability of target i, is the probability that target i is intercepted by launch vehicle l at time t, is a decision variable, 1 means that target i is intercepted by the interception resource launched by launch vehicle l at time t, otherwise it is 0. I is the set of targets to be attacked. , n1 is the total number of targets to be attacked, n2 is the total number of launch vehicles, is the amount of resources launched to target i at time t, is the maximum flight time of target i; Maximize interception value: ； L is the launch vehicle set, , is the interception value of target i; Minimize interception costs: ； The value of interception resources equipped for launch vehicle l; A typical constraint is expressed as: The number of interception resources launched at target i at time t is ; is the set of interceptable time nodes of target i; The number of interception resources launched to target i does not exceed the upper limit of the constraint ; The maximum number of interception resources launched against the target; The number of interception resources launched to launch vehicle 1 cannot exceed the upper limit ; The number of interception resources equipped for launch vehicle l; The time interval constraint between multiple interception resources launched at target i is ; is the moment when the pth interception resource hits target i, is the moment when the p+1th interception resource hits target i, It is the maximum distance between multiple interception resources launched at the target at the moment of impact; The number of channels used by radar r at any time t does not exceed the upper limit: ； is the moment when the pth interceptor resource is launched towards target i, R is the radar set, , n3 is the total number of radars, is the time discretization criterion, is the number of channels occupied by target i on radar r, is the maximum number of channels of radar r; The radar does not switch before the target is intercepted to avoid interference with the terminal interception: ； is a decision variable, which is 1 if target i is tracked by radar r at time t, and 0 otherwise.

7. The multi-objective resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: Neighborhood-based parameter migration strategies include: Since the subproblems are modeled as deep neural networks, the The neural network model parameters of the sub-problem are expressed as , Represents the optimal neural network model parameters, Represents the neural network model parameters that have not been optimized; assuming that The subproblem is solved, that is, the neural network parameters are optimized to near optimality, then the Model parameters for the subproblem Set to The initial parameters of the neural network model of each sub-problem are obtained, and training starts from the initial parameters, that is, the neural network model parameters of the neighborhood sub-problem are transferred to the next adjacent sub-problem, thereby reducing the model training time.

Images