A multi-agent collaborative resource allocation method based on deep reinforcement learning

Abstract

The application discloses a kind of multi-agent collaborative resource allocation methods based on deep reinforcement learning, belong to resource allocation technical field.The method includes: S1, the task scene containing multiple execution unit cluster is initialized configuration;S2, resource allocation decision task is modeled as sequence decision process, constructs single execution unit cluster level and global level optimization target;S3, each execution unit cluster is regarded as independent agent, constructs multi-agent collaborative resource allocation decision model based on deep reinforcement learning and carries out parameter initialization;S4, at each time step, each agent generates resource allocation decision according to its own observation state, and stores interactive experience;S5, data is sampled from experience cache, and loss function is constructed by introducing collaborative credibility, and model parameters are updated;S6, after training, the model is used to guide multiple execution unit cluster to implement collaborative resource allocation.The application improves the collaborative decision-making capability and overall processing efficiency of multiple execution unit cluster in complex operational environment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent decision-making technology, specifically relating to a multi-agent collaborative resource allocation method based on deep reinforcement learning. Background Technology

[0002] In complex and dynamic environments, the problem of collaborative decision-making and resource allocation among multiple execution units is widespread in scenarios such as collaborative operations of intelligent robots, multi-agent systems, and unmanned systems. With the expansion of task scale and the diversification of task types, achieving efficient collaboration and rational resource allocation among multiple execution units under conditions of incomplete information and partial observability has become a critical issue that urgently needs to be addressed.

[0003] Existing resource allocation methods mainly include centralized optimization methods and rule-based allocation strategies. Centralized methods suffer from high computational complexity when the number of execution units is large, making it difficult to meet real-time requirements; rule-based methods lack adaptability and struggle to cope with dynamic environments and task uncertainties. Furthermore, in multi-agent systems, each execution unit typically only possesses local observation capabilities, making it difficult for existing methods to effectively characterize their collaborative relationships, thus limiting overall task execution efficiency. Therefore, there is an urgent need for a resource allocation method capable of enabling collaborative decision-making among multiple execution units under distributed conditions, in order to improve the overall performance and collaborative efficiency of the system in multi-objective task scenarios. Summary of the Invention

[0004] To address the shortcomings of existing resource allocation methods in collaborative processing scenarios involving multiple execution unit clusters, such as insufficient modeling capabilities and limited collaborative decision-making effectiveness, this invention proposes a multi-agent collaborative resource allocation method based on deep reinforcement learning to improve the collaborative processing efficiency of multi-execution unit clusters.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A multi-agent cooperative resource allocation method based on deep reinforcement learning includes the following steps: S1: For task scenarios involving multiple execution unit clusters, perform configuration initialization; S2: Model the resource allocation decision-making task as a sequential decision-making process, and construct the single-execution-unit cluster-level optimization objective and the global-level optimization objective respectively; S3: Treat each execution unit cluster as an agent, construct a multi-agent collaborative resource allocation decision model based on deep reinforcement learning, and initialize the parameters; S4: At each time step, based on the current observation state of each execution unit cluster, each agent generates a corresponding resource allocation decision action, and after executing the action, obtains the state and real-time reward value of the next time step, and stores the interaction data in its own experience cache. S5: Sample small batches of data from the experience cache corresponding to each execution unit cluster, calculate the loss function value that introduces collaborative credibility, and use this value to train the multi-agent collaborative resource allocation decision model and update the model parameters. S6: Repeat steps S4–S5 until training ends, and use the trained model to guide the resource allocation of the multi-execution unit cluster to achieve collaborative resource allocation for multi-task objectives.

[0006] Furthermore, in S1, the task scenario includes a cluster of K execution units and Each task objective, each execution unit cluster equipped with Resource units, task objectives include There are several types of execution units, each with a different probability of completing different objectives. Furthermore, each execution unit cluster can only observe and process a subset of the objectives. The number of objectives that execution unit cluster k can observe is represented as... .

[0007] Furthermore, in S2, the resource allocation decision needs to be made at every time step. Each execution unit cluster generates a resource allocation decision action, and modeling the sequential decision-making process requires analysis of each time step. The corresponding actions and states are defined.

[0008] Furthermore, based on the execution unit cluster's own observable state and the observable target state, the execution unit cluster... At time step observable state under Defined as: in Represents an execution unit cluster At time step The status of resource units; Represents an execution unit cluster At time step The observable target state; Resource unit status Specifically defined as: in Represents an execution unit cluster The resource unit types to be allocated are represented using one-hot encoding, and their dimensions are... , Indicates the current resource unit type The probability vector for completing various objectives has the following dimension: , Represents an execution unit cluster The remaining proportion vectors of various resource units are in the following form: in Represents an execution unit cluster The Resource units in time step The remaining quantity, Represents an execution unit cluster The The initial maximum number of resource units; Observable target state It is composed of the states of its fixed set of observable targets: in Represents an execution unit cluster The first observable j The goal is in time step The state is specifically defined as: in Represents an execution unit cluster Target within observation range j The type information is represented using one-hot encoding, and its dimension is... , This serves as a reference value for the priority weight of the objective. Indicates the time step up to Execution Unit Cluster To the target j The vector of the number of various resource units allocated, with dimension [missing information]. , and Representing the target j Equivalent to an execution unit cluster Distance and azimuth; The time steps are obtained by merging the observable states of a cluster of K execution units. The overall state below : .

[0009] Furthermore, based on the execution unit cluster's own observable state and the observable target state, the execution unit cluster... At time step Action vectors under Defined as: in Represents an execution unit cluster At time step Next, the resource unit to be allocated will be assigned to the first unit in its observable target set. j One goal, j The value ranges from 1 to By defining actions as goal-oriented resource allocation choices, a structured expression of the resource allocation process can be achieved, thereby reducing decision-making complexity.

[0010] Furthermore, in S2, the single execution unit cluster-level optimization objective and the global-level optimization objective are constructed as follows: First, we construct a cluster-level optimization objective for a single execution unit, including the reward for completing the target task and the penalty for resource consumption, which are defined as follows: in For execution unit cluster At time step The benefits of completing the target task For execution unit cluster At time step Resource consumption penalty This serves as a reference value for the priority weight of the objective. Representing resource units For the target j The probability of completion, For the deadline step Execution Unit Cluster The first one already assigned to target j Number of resource units Represents an execution unit cluster The Resource units in time step The remaining quantity, Represents an execution unit cluster The The initial maximum number of resource units is thus obtained; therefore, the single execution unit cluster-level optimization objective is obtained. for: in , The weights corresponding to the cluster-level optimization objectives of each individual execution unit; Then, a global optimization objective is constructed, including objective coverage and global task completion benefits, defined as follows: in For time steps Target coverage, For time steps Global task completion rewards For indicator functions, For the goal j The priority weight reference value is used to obtain the global optimization objective. for: in , To correspond to the weights of each global optimization objective, the global optimization objective is used to guide each execution unit cluster to achieve collaborative coverage and maximize task benefits, thereby improving the overall system performance; Finally, the execution unit cluster At time step The immediate rewards are constructed from single-execution-unit cluster-level and global-level optimization objectives: in For execution unit cluster At time step Instant rewards To control the impact weight of global rewards on the cluster decision-making of a single execution unit, the integration of local and global benefits can achieve a unification of collaborative optimization and individual optimization within a distributed decision-making framework.

[0011] Furthermore, in S3, the multi-agent collaborative resource allocation decision model constructs a set of independent decision network units for each execution unit cluster. Each decision unit adopts a dual-network structure design, including a main network for current decision evaluation and a target network for stabilizing the training process. The two networks have the same structure but their parameters are independent and do not share each other. During the model parameter initialization phase, the network parameters of the main network corresponding to each execution unit cluster are initialized randomly, and the initialized main network parameters are copied to the corresponding target network parameters to ensure the consistency between the main network parameters and the target network parameters in the initial training phase. in and They represent the first The main network parameters and target network parameters corresponding to each execution unit cluster can reduce the oscillation problem of value estimation during training by introducing the target network, thereby improving the convergence stability of the model.

[0012] Furthermore, in S4, the main network corresponding to each execution unit cluster operates at each time step. Based on the current observable state of the execution unit cluster As input, the output is the expected payoff vector for taking different resource allocation actions in this state. : in, Represents an execution unit cluster In state The following measures were taken. Candidate resource allocation decision-making actions The corresponding expected return, where L is the total number of discrete resource allocation actions that the execution unit cluster can take; Based on the expected return vector, construct a relative expected return scaling factor: in For execution unit cluster At time step The maximum expected return value of all candidate resource allocation actions. Indicates action Relative to the expected reward ratio of the current optimal action, To prevent extremely small positive numbers with a denominator of zero, this relative payoff ratio can reflect the relative merits of each action, thereby enhancing the strategy's ability to identify high-quality actions. Then, the execution unit cluster In state The probability of choosing the resource allocation action is defined as follows: in Represents an execution unit cluster In state Next choice The final probability of each resource allocation action, As a profit concentration adjustment factor, by constructing a probability distribution based on profit ratio, it can maintain a certain degree of exploratory nature while ensuring that high-profit actions are prioritized.

[0013] Furthermore, in S4, the main network of each execution unit cluster operates at time steps. Based on the action selection strategy according to the relative expected return ratio, from the final probability Mid-sampling obtains resource allocation actions in the current state. The actions are then applied synchronously to the environment; after receiving resource allocation actions from all execution unit clusters, the environment jointly updates the overall system state for the next moment. And calculate the corresponding single-step reward value for each execution unit cluster. and with The data is stored in the experience cache of the corresponding execution unit cluster in the form of data to support parallel training and collaborative decision-making of multiple execution unit clusters.

[0014] Furthermore, in S5, small batches of experience data are sampled from the experience cache corresponding to each execution unit cluster as training samples, and then input into the main network and target network corresponding to that execution unit cluster, respectively; for any execution unit cluster Single empirical sample The target network calculates the target reward value based on the state at the next time step: in For execution unit cluster At time step The target return value, hyperparameters As the discount factor, To maximize expected return, Represents an execution unit cluster The target network in state Next, make decisions on allocating resources to each candidate. Expected return value; A collaborative credibility is introduced to characterize the historical contribution of the execution unit cluster to the overall collaborative processing effect under a specific state-action pair, and to construct the execution unit cluster. Training loss function: in For execution unit cluster The loss function of the main network, For the current state-action pair The projected return value, For execution unit cluster State-action pairs Collaborative credibility, hyperparameters This is the credibility modulation coefficient, used to balance immediate gains with collaborative historical contributions; The collaborative credibility is updated recursively according to the following rules. This recursive mechanism can integrate historical and current performance to achieve dynamic evaluation of collaborative capabilities: in To update the credibility coefficient, for Collaborative reliability at time -1 The credibility increment is defined as: in For execution unit cluster State-action pairs Number of visits For execution unit cluster At time step The coordination and consistency function value is determined by the cluster of computation execution units. The cosine similarity with other execution unit clusters is obtained. By calculating the policy similarity between different execution unit clusters, their collaborative consistency can be measured, thereby promoting collaborative optimization.

[0015] Furthermore, in S5, the main network corresponding to each execution unit cluster independently updates its own network parameters at each time step based on the loss function using gradient descent, while the target network parameters do not participate in this update process; after the main network has completed a preset number of parameter updates, the corresponding target network parameters are copied and updated by the main network parameters to improve the stability of the training process.

[0016] Compared with the prior art, the beneficial effects achieved by the present invention are: The multi-agent collaborative resource allocation method based on deep reinforcement learning provided by this invention models the resource allocation decision task as a sequential decision process, thereby achieving a refined characterization of the resource allocation decision task and improving the model's adaptability to complex task scenarios. The multi-agent collaborative resource allocation method based on deep reinforcement learning provided by this invention uses a multi-agent deep reinforcement learning framework to train the model, thereby improving the model's adaptability and robustness in multi-objective and large-scale scenarios. The multi-agent collaborative resource allocation method based on deep reinforcement learning provided by this invention introduces a collaborative credibility component during the training process, guiding agents to take into account long-term collaborative effects, thereby improving the overall performance and generalization ability of multi-execution unit cluster collaborative processing strategies.

[0017] The multi-agent collaborative resource allocation method based on deep reinforcement learning provided by this invention is applicable to various application scenarios such as multi-agent resource scheduling, robot collaborative task allocation, and multi-objective optimization decision-making, and has good versatility and scalability. Attached Figure Description

[0018] Figure 1 Framework diagram of a multi-agent collaborative resource allocation decision-making model.

[0019] Figure 2 Observation range diagram of the execution unit cluster.

[0020] Figure 3 Resource allocation decision results diagram. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to embodiments. It should be understood that the embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0022] S1: For task scenarios involving multiple execution unit clusters, perform configuration initialization; The task scenario includes a cluster of K execution units and Each task objective, each execution unit cluster equipped with Resource units, task objectives include There are several types of execution units, each with a different probability of completing different objectives. Furthermore, each execution unit cluster can only observe and process a subset of the objectives. The number of objectives that execution unit cluster k can observe is represented as... .

[0023] In this example, three execution unit clusters and eight task targets are set up. Each execution unit cluster is equipped with two types of resource units, including six type A resource units and four type B resource units.

[0024] This example sets up a task scenario range of 100km×100km and displays the coordinate information of each execution unit cluster and task target in a sample in Tables 1 and 2.

[0025] Table 1 Execution Unit Cluster Coordinate Information serial number Unit type coordinate 1 Execution Unit Cluster (5,25) 2 Execution Unit Cluster (5,50) 3 Execution Unit Cluster (5,75) Table 2. Task target coordinate information serial number Target type coordinate serial number Target type coordinate 1 Type A (72,42) 5 Type B (78,28) 2 Type A (88 58) 6 Type B (90,68) 3 Type A (85,38) 7 Type B (95,72) 4 Type B (75,32) 8 Type C (82,52) For execution unit clusters ,Target j The visibility is determined by the detection angle constraint. Let the coordinates of the execution unit cluster k in the scene be... ,Target j The coordinates are ,when Time to judge target j Can be executed in a cluster It was observed that, in this example, taking Under this constraint, the targets observable by each execution unit cluster are shown in Figure 2 Among them, execution unit cluster 1 and execution unit cluster 2 can observe all targets, while execution unit cluster 3 can only observe 6 targets. Due to the limitation of the detection angle, it cannot observe 2 of the guard category targets.

[0026] S2: Model the resource allocation decision-making task as a sequential decision-making process, and construct the single-execution-unit cluster-level optimization objective and the global-level optimization objective respectively; The resource allocation decision needs to be made at every time step. Each execution unit cluster generates a resource allocation decision action, and modeling this sequence of decision-making processes requires analysis of each time step. The corresponding actions and states are defined.

[0027] Based on the state of the execution unit cluster itself and the state of its observable targets, the execution unit cluster... At time step observable state under Defined as: in Represents an execution unit cluster At time step The status of resource units; Represents an execution unit cluster At time step The observable target state.

[0028] Resource unit status Specifically, it can be defined as: in Represents an execution unit cluster The resource unit types to be allocated are represented using one-hot encoding, and their dimensions are... , Indicates the current resource unit type The probability vector for completing various objectives has the following dimension: , Represents an execution unit cluster The remaining proportion vectors of various resource units are in the following form: in Represents an execution unit cluster The Resource units in time step The remaining quantity, Represents an execution unit cluster The The initial maximum number of resource units.

[0029] Observable target state It is composed of the states of its fixed set of observable targets: in Represents an execution unit cluster The first observable j The goal is in time step The state can be specifically defined as: in Represents an execution unit cluster The type information of target j within the observation range is represented using one-hot encoding, and its dimension is... , This serves as a reference value for the priority weight of the objective. Indicates the time step up to Execution Unit Cluster To the target j The vector of the number of various resource units allocated, with dimension [missing information]. , and Representing the target j Equivalent to an execution unit cluster The distance and azimuth.

[0030] The time steps are obtained by merging the observable states of a cluster of K execution units. The overall state below : In this example, the completion probability of different resource units for different objectives and the priority weight of each objective are set in Tables 3 and 4.

[0031] Table 3 Reference values ​​for completion probability Type A Target Type B Target Type C target Type A resource unit 0.75 0.50 0.25 Type B resource unit 0.55 0.35 0.15 Table 4 Priority Weight Reference Values Target type Type A Target Type B Target Type C target Priority weight 0.15 0.20 0.45 Based on the state of the execution unit cluster itself and the state of its observable targets, the execution unit cluster... As time goes by Action vectors under Defined as: in Represents an execution unit cluster As time goes by Next, the resource unit to be allocated will be assigned to the first unit in its observable target set. j One goal, j The value ranges from 1 to .

[0032] In S2, the single execution unit cluster-level optimization objective and the global-level optimization objective are constructed as follows: First, we construct a cluster-level optimization objective for a single execution unit, including the reward for completing the target task and the penalty for resource consumption, which are defined as follows: in For execution unit cluster At time step The benefits of completing the target task For execution unit cluster At time step Resource consumption penalty Representing resource units For the target j The probability of completion, For the deadline step Execution Unit Cluster To the target j The allocation of the first The number of resource units. Therefore, the single execution unit cluster-level optimization objective is obtained. for: in , In this example, the weights corresponding to the cluster-level optimization objectives of each individual execution unit are set to 0.9 and 0.1 respectively.

[0033] Then, a global optimization objective is constructed, including objective coverage and global task completion benefits, defined as follows: in For time step Target coverage, For time step Global task completion rewards For indicator functions, For the goal j The priority weight reference values ​​are used to obtain the global optimization objective. for: in , In this example, the weights corresponding to the global optimization objectives are set to 0.3 and 0.7 respectively.

[0034] Finally, the execution unit cluster At time step The immediate rewards are constructed from single-execution-unit cluster-level and global-level optimization objectives: in For execution unit cluster At time step Instant rewards To control the weight of the global reward on the decision-making of a single execution unit cluster, it is set to 0.4 in this example.

[0035] S3: Treat each execution unit cluster as an agent, construct a multi-agent collaborative resource allocation decision model based on deep reinforcement learning, and initialize the parameters; like Figure 1 As shown, the multi-agent collaborative resource allocation decision model constructs a set of independent decision network units for each execution unit cluster. Each decision unit adopts a dual-network structure design, including a main network for current decision evaluation and a target network for stabilizing the training process. The two networks have the same structure but their parameters are independent and not shared.

[0036] During the model parameter initialization phase, the network parameters of the main network corresponding to each execution unit cluster are initialized randomly, and the initialized main network parameters are copied to the corresponding target network parameters to ensure the consistency between the main network parameters and the target network parameters in the initial training phase. in and They represent the first The main network parameters and target network parameters corresponding to each execution unit cluster.

[0037] S4: At each time step, based on the current observation state of each execution unit cluster, each agent generates a corresponding resource allocation decision action, and after executing the action, obtains the state and real-time reward value of the next time step, and stores the interaction data in its own experience cache. In each execution unit cluster, the main network corresponds to each time step. Based on the current observable state of the execution unit cluster As input, the output is the expected payoff vector for taking different resource allocation actions in this state. : in, Represents an execution unit cluster In state The following measures were taken. Candidate resource allocation decision-making actions The corresponding expected return is L, which is the total number of discrete resource allocation actions that the execution unit cluster can take.

[0038] Based on the expected return vector, construct a relative expected return scaling factor: in For execution unit cluster At time step The maximum expected return value of all candidate resource allocation actions. Indicates action Relative to the expected reward ratio of the current optimal action, To prevent extremely small positive numbers with a denominator of zero, this example sets it to... .

[0039] Then, the execution unit cluster In state The probability of choosing the resource allocation action is defined as follows: in Represents an execution unit cluster In state Next choice The final probability of a resource allocation action. This is the profit concentration adjustment factor, which is set to 1.5 in this example.

[0040] The main network of each execution unit cluster at time step Based on the action selection strategy based on the relative expected return ratio, respectively, from the final probability Mid-sampling obtains resource allocation actions in the current state. The actions are then applied synchronously to the environment; after receiving resource allocation actions from all execution unit clusters, the environment jointly updates the overall system state for the next moment. And calculate the corresponding single-step reward value for each execution unit cluster. and with The data is stored in the experience cache of the corresponding execution unit cluster in the form of data to support parallel training and collaborative decision-making of multiple execution unit clusters.

[0041] S5: Sample small batches of data from the experience cache corresponding to each execution unit cluster, calculate the loss function value that introduces collaborative credibility, and use this value to train the multi-agent collaborative resource allocation decision model and update the model parameters. Small batches of experience data are sampled from the experience cache corresponding to each execution unit cluster as training samples, and then input into the main network and target network corresponding to that execution unit cluster respectively; for any single experience sample of execution unit cluster k The target network calculates the target reward value based on the state at the next time step: in For execution unit cluster At time step The target return value, hyperparameters The discount factor is set to 0.99 in this example. To maximize expected return, Represents an execution unit cluster The target network in state Next, make decisions on allocating resources to each candidate. The expected return value.

[0042] Building upon this, a collaborative credibility is introduced to characterize the historical contribution of the execution unit cluster to the overall collaborative processing effect under a specific state-action pair, and a training loss function for the execution unit cluster k is constructed: in For execution unit cluster The loss function of the main network, For the current state-action pair The projected return value, For execution unit cluster State-action pairs Collaborative credibility, hyperparameters This is the credibility modulation coefficient, used to balance immediate gains with collaborative historical contributions; in this example, it is set to 0.1.

[0043] The collaborative credibility potential is updated recursively according to the following rules: in The credibility update coefficient is set to 0.05 in this example. for Collaborative reliability at time -1 The credibility increment is defined as: in For execution unit cluster State-action pairs Number of visits For execution unit cluster At time step The coordination and consistency function value is determined by the cluster of computation execution units. The cosine similarity with other execution unit clusters is obtained.

[0044] Each execution unit cluster's corresponding main network independently updates its own network parameters at each time step based on the loss function using gradient descent, while the target network parameters do not participate in this update process. After the main network has completed a preset number of parameter updates, the corresponding target network parameters are copied and updated by the main network parameters to improve the stability of the training process.

[0045] S6: Repeat steps S4–S5 until training ends, and use the trained model to guide the resource allocation of the multi-execution unit cluster to achieve collaborative resource allocation for multi-task objectives.

[0046] In this example, the trained model is used to process the aforementioned samples, and the resource allocation results are shown in Table 5. The resources allocated to each objective and the probability of completion are shown in Table 6. The resource allocation visualization results are shown in Table 7. Figure 3 .

[0047] Table 5 Resource Allocation Results Table 6 Target Processing Status Target number Target type Type A resource unit Type B resource unit Completion probability 1 Type A 2 0 0.9375 2 Type A 3 1 0.9929 3 Type A 1 2 0.9494 4 Type B 1 1 0.6750 5 Type B 1 0 0.5000 6 Type B 3 2 0.9472 7 Type B 2 2 0.8944 8 Type C 5 4 0.8759 As can be seen from the resource allocation decision results, the resource allocation decision strategy designed by the method of the present invention can ensure that all task objectives are covered by firepower, and each objective has a high probability of completion. In addition, the effect of resource allocation decision can also show the synergistic effect of multi-agents. For objectives 4 and 5 that cannot be observed by execution unit cluster 3, they are handed over to execution unit cluster 1 for key processing, and higher threat task objectives are often allocated more firepower. The effectiveness of the multi-agent collaborative resource allocation method based on deep reinforcement learning in dealing with practical resource allocation problems is evident.

Claims

1. A multi-agent collaborative resource allocation method based on deep reinforcement learning, characterized in that, Includes the following steps: S1: For task scenarios involving multiple execution unit clusters, perform configuration initialization; S2: Model the resource allocation decision-making task as a sequential decision-making process, and construct the single-execution-unit cluster-level optimization objective and the global-level optimization objective respectively; S3: Treat each execution unit cluster as an agent, construct a multi-agent collaborative resource allocation decision model based on deep reinforcement learning, and initialize the parameters; S4: At each time step, based on the current observation state of each execution unit cluster, each agent generates a corresponding resource allocation decision action, and after executing the action, obtains the state and real-time reward value of the next time step, and stores the interaction data in its own experience cache. S5: Sample small batches of data from the experience cache corresponding to each execution unit cluster, calculate the loss function value that introduces collaborative credibility, and use this value to train the multi-agent collaborative resource allocation decision model and update the model parameters. S6: Repeat steps S4–S5 until training ends, and use the trained model to guide the resource allocation of the multi-execution unit cluster to achieve collaborative resource allocation for multi-task objectives.

2. The multi-agent cooperative resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In S1, the task scenario includes a cluster of K execution units and Each task objective, each execution unit cluster equipped with Resource units, task objectives include There are several types of execution units, each with a different probability of completing different objectives. Furthermore, each execution unit cluster can only observe and process a subset of the objectives. The number of objectives that execution unit cluster k can observe is represented as... .

3. The multi-agent cooperative resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In S2, the resource allocation decision needs to be made at every time step. Each execution unit cluster generates a resource allocation decision action, and modeling the sequential decision-making process requires analysis of each time step. The corresponding actions and states are defined.

4. The multi-agent cooperative resource allocation method based on deep reinforcement learning according to claim 3, characterized in that, Based on the state of the execution unit cluster itself and the state of its observable targets, the execution unit cluster... At time step observable state under Defined as: in Represents an execution unit cluster At time step The status of resource units; Represents an execution unit cluster At time step The observable target state; Resource unit status Specifically defined as: in Represents an execution unit cluster The resource unit types to be allocated are represented using one-hot encoding, and their dimensions are... , Indicates the current resource unit type The probability vector for completing various objectives has the following dimension: , Represents an execution unit cluster The remaining proportion vectors of various resource units are in the following form: in Represents an execution unit cluster The Resource units in time step The remaining quantity, Represents an execution unit cluster The The initial maximum number of resource units; Observable target state It is composed of the states of its fixed set of observable targets: in Represents an execution unit cluster The first observable j The goal is in time step The state is specifically defined as: in Represents an execution unit cluster Target within observation range j The type information is represented using one-hot encoding, and its dimension is... , This serves as a reference value for the priority weight of the objective. Indicates the time step up to Execution Unit Cluster To the target j The vector of the number of various resource units allocated, with dimension [missing information]. , and Representing the target j Equivalent to an execution unit cluster Distance and azimuth; The time steps are obtained by merging the observable states of a cluster of K execution units. The overall state below : 。 5. The multi-agent cooperative resource allocation method based on deep reinforcement learning according to claim 3, characterized in that, Based on the state of the execution unit cluster itself and the state of its observable targets, the execution unit cluster... At time step Action vectors under Defined as: in Represents an execution unit cluster At time step Next, the resource unit to be allocated will be assigned to the first unit in its observable target set. j One goal, j The value ranges from 1 to .

6. The multi-agent cooperative resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In S2, the single execution unit cluster-level optimization objective and the global-level optimization objective are constructed as follows: First, we construct a cluster-level optimization objective for a single execution unit, including the reward for completing the target task and the penalty for resource consumption, which are defined as follows: in For execution unit cluster At time step The benefits of completing the target task For execution unit cluster At time step Resource consumption penalty This serves as a reference value for the priority weight of the objective. Representing resource units For the target j The probability of completion, For the deadline step Execution Unit Cluster The first one already assigned to target j Number of resource units Represents an execution unit cluster The Resource units in time step The remaining quantity, Represents an execution unit cluster The The initial maximum number of resource units; Therefore, the single execution unit cluster-level optimization objective is obtained. for: in , The weights corresponding to the cluster-level optimization objectives of each individual execution unit; Then, a global optimization objective is constructed, including objective coverage and global task completion benefits, defined as follows: in For time step Target coverage, For time steps Global task completion rewards For indicator functions, For the goal j The priority weight reference value is used to obtain the global optimization objective. for: in , The weights corresponding to each global-level optimization objective; Finally, the execution unit cluster At time step The immediate rewards are constructed from single-execution-unit cluster-level and global-level optimization objectives: in For execution unit cluster At time step Instant rewards To control the weight of global-level rewards on the decision-making of a single execution unit cluster.

7. The multi-agent cooperative resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In S3, the multi-agent collaborative resource allocation decision model constructs a set of independent decision network units for each execution unit cluster. Each decision unit adopts a dual-network structure design, including a main network for current decision evaluation and a target network for stable training. The two networks have the same structure but their parameters are independent and do not share each other. During the model parameter initialization phase, the network parameters of the main network corresponding to each execution unit cluster are initialized randomly, and the initialized main network parameters are copied to the corresponding target network parameters to ensure the consistency between the main network parameters and the target network parameters in the initial training phase. in and They represent the first The main network parameters and target network parameters corresponding to each execution unit cluster.

8. The multi-agent cooperative resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In S4, the main network corresponding to each execution unit cluster operates at each time step. Based on the current observable state of the execution unit cluster As input, the output is the expected payoff vector for taking different resource allocation actions in this state. : in, Represents an execution unit cluster In state The following measures were taken. Candidate resource allocation decision-making actions The corresponding expected return, where L is the total number of discrete resource allocation actions that the execution unit cluster can take; Based on the expected return vector, construct a relative expected return scaling factor: in For execution unit cluster At time step The maximum expected return value of all candidate resource allocation actions. Indicates action Relative to the expected reward ratio of the current optimal action, To prevent extremely small positive numbers with a denominator of zero; Then, the execution unit cluster In state The probability of choosing the resource allocation action is defined as follows: in Represents an execution unit cluster In state Next choice The final probability of each resource allocation action, This is a factor for adjusting the concentration of returns.

9. The multi-agent cooperative resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In S4, the main network of each execution unit cluster operates at time steps. Based on the action selection strategy according to the relative expected return ratio, from the final probability Mid-sampling obtains resource allocation actions in the current state. The actions are then applied synchronously to the environment; after receiving resource allocation actions from all execution unit clusters, the environment jointly updates the overall system state for the next moment. And calculate the corresponding single-step reward value for each execution unit cluster. and with The data is stored in the experience cache of the corresponding execution unit cluster in the form of data to support parallel training and collaborative decision-making of multiple execution unit clusters.

10. The multi-agent cooperative resource allocation method based on deep reinforcement learning according to claim 1, characterized in that, In S5, small batches of experience data are sampled from the experience cache corresponding to each execution unit cluster as training samples, and then input into the main network and target network corresponding to that execution unit cluster respectively; for any execution unit cluster Single empirical sample The target network calculates the target reward value based on the state at the next time step: in For execution unit cluster At time step The target return value, hyperparameters As the discount factor, To maximize expected return, Represents an execution unit cluster The target network in state Next, make decisions on allocating resources to each candidate. Expected return value; A collaborative credibility is introduced to characterize the historical contribution of the execution unit cluster to the overall collaborative processing effect under a specific state-action pair, and to construct the execution unit cluster. Training loss function: in For execution unit cluster The loss function of the main network, For the current state-action pair The projected return value, For execution unit cluster State-action pairs Collaborative credibility, hyperparameters This is a credibility modulation coefficient used to balance immediate gains with collaborative historical contributions; The collaborative trustworthiness is updated recursively according to the following rules: in To update the credibility coefficient, for Collaborative reliability at time -1 The credibility increment is defined as: in For execution unit cluster State-action pairs Number of visits For execution unit cluster At time step The coordination and consistency function value is determined by the cluster of computation execution units. The cosine similarity with other execution unit clusters is obtained; In S5, the main network corresponding to each execution unit cluster independently updates its own network parameters at each time step based on the loss function using gradient descent, while the target network parameters do not participate in this update process; after the main network has completed a preset number of parameter updates, the corresponding target network parameters are copied and updated by the main network parameters to improve the stability of the training process.

Images