A Multi-Agent Cooperative Learning-Based Multi-Node Resource Scheduling Method

Abstract

This invention discloses a multi-node resource scheduling method based on multi-agent collaborative learning. It constructs a sensor node resource state model and a task requirement model, abstracting each sensor node as an agent. A policy optimization framework of centralized training and distributed execution is adopted, designing a joint reward function and introducing a graph neural network to aggregate neighbor information to achieve collaborative policy updates. The scheduling policy is obtained through offline training, and the resource allocation matrix is ​​output and executed in real time during online operation. This invention enables global collaborative optimization of multi-node resources, avoids resource conflicts, and improves system detection efficiency.

Description

Technical Field

[0001] This invention relates to multi-node resource scheduling technology, and in particular to a multi-node resource scheduling method based on multi-agent collaborative learning. Background Technology

[0002] With the widespread application of distributed sensor systems, the collaborative capability among multiple sensor nodes has become a key factor in improving overall performance. In actual operation, each sensor node typically possesses different transmit power capabilities, beam control capabilities, spectrum occupancy, and signal processing capabilities, while also undertaking multiple types of tasks, resulting in heterogeneous and dynamically changing system resources. Existing sensor resource scheduling methods are mostly based on rule-based allocation or heuristic optimization algorithms, usually focusing on the performance of a single node or local metrics as optimization objectives, making it difficult to comprehensively consider the resource coupling relationships and global performance constraints among multiple sensor nodes. When the number of tasks increases or the environment changes rapidly, problems such as resource conflicts, redundant coverage, or insufficient response to high-threat targets easily arise, leading to a decrease in overall system efficiency. Therefore, there is an urgent need for an intelligent resource scheduling method capable of global collaborative optimization and dynamic adaptive adjustment under multi-node conditions to improve the overall detection capability and resource utilization efficiency of distributed sensor systems. Summary of the Invention

[0003] The purpose of this invention is to provide a multi-node resource scheduling method based on multi-agent collaborative learning, so as to improve the overall detection capability and resource utilization efficiency of distributed sensor systems.

[0004] The technical solution to achieve the purpose of this invention is: a multi-node resource scheduling method based on multi-agent cooperative learning, comprising the following steps:

[0005] Step 1: Construct a resource state model for sensor nodes; establish a resource state vector for each sensor node in the distributed sensor system. The resource state vector includes the node's remaining transmit power, available beam time slice, available spectrum bandwidth, available computing power, and network communication bandwidth. The resource state vectors of all nodes constitute the global resource state.

[0006] Step 2: Construct a task requirement model; Based on the sensor task to be executed, establish a task requirement vector, which includes the power resources required by the task, beam dwell time, spectrum bandwidth occupied, computing resource requirements, task time limit, and task priority level. The task requirement vectors of all tasks constitute the task requirement state.

[0007] Step 3: Establish a multi-agent collaborative decision-making model; abstract each sensor node as an independent agent, construct a multi-agent collaborative learning framework with global resource status and task requirement status as input, adopt a strategy optimization mechanism of centralized training and distributed execution, introduce a centralized value evaluation network in the training phase to evaluate the global benefits of the joint actions of multiple nodes, and in the deployment phase, each sensor node independently outputs resource allocation actions according to the local policy network.

[0008] Step 4: Design a joint reward function and a collaborative constraint mechanism; construct a joint reward function that comprehensively considers the probability of successful detection, task completion delay, system energy consumption, and resource conflict penalty. At the same time, construct a graph structure based on the topological connection relationship between sensor nodes, aggregate neighbor node information through a graph neural network, and use the aggregated node embedding representation as the input to the local policy network of each sensor node to realize the collaborative policy update between sensor nodes.

[0009] Step 5: Execute policy training and online dynamic scheduling; In the offline stage, iteratively train the multi-agent model based on historical simulation data or measured data to obtain a converged resource scheduling policy; In the online operation stage, acquire the resource status and task demand status of the multi-sensor nodes in real time, input them into the policy network of each sensor node, output the resource allocation matrix at the current moment, and execute scheduling.

[0010] Furthermore, in step 1, the resource state vector is defined as:

[0011]

[0012] in, This represents the resource state vector of the i-th sensor node at time t; This represents the remaining transmit power, which is the allocatable power after deducting the base sustaining power, system loss power, and safety margin power, and is expressed in watts (W). This represents the available beam time slice resources, which is the total effective beam dwell time that sensor nodes can be allocated within a unit scheduling cycle, in μs. Indicates the available spectrum bandwidth, in MHz; This represents the computing resources available for signal processing and intelligent algorithms, measured in TFLOPS. This indicates the available network communication bandwidth, measured in Mbps.

[0013] Furthermore, in step 2, the task requirement vector is defined as:

[0014]

[0015] in, This represents the demand vector for the j-th task at time t. , , , These represent the power required for the mission, beam dwell time, spectrum bandwidth, and computational resource requirements, respectively, in W, μs, MHz, and TFLOPS. Indicates the task completion time limit, in μs. Indicates the priority level of the task.

[0016] Furthermore, in step 3, the multi-agent collaborative decision-making model adopts a framework of centralized training plus distributed execution, as detailed below:

[0017] (1) Constructing a centralized value function during the training phase Perform centralized training on all nodes:

[0018]

[0019] in, Here are the parameters for a centralized value network, where N is the number of sensor nodes. The global state includes a combination of the resource state vectors of all nodes and the task requirement vector: , where M is the number of tasks; The resource allocation action for the i-th sensor node at time t is determined by the policy function. The calculation yielded:

[0020]

[0021] in, The policy function for the i-th sensor node The parameters include the resource status vector of the current node and a subset of task requirement vectors related to the current node, where k is the set of task indices associated with the i-th sensor node.

[0022] (2) During the deployment phase, each sensor node operates according to its local policy network. Independent output resource allocation action This enables distributed execution.

[0023] Furthermore, in step 4, the joint reward function Defined as:

[0024]

[0025] in This indicates the overall success rate of the system's detection. This indicates the average task completion time. This indicates the total energy consumption of the system. Indicates resource conflict penalties. , , , These are non-negative weighting coefficients;

[0026] The resource conflict penalty item is defined as follows:

[0027]

[0028] in This represents the scheduling allocation amount of the i-th sensor node for the j-th type of resource. This represents the upper limit of the actual available resources of type j for the i-th sensor node.

[0029] Furthermore, in step 4, sensor nodes establish collaborative constraint relationships through a topological graph G=(V, E), where V represents the set of sensor nodes and E represents the set of communication connection edges between sensor nodes. During the policy update process, a graph neural network information aggregation mechanism is used to fuse the states of neighboring nodes, and its node embedding update formula is:

[0030]

[0031] in This represents the embedded representation of the i-th sensor node at layer l. Let represent the set of sensor nodes adjacent to the i-th sensor node. , For trainable weight matrix, It is a non-linear activation function.

[0032] Furthermore, in step 5, the resource allocation action is a resource allocation matrix M(t), which has the following form:

[0033] M(t) = [m ij (t)] N×M

[0034] in This represents the resource ratio coefficient allocated by the i-th sensor node to the j-th task, satisfying:

[0035] 0≤m ij (t)≤1

[0036]

[0037] And it satisfies the resource conservation constraint:

[0038]

[0039]

[0040]

[0041] .

[0042] A multi-node resource scheduling system based on multi-agent cooperative learning includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multi-node resource scheduling method based on multi-agent cooperative learning.

[0043] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-node resource scheduling method based on multi-agent cooperative learning.

[0044] A non-transitory computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the multi-node resource scheduling method based on multi-agent cooperative learning.

[0045] Compared with the prior art, the significant advantages of this invention are: using a sensor node resource state model to describe the resource state distribution of each sensor under multi-node conditions, using a task requirement model to construct a quantifiable mapping relationship between tasks and resources, establishing a multi-agent collaborative decision-making model, constructing a multi-agent collaborative learning framework with global resource state and task requirement state as inputs, and introducing a joint reward function and collaborative constraint mechanism to realize policy collaborative updates among sensor nodes, thereby avoiding local optimal scheduling behavior and improving the overall system benefit. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the multi-node resource scheduling method based on multi-agent collaborative learning of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] like Figure 1 As shown, a multi-node resource scheduling method based on multi-agent collaborative learning is proposed. First, a system model is constructed through steps 1 and 2. Then, a multi-agent decision-making framework is established in step 3. In step 4, a reward function and a GNN collaborative mechanism are designed. Finally, offline training and online scheduling are completed in step 5. The following provides a detailed explanation of each step in conjunction with this process.

[0049] Step 1: Construct a resource state model for sensor nodes. Use a resource state vector to describe the resource state of the i-th sensor node at time t: .in, This represents the resource state vector of the i-th sensor node at time t; This represents the remaining transmit power (in W), which is the allocatable power after deducting the base sustaining power, system loss power, and safety margin power. The available beam time slice resources (in μs) represent the total effective beam dwell time that can be allocated to a sensor node within a unit scheduling cycle. Indicates available spectrum bandwidth (in MHz); This represents the computing resources available for signal processing and intelligent algorithms (in TFLOPS). This indicates the available network communication bandwidth (in Mbps).

[0050] Step 2: Construct the task requirement model. Use a task requirement vector to describe the resource requirements of the j-th task at time t: .in, This represents the demand vector for the j-th task at time t. , , , These represent the power required for the task (in W), beam dwell time (in μs), spectral bandwidth (in MHz), and computational resource requirements (in TFLOPS), respectively. Indicates the task completion time (in μs). This indicates the task priority level. The task requirement vectors of all tasks constitute the task requirement state.

[0051] Step 3: Establish a multi-agent collaborative decision-making model. Abstract each sensor node as an independent agent, construct a multi-agent collaborative learning framework with global resource state and task requirement state as input, and adopt a strategy optimization mechanism of centralized training and distributed execution.

[0052] (1) In the training phase, a centralized value evaluation network is introduced to evaluate the global benefits of joint actions of multiple nodes and a centralized value function is constructed. All nodes are trained centrally, among which, Here are the parameters for a centralized value network, where N is the number of sensor nodes. The global state includes a combination of the resource state vectors of all nodes and the task requirement vector: , where M is the number of tasks; The resource allocation action for the i-th sensor node at time t is determined by the policy function. The calculation yielded: .in, The policy function for the i-th sensor node The parameters include the resource state vector of the current node and a subset of task requirement vectors associated with the current node (e.g., a task is considered associated with the current node only if the task target is within the node's maximum detection range), and k is the set of task indices associated with the i-th sensor node.

[0053] (2) During the deployment phase, each sensor node operates according to its local policy network. Independent output resource allocation action This enables distributed execution. The resource allocation actions are represented in the form of a resource allocation matrix.

[0054] Step 4: Design a joint reward function and a collaborative constraint mechanism. A joint reward function is constructed that comprehensively considers the probability of successful detection, task completion delay, system energy consumption, and resource conflict penalties. Maximizing this joint reward function is the optimization objective. Simultaneously, a graph structure is built based on the topological connections between sensor nodes. A graph neural network is used to aggregate neighbor node information to achieve collaborative policy updates among sensor nodes, thereby avoiding local optimum scheduling behavior and improving overall system benefits.

[0055] Using a joint reward function:

[0056]

[0057] in This indicates the overall success rate of the system's detection. This indicates the average task completion time. This indicates the total energy consumption of the system. Indicates resource conflict penalties. , , , These are non-negative weighting coefficients;

[0058] The resource conflict penalty item is defined as follows:

[0059]

[0060] in This represents the scheduling allocation amount of the i-th sensor node for the j-th type of resource. This represents the upper limit of the actual available resources of type j for the i-th sensor node.

[0061] For the sensor node topology graph, a graph neural network information aggregation mechanism is used to fuse the states of neighboring nodes. The node embedding update formula is as follows:

[0062]

[0063] in This represents the embedded representation of the i-th sensor node at layer l. Let represent the set of sensor nodes adjacent to the i-th sensor node. , For trainable weight matrix, It is a non-linear activation function.

[0064] Step 5: Execute policy training and online dynamic scheduling. In the offline phase, the multi-agent model is iteratively trained based on historical simulation or experimental data to obtain a converged resource scheduling policy. In the online phase, the resource status and task demand status of multiple sensor nodes are acquired in real time, input into the policy network of each sensor node, and the resource allocation matrix at the current moment is output and scheduling is executed to achieve adaptive dynamic reallocation of resources. After scheduling is executed, the resource status of each node is updated according to resource consumption and enters the next scheduling cycle, thereby achieving adaptive dynamic reallocation of resources.

[0065] The resource allocation matrix M(t) has the following form:

[0066] M(t) = [m ij (t)] N×M

[0067] in This represents the resource ratio coefficient allocated by the i-th sensor node to the j-th task, satisfying:

[0068] 0≤m ij (t)≤1

[0069]

[0070] And it satisfies the resource conservation constraint:

[0071]

[0072]

[0073] This invention also proposes a multi-node resource scheduling system based on multi-agent cooperative learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multi-node resource scheduling method based on multi-agent cooperative learning.

[0074] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-node resource scheduling method based on multi-agent cooperative learning.

[0075] A non-transitory computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the multi-node resource scheduling method based on multi-agent cooperative learning.

[0076] Example

[0077] To verify the effectiveness of the present invention, the above method will be explained below using a four-node sensor resource scheduling scenario as an example.

[0078] Step 1: Construct a resource state model for the sensor nodes. Use resource state vectors to describe the resource state of the four sensor nodes at time t: Initial setup of the 4 sensor nodes ( The resource parameters are as follows:

[0079]

[0080]

[0081]

[0082]

[0083] Step 2: Construct the task requirement model. Use a task requirement vector to describe the resource requirements of each task at time t: Design the following 6 tasks, with their initial times ( The parameters are as follows:

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] Step 3: Establish a multi-agent collaborative decision-making model. Abstract the four sensor nodes into four independent agents, construct a multi-agent collaborative learning framework with global resource state and task requirement state as input, and adopt a strategy optimization mechanism of centralized training and distributed execution.

[0091] (1) In the training phase, a centralized value evaluation network is introduced to evaluate the global benefits of joint actions of multiple nodes and a centralized value function is constructed. All nodes are trained centrally, using ReLU as the activation function; The global state includes a combination of the resource state vectors of all nodes and the task requirement vector: ; The resource allocation action for the i-th sensor node at time t is determined by the policy function. The calculation yielded: .in, The policy function for the i-th sensor node The parameters include the resource state vector of the current node and a subset of task requirement vectors related to the current node.

[0092] (2) During the deployment phase, each sensor node operates according to its local policy network. Independent output resource allocation action This enables distributed execution.

[0093] Step 4: Design a joint reward function and a collaborative constraint mechanism. A joint reward function is constructed that comprehensively considers the probability of successful detection, task completion delay, system energy consumption, and resource conflict penalties. Maximizing this joint reward function is the optimization objective. Simultaneously, a graph structure is built based on the topological connections between sensor nodes. A graph neural network is used to aggregate neighbor node information to achieve collaborative policy updates among sensor nodes, thereby avoiding local optimum scheduling behavior and improving overall system benefits.

[0094] Using a joint reward function:

[0095]

[0096] in This indicates the overall success rate of the system's detection. This indicates the average task completion time. This indicates the total energy consumption of the system. Indicates resource conflict penalties. , , , These are non-negative weighting coefficients, with initial values ​​of 1.0, 0.5, 0.1, and 1.0, respectively.

[0097] The resource conflict penalty item is defined as follows:

[0098] .

[0099] in This represents the scheduling allocation amount of the i-th sensor node for the j-th type of resource. This represents the upper limit of the actual available resources of type j for the i-th sensor node.

[0100] For the sensor node topology graph, a graph neural network information aggregation mechanism is used to fuse the states of neighboring nodes, and its node embedding update formula is as follows:

[0101]

[0102] in This represents the embedded representation of the i-th sensor node at layer l. Let represent the set of sensor nodes adjacent to the i-th sensor node. , For trainable weight matrix, This is the ReLU activation function.

[0103] Step 5: Execute policy training and online dynamic scheduling. In the offline phase, the multi-agent model is iteratively trained based on historical simulation data or measured data to obtain a converged resource scheduling policy. In the online operation phase, the resource status and task demand status of multiple sensor nodes are acquired in real time, input into the policy network of each sensor node, output the resource allocation matrix at the current moment, and execute scheduling to achieve adaptive dynamic reallocation of resources.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-node resource scheduling method based on multi-agent cooperative learning, characterized in that, Includes the following steps: Step 1: Construct a resource state model for sensor nodes; establish a resource state vector for each sensor node in the distributed sensor system. The resource state vector includes the node's remaining transmit power, available beam time slice, available spectrum bandwidth, available computing power, and network communication bandwidth. The resource state vectors of all nodes constitute the global resource state. Step 2: Construct a task requirement model; Based on the sensor task to be executed, establish a task requirement vector, which includes the power resources required by the task, beam dwell time, spectrum bandwidth occupied, computing resource requirements, task time limit, and task priority level. The task requirement vectors of all tasks constitute the task requirement state. Step 3: Establish a multi-agent collaborative decision-making model; abstract each sensor node as an independent agent, construct a multi-agent collaborative learning framework with global resource status and task requirement status as input, adopt a strategy optimization mechanism of centralized training and distributed execution, introduce a centralized value evaluation network in the training phase to evaluate the global benefits of the joint actions of multiple nodes, and in the deployment phase, each sensor node independently outputs resource allocation actions according to the local policy network. Step 4: Design a joint reward function and a collaborative constraint mechanism; construct a joint reward function that comprehensively considers the probability of successful detection, task completion delay, system energy consumption, and resource conflict penalty. At the same time, construct a graph structure based on the topological connection relationship between sensor nodes, aggregate neighbor node information through a graph neural network, and use the aggregated node embedding representation as the input to the local policy network of each sensor node to realize the collaborative policy update between sensor nodes. Step 5: Execute policy training and online dynamic scheduling; in the offline stage, iteratively train the multi-agent model based on historical simulation data or measured data to obtain a converged resource scheduling policy; During the online operation phase, the resource status and task requirement status of multiple sensor nodes are acquired in real time, input into the policy network of each sensor node, and the resource allocation matrix at the current moment is output and the scheduling is executed.

2. The multi-node resource scheduling method based on multi-agent cooperative learning according to claim 1, characterized in that, In step 1, the resource state vector is defined as: ； in, This represents the resource state vector of the i-th sensor node at time t; This represents the remaining transmit power, which is the allocatable power after deducting the base sustaining power, system loss power, and safety margin power, and is expressed in watts (W). This represents the available beam time slice resources, which is the total effective beam dwell time that sensor nodes can be allocated within a unit scheduling cycle, in μs. Indicates the available spectrum bandwidth, in MHz; This represents the computing resources available for signal processing and intelligent algorithms, measured in TFLOPS. This indicates the available network communication bandwidth, measured in Mbps.

3. The multi-node resource scheduling method based on multi-agent cooperative learning according to claim 2, characterized in that, In step 2, the task requirement vector is defined as: ； in, This represents the demand vector for the j-th task at time t. , , , These represent the power required for the mission, beam dwell time, spectrum bandwidth, and computational resource requirements, respectively, in W, μs, MHz, and TFLOPS. Indicates the task completion time limit, in μs. Indicates the priority level of the task.

4. The multi-node resource scheduling method based on multi-agent cooperative learning according to claim 3, characterized in that, In step 3, the multi-agent collaborative decision-making model adopts a framework of centralized training plus distributed execution, as follows: (1) Constructing a centralized value function during the training phase Perform centralized training on all nodes: ； in, Here are the parameters for a centralized value network, where N is the number of sensor nodes. The global state includes a combination of the resource state vectors of all nodes and the task requirement vector: , where M is the number of tasks; The resource allocation action for the i-th sensor node at time t is determined by the policy function. The calculation yielded: ； in, The policy function for the i-th sensor node The parameters include the resource status vector of the current node and a subset of task requirement vectors related to the current node, where k is the set of task indices associated with the i-th sensor node. (2) During the deployment phase, each sensor node operates according to its local policy network. Independent output resource allocation action This enables distributed execution.

5. A multi-node resource scheduling method based on multi-agent cooperative learning according to claim 4, characterized in that, In step 4, the joint reward function Defined as: ； in This indicates the overall success rate of the system's detection. This indicates the average task completion time. This indicates the total energy consumption of the system. Indicates resource conflict penalties. , , , These are non-negative weighting coefficients; The resource conflict penalty item is defined as follows: ； in This represents the scheduling allocation amount of the i-th sensor node for the j-th type of resource. This represents the upper limit of the actual available resources of type j for the i-th sensor node.

6. A multi-node resource scheduling method based on multi-agent cooperative learning according to claim 5, characterized in that, In step 4, sensor nodes establish collaborative constraint relationships through a topological graph G=(V, E), where V represents the set of sensor nodes and E represents the set of communication connection edges between sensor nodes. During the policy update process, a graph neural network information aggregation mechanism is used to fuse the states of neighboring nodes, and the node embedding update formula is as follows: ； in This represents the embedded representation of the i-th sensor node at layer l. Let represent the set of sensor nodes adjacent to the i-th sensor node. , For trainable weight matrix, It is a non-linear activation function.

7. A multi-node resource scheduling method based on multi-agent cooperative learning according to claim 6, characterized in that, In step 5, the resource allocation action is a resource allocation matrix M(t), which has the following form: M(t)=[m ij (t)] N×M ; in This represents the resource ratio coefficient allocated by the i-th sensor node to the j-th task, satisfying: 0≤m ij (t)≤1; ； And it satisfies the resource conservation constraint: ； ； ； 。 8. A multi-node resource scheduling system based on multi-agent cooperative learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-node resource scheduling method based on multi-agent cooperative learning as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-node resource scheduling method based on multi-agent cooperative learning as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-node resource scheduling method based on multi-agent cooperative learning as described in any one of claims 1 to 7.

Images