An unmanned cluster task resource allocation method based on deep reinforcement learning

Abstract

The application discloses a kind of based on deep reinforcement learning's unmanned cluster task resource allocation method, belong to artificial intelligence technical field, the method includes: based on unmanned cluster task resource allocation architecture, the Markov model of unmanned cluster task resource allocation problem is established;And design state representation mode, with image to describe system state;Image state input convolutional neural network is designed based on convolutional neural network;With the image state input convolutional neural network as the current network and target network of DQN model, and introduce transfer learning method, construct transfer deep reinforcement learning model;Based on the Markov model and the image description of system state, policy learning is carried out using the transfer deep reinforcement learning model, unmanned cluster task resource allocation is realized.The technical scheme of the application can reduce overall task resource allocation delay and avoid resource waste.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for allocating resources for unmanned swarm tasks based on deep reinforcement learning. Background Technology

[0002] With the rapid development and widespread adoption of unmanned technology, unmanned swarms, also known as multi-agent systems (MAS), have emerged as an important form of intelligent system and are widely used in fields such as emergency rescue, logistics distribution, and military operations. Unmanned systems offer advantages such as low cost, operational flexibility, and immunity to casualties, enabling them to perform missions in harsh and dangerous environments.

[0003] MAS (Unmanned Aerial Vehicles) typically consist of multiple intelligent agents, including drones, unmanned vehicles, and unmanned vessels, which work together to accomplish specific tasks, such as searching for targets, monitoring the environment, or performing coordinated operations. For example, in a swarm of single-type unmanned platforms, a drone swarm can fully leverage the advantages of drone platforms—mobility, low cost, and wide applicability—to efficiently and collaboratively complete diverse tasks through a combination of capabilities, while a surface unmanned vessel swarm can conduct long-duration, large-scale search and rescue operations in complex and dangerous sea conditions.

[0004] In practical applications, task resource allocation, task assignment, and path planning constitute the core challenges in unmanned swarm task planning, directly determining the efficiency, performance, and adaptability of the MAS (Automatic Search System). Traditional unmanned swarm task planning relies on manually defined rules or heuristic algorithms, such as Tabu Search (TS) and Simulated Annealing (SA). While these methods may perform well in static environments or simple tasks, their performance is often limited when facing complex and dynamic task environments. Traditional methods often fall short of optimal resource utilization and task efficiency, impacting overall execution performance. Today, unmanned swarm systems (such as autonomous vehicles and drones) are gradually becoming an important component in the construction of smart logistics, smart cities, and automation exploration. The efficiency of these systems highly depends on their internal task assignment and resource management strategies. However, unmanned swarms face multiple challenges from dynamic environments, such as dynamic changes in resource demands, time sensitivity of task execution, and effective utilization of heterogeneous resources. In such environments, the real-time and variability of requests place enormous pressure on the efficient allocation of resources. Traditional resource allocation methods typically assume that resource requirements are fixed, which simplifies the task resource allocation problem. However, these methods often lack the flexibility and adaptability of solutions, thus necessitating a new approach capable of handling complex and ever-changing environments. Summary of the Invention

[0005] This invention provides a resource allocation method for unmanned swarm tasks based on deep reinforcement learning, in order to solve the technical problems of insufficient resource utilization and task efficiency of traditional resource allocation methods.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] On one hand, the present invention provides a resource allocation method for unmanned swarm tasks based on deep reinforcement learning, the resource allocation method for unmanned swarm tasks based on deep reinforcement learning includes:

[0008] Based on the unmanned cluster task resource allocation architecture, a Markov model for the unmanned cluster task resource allocation problem is established; and a state representation method is designed to describe the system state using images.

[0009] Design an image state input convolutional neural network based on convolutional neural networks;

[0010] The image state is input into a convolutional neural network as the current network and target network of the DQN model, and a transfer learning method is introduced to construct a transfer deep reinforcement learning model.

[0011] Based on the Markov model and the image description of the system state, the transfer deep reinforcement learning model is used for policy learning to achieve resource allocation for unmanned cluster tasks.

[0012] Furthermore, the unmanned cluster task resource allocation architecture includes an unmanned resource management center, which manages multiple unmanned resource bases. Each unmanned resource base contains several unmanned resources of different types and quantities. The unmanned resource management center receives task resource requests from multiple different locations, consisting of different resource requirements. These requests arrive in a discrete time step manner and are added to the request queue of the unmanned resource management center. Each task resource request contains the type of resource required by the current task, the estimated task execution time, the quantity of resources, and the location where the request was initiated. Based on the information contained in the task resource requests, the unmanned resource management center evaluates the remaining available resources in each unmanned resource base, determines which unmanned resource base should respond to which task resource request, and allocates the resources accordingly.

[0013] Furthermore, based on the unmanned cluster task resource allocation architecture, a Markov model for the unmanned cluster task resource allocation problem is established; and a state representation method is designed to describe the system state using images, including:

[0014] Based on the unmanned cluster task resource allocation architecture, the resource allocation problem of unmanned cluster tasks is described;

[0015] Design a state space and use images to describe the system state;

[0016] Design the action space;

[0017] Design a reward function.

[0018] Furthermore, the resource allocation problem for unmanned cluster tasks based on the unmanned cluster task resource allocation architecture is described, including:

[0019] Let M be the number of unmanned resource bases under the jurisdiction of the unmanned resource management center, denoted by set B, where B = {b1, b2, ..., b}. M}, b k Let represent the k-th unmanned resource base, k∈1,...,M; the number of unmanned resource types in each unmanned resource base is N, and the available resource quantity of each unmanned resource is represented by the set Q, where Q={q1,q2,...,q N}, q j This represents the available quantity of the j-th type of unmanned resource, j∈1,...,N; the unmanned resource management center receives n task resource requests, and assigns a unique identifier "task" to the i-th task resource request. i And represented using the following vector:

[0020] task i =[q1,q2,...,q N D i Location], D i ≤T max ,i∈1,...,n

[0021] Among them, D i T represents the estimated execution time of the i-th task under ideal conditions; max This indicates the maximum request duration for a single task resource request; Location indicates the geographic coordinates of the location where the task resource request was initiated; the number of requests for each type of resource in a task resource request cannot exceed the maximum number of that type of resource in each unmanned resource base; the unmanned resource management center will extract information from each task resource request and maintain a mapping table for each task resource request, which includes the resource requirement configuration of the task resource request and the normalized distance of the request initiation location relative to each unmanned resource base.

[0022] The unmanned resource management center maintains a fixed-length queue of pending tasks and a backlog queue for all task resource requests. Incoming pending task resource requests will wait in the queue of pending tasks. When the queue of pending tasks is full, subsequent task resource requests will be stored in the backlog queue.

[0023] If the current unmanned resource base has no resources or insufficient resources, the unmanned resource management center will allocate a pending task resource request to that unmanned resource base from the task allocation queue, to be processed after the currently allocated task resources are released; the pending task resource request will be processed over time, and as long as the task allocation queue is not empty, the unmanned resource management center will continuously allocate new task resource requests to each unmanned resource base; the delay S of the i-th task resource request. i =C i / D i S i ≥1, where C i This represents the actual completion time of the i-th task resource request. The actual completion time includes the waiting time of the task resource request in the backlog queue and the queue of tasks to be assigned, the unmanned cluster travel time, and the estimated task completion time.

[0024] Furthermore, the design state space, which uses images to describe the system state, includes:

[0025] Design a state space, which includes the following elements:

[0026] The usage and remaining quantity of each resource in each unmanned resource base;

[0027] A queue of tasks to be assigned, where each task contains its resource requirements.

[0028] The backlog of task resource requests waiting to be assigned to the task queue;

[0029] The system state is described using images, including state images of the task queue and the unmanned resource base. In both the task queue and the unmanned resource base, each resource corresponds to a state image, and the images are structured like a two-dimensional matrix, with the entire image divided into multiple grids. In each state image of the task queue, the number of grid cells filled horizontally represents the number of resources requested by the current task, and the number of grid cells filled vertically represents the expected usage duration of the corresponding resource. In each state image of the unmanned resource base, the number of grid cells filled horizontally represents the number of resources currently occupied, and the number of grid cells filled vertically represents the expected usage duration of the current resource by the corresponding task. Unfilled areas represent the remaining resources in the unmanned resource base. At the end of each time step, the resource occupancy status in the unmanned resource base's state image shifts upwards by one grid cell, representing a unit of time step advancement.

[0030] Furthermore, the design action space includes:

[0031] The action space is defined as the set of all possible task allocation decisions. Each task allocation decision is described as: allocating a task from a resource request in the task queue to a certain unmanned resource base.

[0032] Furthermore, the design reward function includes:

[0033] The reward function is designed into two parts: a core reward part and a secondary reward part, expressed as follows:

[0034] Reward = R Main +R Aux

[0035] Where Reward represents the reward function; R Main Indicates core reward; R Aux Indicates supplementary rewards;

[0036] The mathematical expression for core rewards is:

[0037] R Main =-(R) Basement +R Queue +R Backlog )

[0038] in, k is the base identifier, p k For all planned operations in base k; t pk The execution time required for the planned operation; t i Let be the estimated execution time of the i-th request in the queue of tasks to be assigned; t g α is the estimated execution time for the g-th request in the backlog queue; k =β=γ=1;

[0039] The mathematical expression for supplementary rewards is:

[0040] R Aux =R Load +R Distance

[0041] Among them, R Load Rewards for base load balancing; R Distance Allocate rewards based on the nearest available task resources;

[0042] count k This is a count of the number of tasks assigned to base k at the current moment; Lb k and Ltask i Base b kAvailable load and tasks i The required load; For task i The normalized distance from base k; η1 and η2 are unit reward parameters.

[0043] Furthermore, the first layer of the image state input convolutional neural network is a convolutional layer with 16 convolutional kernels, each kernel being 3×3 in size. This layer is responsible for extracting primary feature representations from the input data and outputting feature maps with unchanged height and width but a depth of 16. Subsequently, a max pooling layer using a 2×2 window is used to reduce the spatial dimension of each feature map to half its original size. Then, a Dropout layer is applied. Subsequently, a flattening layer is used to flatten the high-dimensional feature data output by the convolutional layer into a one-dimensional array, so that the output of the convolutional layer is passed to two standard fully connected layers. In the two fully connected layers, the first fully connected layer has 256 neurons and uses the ReLU activation function, which is responsible for further feature processing and integration, while the second fully connected layer outputs the final task resource allocation decision.

[0044] Furthermore, the Dropout layer randomly sets 20% of the activation units to zero during training.

[0045] Furthermore, based on the Markov model and the image description of the system state, the transfer deep reinforcement learning model is used for policy learning to achieve resource allocation for unmanned swarm tasks, including:

[0046] Initialize the state space and action space, load the transfer deep reinforcement learning model or initialize a new transfer deep reinforcement learning model. Specifically: Define multiple task sets, each containing multiple task resource requests. For training each task set, initialize its state space and action space. If valid model parameters have been obtained in the previous task training, directly load the previous transfer deep reinforcement learning model to skip the initial learning phase; if there is no previous model, initialize a new transfer deep reinforcement learning model with random parameters.

[0047] The agent selects an action;

[0048] The model observes the state and reward, and stores them in the experience cache.

[0049] Update model parameters by sampling small batches of data from the experience cache.

[0050] In another aspect, the present invention also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method.

[0051] In another aspect, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the above method.

[0052] The beneficial effects of the technical solution provided by this invention include at least the following:

[0053] This invention, based on an unmanned swarm task resource allocation architecture, establishes a Markov model, quantifies resource state representation using images, designs an image state input convolutional neural network, and employs the DQN algorithm to solve the unmanned swarm task resource allocation problem. Furthermore, it introduces transfer learning to improve algorithm performance. This method can be applied to unmanned swarm task resource allocation systems, addressing issues such as allocation efficiency and fairness; reducing task resource allocation latency; and avoiding resource waste. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of the execution flow of the unmanned cluster task resource allocation method based on deep reinforcement learning provided in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the process for establishing a Markov model provided in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the unmanned cluster task resource allocation architecture provided in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the image representation of the state space provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of the DQN algorithm structure provided in an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of an image state input convolutional neural network provided in an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the neural network structure provided in an embodiment of the present invention;

[0062] Figure 8 This is a schematic diagram of the algorithm training process provided in an embodiment of the present invention;

[0063] Figure 9 This is a system block diagram of the electronic device provided in the embodiments of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0065] First, it should be noted that in the embodiments of the present invention, the words "exemplarily," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplarily" is intended to present the concept in a specific manner. Furthermore, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either one or the other.

[0066] First Embodiment

[0067] This embodiment addresses the challenges of unmanned swarm task planning by introducing deep reinforcement learning theory and providing a resource allocation method for unmanned swarm tasks based on deep reinforcement learning. This method can be implemented by an electronic device, which can be a terminal or a server. The execution flow of this method is as follows: Figure 1 As shown, it includes the following steps:

[0068] S1. Based on the unmanned cluster task resource allocation architecture, a Markov model for the unmanned cluster task resource allocation problem is established; and a state representation method is designed to describe the system state using images.

[0069] Specifically, the implementation process of S1 described above is as follows: Figure 2 As shown, it includes:

[0070] S11, Description of the resource allocation problem for drone swarm missions.

[0071] Specifically, this invention considers establishing an unmanned resource management center (hereinafter referred to as the management center) in a certain region, which manages multiple unmanned resource bases (hereinafter referred to as bases). Each base contains several different types and quantities of unmanned resources, such as drones and unmanned vehicles. The management center will receive many task resource requests (hereinafter referred to as requests) from different locations, consisting of different resource requirements. Requests arrive in a discrete time step manner and are added to the management center's request queue. The request includes the type of resource required for the task, the estimated task execution time, the quantity of resources, and the location where the resource request was initiated. Based on the above information, the management center evaluates the remaining available resources in each base, determines which base to respond to which request, and allocates the resources. The unmanned cluster task resource allocation architecture is as follows: Figure 3As shown.

[0072] This invention is based on Figure 3 Based on the existing architecture, the problem is mathematically defined. Let the number of bases under the management center be M, represented by a set B, where B = {b1, b2, ..., b...}. M Let N be the number of heterogeneous unmanned resource types within each base, and let Q represent the available resources for each type of unmanned resource, where Q = {q1, q2, ..., q...} N}, j∈1,...,N, the management center receives n requests, and assigns a unique identifier "task" to each request. i And represented using the following vector:

[0073] task i =[q1,q2,...,q N D i Location], D i ≤T max ,i∈1,...,n(1)

[0074] Among them, D i T represents the expected execution time of the task under ideal conditions. max This represents the maximum request duration limit for each request, designed to prevent unnecessary consumption of task resources. Location represents the geographic coordinates of the request's originating location. The number of requests for each type of resource in a request cannot exceed the maximum number of that type of resource in each base. The management center will extract key information from each request and maintain a mapping table for each request, containing the resource requirement configuration of the request and the normalized distance of the request's originating location relative to each base. (by Location and each base b) k (Transformed from distance).

[0075] The management center maintains a fixed-length "task queue to be assigned" and a "backlog queue" for all requests. Incoming resource requests to be assigned will wait in the task queue to be assigned, and when the queue is full, subsequent request information will be stored in the backlog queue.

[0076] If a base currently has no resources or insufficient resources, the management center will allocate a pending request to that base from the task queue, pending processing once the currently allocated task resources are released. Pending requests will be processed over time, and as long as the task queue is not empty, the management center will continuously allocate new requests to each base.

[0077] Since the allocation of resources in unmanned clusters is not sensitive to the specific execution details of tasks, the main objective of this invention is to minimize overall task execution time and avoid resource waste by optimizing resource allocation. Therefore, the concept of task latency in server resource scheduling is introduced to measure the impact of the task resource allocation algorithm on the overall task execution efficiency. Let the task latency be S, and the actual request completion time be C. i This time includes the waiting time for requests in the backlog queue, the waiting time for assigned tasks, the unmanned cluster travel time, and the estimated task completion time. Therefore, S... i =C i / D i S i ≥1.

[0078] S12, State-space design.

[0079] Each request task i =[q1,q2,...,q N D i We break down [q1, q2, ..., q...] into its location and extract the specific resource requirements for each request, i.e., [q1, q2, ..., q...]. N D i This is then converted into a graphical representation of the state space. In the problem addressed in this invention, the state space should contain the following elements:

[0080] 1) Resource occupancy status of each base. That is, the usage and remaining quantity of each type of resource in the base.

[0081] 2) The queue of tasks to be assigned. Each task contains its specific resource configuration requirements.

[0082] 3) The backlog of resource requests waiting to be assigned to the task queue.

[0083] Therefore, in order to flexibly represent the state space, this invention uses different images to describe the state of the system, which includes the current request processing state of the base, the state of available resources, the resource demand of tasks in the task queue to be assigned, and the state of the backlog queue, thus avoiding complex state space description.

[0084] Figure 4For the specific design of the state space, the image structure resembles a two-dimensional matrix. Multiple colors are used in the image for ease of explanation, but in the actual algorithm implementation, color channels do not need to be distinguished. Each resource type has a state representation image. The leftmost column shows the current resource occupancy status of different requests in different bases, as well as the available quantity of each resource. The queue of tasks to be assigned displays the resource configurations required by requests currently waiting for a response. The capacity of the queue of tasks to be assigned is set to an upper limit of C, meaning that only C state images representing the required resource configurations are maintained for each request to be assigned. This is because a fixed number of states facilitates input to the neural network, and because the system is based on the premise that requests are insensitive to allocation timing, maintaining a limited number of state representations for tasks to be assigned allows control over the dimensionality of the action space.

[0085] In each status image of the task queue, each row represents the amount of resources requested by the task, and each column represents the expected duration of resource usage. Figure 4 In the diagram, the resources required by a request in taskslot 1 are represented by a color, indicating that the task requires 2 units of resource 'a', 1 unit of resource 'b', and 2 units of time. The base's status image follows the same rule. On the left side of the diagram, the two colors in the status image of base 2 represent two different cluster tasks currently occupying resources of types 'a' and 'b' in the base, while the remaining white area represents the current available resource scale of the base. After each time step, the resource occupancy status representation in the base's status image shifts up one unit, representing a unit advancement in the time step. When the management center assigns a waiting request to a base in a taskslot, the resource configuration representation for that task is filled into the base's status image, and the task's status representation is removed from that taskslot. The management center then selects requests from the backlog queue according to their arrival order, generates their resource configuration image status representations, and fills them into empty taskslots. The backlog queue is not displayed in the status representation. The management center maintains a database mapping for each request arriving in the backlog queue, containing the required resource configuration and location information for that request.

[0086] The input to the state space is essentially a set of W×H×N three-dimensional tensor images, where W represents the width of the image, which is used to represent the upper limit of the number of unmanned resources; H represents the length of the image, which is used to represent the upper limit of the processing time required for the task; and N represents the number of channels of the image, which is used to represent the number of unmanned resource types.

[0087] S13, motion space design.

[0088] Specifically, for the unmanned cluster task resource allocation problem, the action space is defined as the set of all possible task allocation decisions, described as "arranging the task slots in the queue of tasks to be allocated".c Tasks in C, c∈1,..., are assigned to base b. k The management center synchronously assigns tasks to each base at each time step; therefore, the entire system's execution can be considered as a joint action, defined as follows:

[0089] a={ψ,b k ,taskslot1,...,taskslot C}(2)

[0090] Here, ψ indicates whether a task is assigned to the base in the current time step; ψ = 1 when resource allocation is performed, and ψ = 0 otherwise. The global action space consists of the possible scheduling action space of the management center for multiple bases, so the size of the action space is (MC+1), where M is the number of bases and C is the capacity of the task queue to be assigned.

[0091] Within each time step, the management center can take multiple actions until it chooses not to allocate any further requests at the current step (ψ = 0) or to perform an invalid allocation operation (e.g., attempting to allocate requests to the base that do not meet the current remaining time resources). This is a stochastic exploration mechanism. In the actual task resource allocation process, this exploration can be achieved by trying to allocate requests to the base that currently do not seem suitable, or by pausing allocation to the base to wait for a better opportunity to allocate resource requests, although this may not conform to the current optimal strategy. This helps the algorithm to explore new and potentially more effective strategies while ensuring that known effective strategies are utilized. For effective allocation, a task is allocated at the first available time step of the base, and the system load state changes accordingly. After choosing "no task allocation" or "invalid task allocation," time continues so that new requests can enter the queue of tasks to be allocated, while requests in the base are processed.

[0092] S14, Reward Function Design.

[0093] Specifically, the present invention designs the reward into two parts: a core reward part and a supplementary reward part.

[0094] Reward = R Main +R Aux (3)

[0095] The core reward component prioritizes average task latency as the primary concern in allocating resources for unmanned cluster tasks, while the secondary reward component focuses more on fairness in the allocation process.

[0096] 1) Core Rewards

[0097] In this invention, the reward is designed to enable the algorithm to learn to minimize task latency. Therefore, the reward function is linked to the time resource requirement of each request, with the reciprocal of the request's time resource length serving as a core component of the reward. The mathematical expression for the reward is as follows:

[0098] R Main =-(R) Basement +R Queue +R Backlog (4)

[0099] Among them, R Basement R Queue R Backlog These represent base rewards, pending task queue rewards, and backlog queue rewards, respectively. By introducing a negative reward mechanism, this reward function explicitly sets the optimization goal of reducing task waiting time and total execution time, aligning with the overall goal of reducing the average task latency S of the unmanned swarm system. Each reward is defined as follows:

[0100] (1) Base Rewards

[0101]

[0102] In the formula, k is the base identifier. p represents all planned jobs in the current base (including invalid task resources). By using the reciprocal of the sum of the execution times of the assigned tasks on each base as part of the reward, the system can be made to favor rewarding multiple short tasks that can be completed quickly, rather than a few long tasks, thereby improving the system's throughput in the same amount of time.

[0103] (2) Rewards for the task queue to be assigned

[0104]

[0105] Among them, t i Let be the estimated execution time of the i-th request in the task queue. The actual task completion time C is reduced by encouraging the algorithm to decrease the waiting time of a single request in the task queue. i This also improves the base's fairness in treating all requests, preventing some requests from waiting for extended periods.

[0106] (3) Backlog queue rewards

[0107]

[0108] Among them, t gThis represents the estimated execution time of the g-th request in the backlog queue. The purpose of this reward is to encourage the processing of tasks in the backlog queue, which helps manage and reduce tasks that cannot be processed immediately due to resource constraints, optimizes resource utilization, avoids waste of system resources, and ensures that requests in the backlog queue can also be processed.

[0109] In summary, the final total core reward function is:

[0110]

[0111] In this model, if all weight factors are set to α k Since β = γ = 1, and without considering other forms of reward, the absolute value of the cumulative reward will become consistent with the sum of the average task latency over time. Therefore, this reward objective helps to minimize the average task latency. If you want requests to be processed as quickly as possible, you can assign a slightly larger value to the parameter β to impose a greater penalty on requests in the task queue.

[0112] 2) Auxiliary Rewards

[0113] Beyond the core reward mechanism, this invention also addresses two other issues while processing requests: firstly, load balancing between bases. To prevent a particular base from significantly exceeding the load of other bases at a given time step, the reward function is designed based on load balancing between bases; secondly, the spatial rationality of task allocation. Therefore, this invention designs auxiliary reward functions for these two issues.

[0114] R Aux =R Load +R Distance (9)

[0115] (1) Base load balancing reward

[0116]

[0117] In the formula, count k This counts the number of tasks assigned to base k at the current time step. The reward function is designed to balance the load on each base in each time step, encouraging the assignment of tasks with higher load requirements to bases with higher available load. Lb k and Ltask i Base b k Available load and tasks i The following describes the methods for assessing the load required for the base and the load required for each request.

[0118] Based on the available quantities of each resource in each base, q1,q2,...,q Nand the current maximum allocatable duration t remain =T max -∑ p t p Construct an M×(N+1) decision matrix B = (b ki ) M×(N+1) Each row represents the sample to be evaluated from one of the bases, denoted as:

[0119]

[0120] Standardizing the elements of matrix B, we can obtain

[0121]

[0122] Use z ij Constructing the standardized matrix Z:

[0123]

[0124] The maximum value of matrix Z is defined as

[0125]

[0126] Minimum value is

[0127]

[0128] Calculate the distance between the i-th base and the maximum value, and the distance between the i-th base and the minimum value.

[0129]

[0130] In the formula, λ j The weighting coefficients for different resource metrics are all set to 1 in the training environment. Therefore, the available load of the i-th base is calculated as follows:

[0131]

[0132] In the formula, Lb k ∈[0,1], it can be seen that Lb k The larger the value, the more resources the base currently has available. For each request, the resource request configuration information maintained by the management center can be used to determine the required load (Ltask) for each request using the same evaluation method as the base. i .

[0133] (2) Rewards for resource allocation based on proximity

[0134]

[0135] in, Let i be the normalized distance from each base. This formula aims to reward the decision to assign the task to the nearest base at each time step; the smaller the normalized distance between the base and the request in each assignment, the greater the reward.

[0136] In summary, the total assistance reward per episode is:

[0137]

[0138] S2, Design an image state input convolutional neural network based on a convolutional neural network;

[0139] It should be noted that, in order to flexibly represent the state space, this invention uses different images to describe the system's state; therefore, the network input is an image. Figure 6 This demonstrates how to input visualized state into a CNN. Through this method, the policy network can capture all state information and provide a series of expected Q-values ​​at the output layer for the various task assignment decisions the management center might make regarding the base in the current state.

[0140] The network structure designed in this invention is as follows: Figure 7 As shown.

[0141] The first layer of the network architecture is a convolutional layer (Conv2D) with 16 kernels, each 3×3 in size. This layer extracts primary feature representations from the input data and outputs a feature map with unchanged height and width but a depth of 16. Subsequently, a max-pooling layer (MaxPooling2D) using a 2×2 window is employed to reduce the spatial dimension of each feature map to half its original size. This not only reduces the complexity of subsequent computations but also helps prevent the model from becoming overly reliant on the exact location of the input features. Following this is a dropout layer, which randomly sets 20% of the activation units to zero during training (randomly disabling the output of a portion of neurons). This allows the CNN to learn more robust feature representations, reducing the model's dependence on single data points and preventing overfitting. To flatten the high-dimensional feature data output from the convolutional layer into a one-dimensional array for subsequent processing, a flattening layer is used. This allows the output of the convolutional layer to be passed to two standard fully connected (Dense) layers. In the two fully connected layers, the first fully connected layer has 256 neurons and uses the ReLU activation function, which is responsible for further feature processing and integration, while the second fully connected layer outputs the final task resource allocation decision.

[0142] S3, the image state is input into the convolutional neural network as the current network and target network of the DQN model, and a transfer learning method is introduced to construct a transfer deep reinforcement learning model.

[0143] S4. Based on the Markov model and the image description of the system state, the transfer deep reinforcement learning model is used to learn the policy and realize the allocation of unmanned cluster task resources.

[0144] It should be noted that this invention uses the transfer deep reinforcement learning model TL-DQN to train the neural network, and then uses the TL-DQN algorithm to update the strategy. Figure 5 This is a schematic diagram of the DQN algorithm. This invention combines this algorithm with a transfer learning method. Combining transfer learning with the DQN algorithm can improve the learning efficiency of the target task by reusing strategies learned on the source task.

[0145] Figure 8 A schematic diagram of the algorithm training process designed for this invention includes:

[0146] S41, initialize the state space and action space, load the model or initialize a new DQN model, and the network adopts the image state input convolutional neural network designed in this invention.

[0147] Specifically, multiple task sets are defined. i Let i ∈ 1, ..., E, and each task set contain N requests. i The training initializes the state and action spaces of the DQN model. If valid model parameters θ′ have been obtained from previous task training, the model is directly loaded to skip the initial learning phase. If no previous model exists, a new DQN model is initialized with random parameters θ. In the DQN model, both the current network and the target network use... Figure 7 The image state shown is input into the convolutional neural network.

[0148] S42, the agent selects an action.

[0149] Specifically, the model enters a training loop that spans M episodes. At the start of each episode, the system initializes state s1 and determines the initial action a1 based on the εgreedy policy. In time step t of each episode, the algorithm executes different policies based on the current task index. For the first task set, taskset1, the εgreedy policy is preferentially used for action selection. For subsequent task sets, a transfer parameter λ is introduced to control the proportion of exploration based on the previous policy π′ or the current model parameters θ.

[0150] S43, Model observation state and reward are stored in experience cache D.

[0151] Specifically, when action a is performed tAfterwards, the model will observe the new state s. t+1 and reward r t All observed transformations <s t ,a t ,r t ,s t+1 > It is stored in the experience buffer D for use in subsequent training.

[0152] S44, sample a small batch of data from D to update the model parameters.

[0153] Specifically, for the experience stored in D, sample small batches of data and calculate the target Q value y. j This is to update the Q function parameter θ. The target Q value is y. j The calculation follows the objective value update principle of standard DQN, but introduces the transfer learning parameter θ′.

[0154] In summary, this embodiment provides a resource allocation method for unmanned swarm tasks based on deep reinforcement learning. Based on the unmanned swarm task resource allocation architecture, a Markov model is established, resource state representation is quantified using images, an image state input convolutional neural network is designed, and the DQN algorithm is used to solve the unmanned swarm task resource allocation problem. Furthermore, transfer learning is introduced to improve algorithm performance. This method can be applied to unmanned swarm task resource allocation systems, solving problems related to allocation efficiency and fairness in unmanned aerial vehicle (UAV) swarm task resource allocation; reducing overall task resource allocation delays and avoiding resource waste.

[0155] Second Embodiment

[0156] This embodiment provides an electronic device, such as... Figure 9 As shown, the electronic device includes a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment described above. Furthermore, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0157] Below, in conjunction with Figure 9 A detailed introduction to each component of this electronic device is provided below:

[0158] The processor is the control center of the electronic device. The electronic device may include multiple processors, each of which can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The term "processor" can refer to a single processor or a collective term for multiple processing elements. For example, a processor can be one or more central processing units (CPUs), other general-purpose processors, application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), one or more field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0159] In a specific implementation, as one example, the processor may include one or more CPUs, for example... Figure 9 CPU0 and CPU1 shown are, of course, merely illustrative examples.

[0160] The memory is used to store the software program that executes the solution of the present invention, and the processor controls its execution. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0161] Optionally, the memory may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may be integrated with the processor or may exist independently, and may be accessed through the interface circuit of the electronic device (…). Figure 9 (Not shown in the image) is coupled to the processor; however, this embodiment of the invention does not impose specific limitations on this.

[0162] The transceiver may include a receiver and a transmitter. Figure 9 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function. The transceiver can be integrated with the processor or exist independently, and is connected through the interface circuit of the electronic device (…). Figure 9 (Not shown in the image) is coupled to the processor, and this embodiment of the invention does not specifically limit this.

[0163] In addition, it should be noted that, Figure 9 The structure of the electronic device shown is not intended to limit the device. Actual devices may include more or fewer components than shown, or combine certain components, or have different component arrangements. Furthermore, the technical effects achieved by this electronic device when performing the method of the first embodiment described above can be referenced to the technical effects described in the first embodiment; therefore, they will not be repeated here.

[0164] Third Embodiment

[0165] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc. The instruction stored therein can be loaded and executed by a processor in a terminal.

[0166] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely or partially hardware embodiment, a completely or partially software embodiment, or an embodiment combining software and hardware aspects. Moreover, when implemented in software, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any usable medium accessible to a computer or a data storage device such as a server or data center containing one or more sets of usable media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0167] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0169] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element. Furthermore, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Additionally, the character " / " in this text generally indicates an "or" relationship between the preceding and following objects, but it can also indicate an "AND / OR" relationship. Please refer to the context for specific interpretations. "At least one" refers to one or more items, while "more than" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be represented as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0170] Furthermore, it is understood that in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0171] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0172] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of functional modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional units in the various embodiments of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0173] If the method is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments of the present invention have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make several improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A resource allocation method for unmanned swarm tasks based on deep reinforcement learning, characterized in that, include: Based on the unmanned cluster task resource allocation architecture, a Markov model is established for the unmanned cluster task resource allocation problem. The system also designs a state representation method to describe the system state using images, including: describing the resource allocation problem of unmanned cluster tasks based on the unmanned cluster task resource allocation architecture; designing a state space to describe the system state using images; designing an action space; and designing a reward function. Design an image state input convolutional neural network based on convolutional neural networks; The image state is input into a convolutional neural network as the current network and target network of the DQN model, and a transfer learning method is introduced to construct a transfer deep reinforcement learning model. Based on the Markov model and the image description of the system state, the transfer deep reinforcement learning model is used to learn policies and realize resource allocation for unmanned cluster tasks. The design state space uses images to describe the system state, including: Design a state space, which includes the following elements: The usage and remaining quantity of each resource in each unmanned resource base; A queue of tasks to be assigned, where each task contains its resource requirements. The backlog of task resource requests waiting to be assigned to the task queue; The system state is described using images, including state images of the task queue and the unmanned resource base. In both the task queue and the unmanned resource base, each resource corresponds to a state image, and the images are structured like a two-dimensional matrix, with the entire image divided into multiple grids. In each state image of the task queue, the number of grid cells filled horizontally represents the number of resources requested by the current task, and the number of grid cells filled vertically represents the expected usage duration of the corresponding resource. In each state image of the unmanned resource base, the number of grid cells filled horizontally represents the number of resources currently occupied, and the number of grid cells filled vertically represents the expected usage duration of the current resource by the corresponding task. Unfilled areas represent the remaining resources in the unmanned resource base. At the end of each time step, the resource occupancy status in the unmanned resource base's state image shifts upwards by one grid cell, representing a unit of time step advancement.

2. The resource allocation method for unmanned swarm tasks based on deep reinforcement learning as described in claim 1, characterized in that, The unmanned cluster task resource allocation architecture includes an unmanned resource management center, which manages multiple unmanned resource bases. Each unmanned resource base contains several unmanned resources of different types and quantities. The unmanned resource management center receives task resource requests from multiple different locations, consisting of different resource requirements. These requests arrive in a discrete time step manner and are added to the request queue of the unmanned resource management center. Each task resource request contains the type of resource required by the current task, the estimated task execution time, the quantity of resources, and the location where the request was initiated. Based on the information contained in the task resource requests, the unmanned resource management center evaluates the remaining available resources in each unmanned resource base, determines which unmanned resource base should respond to which task resource request, and allocates the resources accordingly.

3. The resource allocation method for unmanned swarm tasks based on deep reinforcement learning as described in claim 1, characterized in that, The aforementioned unmanned cluster task resource allocation architecture describes the unmanned cluster task resource allocation problem, including: Let M be the number of unmanned resource bases under the jurisdiction of the unmanned resource management center, denoted by set B, where B = {b1, b2, ..., b}. M }, b k Let represent the k-th unmanned resource base, k∈1,...,M; the number of unmanned resource types in each unmanned resource base is N, and the available resource quantity of each unmanned resource is represented by the set Q, where Q={q1,q2,...,q N }, q j This represents the available quantity of the j-th type of unmanned resource, j∈1,...,N; the unmanned resource management center receives n task resource requests, and assigns a unique identifier "task" to the i-th task resource request. i And represented using the following vector: task i =[q1,q2,...,q N ,D i ,Location],D i ≤T max ,i∈1,...,n Among them, D i T represents the estimated execution time of the i-th task under ideal conditions; max This indicates the maximum request duration for a single task resource request; Location indicates the geographic coordinates of the location where the task resource request was initiated; the number of requests for each type of resource in a task resource request cannot exceed the maximum number of that type of resource in each unmanned resource base; the unmanned resource management center will extract information from each task resource request and maintain a mapping table for each task resource request, which includes the resource requirement configuration of the task resource request and the normalized distance of the request initiation location relative to each unmanned resource base; The unmanned resource management center maintains a fixed-length queue of pending tasks and a backlog queue for all task resource requests. Incoming pending task resource requests will wait in the queue of pending tasks. When the queue of pending tasks is full, subsequent task resource requests will be stored in the backlog queue. If the current unmanned resource base has no resources or insufficient resources, the unmanned resource management center will allocate a pending task resource request to that unmanned resource base from the task allocation queue, to be processed after the currently allocated task resources are released; the pending task resource request will be processed over time, and as long as the task allocation queue is not empty, the unmanned resource management center will continuously allocate new task resource requests to each unmanned resource base; the delay S of the i-th task resource request. i =C i / D i S i ≥1, where C i This represents the actual completion time of the i-th task resource request. The actual completion time includes the waiting time of the task resource request in the backlog queue and the queue of tasks to be assigned, the unmanned cluster travel time, and the estimated task completion time.

4. The resource allocation method for unmanned swarm tasks based on deep reinforcement learning as described in claim 3, characterized in that, The design action space includes: The action space is defined as the set of all possible task allocation decisions. Each task allocation decision is described as: allocating a task from a resource request in the task queue to a certain unmanned resource base.

5. The resource allocation method for unmanned swarm tasks based on deep reinforcement learning as described in claim 3, characterized in that, The design reward function includes: The reward function is designed into two parts: a core reward part and a secondary reward part, expressed as: Reward=R Main +R Aux Where Reward represents the reward function; R Main Indicates core reward; R Aux Indicates supplementary rewards; The mathematical expression for core rewards is: R Main =-(R Basement +R Queue +R Backlog ) in, k is the base identifier, p k For all planned operations in base k; The execution time required for the planned operation; t i Let be the estimated execution time of the i-th request in the queue of tasks to be assigned; t g α is the estimated execution time for the g-th request in the backlog queue; k =β=γ=1; The mathematical expression for supplementary rewards is: R Aux =R Load +R Distance Among them, R Load Rewards for base load balancing; R Distance Allocate rewards based on the nearest available task resources; count k This is a count of the number of tasks assigned to base k at the current moment; Lb k and Ltask i Base b k Available load and tasks i The required load; For task i The normalized distance from base k; η1 and η2 are unit reward parameters.

6. The resource allocation method for unmanned swarm tasks based on deep reinforcement learning as described in claim 1, characterized in that, The first layer of the image state input convolutional neural network is a convolutional layer with 16 convolutional kernels, each 3×3 in size. This layer is responsible for extracting primary feature representations from the input data and outputting a feature map with unchanged height and width but a depth of 16. Subsequently, a max pooling layer using a 2×2 window is used to reduce the spatial dimension of each feature map to half its original size. Then, a Dropout layer is applied. Following this, a flattening layer is used to flatten the high-dimensional feature data output by the convolutional layer into a one-dimensional array, so that the output of the convolutional layer is passed to two standard fully connected layers. In the two fully connected layers, the first fully connected layer has 256 neurons and uses the ReLU activation function, which is responsible for further feature processing and integration, while the second fully connected layer outputs the final task resource allocation decision.

7. The resource allocation method for unmanned swarm tasks based on deep reinforcement learning as described in claim 6, characterized in that, The Dropout layer randomly sets 20% of the activation units to zero during training.

8. The resource allocation method for unmanned swarm tasks based on deep reinforcement learning as described in claim 1, characterized in that, Based on the Markov model and the image description of the system state, the transfer deep reinforcement learning model is used for policy learning to achieve resource allocation for unmanned swarm tasks, including: Initialize the state space and action space, load the transfer deep reinforcement learning model or initialize a new transfer deep reinforcement learning model. Specifically: Define multiple task sets, each containing multiple task resource requests. For training each task set, initialize its state space and action space. If valid model parameters have been obtained in the previous task training, directly load the previous transfer deep reinforcement learning model to skip the initial learning phase; if there is no previous model, initialize a new transfer deep reinforcement learning model with random parameters. The agent selects an action; The model observes the state and reward, and stores them in the experience cache. Update model parameters by sampling small batches of data from the experience cache.

Images