Unmanned cluster task resource allocation method based on deep reinforcement learning

Abstract

The invention discloses an unmanned cluster task resource allocation method based on deep reinforcement learning, and belongs to the technical field of artificial intelligence, and the method comprises the steps: building a Markov model of an unmanned cluster task resource allocation problem based on an unmanned cluster task resource allocation architecture; designing a state representation mode, and describing a system state by using an image; designing an image state based on the convolutional neural network, and inputting the image state into the convolutional neural network; taking the image state input convolutional neural network as a current network and a target network of a DQN model, and introducing a transfer learning method to construct a transfer deep reinforcement learning model; and based on the Markov model and the image description of the system state, adopting the migration deep reinforcement learning model to carry out strategy learning, and realizing unmanned cluster task resource allocation. By adopting the technical scheme of the invention, the overall task resource allocation delay can be reduced, and the resource waste can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an unmanned cluster task resource allocation method based on deep reinforcement learning. Background Art

[0002] With the rapid development and popularization of unmanned technology, unmanned swarms, also known as multi-agent systems (MAS), as an important form of intelligent system, have been widely used in emergency rescue, logistics distribution, military operations and other fields. Unmanned systems have the advantages of low cost, flexible operation, no fear of casualties, etc., and can perform tasks in harsh and dangerous environments.

[0003] MAS is generally composed of multiple intelligent agents, including drones, unmanned vehicles, and unmanned boats, which work together to complete specific tasks, such as searching for targets, monitoring the environment, or performing collaborative operations. For example, in a cluster composed of a single type of unmanned platform, a drone cluster can give full play to the advantages of the drone platform's flexibility, low cost, and wide application, and complete a variety of tasks in a collaborative and efficient manner in the form of a combination of capabilities, while a surface unmanned boat cluster can conduct long-term, large-scale search and rescue in complex and dangerous sea conditions.

[0004] In actual application scenarios, task resource allocation, task allocation, and path planning together constitute the core challenges in unmanned swarm task planning, which directly determine the efficiency, performance, and adaptability of MAS. Traditional unmanned swarm task planning relies on manually formulated rules or heuristic algorithms, such as Tabu Search (TS) and Simulated Annealing (SA). Although these methods may perform well in static environments or simple tasks, their performance is usually limited when faced with complex and dynamic task environments. Traditional methods are often suboptimal in terms of resource utilization and task efficiency, affecting the overall execution performance. Today, unmanned swarm systems (such as unmanned vehicles, drones, etc.) are gradually becoming an important component in the construction of smart logistics, smart cities, and automated exploration. The efficiency of these systems is highly dependent on their internal task allocation and resource management strategies. However, unmanned swarms face multiple challenges from dynamic environments, such as dynamic changes in resource requirements, time sensitivity of task execution, and effective utilization of heterogeneous resources. In such an environment, the real-time and variability of requests puts tremendous pressure on the efficient allocation of resources. Traditional resource allocation methods usually assume that resource requirements are fixed, which simplifies the problem of task resource allocation. However, these methods usually lack the flexibility and adaptability of solution methods, so a new method that can cope with complex and changing environments is needed. Summary of the invention

[0005] The present invention provides an unmanned cluster task resource allocation method based on deep reinforcement learning to solve the technical problems of insufficient resource utilization and task efficiency of traditional resource allocation methods.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] On the one hand, the present invention provides an unmanned cluster task resource allocation method based on deep reinforcement learning, and the unmanned cluster task resource allocation method based on deep reinforcement learning includes:

[0008] Based on the unmanned cluster task resource allocation architecture, a Markov model for the unmanned cluster task resource allocation problem is established; and a state representation method is designed to use images to describe the system state;

[0009] Design image state input convolutional neural network based on convolutional neural network;

[0010] The image state is input into a convolutional neural network as the current network and target network of the DQN model, and a transfer learning method is introduced to construct a transfer deep reinforcement learning model;

[0011] Based on the Markov model and the graphical description of the system state, the transfer deep reinforcement learning model is used to perform strategy learning to achieve unmanned cluster task resource allocation.

[0012] Furthermore, an unmanned resource management center is provided in the unmanned cluster task resource allocation architecture, which manages multiple unmanned resource bases. Each unmanned resource base has a number of unmanned resources of different types and quantities. The unmanned resource management center will receive task resource requests consisting of different resource requirements from multiple different locations. The task resource requests arrive in discrete time steps and are added to the request queue of the unmanned resource management center; the task resource request includes the resource type required for the current task, the estimated task execution time, the number of resources, and the location where the resource request is initiated. The unmanned resource management center evaluates the remaining available resources in each unmanned resource base based on the information contained in the task resource request, determines which unmanned resource base is assigned to respond to which task resource request, and allocates resources.

[0013] Furthermore, based on the unmanned cluster task resource allocation architecture, a Markov model for the unmanned cluster task resource allocation problem is established; and a state representation method is designed to use images to describe the system state, including:

[0014] Based on the unmanned cluster task resource allocation architecture, the unmanned cluster task resource allocation problem is described;

[0015] Design state space and use images to describe the system state;

[0016] Design action spaces;

[0017] Design a reward function.

[0018] Furthermore, the unmanned cluster task resource allocation problem is described based on the unmanned cluster task resource allocation architecture, including:

[0019] Assume that the number of unmanned resource bases under the jurisdiction of the unmanned resource management center is M, represented by a set B, where B = {b1, b2, ..., b M}, b k represents the kth unmanned resource base, k∈1,...,M; the number of unmanned resource types in each unmanned resource base is N, and the available resources of each unmanned resource are represented by the set Q, where Q={q1,q2,...,q N}, q j represents the number of available resources of the jth unmanned resource, j∈1,...,N; the number of task resource requests received by the unmanned resource management center is n, and the unmanned resource management center assigns a unique identifier task to the i-th task resource request i , and is represented by the following vector:

[0020] task i =[q1,q2,...,q N ,D i ,Location],D i ≤T max ,i∈1,...,n

[0021] Among them, D i represents the estimated task execution time under ideal conditions corresponding to the i-th task resource request; T max Indicates the maximum request duration of a single task resource request; Location indicates the geographic coordinates of the location where the task resource request is initiated; The number of requests for each resource in the task resource request cannot exceed the maximum number of resources of this category in each unmanned resource base; The unmanned resource management center will extract the information in each task resource request and maintain a mapping table for each task resource request, which contains the resource demand configuration of the task resource request and the normalized distance of the request initiation location relative to each unmanned resource base;

[0022] The unmanned resource management center will maintain a fixed-length waiting-to-be-allocated task queue and a backlog queue for all task resource requests. Incoming waiting-to-be-allocated task resource requests will wait in the waiting-to-be-allocated task queue. When the waiting-to-be-allocated task queue is full, subsequent arriving task resource requests will be saved in the backlog queue.

[0023] If the current unmanned resource base has no resources or the remaining resources are insufficient, the unmanned resource management center will allocate a pending task resource request to the unmanned resource base from the pending task queue, waiting for the currently allocated task resources to be released and processed; the pending task resource request will be processed over time, and as long as the pending task queue is not empty, the unmanned resource management center will continue to allocate new task resource requests to each unmanned resource base; the delay S of the i-th task resource request i =C i / D i , S i ≥1, where C i It represents the actual completion time of the i-th task resource request. The actual completion time includes the waiting time of the task resource request in the backlog queue and the task queue to be assigned, the unmanned cluster travel time and the estimated task completion time.

[0024] Furthermore, the design state space uses images to describe the system state, including:

[0025] Design a state space, which includes the following elements:

[0026] The usage and surplus of each resource in each unmanned resource base;

[0027] A queue of tasks to be assigned, each of which contains the resource requirements of the current task;

[0028] A backlog queue of task resource requests waiting to enter the task queue to be assigned;

[0029] An image is used to describe the system status, including a status image of a task queue to be assigned and a status image of an unmanned resource base, wherein in the task queue to be assigned and the unmanned resource base, each resource corresponds to a status image, and the image structure is similar to a two-dimensional matrix, and the entire image is evenly divided into multiple grids; in each status image of the task queue to be assigned, the number of grids filled in the horizontal direction of the image represents the number of corresponding resources used by the current task resource request, and the number of grids filled in the vertical direction of the image represents the estimated usage time of the corresponding resources; in each status image of the unmanned resource base, the number of grids filled in the horizontal direction of the image represents the number of currently occupied resources, and the number of grids filled in the vertical direction of the image represents the estimated usage time of the current resources by the corresponding tasks, and the unfilled area represents the remaining amount of the corresponding resources in the current unmanned resource base; after each time step, the resource occupancy status representation in the status image of the unmanned resource base will move up one grid as a whole, representing that the time step has advanced by one unit.

[0030] Furthermore, the designed action space includes:

[0031] The action space is defined as the set of all possible task allocation decisions. Each task allocation decision is described as: allocating a task in a task resource request in the task queue to be allocated to a certain unmanned resource base.

[0032] Furthermore, the design reward function includes:

[0033] The reward function is designed to be divided into two parts: the core reward part and the auxiliary reward part, expressed as:

[0034] Reward=R Main +R Aux

[0035] Among them, Reward represents the reward function; R Main Represents core reward; R Aux Indicates auxiliary reward;

[0036] The mathematical expression of the core reward is:

[0037] R Main =-(R Basement +R Queue +R Backlog )

[0038] in, k is the base identifier, p k for all planned jobs in base k; t pk The execution time required for the planned job; t i is the estimated execution time of the i-th request in the task queue to be assigned; t g is the estimated execution time of the g-th request in the backlog queue; α k =β=γ=1;

[0039] The mathematical expression of auxiliary reward is:

[0040] R Aux =R Load +R Distance

[0041] Among them, R Load Reward for base load balancing; R Distance Allocate rewards for nearby mission resources;

[0042] count k Lb is the number of tasks assigned to base k at the current moment; k and Ltask i Base b kAvailable load and tasks i The required load; For task i The normalized distance from the base k; η1 and η2 are unit reward parameters.

[0043] Furthermore, the first layer of the image state input convolutional neural network is a convolutional layer, which is provided with 16 convolution kernels, each of which has a size of 3×3. This layer is responsible for extracting primary feature representations from the input data and outputting a feature map with unchanged height and width and a depth of 16; then, a maximum pooling layer using a 2×2 window is used to reduce the spatial dimension of each feature map to half of its original size; then a Dropout layer is connected; then a flattening layer is used to flatten the high-dimensional feature data output by the convolution layer into a one-dimensional array, so that the output of the convolution layer is passed to two standard fully connected layers, among which the first fully connected layer has 256 neurons and uses a ReLU activation function, which is responsible for further feature processing and integration, while the second fully connected layer outputs the final task resource allocation decision.

[0044] Furthermore, the Dropout layer randomly sets 20% of the activation units to zero during training.

[0045] Furthermore, based on the Markov model and the image description of the system state, the transfer deep reinforcement learning model is used to perform strategy learning to achieve unmanned cluster task resource allocation, including:

[0046] Initialize the state space and action space, load the transfer deep reinforcement learning model or initialize a new transfer deep reinforcement learning model, specifically: define multiple task sets, each task set contains multiple task resource requests, and for the training of each task set, initialize its state space and action space. If valid model parameters have been obtained in the previous task training, directly load the previous transfer deep reinforcement learning model to skip the initial learning stage; if there is no previous model, use random parameters to initialize the new transfer deep reinforcement learning model;

[0047] The agent chooses an action;

[0048] The model observes the state and reward and stores them in the experience buffer;

[0049] Sample small batches of data from the experience buffer to update model parameters.

[0050] On the other hand, the present invention further provides an electronic device, comprising a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above method.

[0051] In yet another aspect, the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the above method.

[0052] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0053] Based on the unmanned cluster task resource allocation architecture, the present invention establishes a Markov model, uses images to quantify resource state representation, designs an image state input convolutional neural network, uses the DQN algorithm to solve the unmanned cluster task resource allocation problem, and introduces a transfer learning method to improve the algorithm performance. This method can be applied to the unmanned cluster task resource allocation system, and can solve the problems of allocation efficiency and fairness in the unmanned aerial vehicle cluster task resource allocation problem; reduce task resource allocation delays and avoid resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 It is a schematic diagram of the execution flow of the unmanned cluster task resource allocation method based on deep reinforcement learning provided by an embodiment of the present invention;

[0056] Figure 2 It is a schematic diagram of a process of establishing a Markov model provided by an embodiment of the present invention;

[0057] Figure 3 It is a schematic diagram of an unmanned cluster task resource allocation architecture provided by an embodiment of the present invention;

[0058] Figure 4 is a schematic diagram of an image representation of a state space provided by an embodiment of the present invention;

[0059] Figure 5 is a schematic diagram of the DQN algorithm structure provided by an embodiment of the present invention;

[0060] Figure 6 is a schematic diagram of an image state input convolutional neural network provided by an embodiment of the present invention;

[0061] Figure 7 is a schematic diagram of a neural network structure provided by an embodiment of the present invention;

[0062] Figure 8 It is a schematic diagram of the algorithm training process provided by an embodiment of the present invention;

[0063] Fig. 9 It is a system block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0065] First of all, it should be noted that in the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplarily" is intended to present the concept in a concrete way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0066] First embodiment

[0067] This embodiment introduces deep reinforcement learning theory to address the challenges of unmanned cluster task planning, and provides an unmanned cluster task resource allocation method based on deep reinforcement learning. The method can be implemented by an electronic device, which can be a terminal or a server. The execution process of the method is as follows: Figure 1 As shown, the following steps are included:

[0068] S1, based on the unmanned cluster task resource allocation architecture, establish a Markov model for the unmanned cluster task resource allocation problem; and design a state representation method to use images to describe the system state;

[0069] Specifically, the implementation process of the above S1 is as follows: Figure 2 As shown, including:

[0070] S11, description of the resource allocation problem for UAV swarm tasks.

[0071] Specifically, the present invention considers setting up an unmanned resource management center (hereinafter referred to as the management center) in a certain area, which manages multiple unmanned resource bases (hereinafter referred to as bases). Each base has several unmanned resources of different types and quantities, such as drones, unmanned vehicles, etc. The management center will receive task resource requests (hereinafter referred to as requests) consisting of different resource requirements from many different locations. The requests arrive in discrete time steps and are added to the request queue of the management center. The request includes the resource type required for the task, the estimated task execution time, the number of resources, and the location where the resource request is initiated. The management center evaluates the remaining available resources in each base based on the above information, determines which base is assigned to respond to which request, and allocates resources. The unmanned cluster task resource allocation architecture is as follows: Figure 3shown.

[0072] The present invention Figure 3 Based on the framework, the problem is mathematically defined. Assume that the number of bases under the management center is M, represented by a set B, where B = {b1, b2, ..., b M}, k∈1,...,M, the number of heterogeneous unmanned resources in each base is N, and the available resources of each unmanned resource are represented by the set Q, where Q = {q1,q2,...,q N},j∈1,...,N, the number of requests received by the management center is n, and each request is given a unique identifier task i And use the following vector to represent it:

[0073] task i =[q1,q2,...,q N ,D i ,Location],D i ≤T max ,i∈1,...,n(1)

[0074] Among them, D i represents the expected task execution time under ideal conditions, T max Represents the maximum request duration limit for each request, which is to prevent unnecessary occupation of task resources. Location represents the geographic coordinates of the location where the request is initiated. The number of requests for each resource in the request cannot exceed the maximum number of resources of this category in each base. The management center will extract the key information in each request and maintain a mapping table for each request, which contains the resource requirement configuration of the request and the normalized distance of the location where the request is initiated relative to each base. (by Location and each base b k distance converted).

[0075] The management center will maintain a fixed-length "task queue to be assigned" and a "backlog queue" for all requests. Incoming resource requests to be assigned will wait in the task queue to be assigned. When the queue is full, subsequent request information will be saved in the backlog queue.

[0076] If the current base has no resources or insufficient resources, the management center will assign a pending request to the base from the pending task queue to be processed after the currently allocated task resources are released. The pending requests will be processed over time, and as long as the pending task queue is not empty, the management center will continue to assign new requests to each base.

[0077] Since the allocation of unmanned cluster resources is not sensitive to the specific execution details of the task, the main goal of this invention is to reduce the overall task execution time and avoid resource waste as much as possible by optimizing resource allocation. Therefore, the concept of task delay in server resource scheduling is introduced to measure the impact of the task resource allocation algorithm on the overall task execution efficiency. Let the task delay be S and the actual request completion time be C. i , which includes the waiting time of the request in the backlog queue, the waiting task queue, the unmanned cluster travel time and the estimated task completion time, then S i =C i / D i , S i ≥1.

[0078] S12, state-space design.

[0079] Each request task i =[q1,q2,...,q N ,D i , Location] to extract the specific resource requirements of each request, namely [q1,q2,...,q N ,D i ] and convert it into a graphical representation of the state space. In the problem involved in this invention, the state space should contain the following elements:

[0080] 1) Resource occupancy status of each base, that is, the usage and surplus of each resource in the base.

[0081] 2) A queue of tasks to be assigned, each of which contains the specific resource allocation requirements for the task.

[0082] 3) A backlog queue where resource requests waiting to enter the task queue to be assigned temporarily exist.

[0083] Therefore, in order to flexibly represent the state space, the present invention uses different images to describe the state of the system, which includes the request processing status of the current base, the status of available resources, the task resource demand in the task queue to be assigned, and the status of the backlog queue, avoiding complex state space description.

[0084] Figure 4For the specific design of the state space, the image structure is similar to a two-dimensional matrix. Multiple colors are used in the figure to facilitate the introduction, but in the actual algorithm implementation, it is not necessary to distinguish color channels. Each resource type has a state representation image. The leftmost column shows the occupation of current resources by different requests in different bases, as well as the available quantity of each resource; the task queue to be assigned shows the resource configuration required for the requests currently waiting in line for response. The upper limit of the capacity of the task queue to be assigned is set to C, that is, the state image of the required resource configuration is maintained only for C requests to be assigned. Firstly, the fixed number of states is convenient as the input of the neural network. Secondly, since the system is built on the premise that the request is not sensitive to the allocation timing, maintaining a limited number of task state representations to be assigned can control the dimension of the action space.

[0085] In each status image of the task queue to be assigned, each row represents the amount of resources requested for use by the task, and each column represents the estimated usage time of the resources. Figure 4 In the figure, the resources required by the request in task slot 1 are represented by a certain color, which means that the task needs to use 2 units of a resource, 1 unit of b resource and 2 units of time resource. The same rule is followed in the status image of the base. On the left side of the figure, the two colors in the status image of base 2 indicate that two different cluster tasks are currently occupying the a and b resources in the base, and the remaining white area represents the current available resource scale of the base. After each time step, the resource occupation status representation in the base status image will move up one grid as a whole, indicating that the time step has moved forward one unit. When the management center allocates a request waiting in a task slot to the base, the resource configuration representation of the task will be filled into the status image of the base, and the status representation of the task will be deleted from the task slot. The management center will select the request in the backlog queue according to the order of request arrival to generate the image status representation of its resource configuration and fill it into the empty task slot. The backlog queue is not displayed in the status representation. The management center maintains a mapping in the database for each request arriving in the backlog queue, which contains the required resource configuration and location information of the request.

[0086] The input of the state space is essentially multiple sets of W×H×N three-dimensional tensor images, where W represents the width of the image, which is used to indicate the upper limit of the number of unmanned resources; H represents the length of the image, which is used to indicate the upper limit of the processing time required for the task; and N represents the number of channels of the image, which is used to indicate the number of unmanned resource types.

[0087] S13, action space design.

[0088] Specifically, for the unmanned cluster task resource allocation problem, the action space is defined as the set of all possible task allocation decisions, which is described as "assigning task slots in the task queue to be allocated to each task slot ...c ,c∈1,..., the tasks in C are assigned to base b k ”, the management center synchronously assigns tasks to each base in each time step, so the execution of the entire system can be considered as a joint action representation, defined as

[0089] a={ψ,b k ,taskslot1,...,taskslot C}(2)

[0090] Among them, ψ indicates whether the base is assigned a task in the current time step. When resource allocation is performed, ψ = 1, otherwise ψ = 0. The global action space is composed of the possible scheduling action space of the management center for multiple bases, so the size of the action space is (MC+1), where M is the number of bases and C is the capacity of the task queue to be assigned.

[0091] In each time step, the management center can take multiple actions until it chooses not to make further request allocations (ψ = 0) at the current step or to make an invalid allocation operation (for example, trying to allocate requests that do not meet the current remaining time resources of the base to the base). This is a random exploration mechanism. In the actual task resource allocation process, this exploration can be achieved by trying to allocate requests that do not seem to be suitable for the base at present, or by suspending the allocation to the base to wait for a better resource request allocation opportunity, although this may not be consistent with the current optimal strategy. This helps the algorithm to explore new and potentially more effective strategies while ensuring the use of known effective strategies. For effective allocation, a task is allocated at the first available time step of the base, and the system load state changes accordingly. After selecting "no task allocation" or "invalid task allocation", time continues so that new requests enter the task queue to be allocated and the requests in the base are processed.

[0092] S14, reward function design.

[0093] Specifically, the present invention designs the reward into two parts: a core reward part and an auxiliary reward part.

[0094] Reward=R Main +R Aux (3)

[0095] The core reward part takes the average task delay as the most important concern in the allocation of unmanned cluster task resources, while the auxiliary reward part focuses more on fairness in the allocation.

[0096] 1) Core rewards

[0097] In this invention, the reward is designed to allow the algorithm to learn to minimize task delays. Therefore, the reward function is linked to the time resource requirements of each request, and the inverse of the time resource length of the request is used as the core component of the reward. The mathematical expression of the reward is as follows:

[0098] R Main =-(R Basement +R Queue +R Backlog ) (4)

[0099] Among them, R Basement , R Queue , R Backlog They represent the base reward, the task queue reward to be assigned, and the backlog queue reward. By introducing a negative reward mechanism, the reward function setting clearly defines the optimization goal of reducing task waiting and total execution time, which is consistent with the overall goal of reducing the average task delay S of the unmanned cluster system. Each reward is set as follows:

[0100] (1) Base Rewards

[0101]

[0102] Where k is the base identifier. p is all the planned jobs in the current base (including invalid task resources). By taking the inverse of the sum of the execution time of the assigned tasks on each base as part of the reward, the system can be made to tend to reward multiple short tasks that can be completed quickly rather than a few long tasks, so as to improve the throughput of the system in the same time.

[0103] (2) Rewards for the task queue to be assigned

[0104]

[0105] Among them, t i is the estimated execution time of the i-th request in the task queue to be assigned. By encouraging the algorithm to reduce the waiting time of a single request in the task queue to be assigned, the actual completion time of the task can be reduced. i This also improves the fairness of the base in treating each request and avoids some requests waiting for a long time.

[0106] (3) Backlog Queue Rewards

[0107]

[0108] Among them, t gis the estimated execution time of the gth request in the backlog queue. The purpose of this reward is to encourage processing tasks in the backlog queue, which helps manage and reduce tasks that cannot be processed immediately due to resource constraints, optimize resource utilization, avoid waste of system resources, and ensure that requests in the backlog queue can also be executed.

[0109] To sum up, the final total core reward function is:

[0110]

[0111] In this model, if all weight factors are set to α k =β=γ=1. Without considering other forms of rewards, the absolute value of the cumulative reward will be consistent with the sum of the average task delays over time, so this reward goal helps minimize the average task delay. If you want to complete the request as quickly as possible, you can give the parameter β a slightly larger value to impose a greater penalty on the requests in the assigned task queue.

[0112] 2) Auxiliary rewards

[0113] In addition to the core part of the reward, we also hope to pay attention to two other issues while processing requests: one is the load balancing problem of the base. In order to prevent the load of a specific base from far exceeding that of other bases in a certain time step, the reward function is designed based on the load balancing between bases; the second is the rationality of task allocation in space. Therefore, the present invention designs auxiliary reward functions for these two problems.

[0114] R Aux =R Load +R Distance (9)

[0115] (1) Base load balancing rewards

[0116]

[0117] Where count k Counts the number of tasks assigned to base k at the current moment. This reward function is designed to balance the load of each base in the assignment at each time step. When the task requires a large load, it is encouraged to be assigned to a base with a large available load. Lb k and Ltask i Base b k Available load and tasks i The required load for the base and the required load for each request are given below.

[0118] According to the available quantity q1,q2,...,q of each resource in each base NAnd the current maximum allocatable duration t remain =T max -∑ p t p Construct an M×(N+1) decision matrix B=(b ki ) M×(N+1) , where each line is a sample to be evaluated in one of the bases, expressed as:

[0119]

[0120] Normalizing the elements of matrix B, we get

[0121]

[0122] Using z ij Construct the normalization matrix Z:

[0123]

[0124] Define the maximum value of the matrix Z to be

[0125]

[0126] The minimum value is

[0127]

[0128] Calculate the distance between the i-th base and the maximum value and the minimum value,

[0129]

[0130] In the formula, λ j is the weight coefficient assigned to different resource indicators, which is set to 1 in the training environment. The available load of the i-th base is calculated as follows:

[0131]

[0132] Where, Lb k ∈[0,1], it can be seen that Lb k The larger it is, the more resources are currently available in the base. For each request, the resource request configuration information maintained by the management center can be used to obtain the required load Ltask for each request using the same evaluation method as the base. i .

[0133] (2) Nearby task resource allocation rewards

[0134]

[0135] in, is the normalized distance between request i and each base. This formula aims to reward the decision to assign tasks to the nearest base in each time step. The smaller the normalized distance between the base and the request in each assignment, the greater the reward will be.

[0136] To sum up, the total auxiliary reward for each episode is:

[0137]

[0138] S2, designing image state input convolutional neural network based on convolutional neural network;

[0139] It should be noted that in order to flexibly represent the state space, the present invention uses different images to describe the state of the system. Therefore, the input of the network is an image. Figure 6 It shows how to input the graphical state into the CNN. In this way, the policy network can capture all state information and provide a series of expected Q values ​​of various task allocation decisions that the management center may take to the base under the current state at the output layer.

[0140] The network structure designed by the present invention is as follows Figure 7 shown.

[0141] The first layer of the network architecture is a convolutional layer (Conv2D), which is equipped with 16 convolution kernels, each of which is 3×3 in size. This layer is responsible for extracting primary feature representations from the input data and outputting feature maps with unchanged height and width and a depth of 16. Subsequently, a maximum pooling layer (MaxPooling2D) using a 2×2 window is used to reduce the spatial dimension of each feature map to half of the original, which not only reduces the complexity of subsequent calculations, but also helps prevent the model from being too dependent on the exact location of the input features. A Dropout layer is then connected, which randomly sets 20% of the activation units to zero (randomly turns off the output of a part of the neurons) during training, so that CNN learns more robust feature representations, reduces the model's dependence on a single data point, and prevents overfitting. In order to flatten the high-dimensional feature data output by the convolutional layer into a one-dimensional array for subsequent processing, the present invention uses a flatten layer. This enables the output of the convolutional layer to be passed to two standard fully connected layers (Dense). Among the two fully connected layers, the first fully connected layer has 256 neurons and uses the ReLU activation function, which is responsible for further feature processing and integration, while the second fully connected layer outputs the final task resource allocation decision.

[0142] S3, inputting the image state into the convolutional neural network as the current network and target network of the DQN model, and introducing the transfer learning method to construct a transfer deep reinforcement learning model;

[0143] S4, based on the Markov model and the image description of the system state, using the transfer deep reinforcement learning model to perform strategy learning to achieve unmanned cluster task resource allocation;

[0144] It should be noted that the present invention uses the transfer deep reinforcement learning model TL-DQN to train the neural network, thereby using the TL-DQN algorithm to update the strategy; Figure 5 The structure diagram of the DQN algorithm is shown in Figure 2. Based on this, the present invention combines the transfer learning method. The transfer learning method can improve the learning efficiency of the target task by reusing the strategy learned on the source task.

[0145] Figure 8 The algorithm training flow diagram designed for the present invention includes:

[0146] S41, initialize the state space and action space, load the model or initialize a new DQN model, and the network adopts the image state designed by the present invention to input the convolutional neural network.

[0147] Specifically, multiple task sets taskset are defined i ,i∈1,...,E, each task set contains N requests. For each task set taskset i training, initialize its state space and action space. If valid model parameters θ′ have been obtained in previous task training, directly load the model to skip the initial learning phase. If there is no previous model, use random parameters θ to initialize the new DQN model. In the DQN model, both the current network and the target network use Figure 7 The image states shown are input to the convolutional neural network.

[0148] S42, the agent selects an action.

[0149] Specifically, the model enters a training loop that spans M episodes. At the beginning of each episode, the system initializes state s1 and determines the initial action a1 based on the εgreedy strategy. In the time step t of each episode, the algorithm executes different strategies according to the current task index. For the training of the first task set taskset1, the εgreedy strategy is used first for action selection, while for subsequent task sets, the migration parameter λ is introduced to control the proportion of exploration directly using the previous strategy π′ or based on the current model parameters θ.

[0150] S43, the model observes the state and reward and stores them in the experience buffer D.

[0151] Specifically, when executing action a tAfter that, the model will observe the new state s t+1 and reward r t . All observed transitions <s t ,a t ,r t ,s t+1 > is stored in the experience buffer D for subsequent training.

[0152] S44, sample small batches of data from D to update model parameters.

[0153] Specifically, for the experience stored in D, sample a small batch of data and calculate the target Q value y j , in order to update the Q function parameters θ. The target Q value y j The calculation of follows the target value update principle of the standard DQN, but introduces the transfer learning parameter θ′.

[0154] In summary, this embodiment provides an unmanned cluster task resource allocation method based on deep reinforcement learning. Based on the unmanned cluster task resource allocation architecture, a Markov model is established, and the resource state representation is quantified using images. An image state input convolutional neural network is designed, and the DQN algorithm is used to solve the unmanned cluster task resource allocation problem. At the same time, a transfer learning method is introduced to improve the algorithm performance. This method can be applied to the unmanned cluster task resource allocation system, and can solve the problems of allocation efficiency and fairness in the unmanned aerial vehicle cluster task resource allocation problem; reduce the overall task resource allocation delay and avoid resource waste.

[0155] Second embodiment

[0156] This embodiment provides an electronic device, such as Fig. 9 As shown, the electronic device includes: a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method of the first embodiment. In addition, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0157] Next, combine Fig. 9 The following is a detailed introduction to the various components of the electronic device:

[0158] Among them, the processor is the control center of the electronic device, and the electronic device may include multiple processors, each of which may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may be a processor or a general term for multiple processing elements. For example, the processor is one or more central processing units (CPUs), or other general-purpose processors, application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor may execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0159] In a specific implementation, as an embodiment, the processor may include one or more CPUs, such as Fig. 9 The CPU0 and CPU1 shown in the figure are, of course, only exemplary.

[0160] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0161] Optionally, the memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor or exist independently and accessed through the interface circuit ( Fig. 9 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0162] The transceiver may include a receiver and a transmitter ( Fig. 9 The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function. The transceiver can be integrated with the processor or exist independently and communicate with the electronic device through the interface circuit ( Fig. 9 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0163] In addition, it should be noted that Fig. 9 The structure of the electronic device shown in the figure does not constitute a limitation on the device, and the actual device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects achieved by the electronic device when executing the method of the first embodiment above can refer to the technical effects described in the first embodiment above, so they are not repeated here.

[0164] Third embodiment

[0165] This embodiment provides a computer-readable storage medium, which stores at least one instruction, and the instruction is loaded and executed by a processor to implement the method of the first embodiment. The computer-readable storage medium may be a ROM, a random access memory, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. The instructions stored therein may be loaded by a processor in a terminal to execute the method.

[0166] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiment of the present invention can be in the form of a full or partial hardware embodiment, a full or partial software embodiment or an embodiment combining software and hardware. Moreover, when implemented using software, the embodiment of the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center containing one or more available media sets. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state hard disk.

[0167] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements. In addition, the term "and / or" is only an association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding. "At least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can be represented by: a, b, c, ab, ac, bc or abc, where a, b, c can be single or plural.

[0170] In addition, it can be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0171] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0172] In several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of functional modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0173] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0174] Finally, it should be noted that the above is only a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for ordinary technicians in this technical field, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A method for allocating unmanned cluster task resources based on deep reinforcement learning, characterized in that: include: Based on the unmanned swarm task resource allocation architecture, a Markov model for the unmanned swarm task resource allocation problem is established; And design a state representation method to describe the system state with images; Design image state input convolutional neural network based on convolutional neural network; The image state is input into a convolutional neural network as the current network and target network of the DQN model, and a transfer learning method is introduced to construct a transfer deep reinforcement learning model; Based on the Markov model and the graphical description of the system state, the transfer deep reinforcement learning model is used to perform strategy learning to achieve unmanned cluster task resource allocation.

2. The unmanned cluster task resource allocation method based on deep reinforcement learning as claimed in claim 1, characterized in that: In the unmanned cluster task resource allocation architecture, an unmanned resource management center is provided, which manages multiple unmanned resource bases. Each unmanned resource base has a number of unmanned resources of different types and quantities. The unmanned resource management center will receive task resource requests consisting of different resource requirements from multiple different locations. The task resource requests arrive in discrete time steps and are added to the request queue of the unmanned resource management center. The task resource request contains the resource type required for the current task, the estimated task execution time, the number of resources, and the location where the resource request is initiated. The unmanned resource management center evaluates the remaining available resources in each unmanned resource base based on the information contained in the task resource request, determines which unmanned resource base is assigned to respond to which task resource request, and allocates resources.

3. The unmanned cluster task resource allocation method based on deep reinforcement learning as claimed in claim 2, characterized in that: Based on the unmanned cluster task resource allocation architecture, a Markov model for the unmanned cluster task resource allocation problem is established; and a state representation method is designed to describe the system state with images, including: Based on the unmanned cluster task resource allocation architecture, the unmanned cluster task resource allocation problem is described; Design state space and use images to describe the system state; Design action spaces; Design a reward function.

4. The unmanned cluster task resource allocation method based on deep reinforcement learning as claimed in claim 3, characterized in that: The unmanned cluster task resource allocation problem is described based on the unmanned cluster task resource allocation architecture, including: Assume that the number of unmanned resource bases under the jurisdiction of the unmanned resource management center is M, represented by a set B, where B = {b1, b2, ..., b M }, b k represents the kth unmanned resource base, k∈1,...,M; the number of unmanned resource types in each unmanned resource base is N, and the available resources of each unmanned resource are represented by the set Q, where Q={q1,q2,...,q N }, q j represents the number of available resources of the jth unmanned resource, j∈1,...,N; the number of task resource requests received by the unmanned resource management center is n, and the unmanned resource management center assigns a unique identifier task to the i-th task resource request i , and is represented by the following vector: task i =[q1,q2,...,q N ,D i ,Location],D i ≤T max ,i∈1,...,n Among them, D i represents the estimated task execution time under ideal conditions corresponding to the i-th task resource request; T max Indicates the maximum request duration of a single task resource request; Location indicates the geographic coordinates of the location where the task resource request is initiated; The number of requests for each resource in the task resource request cannot exceed the maximum number of resources of this category in each unmanned resource base; The unmanned resource management center will extract the information in each task resource request and maintain a mapping table for each task resource request, which contains the resource demand configuration of the task resource request and the normalized distance of the request initiation location relative to each unmanned resource base; The unmanned resource management center will maintain a fixed-length waiting-to-be-allocated task queue and a backlog queue for all task resource requests. Incoming waiting-to-be-allocated task resource requests will wait in the waiting-to-be-allocated task queue. When the waiting-to-be-allocated task queue is full, subsequent arriving task resource requests will be saved in the backlog queue. If the current unmanned resource base has no resources or the remaining resources are insufficient, the unmanned resource management center will allocate a pending task resource request to the unmanned resource base from the pending task queue, waiting for the currently allocated task resources to be released and processed; the pending task resource request will be processed over time, and as long as the pending task queue is not empty, the unmanned resource management center will continue to allocate new task resource requests to each unmanned resource base; the delay S of the i-th task resource request i =C i / D i , S i ≥1, where C i It represents the actual completion time of the i-th task resource request. The actual completion time includes the waiting time of the task resource request in the backlog queue and the task queue to be assigned, the unmanned cluster travel time and the estimated task completion time.

5. The unmanned cluster task resource allocation method based on deep reinforcement learning as claimed in claim 4, characterized in that: The design state space uses images to describe the system state, including: Design a state space, which includes the following elements: The usage and surplus of each resource in each unmanned resource base; A queue of tasks to be assigned, each of which contains the resource requirements of the current task; A backlog queue of task resource requests waiting to enter the task queue to be assigned; An image is used to describe the system status, including a status image of a task queue to be assigned and a status image of an unmanned resource base, wherein in the task queue to be assigned and the unmanned resource base, each resource corresponds to a status image, and the image structure is similar to a two-dimensional matrix, and the entire image is evenly divided into multiple grids; in each status image of the task queue to be assigned, the number of grids filled in the horizontal direction of the image represents the number of corresponding resources used by the current task resource request, and the number of grids filled in the vertical direction of the image represents the estimated usage time of the corresponding resources; in each status image of the unmanned resource base, the number of grids filled in the horizontal direction of the image represents the number of currently occupied resources, and the number of grids filled in the vertical direction of the image represents the estimated usage time of the current resources by the corresponding tasks, and the unfilled area represents the remaining amount of the corresponding resources in the current unmanned resource base; after each time step, the resource occupancy status representation in the status image of the unmanned resource base will move up one grid as a whole, representing that the time step has advanced by one unit.

6. The unmanned cluster task resource allocation method based on deep reinforcement learning as claimed in claim 4, characterized in that: The designed action space includes: The action space is defined as the set of all possible task allocation decisions. Each task allocation decision is described as: allocating a task in a task resource request in the task queue to be allocated to a certain unmanned resource base.

7. The unmanned cluster task resource allocation method based on deep reinforcement learning as claimed in claim 4, characterized in that: The design reward function includes: The reward function is designed to be divided into two parts: the core reward part and the auxiliary reward part, expressed as: Reward=R Main +R Aux Among them, Reward represents the reward function; R Main Represents core reward; R Aux Indicates auxiliary reward; The mathematical expression of the core reward is: R Main =-(R Basement +R Queue +R Backlog ) in, k is the base identifier, p k For all planned jobs in base k; The execution time required for the planned job; t i is the estimated execution time of the i-th request in the task queue to be assigned; t g is the estimated execution time of the g-th request in the backlog queue; α k =β=γ=1; The mathematical expression of auxiliary reward is: R Aux =R Load +R Distance Among them, R Load Reward for base load balancing; R Distance Allocate rewards for nearby mission resources; count k Lb is the number of tasks assigned to base k at the current moment; k and Ltask i Base b k Available load and tasks i The required load; For task i The normalized distance from the base k; η1 and η2 are unit reward parameters.

8. The unmanned cluster task resource allocation method based on deep reinforcement learning as claimed in claim 1, characterized in that: The first layer of the image state input convolutional neural network is a convolutional layer, which is equipped with 16 convolution kernels, each of which has a size of 3×3. This layer is responsible for extracting primary feature representations from the input data and outputting a feature map with a height and width unchanged and a depth of 16; then, a maximum pooling layer using a 2×2 window is used to reduce the spatial dimension of each feature map to half of the original; then a Dropout layer is connected; then a flattening layer is used to flatten the high-dimensional feature data output by the convolution layer into a one-dimensional array, so that the output of the convolution layer is passed to two standard fully connected layers. Among the two fully connected layers, the first fully connected layer has 256 neurons and uses the ReLU activation function, which is responsible for further feature processing and integration, while the second fully connected layer outputs the final task resource allocation decision.

9. The unmanned cluster task resource allocation method based on deep reinforcement learning as claimed in claim 8, characterized in that: The Dropout layer randomly sets 20% of the activation units to zero during training.

10. The unmanned cluster task resource allocation method based on deep reinforcement learning according to claim 1, characterized in that: Based on the Markov model and the image description of the system state, the migration deep reinforcement learning model is used to perform strategy learning to achieve unmanned cluster task resource allocation, including: Initialize the state space and action space, load the transfer deep reinforcement learning model or initialize a new transfer deep reinforcement learning model, specifically: define multiple task sets, each task set contains multiple task resource requests, and for the training of each task set, initialize its state space and action space. If valid model parameters have been obtained in the previous task training, directly load the previous transfer deep reinforcement learning model to skip the initial learning stage; if there is no previous model, use random parameters to initialize the new transfer deep reinforcement learning model; The agent chooses an action; The model observes the state and reward and stores them in the experience buffer; Sample small batches of data from the experience buffer to update model parameters.

Images