A smart coal mine computing power scheduling method and system based on deep reinforcement learning

Abstract

The present invention proposes a method and system for intelligent coal mine computing power scheduling based on deep reinforcement learning, the method includes: the equipment terminal of the mining face generates real-time computing tasks and sends real-time status information to the cloud; the cloud establishes a computing power scheduling optimization model that minimizes the computing time based on the status information, adopts a deep reinforcement learning algorithm, considers the location of the terminal device, task requirements, task calculation amount, its own computing power, channel conditions and edge computing server, forms a cloud-edge collaborative computing strategy and a scheduling plan for drones, and sends the task splitting method, unloading strategy, computing power allocation plan and drone hovering position information to the drone and edge computing server. After the deep reinforcement learning training is completed, the present invention can form an efficient task unloading and resource allocation plan, effectively reducing the delay of computing tasks on the mining face.

Description

Technical Field

[0001] The present invention relates to the field of computing power scheduling, and in particular to a smart coal mine computing power scheduling method and system based on deep reinforcement learning under a mining working face. Background Art

[0002] The mining face of a smart mine uses panoramic cameras, microphone arrays, and various sensors for location, environment, equipment, personnel, and geology to collect and integrate data. These terminal devices collect information about the surrounding environment and provide the remote monitoring center with the real-time status of the mining face, and make corresponding operations and responses based on the status information. Therefore, in the face of the massive data of smart mines, how to achieve ultra-low latency computing and ensure the freshness of environmental information and the timeliness of operations has become a key issue that needs to be considered.

[0003] In order to meet the computing needs of data-intensive and latency-sensitive tasks, Mobile Edge Computing (MEC) has become one of the current research hotspots. Edge computing enables terminal devices to offload their own computing tasks to edge computing servers by sinking computing resources to edge computing servers closer to the terminal. On the one hand, it can make up for the limitation of insufficient computing power, and on the other hand, it greatly reduces the latency of data transmission to the cloud, effectively improving computing efficiency. In the edge computing architecture, the cloud platform, as the brain, focuses on the processing of long-term, global big data, while the edge side, as the extension of the central cloud, focuses on the processing of real-time, small data, flexibly solving near-real-time business needs, and the terminal side focuses on intelligent perception, data collection, and command execution. However, in the complex scenario of coal mine working faces, there are blind spots in communication signal coverage, and the communication effect is poor during mining. Therefore, drones are used as data forwarding relays and small computing servers, and the flexibility and line-of-sight communication capabilities of drones are used to provide an effective solution for edge computing data transmission in mines.

[0004] Edge computing can realize collaborative computing between the cloud, the edge and the terminal. However, the way to split tasks in the system, the offloading decision of the terminal and the joint allocation of limited computing resources are urgent problems to be solved. At the same time, due to the flexibility of drones, finding the optimal drone deployment location and service method has also become a key issue in the system. Existing research rarely studies the use of drones to assist edge computing in mining faces, and the existing computing resource scheduling methods cannot achieve joint scheduling and allocation of resources. Summary of the invention

[0005] Purpose of the invention: In view of the above-mentioned problems existing in the prior art, the purpose of the present invention is to propose a smart coal mine computing power scheduling method and system based on deep reinforcement learning, which can realize the joint scheduling of cloud-edge computing power by reasonably splitting the terminal computing tasks of the smart mine mining face to meet the computing needs of data-intensive and delay-sensitive tasks.

[0006] Technical solution: To achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a smart coal mine computing power scheduling method based on deep reinforcement learning, comprising the following steps:

[0008] The terminal equipment at the mining face generates real-time computing tasks and sends real-time status information to the cloud; the status information includes the equipment location, task requirements, task computing amount, its own computing power, and communication noise;

[0009] The cloud periodically obtains the status information of the working surface, establishes a communication and computing model including terminal equipment, drones and edge computing servers, and then establishes a computing power scheduling optimization model that minimizes computing time;

[0010] A Markov decision process is established in the cloud to describe the computing power scheduling process and train a deep reinforcement learning model;

[0011] The cloud uses the trained deep reinforcement learning model to form task splitting methods, offloading strategies, computing power allocation plans and drone hovering position information based on the real-time status information of the terminal device, and sends it to the drone and edge computing server.

[0012] Furthermore, the optimization problem of the computing power scheduling optimization model that minimizes computing time is expressed as:

[0013] ;

[0014] in: ;

[0015] t Indicates time slot, T MAX The upper limit of time considered for the optimization problem, N Indicates the number of terminal devices. Indicates terminal equipment n mission requirements, The larger the value, the higher the delay sensitivity of the task. Represents an edge computing server m The computing resources are allocated to the terminal devices n The proportion of Indicates terminal equipment n The proportion of tasks that are offloaded, Indicates terminal equipment n The time calculated locally, is the indicator function, and its value is , represents the uninstall decision, Indicates terminal equipment n Directly transfer tasks to edge computing servers, Indicates terminal equipment n The task is offloaded to the edge computing server through the drone relay. Indicates terminal equipment n Offload the task to the drone for computing. , and Respectively represent terminal devices n With edge computing servers m Communication time between terminal devices n By drone w Relay to edge computing server m Communication time, and terminal equipment n With drones w The communication time between Indicates terminal equipment n Offloading tasks to edge computing servers m The required computation time, Indicates terminal equipment n Offloading tasks to drones w The required computation time, for N dimensional vector, corresponding to N The task splitting method of each terminal device is as follows: for N dimensional vector, corresponding to N The task offloading strategy of the terminal device is MN dimensional vector, corresponding to M The computing power allocation scheme for edge computing servers. For 3 W dimensional vector, corresponding to W The best hovering position for a drone.

[0016] Furthermore, for terminal devices n , the time for the task to be calculated locally is expressed as:

[0017] ;

[0018] in, represents the task computation amount, Indicates terminal equipment n Computational capacity;

[0019] when or When the terminal device n Offloading tasks to edge computing servers m The required computation time is expressed as:

[0020] ;

[0021] in, For edge computing servers m Computational capacity;

[0022] when When the terminal device n Offloading tasks to drones w The required computation time is expressed as:

[0023] ;

[0024] in, For drones w Computing power.

[0025] Furthermore, the communication time is the ratio of the offloaded task computational load to the communication rate, wherein the communication rate is calculated based on the signal-to-noise ratio, the signal-to-noise ratio is calculated based on the channel gain and the channel noise power, and the channel gain is determined based on the positions of the communicating parties.

[0026] Furthermore, the state space of the Markov decision process includes the location of the terminal device, the computing power of the terminal device itself, the location of the edge computing server, the computing power of the edge computing server, the computing power of the drone, the communication transmission power of the drone, the task computing amount of the terminal device, the task requirements, the communication transmission power of the terminal device, and the transmission noise power in its own environment;

[0027] The action space of the Markov decision process Expressed as ;in, For 4 N dimensional vector, corresponding to N One-hot encoding of the uninstallation strategy of each terminal device;

[0028] The reward function of the Markov decision process is the computing time saved after adopting the selected computing power scheduling scheme relative to the task being completely calculated locally by the terminal device.

[0029] Furthermore, the deep reinforcement learning model is trained in the cloud based on the deep deterministic policy gradient algorithm.

[0030] In a second aspect, the present invention provides a smart coal mine computing power scheduling system based on deep reinforcement learning, including a cloud, a terminal device deployed on a mine working face, an edge computing server, and a drone;

[0031] The terminal device is used to generate real-time computing tasks and send real-time status information to the cloud; the status information includes device location, task requirements, task computing amount, its own computing power and communication noise;

[0032] The cloud is used to periodically obtain the status information of the working surface, establish a communication and computing model including terminal equipment, drones and edge computing servers, and then establish a computing power scheduling optimization model that minimizes computing time; establish a Markov decision process to describe the computing power scheduling process, and train a deep reinforcement learning model; and use the trained deep reinforcement learning model to form a task splitting method, unloading strategy, computing power allocation plan and drone hovering position information according to the real-time status information of the terminal device, and send it to the drone and edge computing server.

[0033] Beneficial effects: The present invention proposes a smart coal mine computing power scheduling method based on deep reinforcement learning. The method targets the cloud-edge computing power architecture assisted by drones. In the application scenario of smart coal mines, the method reasonably divides the data computing requirements generated by the sensing equipment of the mining face, realizes the joint allocation of cloud-edge computing power, optimizes the hovering position of the drone, and meets the computing requirements of data-intensive and delay-sensitive tasks. The edge computing server in the system can be deployed in the tunnel to provide computing resources for underground working equipment. The drone equipment, as a relay device and a small computing server, can solve the problems of complex and changeable mine environment and limited signal coverage, and provide communication and computing services for underground equipment. The method of computing power scheduling using deep reinforcement learning used in the present invention can effectively reduce the computing delay of user terminal tasks, and achieves a good balance between complexity and performance. In summary, in the edge computing scenario assisted by drones in the mining face of smart mines, the computing power scheduling method based on deep reinforcement learning proposed in the present invention is superior in reducing the computing delay of user terminal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0035] Figure 2 Schematic diagram of the deep reinforcement learning algorithm training process in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0037] The core idea of ​​the present invention is to use drones to assist in data transmission and task calculation in the application scenario of edge computing of mining faces in smart mines, propose a smart coal mine computing power scheduling method based on deep reinforcement learning, reasonably split the data computing requirements generated by the sensing equipment of the mining face, and optimize the task offloading strategy and computing power allocation plan, optimize the hovering position of drones, and meet the computing requirements of data-intensive and delay-sensitive tasks. Specifically, an edge computing system architecture including a central cloud platform, edge computing servers, drones, and terminal devices is deployed in smart mines, where the cloud platform is responsible for system management and long-cycle big data processing, the edge computing server is responsible for unloading and calculating real-time computing tasks of terminal devices, and drones serve as data transmission intermediaries and small computing servers to assist in unloading computing tasks in complex mine environments, and terminal devices perceive real-time data and generate computing data.

[0038] like Figure 1 As shown, an intelligent coal mine computing power scheduling method based on deep reinforcement learning disclosed in an embodiment of the present invention comprises the following steps:

[0039] (1) The terminal equipment at the mining face generates real-time computing tasks and sends real-time status information to the cloud; the status information includes the device location, task requirements, task computing amount, its own computing power, and communication noise.

[0040] (2) The cloud periodically obtains status information of the working surface.

[0041] (3) A communication and computing model including terminal devices, drones, and edge computing servers is established in the cloud, and then a computing power scheduling optimization model that minimizes computing time is established.

[0042] (4) A Markov decision process is established in the cloud to describe the computing power scheduling process and define its state space, action space, and reward function.

[0043] (5) Train the deep reinforcement learning model based on the Deep Deterministic Policy Gradient (DDPG) algorithm in the cloud.

[0044] (6) During the execution phase, the cloud uses the trained deep reinforcement learning model to form task splitting methods, offloading strategies, computing power allocation plans, and drone hovering position information based on the real-time status information of the terminal device, and sends it to the drone and edge computing server.

[0045] (7) The drone and edge computing server execute commands.

[0046] In this embodiment, step (3) includes the following specific steps:

[0047] (3a) Establish an edge computing model under the mining face. The system includes N Terminal devices, W drones and M edge computing servers, represented as a set , , , terminal equipment n The position is represented by , the communication transmission power is , the computing power is , in bit / s. w The position is represented by , the communication transmission power is , the computing power is . Edge computing server m The position is represented by , the computing power is . Discretize time into pieces of length The terminal device is in the time slot t The amount of computational effort generated is recorded as , the unit is bit, and the task requirement is defined as ,in, Indicates the urgency of the task. The larger the value, the higher the delay sensitivity of the current task, and the more it is hoped that the calculation can be completed as soon as possible. In the task offloading stage, the computing task can be partially offloaded to the edge computing server for calculation, and partially left on the terminal device for calculation. The task splitting method is express, Indicates terminal equipment n The proportion of tasks that are offloaded. express, ;in, Indicates terminal equipment n All computing tasks are performed locally. Indicates terminal equipment n Directly transfer tasks to edge computing servers, Indicates terminal equipment n The task is offloaded to the edge computing server through the drone relay. Indicates terminal equipment n Offload the task to the drone for calculation. Since the computing power that the drone can carry is limited, this embodiment assumes that the drone can only undertake one computing task at the same time. For edge computing servers, since multiple edge computing servers have performed task offloading, computing resources need to be allocated. m , the allocation ratio of computing power resources is expressed as In order to make full use of computing resources, there are .

[0048] (3b) Establish system communication model, terminal equipment n With edge computing servers m The communication rate between is expressed as:

[0049] ;

[0050] in, B is the communication bandwidth, For terminal devices n With edge computing servers m The signal-to-noise ratio between is expressed as:

[0051] ;

[0052] in, Represents the transmission channel noise power between the terminal device and the edge computing server, represents the channel gain, which is calculated as follows: ;

[0053] in, is the attenuation coefficient, is the path loss exponent, represents the two-norm. Therefore, for , Mission Terminal Equipment n With edge computing servers m The communication time between them is expressed as:

[0054] ;

[0055] Similarly, the terminal device can be calculated n By drone w With edge computing servers m Between, terminal equipment n With drones w The communication time between and .

[0056] Since the amount of data in the calculation result is smaller than the original data, the time it takes to return the calculation result can be ignored.

[0057] (3c) Establish a system calculation model for terminal equipment n , the time for local computation of a task can be expressed as:

[0058] ;

[0059] when or When the terminal device n Offloading tasks to edge computing servers m The required computation time is expressed as:

[0060] ;

[0061] when When the terminal device n Offloading tasks to drones w The time required for the calculation is expressed as:

[0062] ;

[0063] When all parts of the task are calculated, the task is considered to be completed. Therefore, this embodiment defines the completion time of the task as the maximum completion time of all parts of the task, which is expressed as:

[0064] ;

[0065] in, is the indicator function, and its value is .

[0066] (3d) Establish a computing power scheduling optimization model that minimizes computing time. In summary, the optimization problem can be expressed as:

[0067] ;

[0068] in, for N dimensional vector, corresponding to N The task splitting method of each terminal device is as follows: for N dimensional vector, corresponding to N The task offloading strategy of the terminal device is MN dimensional vector, corresponding to M The computing power allocation scheme for edge computing servers. For 3 W dimensional vector, corresponding to W The best hovering position for a drone. T MAX The upper limit of time considered for the optimization problem.

[0069] In this embodiment, step (4) includes the following specific steps:

[0070] (4a) Define the state space of the Markov decision process , the state space includes the location of the terminal device, the computing power of the terminal device itself, the location of the edge computing server, the computing power of the edge computing server, the computing power of the drone, the communication transmission power of the drone, the task computing amount of the terminal device, the task requirements, the communication transmission power of the terminal device, and the transmission noise power in its own environment. Therefore, the state space is expressed as:

[0071] ;

[0072] in, They are N A vector consisting of the location of each terminal device, its own computing power, computing task volume, task requirements, communication transmission power, and transmission noise power. They are M A vector composed of the location and computing power of edge computing servers. They are W A vector consisting of the computing power and communication transmission power of each drone.

[0073] (4b) Define the action space of the Markov decision process. In this system, the task splitting method, offloading strategy, computing power allocation scheme and drone hovering position are optimized. Therefore, the action space is expressed as:

[0074] ;

[0075] in, The vector length is 4 N ,correspond N One-hot encoding of the uninstallation decision of each terminal.

[0076] (4c) Define the reward function of the Markov decision process In order to minimize the computing time, the reward function is defined as the computing time saved for the current task after adopting the computing power scheduling scheme output by the current action network:

[0077] ;

[0078] in, It is the computing time required when all tasks are calculated locally without offloading, expressed as:

[0079] .

[0080] In this embodiment, step (5) trains the deep reinforcement learning model based on the DDPG algorithm in the cloud. Figure 2The process of updating the deep reinforcement learning network in the training phase is described in the algorithm, including four networks: action network, evaluation network, target action network and target evaluation network. In some other embodiments, other deep reinforcement learning algorithms can also be used to solve the Markov decision process described in step (4).

[0081] Training a deep reinforcement learning model based on the DDPG algorithm includes the following specific steps:

[0082] (5a) Initialize the data buffer , the evaluation network, action network, target evaluation network, target action network, the network parameters are expressed as , , and .

[0083] (5b) Initialize the number of rounds EPISODE and the time slots in a round t .

[0084] (5c) Multiple agents output actions based on the output of the current action network and the exploration mechanism.

[0085] (5d) The terminal device, drone, and edge computing server execute actions, and the cloud generates a quadruple based on the execution results. ,in s is the current environment status, To perform an action, r is the reward function obtained after execution, is the environmental state of the next time slot, and the quaternary is stored in middle.

[0086] (5e) The current state is updated to the state of the next time slot.

[0087] (5f) If If the data in is full, the quadruple of the set batch size BATCH_SIZE is sampled as the training set to start training the deep reinforcement learning model.

[0088] (5g) will time slot t Environmental status and actions Input the evaluation network and get the time slot t The predicted value of the value function , which is simplified as .

[0089] (5h) Calculate the next time slot using the target action network t+ Predicted Action 1 ,use Represents, and uses the target evaluation network to get the next time slot t+ Expected value function of 1 ,use express.

[0090] (5i) Update the evaluation network, and the loss function is expressed as:

[0091] ;

[0092] in, S is the buffer size, MSE represents the mean square error function, For time slot t The actual reward function is is the attenuation factor.

[0093] (5j) Use action network according to time slot t Environmental status Output a set of actions, using Represented, and the value function corresponding to the predicted action given by the evaluation network output and the action network is used. To distinguish it from the output when the real environment and action are used as input, the output is represented as , which is simplified as .

[0094] (5k) Update the action network. The loss function of the action network is the output of the evaluation network in step (5j). , through the input to the action network Find the partial derivatives to get the update direction of network parameters.

[0095] (5l) In order to ensure the stability of training, the target network parameters and Perform soft update, and the update step coefficient is , the updated network parameters are:

[0096] .

[0097] (5m) Determine EPISODE and time slot t Is it less than the total training time? If so, go to step (5c), otherwise end the training.

[0098] In summary, the computing power scheduling method of the embodiment of the present invention can ensure low-latency computing of massive data through the rational use of computing resources in a mining environment and the rational deployment of drones, thereby improving the safety and production efficiency of the mining face. In a specific embodiment, the deep reinforcement learning network can be deployed on a remote cloud platform, and the deep reinforcement learning algorithm can be used to collect status information of the terminal equipment and edge computing servers of the smart mine face to generate task splitting methods, unloading strategies, drone hovering positions, and computing power allocation plans. After the deep reinforcement learning training is completed, an efficient cloud-edge collaborative computing strategy and drone scheduling plan are formed to effectively reduce the latency of computing tasks on the mining face.

[0099] Based on the description of the present invention, it should not be difficult for technical personnel in the field to see that the smart coal mine computing power scheduling method based on deep reinforcement learning proposed in the present invention can effectively reduce the computing delay of terminal equipment tasks and achieve a good balance between complexity and performance.

[0100] Based on the same inventive concept, an embodiment of the present invention discloses a smart coal mine computing power scheduling system based on deep reinforcement learning, including a cloud, a terminal device deployed on a mine working face, an edge computing server and a drone; the terminal device is used to generate real-time computing tasks and send real-time status information to the cloud; the status information includes device location, task requirements, task calculation amount, its own computing power and communication noise; the cloud is used to periodically obtain the status information of the working face, establish a communication and computing model including terminal devices, drones and edge computing servers, and then establish a computing power scheduling optimization model that minimizes computing time; establish a Markov decision process to describe the computing power scheduling process, and train a deep reinforcement learning model; and use the trained deep reinforcement learning model to form a task splitting method, unloading strategy, computing power allocation plan and drone hovering position information according to the real-time status information of the terminal device, and send it to the drone and edge computing server.

[0101] The specific process of establishing and solving the computing power scheduling optimization model that minimizes the computing time is described in the above method embodiment, which will not be repeated here. The contents not described in detail in the present invention belong to the prior art known to professional and technical personnel in this field.

Claims

1. A smart coal mine computing power scheduling method based on deep reinforcement learning, characterized in that: The steps include: The terminal equipment at the mining face generates real-time computing tasks and sends real-time status information to the cloud; the status information includes the equipment location, task requirements, task computing amount, its own computing power, and communication noise; The cloud periodically obtains the status information of the working surface, establishes a communication and computing model including terminal equipment, drones and edge computing servers, and then establishes a computing power scheduling optimization model that minimizes computing time; A Markov decision process is established in the cloud to describe the computing power scheduling process and train a deep reinforcement learning model; The cloud uses the trained deep reinforcement learning model to form task splitting methods, offloading strategies, computing power allocation plans, and drone hovering position information based on the real-time status information of the terminal device, and sends it to the drone and edge computing server; The optimization problem of the computing power scheduling optimization model that minimizes computing time is expressed as: ； in: ; t Indicates time slot, T MAX The upper limit of time considered for the optimization problem, N Indicates the number of terminal devices. Indicates terminal equipment n mission requirements, The larger the value, the higher the delay sensitivity of the task. Represents an edge computing server m The computing resources are allocated to the terminal devices n The proportion of Indicates terminal equipment n The proportion of tasks that are offloaded, Indicates terminal equipment n The time calculated locally, is the indicator function, and its value is , represents the uninstall decision, Indicates terminal equipment n All computing tasks are performed locally. Indicates terminal equipment n Directly transfer tasks to edge computing servers, Indicates terminal equipment n The task is offloaded to the edge computing server through the drone relay. Indicates terminal equipment n Offload the task to the drone for computing. , and Respectively represent terminal devices n With edge computing servers m Communication time between terminal devices n By drone w Relay to edge computing server m Communication time, and terminal equipment n With drones w The communication time between Indicates terminal equipment n Offloading tasks to edge computing servers m The computation time required, Indicates terminal equipment n Offloading tasks to drones w The computation time required, for N dimensional vector, corresponding to N The task splitting method of each terminal device is as follows: for N dimensional vector, corresponding to N The task offloading strategy of the terminal device is MN dimensional vector, corresponding to M The computing power allocation scheme for edge computing servers. For 3 W dimensional vector, corresponding to W The best hovering position for a drone.

2. According to claim 1, a smart coal mine computing power scheduling method based on deep reinforcement learning is characterized in that: For terminal equipment n , the time for the task to be calculated locally is expressed as: ； in, represents the task computation amount, Indicates terminal equipment n Computational capacity; when or When the terminal device n Offloading tasks to edge computing servers m The required computation time is expressed as: ； in, For edge computing servers m Computational capacity; when When the terminal device n Offloading tasks to drones w The required computation time is expressed as: ； in, For drones w Computing power.

3. According to claim 1, a method for scheduling intelligent coal mine computing power based on deep reinforcement learning is characterized in that: The communication time is the ratio of the offloaded task computational load to the communication rate, where the communication rate is calculated based on the signal-to-noise ratio, the signal-to-noise ratio is calculated based on the channel gain and channel noise power, and the channel gain is determined based on the locations of the communicating parties.

4. According to the method of intelligent coal mine computing power scheduling based on deep reinforcement learning in claim 1, it is characterized in that: The state space of the Markov decision process includes the location of the terminal device, the computing power of the terminal device itself, the location of the edge computing server, the computing power of the edge computing server, the computing power of the drone, the communication transmission power of the drone, the task computing amount of the terminal device, the task requirements, the communication transmission power of the terminal device, and the transmission noise power in its own environment; The action space of the Markov decision process Expressed as ;in, For 4 N dimensional vector, corresponding to N One-hot encoding of the uninstallation strategy of each terminal device; The reward function of the Markov decision process is the computing time saved after adopting the selected computing power scheduling scheme relative to the task being completely calculated locally by the terminal device.

5. According to the method of intelligent coal mine computing power scheduling based on deep reinforcement learning in claim 1, it is characterized in that: Train deep reinforcement learning models using deep deterministic policy gradient algorithms in the cloud.

6. The method for scheduling intelligent coal mine computing power based on deep reinforcement learning according to claim 1 is characterized in that: The steps for training a deep reinforcement learning model based on the deep deterministic policy gradient algorithm include: (1) Initialize the data buffer , the evaluation network, action network, target evaluation network, target action network, the network parameters are expressed as , , and ; (2) Initialize the number of rounds and the time slots in the round t ; (3) Multiple agents output actions based on the output of the current action network and the exploration mechanism; (4) The terminal device, drone, and edge computing server execute actions, and the cloud generates a quadruple based on the execution results. ,in s is the current environment status, To perform an action, r is the reward function obtained after execution, is the environmental state of the next time slot, and the quaternary is stored in middle; (5) The current state is updated to the state of the next time slot; (6) If When the data in the data is full, the quadruples of the set batch number are sampled as the training set to start training the deep reinforcement learning model; (7) Set the time slot t Environmental status and actions Input the evaluation network and get the time slot t The predicted value of the value function , which is simplified as ; (8) Use the target action network to calculate the next time slot t+ Predicted Action 1 ,use Represents, and uses the target evaluation network to get the next time slot t+ Expected value function of 1 ,use express; (9) Update the evaluation network, and the loss function is expressed as: ； in, S is the buffer size, MSE represents the mean square error function, For time slot t The actual reward function is is the attenuation factor; (10) Use action network according to time slot t Environmental status Output a set of actions, using Represented, and the value function corresponding to the predicted action given by the evaluation network output and the action network is used. To distinguish it from the output when the real environment and action are used as input, the output is represented as , which is simplified as ; (11) Update the action network. The loss function of the action network is the output of the evaluation network in step (10). , through the input to the action network Find the partial derivative to get the update direction of network parameters; (12) Target network parameters and Perform soft update, and the update step coefficient is , the updated network parameters are: ； (13) Determine the number of rounds and time slots t Is it less than the total training time? If so, go to step (3); otherwise, end the training.

7. A smart coal mine computing power scheduling system based on deep reinforcement learning, characterized in that: Including the cloud, terminal devices deployed at the mine face, edge computing servers and drones; The terminal device is used to generate real-time computing tasks and send real-time status information to the cloud; the status information includes device location, task requirements, task computing amount, its own computing power and communication noise; The cloud is used to periodically obtain the status information of the working surface, establish a communication and computing model including terminal equipment, drones and edge computing servers, and then establish a computing power scheduling optimization model that minimizes computing time; establish a Markov decision process to describe the computing power scheduling process, and train a deep reinforcement learning model; and use the trained deep reinforcement learning model to form a task splitting method, unloading strategy, computing power allocation plan and drone hovering position information according to the real-time status information of the terminal equipment, and send it to the drone and edge computing server; The optimization problem of the computing power scheduling optimization model that minimizes computing time is expressed as: ； in: ; t Indicates time slot, T MAX The upper limit of time considered for the optimization problem, N Indicates the number of terminal devices. Indicates terminal equipment n mission requirements, The larger the value, the higher the delay sensitivity of the task. Represents an edge computing server m The computing resources are allocated to the terminal devices n The proportion of Indicates terminal equipment n The proportion of tasks that are offloaded, Indicates terminal equipment n The time calculated locally, is the indicator function, and its value is , represents the uninstall decision, Indicates terminal equipment n All computing tasks are performed locally. Indicates terminal equipment n Directly transfer tasks to edge computing servers, Indicates terminal equipment n The task is offloaded to the edge computing server through the drone relay. Indicates terminal equipment n Offload the task to the drone for computing. , and Respectively represent terminal devices n With edge computing servers m Communication time between terminal devices n By drone w Relay to edge computing server m Communication time, and terminal equipment n With drones w The communication time between Indicates terminal equipment n Offloading tasks to edge computing servers m The computation time required, Indicates terminal equipment n Offloading tasks to drones w The computation time required, for N dimensional vector, corresponding to N The task splitting method of each terminal device is as follows: for N dimensional vector, corresponding to N The task offloading strategy of the terminal device is MN dimensional vector, corresponding to M The computing power allocation scheme for edge computing servers. For 3 W dimensional vector, corresponding to W The best hovering position for a drone.

8. The intelligent coal mine computing power scheduling system based on deep reinforcement learning according to claim 7 is characterized in that: The state space of the Markov decision process includes the location of the terminal device, the computing power of the terminal device itself, the location of the edge computing server, the computing power of the edge computing server, the computing power of the drone, the communication transmission power of the drone, the task computing amount of the terminal device, the task requirements, the communication transmission power of the terminal device, and the transmission noise power in its own environment; The action space of the Markov decision process Expressed as ;in, For 4 N dimensional vector, corresponding to N One-hot encoding of the uninstallation strategy of each terminal device; The reward function of the Markov decision process is the computing time saved after adopting the selected computing power scheduling scheme relative to the task being completely calculated locally by the terminal device.

Images