Task scheduling method and device based on reinforcement learning, equipment and storage medium

Abstract

The invention discloses a task scheduling method, device and equipment based on reinforcement learning and a storage medium, and the method comprises the steps: inputting an obtained to-be-allocated task, the current tower crane information of each tower crane in a preset region and the current environment information of the preset region into a first reinforcement learning network, obtaining to-be-executed initial subtasks, output by the first reinforcement learning network, of the tower cranes; according to the current tower crane information of each tower crane in the preset area, the current environment information of the preset area and the to-be-executed initial sub-task of each tower crane, predicting a task execution result of executing the corresponding to-be-executed initial sub-task by each tower crane; and according to a task execution result of executing the corresponding to-be-executed initial sub-task by each tower crane, adjusting the to-be-executed initial sub-task of each tower crane output by the first reinforcement learning network to obtain a to-be-executed target sub-task of each tower crane. Therefore, the execution efficiency and accuracy of task scheduling can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tower cranes, and in particular to a task scheduling method based on reinforcement learning, a task scheduling device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] A tower crane (referred to as a tower crane) is a vertical transportation mechanical equipment, which is widely used in the fields of construction, port logistics, industrial manufacturing, etc. Its main function is to realize the vertical lifting and horizontal carrying of heavy objects through a hook, so as to improve the construction efficiency and reduce the labor cost. In modern construction engineering, with the increasing of the building scale and the improvement of the construction complexity, the role of the tower crane becomes more and more important, and in the case of multi-task parallel and multi-device operation, the application scenario of multi-tower crane cooperation is also required more and more.

[0003] The operation of the traditional tower crane needs a professional driver to manually control in the cockpit, but this way has the following problems: the tower crane driver needs to receive professional training, and the long-term high work will increase the labor cost; the scheduling efficiency of the manually operated tower crane is low, and is easily limited by the experience and physical strength of the operator; in some scenarios (such as blind hoisting, cross-hoisting, etc.), due to the limited vision of the tower crane driver, there are potential risks such as tower crane collision. SUMMARY

[0004] The technical problem solved by the present application is to provide a task scheduling method based on reinforcement learning, a task scheduling device, an electronic device and a computer readable storage medium, which can improve the automation and accuracy of task scheduling.

[0005] To solve the above technical problems, one technical solution adopted by the present application is to provide a task scheduling method based on reinforcement learning, which comprises: acquiring a to-be-allocated task, current tower crane information of each tower crane in a preset area and current environment information of the preset area; inputting the to-be-allocated task, the current tower crane information of each tower crane in the preset area and the current environment information of the preset area into a first reinforcement learning network to obtain initial sub-tasks to be executed by each tower crane output by the first reinforcement learning network; predicting task execution results of each tower crane executing the corresponding initial sub-tasks to be executed according to the current tower crane information of each tower crane in the preset area, the current environment information of the preset area and the initial sub-tasks to be executed by each tower crane; and adjusting the initial sub-tasks to be executed by each tower crane output by the first reinforcement learning network according to the task execution results of each tower crane executing the corresponding initial sub-tasks to be executed, to obtain target sub-tasks to be executed by each tower crane.

[0006] To solve the above technical problems, another technical scheme adopted by the present application is to provide a task scheduling device based on reinforcement learning, comprising an acquisition module, a learning network processing module, a prediction module and an adjustment module; the acquisition module is configured to acquire a to-be-allocated task, current crane information of each crane in a preset area and current environment information of the preset area; the learning network processing module is configured to input the to-be-allocated task, the current crane information of each crane in the preset area and the current environment information of the preset area into a first reinforcement learning network to obtain initial to-be-executed sub-tasks of each crane output by the first reinforcement learning network; the prediction module is configured to predict task execution results of each crane in executing corresponding initial to-be-executed sub-tasks according to the current crane information of each crane in the preset area, the current environment information of the preset area and the initial to-be-executed sub-tasks of each crane; and the adjustment module is configured to perform adjustment processing on the initial to-be-executed sub-tasks of each crane output by the first reinforcement learning network according to the task execution results of each crane in executing corresponding initial to-be-executed sub-tasks to obtain target to-be-executed sub-tasks of each crane.

[0007] To solve the above technical problems, another technical scheme adopted by the present application is to provide an electronic device comprising a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above-mentioned task scheduling method based on reinforcement learning.

[0008] To solve the above technical problems, another technical scheme adopted by the present application is to provide a computer-readable storage medium comprising stored program data, wherein the program data is executed by a processor to implement the above-mentioned task scheduling method based on reinforcement learning.

[0009] The present application has the following beneficial effects: the embodiments of the present application acquire a to-be-allocated task, current crane information of each crane in a preset area and current environment information of the preset area; input the to-be-allocated task, the current crane information of each crane in the preset area and the current environment information of the preset area into a first reinforcement learning network to obtain initial to-be-executed sub-tasks of each crane output by the first reinforcement learning network; predict task execution results of each crane in executing corresponding initial to-be-executed sub-tasks according to the current crane information of each crane in the preset area, the current environment information of the preset area and the initial to-be-executed sub-tasks of each crane; and perform adjustment processing on the initial to-be-executed sub-tasks of each crane output by the first reinforcement learning network according to the task execution results of each crane in executing corresponding initial to-be-executed sub-tasks to obtain target to-be-executed sub-tasks of each crane. Thus, the learning capability of reinforcement learning is used to optimize task scheduling, and the execution efficiency and accuracy are higher than those of traditional manual scheduling. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0011] Figure 1 is a flowchart of an exemplary embodiment of the task scheduling method shown in the present application;

[0012] Figure 2 is a flowchart of another exemplary embodiment of the task scheduling method shown in the present application;

[0013] Figure 3 is a structural diagram of an exemplary embodiment of the task scheduling device shown in the present application;

[0014] Figure 4 is a structural diagram of an embodiment of the electronic device provided by the present application;

[0015] Figure 5 is a structural diagram of an embodiment of the computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the present application, not to limit the present application. In addition, it should be noted that, for the sake of description, only the parts related to the present application are shown in the drawings, not all the structures. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0017] First of all, it should be noted that in the multi-crane working scene, the task scheduling needs to consider the allocation of multi-crane tasks, the priority of tasks, time constraints and the optimal use of resources. The traditional task scheduling method relies on manual command signals, and it is difficult to handle complex tasks and dynamic scene changes in real time.

[0018] Based on this, the present application provides a task scheduling method based on reinforcement learning, electronic equipment and computer readable storage medium, which can automatically allocate tasks to be allocated through a reinforcement learning network, improve the cooperation efficiency and hoisting safety of multi-crane, the following can be referred to as task scheduling method based on reinforcement learning. Details can be found in Figure 1 , Figure 1 is a flowchart of an exemplary embodiment of the task scheduling method shown in the present application.

[0019] The execution subject of the task scheduling method can be a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc., and the server can be an HLS (HTTP Live Streaming, HTTP-based adaptive bit rate streaming protocol). The execution subject of the task scheduling method can also be a task scheduling apparatus. In some possible implementation manners, the task scheduling method can be implemented by a processor invoking computer readable instructions stored in a memory.

[0020] Specifically, the task scheduling method of the embodiment includes the following steps:

[0021] S110: Obtain a to-be-assigned task, current tower crane information of each tower crane in a preset area, and current environment information of the preset area.

[0022] The to-be-assigned task can be a task to be executed by a tower crane in the preset area. The to-be-assigned task can include one or more to-be-assigned sub-tasks. In some application scenarios, each to-be-assigned sub-task in the to-be-assigned task can have a task priority, a task deadline, and a task execution location, etc.

[0023] The preset area can be determined according to a geographical area or a tower crane operating environment. For example, the preset area can be a freight port, a construction site, etc. With the increasing size of buildings and the increasing complexity of construction, there can be multiple tower cranes in the preset area, and the tasks executed by the multiple tower cranes are different, so it is necessary to plan the execution tasks and execution paths of the tower cranes to ensure the execution efficiency and safety of the tower cranes.

[0024] The tower crane is a heavy mechanical equipment widely used in construction sites, ports, wharfs, etc., and its main function is to lift and move heavy objects. In some embodiments, the tower crane feeds back current tower crane information and current environment information to the task scheduling apparatus in real time, and the task scheduling apparatus uniformly manages all tower cranes in the preset area. A plurality of sensors can be installed on the tower crane to detect the current tower crane information and the current environment information in real time, including but not limited to image sensors, ranging sensors, etc.

[0025] The current tower crane information includes, but is not limited to, the current working state of the tower crane (for example, busy or idle), current position information, and a current task being performed. Illustratively, the current position information of the tower crane can be extracted from a real-time image of the global tower crane working environment. The current working state and the current task being performed of each tower crane can be determined from historical task allocation information, or the current working state of the tower crane can be determined by monitoring whether the tower crane is running.

[0026] The current environment information can refer to environment information related to the working of the tower crane in the preset area, including, but not limited to, obstacle information, a global map, and working area restrictions. Illustratively, the current environment information of the tower crane can be extracted from a real-time image of the global tower crane working environment.

[0027] S120: inputting the task to be allocated, the current tower crane information of each tower crane in the preset area, and the current environment information of the preset area into the first reinforcement learning network to obtain an initial subtask to be performed of each tower crane output by the first reinforcement learning network.

[0028] Reinforcement learning is that at each time, an agent (in the embodiments of the present application, the tower crane is the agent) observes the current state st in the environment it is in, makes an action, so that the state is transferred to st+1, and the agent gets a reward from the environment accompanying the state transition. The ultimate goal is to find a policy π(st) to maximize the cumulative reward value. The task scheduling model of the present application is a model established based on reinforcement learning. The first reinforcement learning network is a module for generating task scheduling instructions in the task scheduling model, and is mainly used for allocating the task to be allocated to each tower crane according to the task to be allocated, the current tower crane information of each tower crane in the preset area, and the current environment information of the preset area. By optimizing the first reinforcement learning network, the task scheduling instructions output by the first reinforcement learning network are more in line with the current task demand and the scheduling environment. It should be noted that each time the next operation is performed, the task to be allocated, the current tower crane information of each tower crane in the preset area, and the current environment information of the preset area need to be reacquired to determine whether the environment has changed, so that the task scheduling is reperformed according to the updated information. Thus, it is ensured that the task scheduling scheme can be quickly adjusted when the task demand or the environmental condition changes. Illustratively, the algorithm design of the task scheduling model can be Multi-Agent Deep Deterministic Policy Gradient (MADDPG).

[0029] The to-be-executed initial sub-tasks are the execution tasks of the cranes output by the first reinforcement learning network. Exemplarily, the to-be-allocated sub-tasks in the to-be-allocated tasks are allocated to the cranes to obtain the to-be-executed initial sub-tasks of the cranes. It should be noted that one crane can be allocated zero, one or more to-be-executed initial sub-tasks, for example, when the number of to-be-allocated sub-tasks is greater than the number of cranes, some cranes can be allocated to multiple to-be-executed initial sub-tasks, and some cranes can not be allocated to to-be-executed initial sub-tasks.

[0030] The task scheduling device inputs the to-be-allocated tasks, the current crane information of the cranes in the preset area and the current environment information of the preset area into the first reinforcement learning network, and processes the first reinforcement learning network to determine the to-be-executed initial sub-tasks of the cranes.

[0031] S130: According to the current crane information of the cranes in the preset area, the current environment information of the preset area and the to-be-executed initial sub-tasks of the cranes, the task execution results of the cranes executing the corresponding to-be-executed initial sub-tasks are predicted.

[0032] After the task scheduling device outputs the to-be-executed initial sub-tasks of the cranes by the first reinforcement learning network, the task execution results of the to-be-executed initial sub-tasks of the cranes are predicted. The task execution results include task execution state, task execution time, resource utilization rate in task execution process and the like. Through the task execution results, it can be judged whether the first reinforcement learning network meets the preset requirements, and the first reinforcement learning network is optimized by the task execution results.

[0033] S140: According to the task execution results of the cranes executing the corresponding to-be-executed initial sub-tasks, the to-be-executed initial sub-tasks of the cranes output by the first reinforcement learning network are adjusted to obtain the to-be-executed target sub-tasks of the cranes.

[0034] In some embodiments, the to-be-executed target sub-tasks of the cranes are obtained by optimizing the first reinforcement learning network. Specifically, if the to-be-executed initial sub-tasks of the cranes output by the first reinforcement learning network do not meet the preset conditions, the network parameters of the first reinforcement learning network are adjusted, and the to-be-allocated tasks, the current crane information of the cranes in the preset area and the current environment information of the preset area are re-processed based on the adjusted first reinforcement learning network to obtain the to-be-executed initial sub-tasks of the cranes output by the adjusted first reinforcement learning network; if the preset conditions are met, the output to-be-executed initial sub-tasks of the cranes are directly determined as the to-be-executed target sub-tasks of the corresponding cranes.

[0035] In some embodiments, the task scheduling model can be modeled as a Markov decision process, each tower crane as an agent, the state space designed to include all information affecting task scheduling, such as current tower crane information (including the position information of each tower crane, the current task being executed, and the idle or busy working state), to-be-assigned tasks (including the execution location of each to-be-assigned subtask in the to-be-assigned task, the task priority, the task deadline, and the resources required by the task), and current environmental information (including obstacle information, a global map, and work area restrictions, etc.); the action space is designed as executable operations of each tower crane, including moving (the tower crane moving to a certain location), picking / placing (the tower crane starting or ending the lifting task), and waiting (the tower crane remaining at the current location); the reward function is used to guide the tower crane to learn the optimization goal, such as positive rewards including successful completion of tasks, reduction of task execution time, and improvement of resource utilization, and negative rewards including task delays, tower crane collisions, and resource idling, etc.; the state transition model describes how the system transitions to the next state after performing an action from a certain state. The algorithm is designed as a multi-agent deep deterministic policy gradient algorithm, which is a centralized training and decentralized execution framework, solving the problem of non-stationary environment while maintaining the distribution characteristics of execution. The multi-agent deep deterministic policy gradient algorithm is continuously optimized to obtain a task scheduling strategy that meets the requirements. Thus, through the reinforcement learning technology, the task allocation result with the highest target task reward value under the current state is obtained through continuous interaction with the environment, so as to improve the rationality and accuracy of task allocation.

[0036] In other embodiments, the first reinforcement learning network can also be used as an auxiliary. After obtaining the to-be-executed initial subtasks of each tower crane output by the first reinforcement learning network, if the predicted task execution result is not optimistic, the to-be-executed initial subtasks of each tower crane can be adjusted through a preset rule, for example, manually adjusted.

[0037] In other embodiments, the to-be-assigned tasks can also be scheduled and processed through priority scheduling algorithms, shortest job first scheduling algorithms, genetic algorithms, and simulated annealing algorithms, etc. to obtain the to-be-executed target subtasks of each tower crane.

[0038] In some embodiments, the adjustment processing of the to-be-executed initial subtasks of each tower crane includes using the reward value as an evaluation of task scheduling to determine whether the preset requirements are met. Specifically, the target task reward value is determined according to the task execution result of each tower crane executing the corresponding to-be-executed initial subtask; in response to the target task reward value not meeting the first preset requirement, the network parameters of the first reinforcement learning network are optimized according to the target task reward value until the target task reward value meets the first preset requirement, obtaining the first reinforcement learning network that meets the requirements; and the to-be-executed initial subtasks of each tower crane output by the first reinforcement learning network that meets the requirements are determined as the to-be-executed target subtasks of each tower crane.

[0039] In the task scheduling problem based on reinforcement learning, the goal is to optimize the first reinforcement learning network to maximize the target task reward value to improve the rationality of task scheduling. The reward and punishment rules are generally determined according to the task requirements. Generally, if the task is completed, a reward is given, otherwise a punishment is given; if the task execution time is reduced, a reward is given, otherwise a punishment is given; if the resource utilization rate of task execution is improved, a reward is given, otherwise a punishment is given. Specifically, the task execution result includes task completion, task completion time and resource utilization rate, the first task reward value of each tower crane executing the corresponding initial sub-task of the to-be-executed task is determined according to the task completion; the second task reward value of each tower crane executing the corresponding initial sub-task of the to-be-executed task is determined according to the task completion time; the third task reward value of each tower crane executing the corresponding initial sub-task of the to-be-executed task is determined according to the resource utilization rate; and the target task reward value is determined according to the first task reward value, the second task reward value and the third task reward value.

[0040] The first task reward value can be determined according to the task completion. As an example, when the task is completed, a fixed positive value is given, and when the task is not completed, a fixed negative value is given.

[0041] The second task reward value can be determined according to the task completion time. As an example, when the task completion time is less than the preset time, a fixed positive value is given, and when the task completion time is greater than or equal to the preset time, a fixed negative value is given. As another example, the task completion time is negatively correlated with the second task reward value, the longer the task completion time, the smaller the second task reward value, and vice versa.

[0042] The third task reward value can be determined according to the resource utilization rate. As an example, when the resource utilization rate is greater than the preset resource utilization rate, a fixed positive value is given, and when the resource utilization rate is less than or equal to the preset resource utilization rate, a fixed negative value is given. As another example, the resource utilization rate is positively correlated with the third task reward value, the greater the resource utilization rate, the greater the third task reward value, and vice versa.

[0043] After determining the first task reward value, the second task reward value and the second task reward value of each tower crane, the task completion, the task completion time and the resource utilization rate of each tower crane are calculated by summation or weighted summation to obtain the target task sub-reward value of each tower crane; then the target task sub-reward value of each tower crane is summed or weighted summed to obtain the target task reward value.

[0044] Further, after task scheduling, the target paths of the cranes for executing the corresponding target sub-tasks to be executed also need to be planned. In some embodiments, the target paths of the cranes can also be determined by reinforcement learning. Specifically, the target sub-tasks to be executed of the cranes, the current crane information of the cranes in the preset area, and the current environmental information of the preset area are input into a second reinforcement learning network to obtain the initial paths to be executed of the cranes output by the second reinforcement learning network; the path execution results of the cranes in executing the corresponding target sub-tasks to be executed through the corresponding initial paths to be executed are predicted according to the current crane information of the cranes in the preset area, the current environmental information of the preset area, and the initial paths to be executed of the cranes; and the initial paths to be executed of the cranes output by the second reinforcement learning network are adjusted to obtain the target paths to be executed of the cranes.

[0045] The path planning can also be completed by using a path planning model based on reinforcement learning. The second reinforcement learning network can be a module for generating the initial paths to be executed in the path planning model, and is mainly used for planning the initial paths to be executed for the cranes to execute the corresponding target sub-tasks to be executed according to the target sub-tasks to be executed, the current crane information of the cranes in the preset area, and the current environmental information of the preset area. By optimizing the second reinforcement learning network, the initial paths to be executed output by the second reinforcement learning network are more in line with the current environment and meet the task requirements. It also needs to be noted that when the next step is performed, the target sub-tasks to be executed, the current crane information of the cranes in the preset area, and the current environmental information of the preset area need to be obtained again to determine whether the environment has changed, so that the path planning is performed again according to the updated information. In this way, it is ensured that the planned path can be quickly adjusted when the environmental conditions change, and problems such as collision can be prevented. Exemplarily, the algorithm design of the path planning model can be a DQN (Deep Q-Network) algorithm.

[0046] The initial paths to be executed are the paths of the cranes output by the second reinforcement learning network for executing the corresponding initial sub-tasks to be executed. After the task scheduling device outputs the initial paths to be executed of the cranes by the second reinforcement learning network, the path execution results of the cranes in executing the corresponding initial paths to be executed are predicted. The path execution results include path collision probability and path length, etc. Whether the second reinforcement learning network meets the preset requirements can be determined by the path execution results, and the second reinforcement learning network is optimized by the path execution results.

[0047] In some embodiments, the to-be-executed target paths of the cranes for executing the to-be-executed target sub-tasks are obtained by optimizing the second reinforcement learning network. Specifically, if the to-be-executed target paths of the cranes output by the second reinforcement learning network do not satisfy the preset condition, the network parameters of the second reinforcement learning network are adjusted, and the to-be-allocated tasks, the current crane information of the cranes in the preset area, and the current environment information of the preset area are reprocessed based on the adjusted second reinforcement learning network to obtain the to-be-executed initial paths of the cranes output by the adjusted second reinforcement learning network; if the preset condition is satisfied, the to-be-executed initial paths of the cranes output are directly determined as the to-be-executed target paths of the corresponding cranes.

[0048] In some embodiments, the path planning model can be modeled as a Markov decision process, each crane as an agent, including a state space, an action space, a reward function, and an algorithm design, wherein the state space includes a high-dimensional state vector composed of the crane positions, speeds, and hoisted object sizes of the cranes; the action space includes a high-dimensional action vector composed of the moving directions of the cranes; the reward function includes a combination of the crane collision probability and the path length; and the algorithm adopts a DQN (Deep Q-Network) algorithm supporting discrete states and actions.

[0049] As an example, assuming that there are two cranes in the preset area, the to-be-executed target sub-tasks of the two cranes are determined according to the task scheduling, the No. 1 crane needs to be hoisted from point A to point B, and the No. 2 crane needs to be hoisted from point C to point D; each crane has six action instructions (forward, backward, left, right, up, and down), so the output layer of the entire DQN network has 6*6=36 nodes in total, respectively representing the target path reward values obtained by taking the action instruction combinations in the current state, and the action instruction combination with the maximum target path reward value is issued to the bottom layer control; the input of the DQN network is composed of the crane positions, speeds, hoisted object sizes, global three-dimensional map, and target points of hoisting of the cranes; in the reinforcement learning training phase, after each action instruction is executed, if the hoisted object collides with the surrounding environment or the hoisted object cannot reach the target point, a negative reward is given, and if the hoisted object successfully reaches the target point, a positive reward is given, and then the network parameters of the DQN network are adjusted using the gradient descent algorithm.

[0050] In the path planning problem based on reinforcement learning, the target is also to optimize the second reinforcement learning network to maximize the target path reward value to improve the rationality of path planning. Specifically, the target path reward value is determined according to the path execution results of the cranes; in response to the target path reward value not satisfying the second preset requirement, the network parameters of the second reinforcement learning network are optimized according to the target path reward value until the target path reward value satisfies the second preset requirement, obtaining the second reinforcement learning network satisfying the requirement; and the to-be-executed initial paths of the cranes output by the second reinforcement learning network satisfying the requirement are determined as the to-be-executed target paths of the cranes.

[0051] The target path reward value is used to evaluate the initial path to be executed of each crane output by the second reinforcement learning network. Generally, the higher the target path reward value, the better the second reinforcement learning network, and the lower the target path reward value, the worse the second reinforcement learning network. Exemplarily, the path execution result includes a path length and a path collision probability, the first path reward value of each crane executing the corresponding initial path to be executed is determined according to the path length, the second path reward value of each crane executing the corresponding initial path to be executed is determined according to the path collision probability, and the target path reward value is determined according to the first path reward value and the second path reward value. Thus, the target path reward value is determined by the path collision probability and the path length, and an optimal path with a short path length and a small collision probability can be generated for the crane to execute the target sub-task to be executed.

[0052] The first path reward value can be determined according to the path length. As an example, when the path length is less than a preset length, a fixed positive value is given, and when the path length is greater than or equal to the preset length, a fixed negative value is given. As another example, the path length is negatively correlated with the first path reward value, the longer the path length, the smaller the first path reward value, and vice versa.

[0053] The second path reward value can be determined according to the path collision probability. The path collision probability is the probability of collision between the crane and other cranes or obstacles when the crane executes the corresponding path. As an example, when the path collision probability is less than a preset probability, a fixed positive value is given, and when the path collision probability is greater than or equal to the preset probability, a fixed negative value is given. As another example, the path collision probability is negatively correlated with the second path reward value, the greater the path collision probability, the smaller the second path reward value, and vice versa.

[0054] After the first path reward value and the second path reward value of each crane are determined, the first path reward value and the second path reward value of the same crane can be processed by summation or weighted summation to obtain the target path sub-reward value of each crane, and then the target path sub-reward values of the cranes are summed or weighted summed to obtain the target path reward value.

[0055] After the target path to be executed of each crane is obtained, the corresponding crane is controlled to execute the corresponding target sub-task to be executed according to the corresponding target path to be executed of the crane, and the current crane information of each crane in the task to be allocated and the preset area and the current environment information of the preset area are updated and used for the next task scheduling and the next path planning. Thus, after the crane completes the target sub-task to be executed according to the target path to be executed, the state information is updated in real time to promote the optimization and adjustment of reinforcement learning.

[0056] In order to elaborate the task scheduling method applied in the present application,Figure 2 The flowchart is further illustrated as follows:

[0057] In a multi-crane working environment, task requirements and environmental conditions change dynamically. Traditional task scheduling methods usually use rule-based or heuristic algorithm-based methods, such as priority rule-based scheduling algorithms or genetic algorithms, and traditional path planning methods usually use A* algorithm or Dijkstra algorithm-based shortest path planning. These methods are mostly static calculations, lack adaptability to real-time environments, and algorithm design relies on human experience. As the complexity of tasks and the number of cranes increase, the computational overhead also increases significantly, making it difficult to meet real-time requirements. Based on this, the embodiments of the present application introduce reinforcement learning for task scheduling, aiming to realize intelligent, dynamic and optimal multi-crane task scheduling and path planning.

[0058] Initialization phase: The execution device of the task scheduling method can also be a central controller. The central controller starts the system and initializes the system parameters (current crane information, to-be-assigned tasks and current environment information); acquires data acquisition information of each crane, and updates the current environment information and current crane information perceived by each crane in real time according to the acquired data acquisition information, and waits for task scheduling applications. Each crane acquires the current environment information and current crane information of the crane through sensors (such as laser radar, camera, encoder and torque, etc.).

[0059] Task scheduling phase: in response to receiving the task scheduling request in the task pool, the to-be-assigned task in the task scheduling request, the real-time updated current crane information and the real-time updated current environment information are input into the task scheduling model together, and the to-be-executed target sub-tasks of each crane output by the task scheduling model are obtained;

[0060] Path planning phase: after obtaining the to-be-executed target sub-tasks of each crane, the current crane information, to-be-executed target sub-tasks and current environment information of each crane are input into the path planning model, the to-be-executed target paths for the cranes that need to execute the to-be-executed target sub-tasks are planned, and are sent to the corresponding cranes; thereby dynamically generating the to-be-executed target paths through reinforcement learning, improving the accuracy and safety of the paths.

[0061] Execution phase: control each crane to execute the to-be-executed target sub-tasks according to the corresponding to-be-executed target paths, and feed back the task completion situation to the central controller, promoting the learning and adjustment of the optimization algorithm. Thus, under the same reinforcement learning framework, optimal solutions of task scheduling and path planning are realized at the same time, significantly improving the computational efficiency of task scheduling and path planning, and reducing the computational complexity.

[0062] Please refer to Figure 3 , Figure 3is a structural schematic diagram of an exemplary embodiment of a task scheduling apparatus shown in the present application. The task scheduling apparatus 300 comprises an acquisition module 310, a learning network processing module 320, a prediction module 330 and an adjustment module 340. The acquisition module 310 is configured to acquire a to-be-allocated task, current crane information of each crane in a preset area and current environment information of the preset area. The learning network processing module 320 is configured to input the to-be-allocated task, the current crane information of each crane in the preset area and the current environment information of the preset area into a first reinforcement learning network to obtain initial sub-tasks to be executed by each crane output by the first reinforcement learning network. The prediction module 330 is configured to predict task execution results of each crane in executing corresponding initial sub-tasks to be executed according to the current crane information of each crane in the preset area, the current environment information of the preset area and the initial sub-tasks to be executed by each crane. The adjustment module 340 is configured to perform adjustment processing on the initial sub-tasks to be executed by each crane output by the first reinforcement learning network according to the task execution results of each crane in executing corresponding initial sub-tasks to be executed, to obtain target sub-tasks to be executed by each crane.

[0063] The above scheme, the task scheduling apparatus acquires a to-be-allocated task, current crane information of each crane in a preset area and current environment information of the preset area; inputs the to-be-allocated task, the current crane information of each crane in the preset area and the current environment information of the preset area into a first reinforcement learning network to obtain initial sub-tasks to be executed by each crane output by the first reinforcement learning network; predicts task execution results of each crane in executing corresponding initial sub-tasks to be executed according to the current crane information of each crane in the preset area, the current environment information of the preset area and the initial sub-tasks to be executed by each crane; performs adjustment processing on the initial sub-tasks to be executed by each crane output by the first reinforcement learning network according to the task execution results of each crane in executing corresponding initial sub-tasks to be executed, to obtain target sub-tasks to be executed by each crane. Thus, the learning ability of reinforcement learning is used to optimize task scheduling, which has higher execution efficiency and accuracy compared with traditional manual scheduling.

[0064] The functions of each module can be referred to the task scheduling method embodiments, which will not be described here.

[0065] To implement the task scheduling method of the above embodiments, the present application provides another electronic device, please refer to Figure 4 , Figure 4 is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0066] The electronic device 400 comprises a memory 410 and a processor 420, wherein the memory 410 and the processor 420 are coupled.

[0067] The memory 410 is configured to store program data, and the processor 420 is configured to execute the program data to implement the task scheduling method of the above embodiments.

[0068] In this embodiment, the processor 420 can also be referred to as a CPU (Central Processing Unit). The processor 420 can be an integrated circuit chip having a processing capability of signals. The processor 420 can also be a general processor, a digital signal processor (DSP), an ASIC (Application-Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general processor can be a microprocessor or the processor 420 can also be any conventional processor.

[0069] The present application also provides a computer readable storage medium, such as Figure 5 As shown in the figure, the computer readable storage medium 500 is used to store program data 510, which when executed by a processor, is used to implement the task scheduling method in the method embodiments of the present application.

[0070] The method involved in the task scheduling method embodiments of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0071] The above is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for task scheduling based on reinforcement learning, characterized in that, The method comprises: obtaining a to-be-assigned task, current crane information of each crane in a preset area, and current environment information of the preset area; inputting the to-be-assigned task, the current crane information of each crane in the preset area, and the current environment information of the preset area into a first reinforcement learning network to obtain initial sub-tasks to be executed by each crane output by the first reinforcement learning network; predicting task execution results of each crane in executing corresponding initial sub-tasks to be executed according to the current crane information of each crane in the preset area, the current environment information of the preset area, and the initial sub-tasks to be executed by each crane; adjusting the initial sub-tasks to be executed by each crane output by the first reinforcement learning network according to the task execution results of each crane in executing corresponding initial sub-tasks to be executed, to obtain target sub-tasks to be executed by each crane.

2. The method of claim 1, wherein, The step of adjusting the initial sub-tasks to be executed by each crane output by the first reinforcement learning network according to the task execution results of each crane in executing corresponding initial sub-tasks to be executed, to obtain target sub-tasks to be executed by each crane, comprises: determining a target task reward value according to the task execution results of each crane in executing corresponding initial sub-tasks to be executed; in response to the target task reward value not satisfying a first preset requirement, optimizing network parameters of the first reinforcement learning network according to the target task reward value until the target task reward value satisfies the first preset requirement, to obtain a first reinforcement learning network satisfying the requirement; determining the initial sub-tasks to be executed by each crane output by the first reinforcement learning network satisfying the requirement as the target sub-tasks to be executed by each crane.

3. The method of claim 2, wherein, The task execution results comprise a task completion condition, a task completion time, and a resource utilization rate, and the step of determining a target task reward value according to the task execution results of each crane in executing corresponding initial sub-tasks to be executed comprises: determining a first task reward value of each crane in executing corresponding initial sub-tasks to be executed according to the task completion condition; determining a second task reward value of each crane in executing corresponding initial sub-tasks to be executed according to the task completion time; determining a third task reward value of each crane in executing corresponding initial sub-tasks to be executed according to the resource utilization rate; determining the target task reward value according to the first task reward value, the second task reward value, and the third task reward value.

4. The method of claim 1, wherein, After the step of adjusting the initial sub-tasks to be executed by each crane output by the first reinforcement learning network according to the execution results of each crane in executing corresponding initial sub-tasks to be executed, to obtain target sub-tasks to be executed by each crane, the method further comprises: inputting the target sub-tasks to be executed by each crane, the current crane information of each crane in the preset area, and the current environment information of the preset area into a second reinforcement learning network to obtain initial paths to be executed by each crane output by the second reinforcement learning network; predicting path execution results of each crane in executing corresponding target sub-tasks to be executed through corresponding initial paths to be executed according to the current crane information of each crane in the preset area, the current environment information of the preset area, and the initial paths to be executed by each crane; and Adjust the initial path to be executed of each tower crane output by the second reinforcement learning network according to the path execution result corresponding to each tower crane, to obtain a target path to be executed of each tower crane.

5. The method of claim 4, wherein, The step of adjusting the initial path to be executed of each tower crane output by the second reinforcement learning network according to the path execution result corresponding to each tower crane, to obtain a target path to be executed of each tower crane, comprises: Determining a target path reward value according to the path execution result corresponding to each tower crane; In response to the target path reward value not meeting a second preset requirement, optimizing the network parameter of the second reinforcement learning network according to the target path reward value until the target path reward value meets the second preset requirement, to obtain a second reinforcement learning network meeting the requirement; Determining the initial path to be executed of each tower crane output by the second reinforcement learning network meeting the requirement as the target path to be executed of each tower crane.

6. The method of claim 5, wherein, The path execution result comprises a path length and a path collision probability, and the step of determining a target path reward value according to the path execution result corresponding to each tower crane comprises: Determining a first path reward value of each tower crane executing the corresponding initial path to be executed according to the path length; Determining a second path reward value of each tower crane executing the corresponding initial path to be executed according to the path collision probability; Determining the target path reward value according to the first path reward value and the second path reward value.

7. The method of claim 4, wherein, After the step of adjusting the initial path to be executed of each tower crane output by the second reinforcement learning network according to the path execution result corresponding to each tower crane, to obtain a target path to be executed of each tower crane, the method further comprises: Controlling the corresponding tower crane to execute the corresponding target subtask to be executed according to the target path to be executed corresponding to each tower crane; Updating the to-be-assigned task, the current tower crane information of each tower crane in the preset area and the current environment information of the preset area and using them for next task scheduling and next path planning.

8. A task scheduling apparatus based on reinforcement learning, characterized by, The device comprises: An acquisition module configured to acquire a to-be-assigned task, current tower crane information of each tower crane in a preset area and current environment information of the preset area; A learning network processing module configured to input the to-be-assigned task, the current tower crane information of each tower crane in the preset area and the current environment information of the preset area into a first reinforcement learning network, to obtain an initial subtask to be executed of each tower crane output by the first reinforcement learning network; A prediction module configured to predict a task execution result of each tower crane executing the corresponding initial subtask to be executed according to the current tower crane information of each tower crane in the preset area, the current environment information of the preset area and the initial subtask to be executed of each tower crane; An adjustment module configured to adjust the initial subtask to be executed of each tower crane output by the first reinforcement learning network according to the task execution result of each tower crane executing the corresponding initial subtask to be executed, to obtain a target subtask to be executed of each tower crane.

9. An electronic device, comprising: comprise: A memory and a processor, wherein the memory stores program instructions, and the processor fetches the program instructions from the memory to execute the method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, comprise: program data stored thereon for causing a processor to implement the method of any of claims 1-7 when the program data is executed by the processor.

Images