A robot task management system based on deep reinforcement learning

Through the deep reinforcement learning robot task management system, dynamically allocate and optimize task flow, the problem of idle robots and awkward human-computer interaction is solved, and the utilization rate and interaction smoothness of robots are improved.

CN118644055BActive Publication Date: 2024-11-12XIAMEN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411116566.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-12
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

Existing robots are usually limited to performing a single task in functionality, resulting in idleness when there are no tasks, low utilization, and causing human-computer interaction by filling out forms to post tasks.

Method used

Adopt a robot task management system based on deep reinforcement learning, including a task release subsystem, a task allocation subsystem and a task scheduling subsystem. Through dynamic allocation and optimization of task flow, robots can perform multiple tasks, and pre-trained language models and DQN algorithm improves the smoothness of human-computer interaction.

Benefits of technology

It improves robot utilization, achieves smoother human-computer interaction, solves the problem that robots can only perform a single task, and improves work efficiency through dynamic task scheduling optimization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118644055B_ABST
    Figure CN118644055B_ABST
Patent Text Reader

Abstract

The present application discloses a robot task management system based on deep reinforcement learning, which relates to the field of robot technology and solves the problem that robots currently limited to a single task will be idle when there is no task, resulting in low robot utilization. The system includes: a task publishing subsystem, a task allocation subsystem and a task scheduling subsystem; the task publishing subsystem is used to obtain a task flow and send the task flow; the task allocation subsystem is used to select a target robot for each subtask to be executed in the task flow, generate scheduling information corresponding to each target robot, and assign each scheduling information to the task scheduling subsystem of the corresponding target robot; the task scheduling subsystem is used to determine the task list of the corresponding target robot based on at least one scheduling information, and dynamically optimize the task list so that the target robot performs the task according to the updated task list.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot task management system based on deep reinforcement learning. Background Art

[0002] Mobile robots are increasingly being deployed in campuses, logistics warehousing and other application scenarios. On campus, campus robots are used to pick up express parcels and deliver food, in factories, inspection robots are used for safety inspections, and in hotels, guidance robots are used to assist in guidance. However, these service robots are usually limited in function to perform a single task. For example, in a campus scenario, users publish a single task by filling out a form, and the target robot performs a single task based on the published information. In other words, robots that are limited to a single task will be idle when there is no task, resulting in low robot utilization. At the same time, the way of publishing tasks by filling out forms leads to problems with awkward human-computer interaction. Summary of the invention

[0003] The embodiments of the present application provide a robot task management system based on deep reinforcement learning, the main purpose of which is to solve the problem that robots that are currently limited to a single task will be idle when there is no task, resulting in low robot utilization, and the method of issuing tasks by filling out forms leads to awkward human-computer interaction.

[0004] According to the present application, a robot task management system based on deep reinforcement learning is provided, comprising: a task publishing subsystem, a task allocation subsystem and a task scheduling subsystem;

[0005] The task issuing subsystem is used to obtain the task flow and send the task flow to the task allocating subsystem;

[0006] The task allocation subsystem is used to receive the task flow, select a target robot for each subtask to be executed in the task flow, generate scheduling information corresponding to each target robot based on the task information of the subtask to be executed, and allocate each scheduling information to the task scheduling subsystem corresponding to the target robot;

[0007] The task scheduling subsystem is used to receive at least one of the scheduling information, determine a task list corresponding to the target robot based on at least one of the scheduling information, and dynamically optimize the task list and update the task list in real time so that the target robot performs the task according to the updated task list;

[0008] The task allocation subsystem includes: a second communication module and a robot selection module, wherein the second communication module is connected to the robot selection module;

[0009] The second communication module is used to receive the task flow and send the task flow to the robot selection module;

[0010] The robot selection module is used to obtain environmental state information of multiple robots when receiving the task stream, select a target robot for each subtask to be executed from the multiple robots using the target DQN algorithm based on the environmental state information, determine the system identifier of the task scheduling subsystem corresponding to each target robot, extract task information corresponding to each subtask to be executed from the task stream, generate scheduling information corresponding to each target robot based on the task information and system identifier of the subtask to be executed, and send the scheduling information corresponding to each target robot to the second communication module;

[0011] The second communication module is further used to distribute the scheduling information corresponding to each of the target robots to the corresponding task scheduling subsystem;

[0012] The task scheduling subsystem includes: a third communication module and a task optimization module, wherein the third communication module is connected to the task optimization module;

[0013] The third communication module is used to receive at least one of the scheduling information and send at least one of the scheduling information to the task optimization module;

[0014] The task optimization module is used to extract the task information of the subtasks to be executed from each of the scheduling information, obtain the task information of at least one of the subtasks to be executed, generate a task list corresponding to the target robot based on the task information of at least one of the subtasks to be executed, and use a target proximal strategy optimization algorithm to dynamically optimize the task list, and update the task list in real time so that the target robot performs tasks according to the updated task list.

[0015] Optionally, the task issuing subsystem includes: a task decomposition module and a first communication module, wherein the task decomposition module is connected to the first communication module;

[0016] The task decomposition module is used to respond to the task decomposition instruction, determine the task information to be executed, perform task decomposition on the task information to be executed using the pre-trained language model, determine the task flow, and send the task flow to the first communication module, wherein the task flow includes a plurality of subtasks to be executed with an execution order and task information corresponding to each subtask to be executed, and the task information includes action information and task parameters;

[0017] The first communication module is used to send the task flow to the task allocation subsystem.

[0018] Optionally, the task decomposition module is further used to respond to the task decomposition instruction, obtain the task voice information indicated by the task decomposition instruction, convert the task voice information into task text information using speech recognition technology, and use the task text information as the task information to be executed.

[0019] Optionally, the task allocation subsystem further includes: a first training module;

[0020] The first training module is used to obtain a first training set, use the first training set to train the Q network and the Target network in the DQN algorithm, update the model parameters, and obtain the target DQN algorithm, wherein the first training set includes multiple tasks and task information corresponding to each task, robot type, and robot environment state information, and the DQN algorithm is based on a first state element, a first action element, a first state transition element, and a first reward element as a framework, wherein,

[0021] The first state element includes robot parameters, task parameters and a task allocation function, and the task allocation function is:

[0022] ,

[0023] in, g i Indicates i subtasks, unassigned Indicates unallocated, assigned Indicates allocation,

[0024] Two action types are set in the first action element, and the action type is to assign a task to the robot or not to assign a task to the robot.

[0025] A state transfer function is provided in the first state transfer element.

[0026] An expected benefit function is set in the first reward element, and the expected benefit function is:

[0027] ,

[0028] in, For expected revenue, is the first weight coefficient, is the second weight coefficient, is the third weight coefficient, is the robot's travel cost, Indicates the number of assigned tasks.

[0029] Optionally, the task scheduling subsystem further includes: a second training module;

[0030] The second training module is further used to obtain a second training set, use the second training set to train the Actor network and the Critic network in the proximal policy optimization algorithm, update the model parameters, and obtain the target proximal policy optimization algorithm, the second training set includes environmental information and robot information, and the proximal policy optimization algorithm is based on a second state element, a second action element, a second state transition element, a second reward element, and a discount factor element as a framework, wherein,

[0031] A state function is provided in the second state element, and the state function is:

[0032] ,

[0033] in, POS represents the position of the robot in the environment, b Indicates the current battery level. Indicates an action in progress. Indicates an action to be completed. c Indicates the location information of the charging pile. d Indicates the robot rotation angle,

[0034] The second action element is provided with an executable operation set ,in,

[0035] ,

[0036] ,

[0037] ,

[0038] in, forward Indicates moving forward, backward Indicates retreat, left Indicates left shift, right Indicates right shift, turnleft Indicates a left turn. turnright Indicates a right turn.

[0039] A state transfer function is provided in the second state transfer element, and the state transfer function is:

[0040] ,

[0041] in, is the state after taking action, a For the actions taken, s is the current state, Tr Indicates conversion,

[0042] An expected benefit function is set in the second reward element, and the expected benefit function is:

[0043] ,

[0044] in, is the first parameter, is the second parameter, is the change in task priority, is the change in battery charge,

[0045] A discount factor function is provided in the discount factor element, and the discount factor function is:

[0046] ,

[0047] in, is the target reward, T is the time step, t is the time, is the discount factor.

[0048] Optionally, the robot task management system based on deep reinforcement learning also includes a task monitoring subsystem;

[0049] The task monitoring subsystem is used to detect the task execution status of each target robot, obtain the detection result, and send the detection result to the terminal held by the administrator, so that the administrator can determine whether the robot task management system based on deep reinforcement learning has a fault based on the detection result when the terminal receives the detection result.

[0050] Optionally, the task monitoring subsystem includes: a monitoring module and a fourth communication module, the monitoring module is connected to the fourth communication module;

[0051] The monitoring module is used to obtain a task list of each of the target robots, obtain preset execution action information from the task list, obtain current execution action information of the target robot, generate an initial detection result based on the preset execution action information and the current execution action information, obtain the initial detection result of each of the target robots, generate a detection result based on the robot information of each of the target robots and the corresponding initial detection result, and send the detection result to the fourth communication module;

[0052] The fourth communication module is used to send the detection result to the terminal held by the administrator.

[0053] Optionally, the monitoring module is further configured to, when determining that the detection result of any of the multiple target robots indicates a detection abnormality, obtain target robot information of the target robot, generate warning information indicating a task failure based on the target robot information, and send the warning information to the fourth communication module;

[0054] The fourth communication module is further used to push the warning information to the user for warning.

[0055] By means of the above technical scheme, the present application provides a robot task management system based on deep reinforcement learning. The present application obtains a task flow through a task publishing subsystem, and the task flow includes multiple subtasks to be executed. The task allocation subsystem can select a suitable target robot for each subtask to be executed, and allocate the scheduling information to the task scheduling subsystem of each target robot. The task scheduling subsystem of each target robot can generate a task list for the target robot based on the received scheduling information, and optimize the action of the task list to obtain an updated task list, so that the target robot can perform tasks according to the updated task list, that is, execute the actions with execution order in the updated task list; according to the environmental status information of all available robots, the task allocation subsystem can select a suitable target robot for each subtask to be executed, that is, the target robot can perform multiple types of tasks, which solves the problem in the prior art that the robot can only perform a single task, and greatly improves the utilization rate of the robot. At the same time, the present application automatically obtains the tasks assigned by the user through the task publishing subsystem, and generates a task flow based on the tasks assigned by the user, so that the human-computer interaction is smoother, and the problem of awkward human-computer interaction in the prior art will not occur. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of the structure of a robot task management system based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0057] Figure 2 A schematic diagram of the structure of a task issuing subsystem of a robot task management system based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0058] Figure 3 A schematic diagram of the structure of a task allocation subsystem of a robot task management system based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0059] Figure 4 A structural framework diagram of a target DQN algorithm of a robot task management system based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0060] Figure 5A schematic diagram of the structure of a task scheduling subsystem of a robot task management system based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0061] Figure 6 A structural framework diagram of a target proximal strategy optimization algorithm for a robot task management system based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0062] Figure 7 A comparison diagram of unoptimized task scheduling results and optimized task scheduling results of a robot task management system based on deep reinforcement learning provided by an embodiment of the present application is shown;

[0063] Figure 8 A schematic diagram of the structure of another robot task management system based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0064] Fig. 9 A structural schematic diagram of a task monitoring subsystem of a robot task management system based on deep reinforcement learning provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0065] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0066] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0067] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0068] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0069] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.

[0070] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0071] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0072] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0073] The present application embodiment provides a robot task management system based on deep reinforcement learning, such as Figure 1 As shown, it includes: task publishing subsystem, task allocation subsystem and task scheduling subsystem.

[0074] Specifically, the task publishing subsystem is used to obtain the task flow and send the task flow to the task allocation subsystem; the task allocation subsystem is used to receive the task flow, select the target robot for each subtask to be executed in the task flow, generate the scheduling information corresponding to each target robot based on the task information of the subtask to be executed, and assign each scheduling information to the task scheduling subsystem of the corresponding target robot; the task scheduling subsystem is used to receive at least one scheduling information, determine the task list of the corresponding target robot based on at least one scheduling information, and dynamically optimize the task list, and update the task list in real time so that the target robot performs the task according to the updated task list. In other words, the task scheduling subsystem can update the task list in real time during the scheduling process.

[0075] Furthermore, if Figure 2 As shown, the task release subsystem includes a task decomposition module and a first communication module. The task decomposition module is connected to the first communication module. When receiving the task decomposition instruction, the task decomposition module obtains the task voice information indicated by the task decomposition instruction, converts the task voice information into task text information using speech recognition technology, and uses the processed task text information as the task information to be executed. After that, the task information to be executed is decomposed using the pre-trained language model, the task flow is determined, and the task flow is sent to the first communication module. Here, the task flow includes multiple subtasks to be executed with an execution order and the task information corresponding to each subtask to be executed. The first communication module sends the task flow to the task allocation subsystem.

[0076] It should be noted that the pretrained language model (PLM) can be understood as a trained one. The pretrained language model can be used to identify the intention of user tasks and extract key information. This key information can be task type, action list, time limit, location, etc. In other words, the pretrained language model can process complex natural language input, accurately understand user needs, and convert the identified task intention into a standardized task description, that is, the task flow, to facilitate subsequent task allocation. In the process of training with a large-scale task data set, the pretrained language model can use LoRA fine-tuning technology (preferably Baichuan2-7b model) to assist in task decomposition to cover task scenarios of different types and complexities. It should be further noted that the pretrained language model (PLM) is a prior art and is easily available to those skilled in the art, so it will not be described in detail here. In addition to using pretrained language models, other natural language processing (NLP) technologies can also be used to understand user intentions.

[0077] Exemplarily, in the campus scene, 100 campus robots are configured, and each campus robot corresponds to a task scheduling subsystem. For users, users can publish tasks through relevant applications. The campus application scene has the following objects: robots and users; the following locations: Duxing Dormitory, Yingxue Dormitory, Fengting Canteen, Phase I Cainiao Station, Phase II Cainiao Station, New Engineering R&D Building, Comprehensive Gymnasium, Swimming Pool, Siyuan Canteen, Guoguang Dormitory, Xiping Building, Trash Can, Zhou Longquan Building, Nan'an Dormitory, Furong Dormitory, Jingfeng Canteen, Student Activity Center, Luyinhai Supermarket, Comprehensive Basketball Court, Xuewu Building, Dewang Library, Boxue Dormitory, Zhengxin Building, Wenxuan Building, Kunluan Building; The actions that the robot can perform: cleaning, navigation, maintenance, monitoring, collection , disinfection, patrol, etc.; users publish tasks through related applications, that is, the task voice information of "Please disinfect the parcels at the express station and bring them to me before 7 pm". The task decomposition module of the task publishing subsystem first converts this task voice information into the task text information of "Please disinfect the parcels at the express station and bring them to me before 7 pm" by using speech recognition technology, and inputs this task text information of "Please disinfect the parcels at the express station and bring them to me before 7 pm" into the pre-trained language model, and directly outputs the task flow, that is, Task #1: {Task type: 'Disinfection', Time limit: '18:15-18:30', Equipment: 'Disinfection equipment', Location: {'Equipment station', 'Express station'}, Operation: 'Disinfection', Items: ' '}; Task #2: {Task type: 'delivery', Time limit: '18:30-19:00', Equipment: 'delivery equipment', Location: {'equipment station', 'express station', 'user location'}, Operation: ' ', Item: 'express'}. For the disinfection task, the actions are: go to the equipment station, change disinfection equipment, go to the express station, disinfect. For the delivery task, the actions are: go to the equipment station, change express equipment, go to the express station, pick up the express, go to the user's location, and put down the express.

[0078] Furthermore, if Figure 3As shown, the task allocation subsystem includes a second communication module and a robot selection module, and the second communication module is connected to the robot selection module; the second communication module receives the task flow and sends the task flow to the robot selection module. When receiving the task flow, the robot selection module obtains the environmental state information of multiple robots, selects a target robot for each subtask to be executed from multiple robots using the target DQN algorithm based on the environmental state information, determines the system identifier of the task scheduling subsystem corresponding to each target robot, extracts the task information corresponding to each subtask to be executed from the task flow, generates the scheduling information corresponding to each target robot based on the task information and the system identifier of the subtask to be executed, and sends the scheduling information corresponding to each target robot to the second communication module. The second communication module continues to allocate the scheduling information corresponding to each target robot to the corresponding task scheduling subsystem.

[0079] It should be noted that the task allocation subsystem also includes a first training module. The first training module obtains a first training set, uses the first training set to train the Q network and the Target network in the DQN algorithm, updates the model parameters, and obtains the target DQN algorithm. The first training set here includes multiple tasks and task information corresponding to each task (taking the delivery task as an example, the corresponding execution actions are: go to the equipment station, change the express equipment, go to the express station, pick up the express, go to the user's location, and put down the express), robot environment status information (task type, robot type, robot status), etc. Specifically, Figure 4 As shown, many tasks, the task information corresponding to each task, and the robot environment state information constitute the input state s , input status s The Q value of each action is calculated through the Q network in the DQN (Deep Q Network) algorithm , and select the action corresponding to the maximum Q value , according to the selected action a Affects the environment, and the environment returns to a new state and rewards r ,Will( s , a , r , ) is stored in the experience replay pool and the target Q value is calculated using the Target network , update the Target network by copying the parameters of the Q network, then use the samples in the experience revisit pool to calculate the loss function, and update the parameters of the Q network by gradient descent. Through training with a large amount of training data, the optimal target DQN algorithm can be obtained. The target DQN algorithm can find a suitable target robot for the subtask to be performed.

[0080] It should be noted that the DQN algorithm is based on the first state element, the first action element, the first state transition element, and the first reward element. In other words, the task allocation process needs to be modeled as a Markov decision process (MDP) problem:

[0081] a. The first state element: State is defined as a complex state space, which contains the following information: robot type, robot position, robot energy, task type, task position, and task priority. The spatiotemporal state and position coordinates of each robot and task can be defined as the environment matrix S, where S includes the current time and space of the robot and the task destination, etc. g is used to indicate whether a task is assigned, where:

[0082] ,

[0083] in, g i Indicates i subtasks, unassigned Indicates unallocated, assigned Indicates allocation.

[0084] b. First action element: There are two types of actions. One is to assign a task to the robot. In this case, the robot will go to the specified location to perform the task and obtain the task reward. The other is not to assign a task to the robot, and the robot's state remains unchanged.

[0085] c. The first state transition element: the state transition function is used to change the spatiotemporal state S of the robot according to the action performed.

[0086] d. The first reward element: The reward is defined as the expected benefit of the robot performing the task, which is related to the travel cost of all robots, the completion time of all assigned tasks, and the load balance, where

[0087] ,

[0088] in, For expected revenue, is the first weight coefficient, is the second weight coefficient, is the third weight coefficient, is the robot's travel cost, Indicates the number of assigned tasks.

[0089] Exemplarily, the environmental state information can be understood as environmental observation values, that is, including robot information and surrounding environment information. The robot information may include robot type, robot position, robot task execution status, robot load, robot current equipment, robot power, etc. The surrounding environment information may include charging pile location information, occupancy, etc., equipment library location information, equipment usage, etc. Combined with task information, that is, task type, time limit, task location, execution action, etc., the target DQN algorithm matches the subtask to be executed and the robot to determine which robot is most suitable to perform the task. That is, the target DQN algorithm can be used to select the most suitable target robot based on the above judgment according to the robot's position, power and current task volume. After the task is assigned, the target robot executes these subtasks to be executed. After the target robot completes a subtask to be executed, it will return the reward corresponding to the subtask to be executed, which also includes the reward lost during the execution process. Exemplarily, it can be preset to complete the task +100 (this 100 depends on different types of tasks), and each step -1 (hope that the robot completes the task with the shortest distance). The reward allows the target DQN algorithm to know whether the allocation is reasonable.

[0090] Furthermore, if Figure 5 As shown, the task scheduling subsystem includes a third communication module and a task optimization module. The third communication module is connected to the task optimization module. The third communication module receives at least one scheduling information and sends at least one scheduling information to the task optimization module. The task optimization module extracts the task information of the subtask to be executed from each scheduling information, obtains the task information of at least one subtask to be executed, generates a task list corresponding to the target robot based on the task information of at least one subtask to be executed, and uses the target proximal strategy optimization algorithm to dynamically optimize the task list, and updates the task list in real time so that the target robot performs the task according to the updated task list.

[0091] It should be noted that the task scheduling subsystem also includes a second training module, which obtains a second training set, uses the second training set to train the Actor network and the Critic network in the proximal policy optimization algorithm, updates the model parameters, and obtains the target proximal policy optimization algorithm, wherein the second training set includes multiple robot environment state information. For example, Figure 6 As shown, enter the robot environment state s n, the Actor network in the proximal policy optimization algorithm is based on the input robot environment state s nGenerate action a n, the robot environment state can be determined based on the Critic network in the proximal strategy optimization algorithm s The value function of n , and the next robot environment state s The value function of n+1 , according to the proximal strategy optimization algorithm, the advantage function can also be calculated and TD (Temporal Difference) error , this TD error can be understood as a measure of the difference between the current state value function estimate and the target value calculated by the next state value function estimate and the immediate reward, based on the robot environment state s The value function of n and the next robot environment state s The value function of n+1 determines the value loss function , based on the value loss function Use the Stochastic gradient descent (SGD) method to update the parameters of the Critic network. Furthermore, you can also use the strategy ratio Determine the clipping loss function ,in, For the current strategy, For the old strategy, based on the clipping loss function The parameters of the Actor network are updated using the gradient loss function. The target proximal policy optimization algorithm can be finally obtained through training with a large amount of training data. Then, the target proximal policy optimization algorithm can be used to obtain an updated task list based on the task list. This task list can be understood as a task action list. For example, Figure 7 As shown in the figure, the task list before optimization is: Task 1# Disinfection (1. Go to the equipment station, 2. Replace disinfection equipment, 3. Go to the express station, 4. Perform disinfection task) and Task 2# Navigation (5. Go to the user's location, 6. Navigate the user, 7. Go to the express station); after optimization by the target proximal strategy optimization algorithm, the optimized updated task list is: 1. Go to the equipment station, 2. Replace disinfection equipment, 3. Go to the user's location, 4. Navigate the user, 5. Go to the express station, 6. Go to the express station, 7. Perform disinfection task; It can be seen from the content of the task list that after re-optimization, time can be effectively saved and the efficiency of robot service can be improved.

[0092] It should be noted that the proximal policy optimization algorithm is based on the second state element, the second action element, the second state transition element, the second reward element and the discount factor element. In other words, the task scheduling process needs to be modeled as a Markov decision process (MDP) problem:

[0093] a. The second state element: define the state as ,in, POS represents the position of the robot in the environment, b Indicates the current battery level. Indicates an action in progress. Indicates an action to be completed. c Indicates the location information of the charging pile. d Indicates the robot rotation angle.

[0094] b. Second Action element: Set the actions that the robot can perform in its environment , including forward, backward, move left, move right, turn left, and turn right.

[0095] ,

[0096] ,

[0097] ,

[0098] in, forward Indicates moving forward, backward Indicates retreat, left Indicates left shift, right Indicates right shift, turnleft Indicates a left turn. turnright Indicates a right turn.

[0099] c. The second state transition factor: determined by the robot's actions and the feedback from the environment. The specific formula is:

[0100] ,

[0101] in, is the state after taking action, a For the actions taken, s is the current state, Tr Indicates conversion.

[0102] d. The second reward factor is related to the priority of the task and the battery power. The specific formula is:

[0103] ,

[0104] in, is the first parameter, is the second parameter, is the change in task priority, is the change in battery charge.

[0105] e. Discount Factor: related to the priority of the task and the battery power. The specific formula is:

[0106] ,

[0107] in, is the target reward, T is the time step, t is the time, is the discount factor. The priority of a task exists in the task specification of the task and is divided into 5 levels, namely 3-7. When the steps of the task are completed once, the priority of the task is +1, with an upper limit of 7. In addition, the priority of the task is dynamically adjusted according to the predicted task demand and robot status. For example, when a peak task demand is predicted within a certain time period, the system will assign high priority to critical tasks in advance.

[0108] It should be noted that real-time data can also be transmitted between subsystems via the wireless network Wi-Fi. For example, the task allocation subsystem transmits data to the task scheduling subsystem via the wireless network. The task scheduling subsystem can also pre-process the received data. The pre-processing can be data cleaning, noise reduction and format conversion to obtain processed data, and then schedule based on the processed data. It should also be noted that data synchronization needs to be maintained between the task scheduling subsystem and the task allocation subsystem to ensure the latest task execution status and robot status data.

[0109] Furthermore, since the target robot in this application needs to perform tasks according to the task list updated in real time, and when there is a disinfection task in the task list, the robot needs to go to the equipment station at the target time and replace the disinfection equipment. Therefore, in this application, it is also necessary to configure a shared equipment library based on a reusable mechanism, that is, the robot task management system based on deep reinforcement learning in this application also includes an equipment library management subsystem, which manages the inventory of various types of equipment such as robotic arms, sensors, vacuum cleaners, high-definition cameras, etc., and uses databases and Internet of Things technologies to record and manage the inventory status of each device in real time, including the number, location, and usage status of the equipment. It should be noted that the above-mentioned robotic arm is an integrated high-precision robotic arm equipped with an RFID tag, which enables the robotic arm to quickly identify the required equipment type. For example, when the robotic arm performs a monitoring task, it can automatically identify and install a high-definition camera. The equipment library management subsystem also has an early warning mechanism for automatically sending out-of-stock reminders when the inventory of a certain equipment is lower than the set threshold to ensure the stability of the equipment supply chain. The hardware design of the robot in this application is as follows: the robot is provided with a chassis that emphasizes stability and adaptability, so that the robot can freely attach suitable equipment as needed; it is also provided with a magnetic component to switch various types of detectors, sensors, cameras and similar devices using the magnetic attachment method; it is also provided with a fixed slot to stabilize equipment with load requirements, such as vacuum cleaners and robotic arms using the fixed slot; it is also provided with a communication interface, through which physical connection is achieved, thereby ensuring that different communication connectors such as USB, serial port, CAN, etc. have a unified physical interface, simplifying the plugging and unplugging of cables. The software design of the robot in this application is as follows: first, ROS is used to manage hardware devices, and the devices are registered as ROS nodes. After that, a plug-in is defined for each type of device using the ROS plug-in library "pluginlib". When a new device is connected, the system can dynamically load its driver and functions.

[0110] It should be noted that the robot task management system based on deep reinforcement learning in the embodiment of the present application can use Internet of Things devices to collect real-time weather and environmental data, and use equipment such as lidar for modeling, as well as use a self-built digital twin platform to simulate scenes, collect a large amount of virtual data, and test it in the simulator. It can also use the global camera function to allow users to observe and analyze group robots from different perspectives, and use the task release function (task release subsystem) to allow users to issue instructions to group robots.

[0111] Furthermore, the robot task management system based on deep reinforcement learning of the embodiment of the present application can use the equipment replacement function of the equipment management subsystem to allow the group intelligence robot to automatically go to the equipment station to switch equipment according to the received instructions. And use the task execution function to allow the user to view the entire process of robot task execution. In detail, the robot task management system based on deep reinforcement learning can use the automatic execution subfunction to allow the group intelligence robot to autonomously schedule and optimize the original task assigned to it through the built-in reinforcement learning scheduling algorithm, and automatically execute the task, and use the manual control subfunction to allow the user to directly control the robot's movement direction and viewing direction through the control keys Q, W, E, A, S, and D on the interface, and at the same time use the O key to execute the robot wake-up and the P key to execute the robot sleep special functions, and can also use the route viewing subfunction to allow the user to switch the route viewing perspective, view the robot's current position, the planned path, and how the robot will navigate to the target location.

[0112] Furthermore, the robot task management system based on deep reinforcement learning of the present application can also use the data center function to allow users to view and analyze various data and indicators. In detail, the data center function specifically includes:

[0113] a. Data statistics: Provide comprehensive data on robot task execution, such as the number of completed tasks, task success rate and other indicators.

[0114] b. Task Center: Provides a list of currently active tasks, robot task assignments, and progress updates.

[0115] c. Task details: Provides detailed data viewing for a single task, such as the robot's running path and task execution status.

[0116] d. Map: Provides a global perspective of the robot when performing tasks.

[0117] e. Robot status: Provides status information of each robot, including number of tasks, battery power, operating status, etc.

[0118] f. Data analysis: Provide historical data comparison of robots.

[0119] g. Data analysis: Provides the robot’s performance development trends such as power consumption, execution time, and execution distance.

[0120] h. Data analysis: Provides a detailed prediction model for the robot’s future tasks.

[0121] Exemplarily, when a target robot receives multiple subtasks to be executed, the current task list of each task to be executed can be obtained through the target proximal strategy optimization algorithm. The final scheduling content is all the actions of all subtasks to be executed. Exemplarily, the task of a robot is to go to the express station for disinfection and take someone to the express station. The action list decomposed is: go to the equipment station, change the disinfection equipment, go to the express station, perform disinfection actions, go to the user's location, lead the user, and go to the express station. The action list is sorted based on the environmental observation value through the target proximal strategy optimization algorithm, and the target proximal strategy optimization algorithm is used to determine in what order to complete these tasks so as to minimize the cost, the fastest time, and be able to complete them within the constraints. It should be noted that the constraints here include the time constraint of the task and the power constraint of the robot itself. Similarly, after completing a task, the target proximal strategy optimization algorithm will feedback a reward value. Exemplarily, during the task, the robot will deduct the reward value for the distance it walks, and the time to complete the task will also consider the reward value as its influencing factor. Finally, the robot knows whether the planned path is reasonable through the reward value. At the same time, if the robot's battery is low, it will directly go into standby mode, and the mission will fail. Therefore, after multiple attempts, the robot will ensure that it performs the mission in a powered state. Since the target proximal policy optimization algorithm is pre-trained, it will plan an optimal action sequence. For example, the actions planned for all actions in the above action list are: go to the equipment station, change the disinfection equipment, go to the user's location, lead the user, go to the express station, go to the express station, and perform disinfection actions.

[0122] Exemplarily, when the robot detects that the battery is about to run out, the robot goes to the nearest charging station according to the optimization instructions. In the existing method, since each robot is fixed with a task, its task allocation is carried out in a batch processing manner. Exemplarily, a task is allocated once after accumulating 10 tasks; while the tasks in this application are passed in the form of a task stream, and a robot is assigned when a task comes in. At the same time, compared with the existing scheduling methods (such as the shortest distance priority algorithm and the first-come-first-executed algorithm), the scheduling method in this application can enable the robot to optimize and rearrange its action list to achieve self-optimization scheduling. Exemplarily, the method of this application can reduce the time cost of task completion by 43.11%, the travel distance by 18.36%, the timeout rate by 24.17%, and the response time by 40.94%, while increasing the number of completed tasks by 19.68% and the robot utilization rate by an average of 20.99%. It should be noted that the principle of the shortest distance first algorithm is: assign tasks according to the distance between tasks; continuously iterate the current dynamic task list, and assign each task in order to the nearest robot that can perform the task according to the distance between the task and the robot; it is a prior art and can be easily obtained by those skilled in the art. At the same time, the principle of the first-come-first-executed algorithm is: arrange the action list according to the allocated time, and the robot will complete the operation in the order of arrival time, which is also a prior art and can be easily obtained by those skilled in the art, and will not be described in detail here.

[0123] Furthermore, the robot task management system based on deep reinforcement learning also includes a task monitoring subsystem, such as Figure 8 As shown, the task monitoring subsystem can detect the task execution status of each target robot, obtain the detection result, and send the detection result to the terminal held by the administrator, so that the administrator can determine whether the robot task management system based on deep reinforcement learning has a fault based on the detection result when the terminal receives the detection result. Specifically, Fig. 9As shown, the task monitoring subsystem includes: a monitoring module and a fourth communication module, the monitoring module is connected to the fourth communication module, the monitoring module obtains the task list of each target robot, obtains the preset execution action information from the task list, obtains the current execution action information of the target robot, and generates an initial detection result based on the preset execution action information and the current execution action information. Exemplarily, the preset execution action information can be compared with the current execution action information to obtain the comparison result as the detection result. For example, the preset execution action is to go to the express station, but the actual execution action is to perform a disinfection task. Then, through the comparison, it can be seen that the execution actions of the two are inconsistent. Then, the initial detection result of each target robot is obtained, and the detection result is generated based on the robot information of each target robot and the corresponding initial detection result, and the detection result is sent to the fourth communication module; and the fourth communication module sends the detection result to the terminal held by the administrator. Furthermore, when the monitoring module determines that the detection result of any target robot among multiple target robots indicates a detection abnormality, it obtains the target robot information of the target robot, generates warning information indicating the failure of the task based on the target robot information, and sends the warning information to the fourth communication module; the fourth communication module can also push the warning information to the user for warning. That is to say, when the robot fails to perform a task, the user can promptly understand the information of the task failure, and then can take other measures to recover the loss. For example, the administrator can be called to inform him of the situation in time for troubleshooting or claim.

[0124] It should be noted that the first, second, third and fourth mentioned above have no clear physical meanings and are only used to distinguish communication modules. The communication module may use a wireless communication module or other communication modules that can realize communication functions.

[0125] The system provided by the embodiment of the present application obtains a task flow through a task publishing subsystem, and this task flow includes multiple subtasks to be executed. The task allocation subsystem can select a suitable target robot for each subtask to be executed, and allocate the scheduling information to the task scheduling subsystem of each target robot. The task scheduling subsystem of each target robot can generate a task list for the target robot based on the received scheduling information, and optimize the actions of the task list to obtain an updated task list, so that the target robot can execute tasks according to the updated task list, that is, execute actions with an execution order in the updated task list; according to the environmental status information of all available robots, the task allocation subsystem can select a suitable target robot for each subtask to be executed, that is, the target robot can execute multiple types of tasks, which solves the problem in the prior art that the robot can only execute a single task, and greatly improves the utilization rate of the robot. At the same time, the present application automatically obtains the tasks issued by the user through the task publishing subsystem, and generates a task flow based on the tasks issued by the user, so that the human-computer interaction is smoother and the problem of awkward human-computer interaction in the prior art will not occur.

[0126] Furthermore, in addition to the above effects, in order to adapt to complex environments more quickly, the task scheduling subsystem of each target robot can further dynamically optimize the task list of the target robot after receiving the scheduling information, that is, rescheduling, which can effectively improve the work efficiency of the robot and take into account the hardware constraints of the robot, such as returning to charge when the battery is low, ensuring minimum resource consumption, so that the target robot can find a balance between task rewards and task costs.

[0127] Furthermore, this application also demonstrates a major leap in the efficiency of online streaming task allocation and dynamic scheduling of robots with interchangeable devices. The core lies in its ability to manage robots. Through this robot task management system based on deep reinforcement learning, not only can tasks be effectively assigned, but also the actions of each robot can be dynamically replanned to ensure that the capabilities and resources of each robot are optimally utilized. Taking the campus scene as an example, campus tasks range from routine delivery to emergency response, which requires both speed and adaptability. Therefore, this application innovatively adopts large-scale language models and hot exchange technology, and integrates a digital twin full-process simulation platform to make human-computer interaction more natural and resources reusable. This not only enhances the operational fluency of the system, but also significantly reduces the execution cost.

[0128] Those skilled in the art will appreciate that the modules in the system in the implementation scenario can be distributed in the system of the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more systems different from the implementation scenario. The modules in the above implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0129] The above application serial numbers are for description only and do not represent the advantages or disadvantages of the implementation scenarios.

[0130] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A robot task management system based on deep reinforcement learning, characterized in that: include: Task publishing subsystem, task allocation subsystem and task scheduling subsystem; The task issuing subsystem is used to obtain the task flow and send the task flow to the task allocating subsystem; The task allocation subsystem is used to receive the task flow, select a target robot for each subtask to be executed in the task flow, generate scheduling information corresponding to each target robot based on the task information of the subtask to be executed, and allocate each scheduling information to the task scheduling subsystem corresponding to the target robot; The task scheduling subsystem is used to receive at least one of the scheduling information, determine a task list corresponding to the target robot based on at least one of the scheduling information, and dynamically optimize the task list and update the task list in real time so that the target robot performs the task according to the updated task list; The task allocation subsystem includes: a second communication module and a robot selection module, wherein the second communication module is connected to the robot selection module; The second communication module is used to receive the task flow and send the task flow to the robot selection module; The robot selection module is used to obtain the environmental state information of multiple robots when receiving the task flow, select a target robot for each subtask to be executed from the multiple robots using the target DQN algorithm based on the environmental state information, determine the system identifier of the task scheduling subsystem corresponding to each target robot, extract the task information corresponding to each subtask to be executed from the task flow, generate the scheduling information corresponding to each target robot based on the task information and system identifier of the subtask to be executed, and send the scheduling information corresponding to each target robot to the second communication module, wherein the task information corresponding to the subtask to be executed includes the task type, time limit, task location and execution action, the environmental state information includes the robot information and the surrounding environment information, the robot information includes the robot type, robot location, robot task execution status, robot load status, robot current equipment and robot power, and the surrounding environment information includes the charging pile location information, charging pile occupancy status, equipment warehouse location information and equipment usage status; The second communication module is further used to distribute the scheduling information corresponding to each of the target robots to the corresponding task scheduling subsystem; The task scheduling subsystem includes: a third communication module and a task optimization module, wherein the third communication module is connected to the task optimization module; The third communication module is used to receive at least one of the scheduling information and send at least one of the scheduling information to the task optimization module; The task optimization module is used to extract the task information of the subtasks to be executed from each of the scheduling information, obtain the task information of at least one of the subtasks to be executed, generate a task list corresponding to the target robot based on the task information of at least one of the subtasks to be executed, and dynamically optimize the task list using a target proximal strategy optimization algorithm, and update the task list in real time, so that the target robot performs the task according to the updated task list; The task allocation subsystem also includes: a first training module; The first training module is used to obtain a first training set, use the first training set to train the Q network and the Target network in the DQN algorithm, update the model parameters, and obtain the target DQN algorithm, the first training set includes multiple tasks and task information corresponding to each task, robot type and robot environment state information, and the DQN algorithm is based on the first state element, the first action element, the first state transition element and the first reward element as the framework, wherein, The first state element includes robot parameters, task parameters and a task allocation function, and the task allocation function is: , in, g i Expressed as i subtasks, unassigned Indicates unallocated, assigned Indicates allocation, Two action types are set in the first action element, and the action type is to assign a task to the robot or not to assign a task to the robot. A state transfer function is provided in the first state transfer element. An expected benefit function is set in the first reward element, and the expected benefit function is: , in, For expected revenue, is the first weight coefficient, is the second weight coefficient, is the third weight coefficient, is the robot's travel cost, Indicates the number of tasks assigned; The task scheduling subsystem also includes: a second training module; The second training module is further used to obtain a second training set, use the second training set to train the Actor network and the Critic network in the proximal policy optimization algorithm, update the model parameters, and obtain the target proximal policy optimization algorithm, wherein the second training set includes environmental information and robot information, and the proximal policy optimization algorithm is based on a second state element, a second action element, a second state transition element, a second reward element, and a discount factor element as a framework, wherein, A state function is provided in the second state element, and the state function is: , in, POS represents the position of the robot in the environment, b Indicates the current battery level. Indicates an action in progress. Indicates an action to be completed. c Indicates the location information of the charging pile. d Indicates the robot rotation angle, The second action element is provided with an executable operation set ,in, , , , in, forward Indicates moving forward, backward Indicates retreat, left Indicates left shift, right Indicates right shift, turnleft Indicates a left turn. turnright Indicates a right turn. A state transfer function is provided in the second state transfer element, and the state transfer function is: , in, is the state after taking action, a For the actions taken, s is the current state, Tr Indicates conversion, An expected benefit function is set in the second reward element, and the expected benefit function is: , in, is the first parameter, is the second parameter, is the change in task priority, is the change in battery charge, A discount factor function is provided in the discount factor element, and the discount factor function is: , in, is the target reward, T is the time step, t is the time, is the discount factor.

2. The robot task management system based on deep reinforcement learning according to claim 1 is characterized in that: The task issuing subsystem comprises: a task decomposition module and a first communication module, wherein the task decomposition module is connected to the first communication module; The task decomposition module is used to respond to the task decomposition instruction, determine the task information to be executed, perform task decomposition on the task information to be executed using the pre-trained language model, determine the task flow, and send the task flow to the first communication module, wherein the task flow includes a plurality of subtasks to be executed with an execution order and task information corresponding to each subtask to be executed, and the task information includes action information and task parameters; The first communication module is used to send the task flow to the task allocation subsystem.

3. The robot task management system based on deep reinforcement learning according to claim 2 is characterized in that: The task decomposition module is also used to respond to the task decomposition instruction, obtain the task voice information indicated by the task decomposition instruction, convert the task voice information into task text information using speech recognition technology, and use the task text information as the task information to be executed.

4. The robot task management system based on deep reinforcement learning according to claim 1 is characterized in that: It also includes a mission monitoring subsystem; The task monitoring subsystem is used to detect the task execution status of each target robot, obtain the detection result, and send the detection result to the terminal held by the administrator, so that the administrator can determine whether the robot task management system based on deep reinforcement learning has a fault based on the detection result when the terminal receives the detection result.

5. The robot task management system based on deep reinforcement learning according to claim 4 is characterized in that: The task monitoring subsystem comprises: a monitoring module and a fourth communication module, wherein the monitoring module is connected to the fourth communication module; The monitoring module is used to obtain a task list of each of the target robots, obtain preset execution action information from the task list, obtain current execution action information of the target robot, generate an initial detection result based on the preset execution action information and the current execution action information, obtain the initial detection result of each of the target robots, generate a detection result based on the robot information of each of the target robots and the corresponding initial detection result, and send the detection result to the fourth communication module; The fourth communication module is used to send the detection result to the terminal held by the administrator.

6. The robot task management system based on deep reinforcement learning according to claim 5 is characterized in that: The monitoring module is further configured to obtain target robot information of the target robot when it is determined that the detection result of any target robot among the multiple target robots indicates a detection abnormality, generate warning information indicating a task failure based on the target robot information, and send the warning information to the fourth communication module; The fourth communication module is further used to push the warning information to the user for warning.