Multi-task adaptive scheduling method and device based on reinforcement learning, computer device and computer readable storage medium

Abstract

This application relates to a multi-task adaptive scheduling method, apparatus, computer device, and computer-readable storage medium based on reinforcement learning. The method includes: acquiring task description information of multiple original tasks and determining the task feature vector corresponding to each original task; determining a target scheduling strategy based on the running status information of multiple task execution modules, task execution environment information, task progress information, and a reinforcement learning model; controlling multiple task execution modules to execute multiple original tasks according to the target scheduling strategy; updating the reinforcement learning model in real time based on task execution feedback information; and determining tasks to be rescheduled from multiple original tasks based on task execution feedback information; and rescheduling the tasks to be rescheduled based on the current real-time state vector of the task execution modules and the task feature vector of the tasks to be rescheduled. Using the scheme of this application, an accurate and efficient multi-task scheduling scheme can be achieved.

Description

Technical Field

[0001] This application relates to the field of task processing technology, and in particular to a multi-task adaptive scheduling method, apparatus, computer device, and computer-readable storage medium based on reinforcement learning. Background Technology

[0002] With the increasing maturity of artificial intelligence, the Internet of Things, and automation technologies, intelligent work methods and automated systems have become indispensable components in modern industrial production, logistics warehousing, and smart buildings. In multi-device collaborative operation scenarios, how to achieve efficient scheduling of multiple tasks among multiple execution modules is a key issue for improving overall system efficiency and resource utilization.

[0003] In related technologies, the execution order and resource allocation of tasks are generally arranged through preset scheduling rules or optimization algorithms. Specifically, multi-task scheduling mainly adopts static or semi-static scheduling methods, which suffer from low scheduling efficiency and low resource utilization. Summary of the Invention

[0004] Therefore, it is necessary to provide a multi-task adaptive scheduling method, device, computer equipment, computer-readable storage medium, and computer program product based on reinforcement learning to address the above-mentioned technical problems, which can improve the efficiency of multi-task scheduling and resource utilization.

[0005] Firstly, this application provides a multi-task adaptive scheduling method based on reinforcement learning, including:

[0006] Responding to task execution requests, obtaining task description information for multiple raw tasks, extracting task attributes for each raw task based on the task description information, and determining the task feature vector corresponding to each raw task based on the task attributes;

[0007] Based on the running status information, task execution environment information, and task progress information of multiple task execution modules, a real-time status vector for each task execution module is generated.

[0008] Input the task feature vector and the real-time state vector into the reinforcement learning model to obtain the target scheduling strategy output by the reinforcement learning model for scheduling multiple original tasks among multiple task execution modules.

[0009] According to the target scheduling strategy, multiple task execution modules are controlled to execute multiple original tasks, and task execution feedback information for the original tasks is obtained in real time.

[0010] The reinforcement learning model is updated in real time based on task execution feedback information, and tasks to be rescheduled are determined from multiple original tasks based on task execution feedback information.

[0011] The updated fused features are obtained by fusing the current real-time state vector of the task execution module and the task feature vector of the task to be rescheduled.

[0012] The updated fused features are input into the updated reinforcement learning model to obtain the updated scheduling policy, and the rescheduled tasks are rescheduled according to the updated scheduling policy.

[0013] Secondly, this application provides a multi-task adaptive scheduling device based on reinforcement learning, comprising:

[0014] The response module is used to respond to task execution requests, obtain task description information of multiple raw tasks, extract task attributes of each raw task based on the task description information, and determine the task feature vector corresponding to each raw task based on the task attributes.

[0015] The generation module is used to generate real-time status vectors for each task execution module based on the running status information, task execution environment information, and task progress information of multiple task execution modules.

[0016] The input module is used to input the task feature vector and the real-time state vector into the reinforcement learning model to obtain the target scheduling strategy output by the reinforcement learning model for scheduling multiple original tasks among multiple task execution modules.

[0017] The execution module is used to control multiple task execution modules to execute multiple original tasks according to the target scheduling strategy, and to obtain task execution feedback information for the original tasks in real time.

[0018] The update module is used to update the reinforcement learning model in real time based on task execution feedback information, and to determine the tasks to be rescheduled from multiple original tasks based on task execution feedback information.

[0019] The fusion module is used to fuse the current real-time state vector of the task execution module and the task feature vector of the task to be rescheduled to obtain the updated fused features.

[0020] The scheduling module is used to input the updated fused features into the updated reinforcement learning model to obtain the updated scheduling policy, and to reschedule the rescheduled tasks according to the updated scheduling policy.

[0021] Thirdly, this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the method described above.

[0022] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0023] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described above.

[0024] The aforementioned reinforcement learning-based multi-task adaptive scheduling method, apparatus, computer device, computer-readable storage medium, and computer program product first extract task features and collect state data, then make initial scheduling decisions based on a reinforcement learning model, and collect feedback during task execution, update the model online, and identify tasks to be rescheduled. Finally, the system reschedules the tasks to be rescheduled based on the updated model. The reinforcement learning model can continuously learn and optimize based on actual execution results, making the scheduling strategy increasingly intelligent. At the same time, when anomalies occur during task execution, the system can automatically identify and trigger rescheduling, achieving high efficiency, high real-time performance, and strong adaptability in multi-task scheduling. Attached Figure Description

[0025] Figure 1 An application environment diagram of a multi-task adaptive scheduling method based on reinforcement learning provided in the embodiments of this application;

[0026] Figure 2 A flowchart illustrating a multi-task adaptive scheduling method based on reinforcement learning provided in an embodiment of this application;

[0027] Figure 3 A structural block diagram of a multi-task adaptive scheduling device based on reinforcement learning provided in this application embodiment;

[0028] Figure 4 An internal structural diagram of a computer device provided in an embodiment of this application;

[0029] Figure 5 An internal structural diagram of another computer device provided in an embodiment of this application;

[0030] Figure 6 This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0032] Before describing the embodiments of this application, the relevant technologies and their existing problems will be further explained:

[0033] In related technologies, task scheduling generally employs static or semi-static scheduling schemes. Static scheduling refers to pre-calculating a fixed task execution order and resource allocation scheme based on estimated task execution time and resource requirements before all tasks begin execution, and then executing them sequentially according to this scheme. Semi-static scheduling, building upon static scheduling, allows for limited adjustments at specific points in time, but the frequency and flexibility of adjustments are still significantly limited. For example, in manufacturing execution systems (MES) on industrial production lines, scheduling plans are typically formulated before the start of the shift. When unexpected situations arise during production, such as equipment failures, urgent order insertions, or material delays, manual intervention is required to reschedule. In AGV scheduling systems for logistics and warehousing, rule-based scheduling methods are typically used, such as simple rules like first-come, first-served or shortest processing time priority, to allocate tasks.

[0034] The aforementioned traditional reinforcement learning-based multi-task adaptive scheduling methods have the following problems:

[0035] First, scheduling is inefficient and lacks real-time performance. Static or semi-static scheduling cannot respond to changes in system state in real time. When a task execution module malfunctions, task execution time exceeds expectations, or a new task is suddenly inserted, the original scheduling plan deviates significantly from the actual situation. However, the system cannot automatically and quickly adjust the task execution order and resource allocation, and can only rely on manual intervention to reschedule. Manual intervention is not only time-consuming, but also difficult to find an optimal scheduling solution under complex constraints, leading to long periods of idle task execution modules or large-scale task delays, resulting in overall low scheduling efficiency.

[0036] Second, there is a lack of real-time feedback and adaptive scheduling mechanisms. Traditional scheduling methods mainly rely on open-loop control during execution, meaning that after issuing instructions according to a preset plan, subsequent decisions are not dynamically adjusted based on feedback from the execution status. Although the system can collect equipment status information, this information is mainly used for monitoring and alarms, rather than for real-time optimization of scheduling strategies. Due to the lack of a closed-loop feedback mechanism, the scheduler cannot perceive the deviation between actual execution and the plan, let alone use this deviation information to correct subsequent scheduling decisions, resulting in a disconnect between the scheduling strategy and the actual execution status.

[0037] Third, traditional scheduling methods cannot automatically handle exceptional situations during task execution. In multi-task collaborative execution, various exceptional situations frequently occur, such as task execution module failures causing interruptions, task execution times exceeding time window constraints leading to timeouts, multiple task execution modules simultaneously contending for the same resource causing conflicts, and the urgent insertion of high-priority new tasks. Traditional scheduling methods lack the ability to automatically identify and respond to these exceptional situations. When exceptional situations occur, affected subsequent tasks often become stuck in ineffective waiting due to unmet dependencies, or fail to execute due to resource allocation conflicts, further exacerbating system chaos and efficiency degradation.

[0038] Fourth, scheduling strategies cannot be continuously optimized based on operational experience. Traditional scheduling methods rely on preset rules or algorithms, and their scheduling logic is fixed and deterministic. The system does not learn from past successful or failed task executions and cannot automatically optimize its scheduling strategy as runtime increases. For example, if a certain scheduling pattern consistently causes congestion on specific devices, traditional systems cannot autonomously identify and correct this pattern, leading to repeated occurrences of the same problem. This lack of learning ability limits the system's adaptability and intelligence in complex and dynamic environments.

[0039] Therefore, how to implement a reinforcement learning-based multi-task adaptive scheduling method that can respond to system state changes in real time, continuously optimize scheduling strategies using feedback information, automatically handle abnormal situations during execution, and has self-learning capabilities is an urgent problem to be solved.

[0040] The multi-task adaptive scheduling method based on reinforcement learning provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0041] like Figure 2 As shown, this application provides a multi-task adaptive scheduling method based on reinforcement learning, which is applied to... Figure 1The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0042] Step 202: Respond to the task execution request, obtain the task description information of multiple original tasks, extract the task attributes of each original task based on the task description information, and determine the task feature vector corresponding to each original task based on the task attributes.

[0043] In this context, a task execution request refers to an instruction that triggers the scheduling process. It can be transmitted in real-time from the system front-end (such as a user interface) or external devices (such as the host computer of a production management system). The task execution request carries task description information for multiple raw tasks. The task description information consists of raw data describing the basic attributes of each task; it can be structured data (such as data packets in JSON or XML format) or unstructured text. Task attributes are multi-dimensional information extracted from the task description information to characterize the task's features, including at least one of the following attributes: task type, resource requirements, dependencies, and priority. The task type identifies the nature of the task, such as assembly, handling, inspection, sorting, cleaning, or patrol. Resource requirements describe the conditions required to execute the task, including the required task execution module type (such as a robotic arm or AGV), the amount of computing resources, and the type and number of sensor data. Dependencies describe the execution constraints between this task and other tasks, such as sequential dependencies (task B can only start after task A is completed), parallel execution dependencies (tasks C and D can be executed simultaneously), or mutually exclusive execution dependencies (tasks E and F cannot be executed simultaneously). Priority attributes describe the urgency and importance of a task, including task urgency level, task importance level, and time window constraints. The time window constraints include the earliest start time and the latest completion time for the task.

[0044] For example, suppose we receive task descriptions for three original tasks: Task 1 is "PCB board mounting for urgent orders," Task 2 is "PCB board mounting for regular orders," and Task 3 is "finished product quality sampling inspection." From the description of Task 1, we can extract: Task type is "assembly," resource requirement is "robot mounting arm," dependency is "no prerequisite tasks," and priority attributes are "high urgency, time window: within 10 minutes." From Task 2, we can extract: Task type is "assembly," resource requirement is "robot mounting arm," dependency is "no prerequisite tasks," and priority attribute is "medium urgency." From Task 3, we can extract: Task type is "inspection," resource requirement is "inspection sensor," dependency is "dependent on Task 1 for completion," and priority attribute is "low urgency."

[0045] A task feature vector is a fixed-dimensional numerical vector obtained by numerically encoding task attributes. It represents the complete attribute information of the task, facilitating subsequent processing by the reinforcement learning model. Specifically, the extracted task type identifier, resource requirements, dependencies, and priority attributes are converted into numerical vectors, and then concatenated according to a preset dimensional order to obtain the task feature vector corresponding to the original task. By transforming the original unstructured or semi-structured task description information into a standardized, numerical feature vector representation, the subsequent reinforcement learning model can uniformly process different types of task inputs, thus providing basic data support for intelligent scheduling decisions.

[0046] Step 204: Generate a real-time status vector for each task execution module based on the running status information, task execution environment information, and task progress information of multiple task execution modules.

[0047] The operational status information refers to the current operational status of each task execution module, including at least one of the following: idle status, busy status, fault status, location information, current task progress percentage, and estimated remaining completion time. Specifically, idle status indicates that the module is not currently executing any tasks; busy status indicates that the module is executing tasks; fault status indicates that the module has encountered an error and cannot function properly; location information indicates the module's current spatial coordinates; progress percentage indicates the percentage of the current task completed; and estimated remaining completion time indicates the time required to complete the current task.

[0048] Task execution environment information refers to the environmental parameters of the task execution module, including at least one of the following: temperature, humidity, light intensity, air quality, and path congestion level. For example, in an industrial production line scenario, light intensity may affect the accuracy of visual inspection tasks; path congestion level may affect the AGV's movement speed and path selection.

[0049] Task progress information refers to the progress of a task currently being executed by the task execution module, including at least one of the following: actual elapsed time, estimated remaining completion time, and the ratio of completed subtasks to the total number of subtasks. For example, for an assembly task that requires 10 steps, completing 3 steps indicates a completion rate of 30%.

[0050] For example, suppose there is a robotic arm in the system as a task execution module. Its operating status information may include: current status is "busy", position coordinates are "(2.5m, 3.0m, 1.2m)", current task progress percentage is "60%", and estimated remaining completion time is "2 minutes"; its environmental information may include: ambient temperature "25℃", light intensity "300 lux", and path congestion level "low"; its task progress information may include: actual time used "3 minutes", estimated remaining completion time "2 minutes", and percentage of sub-tasks completed "30%".

[0051] The real-time state vector is a unified numerical vector obtained by aligning and standardizing the three types of information mentioned above using timestamps. Specifically, since the operating status information, environmental information, and progress information come from different sensors, their sampling frequencies and dimensions may differ. Therefore, they need to be aligned according to the same time base, and the data needs to be normalized or standardized to have a uniform value range, ultimately generating the real-time state vector corresponding to each task execution module.

[0052] This application embodiment comprehensively collects the real-time status information of the task execution module and transforms it into a standardized vector representation, providing the reinforcement learning model with a complete perception of the current system status, enabling scheduling decisions to be made based on real and real-time on-site conditions.

[0053] Step 206: Input the task feature vector and the real-time state vector into the reinforcement learning model to obtain the target scheduling strategy output by the reinforcement learning model for scheduling multiple original tasks among multiple task execution modules.

[0054] In this embodiment, the reinforcement learning model learns the optimal decision-making strategy by interacting with the environment and based on reward signals. The reinforcement learning model can employ a Deep Q-Network (DQN) or Actor-Critic architecture. The model is trained with the goal of maximizing cumulative reward. Cumulative reward is a comprehensive score calculated based on task execution results, and its calculation dimensions include at least one of the following: task completion efficiency (the faster the task is completed, the higher the reward), execution latency (the shorter the task waiting time, the higher the reward), resource utilization (the less idle time and the higher the utilization of the task execution module, the higher the reward), and task conflict (the fewer resource or path conflicts between tasks, the higher the reward).

[0055] The task feature vectors of all original tasks can be fused with the real-time state vectors of each task execution module (e.g., concatenating all task feature vectors in task order and then concatenating them with the real-time state vectors) to form a complete input state matrix. This input state matrix contains the attribute information of all tasks currently to be processed and the real-time state information of all task execution modules. After the input state matrix is ​​input into the reinforcement learning model, the model calculates through forward propagation and outputs a target scheduling policy. The target scheduling policy is a decision scheme for scheduling multiple original tasks among multiple task execution modules. This target scheduling policy can include at least one of the following decisions: task execution order adjustment decision (determining the execution order of each task relative to other tasks), resource allocation decision (assigning each task to a specific task execution module), priority adjustment policy (adjusting the priority level of each task relative to the current priority, such as elevating tasks facing timeout risk to higher priority), and network resource allocation policy (allocating communication bandwidth and transmission priority to each task).

[0056] This application's embodiments utilize the decision-making capabilities of reinforcement learning models to automatically generate optimal scheduling strategies based on all current task attributes and the real-time state of the system. Compared to traditional scheduling methods based on fixed rules, this approach can better handle complex dependencies and dynamically changing environments.

[0057] Step 208: Control multiple task execution modules to execute multiple original tasks according to the target scheduling strategy, and obtain task execution feedback information for the original tasks in real time.

[0058] The process involves generating corresponding control commands based on the target scheduling strategy and issuing them to each task execution module. Each control command includes at least a task identifier, action parameters, expected start time, and expected completion time. Upon receiving the control command, the task execution module performs the corresponding task operation according to the command requirements. For example, if the target scheduling strategy indicates that "task A is executed by robotic arm 1, in the first position," then a control command is generated and issued to robotic arm 1, and robotic arm 1 begins executing task A.

[0059] Task execution feedback information refers to data collected in real time during task execution that reflects the actual situation of task execution. This feedback information includes at least one of the following: the actual completion time of the original task (i.e., the actual time taken from the start to the end of the task, which is different from the expected completion time), the real-time status changes of the task execution module during execution (such as a state sequence from "idle" to "busy" and then back to "idle"), the actual amount of resources used during the original task execution (such as the actual CPU time, memory, and communication bandwidth used, which is different from the pre-allocated resources), and abnormal events that occur during the original task execution (such as path conflicts caused by two AGVs arriving at the same intersection point at the same time, task interruption caused by a failure of the task execution module, and task execution time exceeding the time window constraint).

[0060] For example, during task execution, sensors continuously monitor and collect feedback information: the actual completion time of task 1 is 8 minutes (better than the expected 10 minutes), the state of robotic arm 1 changes from "idle" to "busy" and then back to "idle", the actual CPU time occupied by task 1 is 30 seconds and the communication bandwidth is 10Mbps, and no abnormal events occur during the execution; during the execution of task 2, the vibration amplitude of robotic arm 1 exceeds the threshold, triggering a fault warning.

[0061] The embodiments of this application collect feedback information in real time while the task is being executed. This feedback information is used to evaluate the execution effect of the current scheduling strategy on the one hand, and to provide a data basis for subsequent model updates and rescheduling on the other hand, forming a closed loop between execution and feedback.

[0062] Step 210: Update the reinforcement learning model in real time based on the task execution feedback information, and determine the tasks to be rescheduled from multiple original tasks based on the task execution feedback information.

[0063] The process involves calculating task completion efficiency based on the actual completion time in the feedback information; for example, a shorter actual completion time results in higher efficiency. Optionally, execution latency can be calculated based on real-time state changes; for example, a longer task waiting time incurs a greater latency penalty. Optionally, resource utilization can be calculated based on the actual amount of resources used; for example, less idle time and higher utilization result in a greater reward. Optionally, task conflict can be calculated based on abnormal events; for example, a negative reward is given if a conflict occurs. The calculated results are used as reward components, and a weighted sum is obtained to obtain a comprehensive reward signal. This comprehensive reward signal reflects the effectiveness of the current scheduling strategy in this execution. The comprehensive reward signal, along with the input state and selected action corresponding to the generation of the reward signal, are used as training samples. During model training, batches of training samples are randomly sampled, the loss function is calculated using the temporal difference error method, and the network weights of the model are updated using the backpropagation algorithm. In this way, the reinforcement learning model can continuously optimize its decision-making ability based on actual execution results, making the model "smarter with use."

[0064] Based on the actual completion time, original tasks whose current execution progress has not reached a preset progress threshold are marked as incomplete tasks. For example, a task expected to complete in 5 minutes but still not completed after 8 minutes is marked as incomplete. Based on the abnormal events in the feedback information and the actual completion time, it is determined whether the dependencies between the original tasks have been broken. For example, if task B depends on task A, but task A fails to complete on time due to a malfunction, causing task B to be unable to proceed as planned, the task with the broken dependency is marked as a task awaiting rescheduling. Furthermore, if a new high-priority task is inserted during task execution (e.g., a user urgently submits a new order with a higher priority than the currently executing task), it is also marked as a task awaiting rescheduling. The incomplete tasks, tasks with broken dependencies, and newly inserted high-priority tasks are then merged to obtain a set of tasks awaiting rescheduling.

[0065] This application embodiment utilizes feedback information to continuously optimize the reinforcement learning model, enabling the scheduling strategy to evolve continuously with the accumulation of operational experience. Furthermore, by identifying tasks that need to be rescheduled in real time, it provides a set of tasks for subsequent dynamic rescheduling, ensuring that the system can respond promptly to abnormal situations and changes in requirements during the execution process.

[0066] Step 212: Fuse the current real-time state vector of the task execution module and the task feature vector of the task to be rescheduled to obtain the updated fused features.

[0067] When there are tasks awaiting rescheduling, the status acquisition operation is re-executed: The real-time status vectors of each task execution module are obtained, representing the latest running status information, task execution environment information, and task progress information. Simultaneously, the task feature vectors corresponding to each task in the set of tasks awaiting rescheduling are obtained. Then, the task feature vectors of the tasks awaiting rescheduling are fused (e.g., concatenated) with the current real-time status vectors to form an updated fused feature. This updated fused feature represents the complete information of the set of tasks awaiting rescheduling in the latest state.

[0068] Feature fusion can be achieved through concatenation, specifically including: arranging the feature vectors of all tasks to be rescheduled sequentially by task ID to form a first combined vector, the dimension of which is equal to the dimension of a single task feature vector multiplied by the number of tasks to be rescheduled; arranging the real-time state vectors of all task execution modules sequentially by module ID to form a second combined vector, the dimension of which is equal to the dimension of a single module's real-time state vector multiplied by the total number of task execution modules; then concatenating the first and second combined vectors end-to-end to form a complete one-dimensional vector, which is the updated fused feature. The dimension of the updated fused feature is equal to the total dimension of the feature vectors of all tasks to be rescheduled plus the total dimension of the real-time state vectors of all task execution modules.

[0069] Alternatively, feature fusion can also be achieved using a weighted summation method. For the corresponding feature dimensions in the feature vector of the task to be rescheduled and the real-time state vector, different weight coefficients are assigned to them, and then the weighted summation is performed to obtain a fused vector. The weight coefficients can be set according to the importance of the feature dimensions; for example, task priority features can be given higher weights, while ambient temperature features can be given lower weights. The weighted summation method can achieve feature fusion without increasing the vector dimension, making it suitable for scenarios with high computational efficiency requirements.

[0070] For example, the tasks to be rescheduled include Task 1, Task 2, and Task 4. At this point, the real-time status is re-acquired: the robotic arm originally executing Task 1 has returned to idle, but the detection sensors are still in a busy state, while the AGV is idle. The task feature vectors of Task 1, Task 2, and Task 4 are fused with the current real-time status vector to obtain the updated fused features.

[0071] Based on the latest system status and information on tasks to be rescheduled, the embodiments of this application construct updated input features to provide an accurate and real-time data foundation for rescheduling, ensuring that rescheduling decisions can reflect the current real situation.

[0072] Step 214: Input the updated fused features into the updated reinforcement learning model to obtain the updated scheduling policy, and reschedule the task to be rescheduled according to the updated scheduling policy.

[0073] The updated scheduling strategy is a scheduling scheme for the set of tasks to be rescheduled. Based on this updated scheduling strategy, control is restored to each task execution module to execute the tasks to be rescheduled. For example, if the original scheduling plan cannot continue due to a task execution module failure, the updated scheduling strategy can reassign tasks from the failed module to other idle task execution modules, or adjust the execution order of tasks to bypass the failure point; if a newly inserted high-priority task needs to be executed immediately, the updated scheduling strategy can interrupt currently executing low-priority tasks and allocate resources preferentially to high-priority tasks.

[0074] For example, the updated reinforcement learning model outputs an updated scheduling strategy based on the current state (the mounting robot arm is idle, the detection sensor is busy, and the AGV is idle) and the tasks to be rescheduled (task 1, task 2, and task 4): Since task 4 is a high-priority urgent order and the mounting robot arm is currently idle, task 4 will be executed by the mounting robot arm first; task 1 (the original unfinished task) will continue to be executed after task 4 is completed; since the detection sensor is still busy, task 3 (the detection task that depends on task 1) will be temporarily suspended; task 2 (the locking task) will be executed by the idle robot arm after task 1 is completed.

[0075] This application embodiment utilizes an updated reinforcement learning model to dynamically reschedule tasks to be rescheduled based on the latest system state, enabling the system to automatically adapt to various changes during execution (such as equipment failure, task timeout, insertion of new tasks, etc.) without manual intervention, thereby significantly improving the real-time performance, flexibility and robustness of multi-task scheduling.

[0076] This application embodiment first extracts task features and collects state data, then makes an initial scheduling decision based on a reinforcement learning model, and collects feedback during task execution, updates the model online, and identifies tasks to be rescheduled. Finally, it reschedules the tasks to be rescheduled based on the updated model. The reinforcement learning model can continuously learn and optimize based on actual execution results, making the scheduling strategy more and more intelligent over time. At the same time, when anomalies occur during task execution, the system can automatically identify and trigger rescheduling, achieving high efficiency, high real-time performance, and strong adaptability in multi-task scheduling.

[0077] In some embodiments, in response to a task execution request, task description information of multiple original tasks is obtained, task attributes of each original task are extracted based on the task description information, and a task feature vector corresponding to each original task is determined based on the task attributes, including:

[0078] For each original task, the task type field of the original task is parsed from the task description information to obtain the task type identifier of the original task;

[0079] Parse the resource requirement field of the original task from the task description information to obtain the resource requirement of each task in the original task.

[0080] Parse the list of upstream and downstream tasks that depend on the original task from the task description information, and determine the dependency relationship between the current original task and other original tasks based on the list of upstream and downstream tasks.

[0081] Parse the priority attributes of the original task from the task description information; the priority attributes include task urgency, task importance level, and time window constraints;

[0082] The task feature vector of the original task is obtained by concatenating the task type identifier, resource requirements, dependencies, and priority attributes.

[0083] The task type field is a data item in the task description information used to identify the nature of the task, usually existing in the form of a predefined label or code. Specifically, an assembly task refers to the operation of combining multiple parts into a complete product; a handling task refers to the operation of transferring materials or semi-finished products from one location to another; an inspection task refers to the operation of performing quality inspection or performance testing on products; a sorting task refers to the operation of classifying and screening items according to specific rules; a cleaning task refers to the operation of cleaning, dusting, or disinfecting equipment, sites, or products; and a patrol inspection task refers to the operation of performing routine inspections of equipment or the environment according to a predetermined route.

[0084] For example, suppose we receive the task description information of a raw task as: {"task_id": "T001", "task_type": "assembly", "description": "PCB board mounting"}. Parsing the "task_type" field from this description information, we obtain the task type identifier as "assembly", i.e., an assembly task. As another example, if the task description information of another task is: {"task_id": "T002", "task_type": "inspect", "description": "finished product quality inspection"}, parsing it reveals the task type identifier as "inspect", i.e., an inspection task. This embodiment of the application accurately identifies the task type of each raw task from the task description information, providing basic classification information for subsequent resource matching and scheduling decisions.

[0085] The resource requirements field is a data item in the task description information that describes the resource conditions required to execute the task. This field may contain information such as the required task execution module type, computational resources, and sensor data type and quantity. The task execution module type specifies the type of equipment required to execute the task, such as a robotic arm for mounting, a AGV for handling, or detection sensors. The computational resources specify the CPU time, memory capacity, or GPU computing power required to execute the task. The sensor data type and quantity specify the types of environmental perception data required to execute the task and the required quantity of each type of sensor.

[0086] Resource requirement fields can be included in the task description information or associated with a predefined resource requirement template by task type. When a resource requirement field is missing, default resource requirements can be read from a pre-configured resource requirement template based on the task type identifier. For example, for a task of type "assembly", the default resource requirements could be "one mounting robot arm, 10 seconds of CPU time, no sensor required".

[0087] For example, suppose the task description information of an original task includes a resource requirement field: "resource_requirement": {"module_type": "pick_and_place_robot", "cpu_time": "15s", "sensor_type": "vision", "sensor_count": 1}. Parsing this field reveals that the required task execution module type is "pickup robot arm", the required computational resources are "15 seconds of CPU time", the required sensor type is "vision sensor", and the required number of sensors is "1". If another task's task description information does not have an explicit resource requirement field, but the task type is identified as "transportation", then default resource requirements can be read from a pre-configured resource requirement template, such as "required module type is AGV, required computational resources are 5 seconds of CPU time". This application embodiment, by clearly defining the resource conditions required for the execution of each task, provides a basis for subsequent resource allocation decisions, avoiding resource conflicts and resource shortages.

[0088] Upstream and downstream task lists are data items in the task description information used to describe the execution order constraints between this task and other tasks. The upstream task list, also known as the predecessor task list, represents the set of tasks that need to be completed before the current task begins execution. The downstream task list, also known as the successor task list, represents the set of tasks that need to be completed before the current task can begin execution. By parsing the upstream and downstream task lists, the dependencies between tasks can be determined. These dependencies can include sequential dependencies, parallel execution dependencies, or mutually exclusive execution dependencies. Sequential dependencies mean that one task can only begin after another task has completed; for example, if task B depends on task A, then task A must be executed before task B. Parallel execution dependencies mean that two or more tasks can be executed simultaneously without affecting each other; for example, tasks C and D can be scheduled to be processed in parallel by different task execution modules within the same time period. Mutually exclusive execution dependencies mean that two or more tasks cannot be executed simultaneously, usually because they share the same resource or there is resource contention; for example, tasks E and F both require the same scarce equipment, so they cannot be executed simultaneously. This application embodiment accurately parses the execution constraint relationships between tasks, providing constraints for subsequent execution order adjustment decisions, ensuring that the scheduling strategy meets the dependency requirements between tasks, and avoiding execution failure due to incorrect order.

[0089] Priority attributes are data items in task description information used to describe the urgency and importance of task execution. Priority attributes include at least one of task urgency, task importance level, and time window constraints. Task urgency indicates the time-sensitive nature of the task, which can be expressed in a hierarchical manner, such as high urgency, medium urgency, and low urgency, or numerically, with higher values ​​indicating greater urgency. Task importance level indicates the task's importance in terms of business value; for example, high importance means the task is related to core business or customer satisfaction, while low importance means the task is auxiliary. Time window constraints specify the time range for task execution, including the earliest start time and the latest completion time. Tasks not completed by the latest completion time are considered overdue tasks, potentially affecting subsequent tasks or causing business losses. Priority attributes can be explicitly included in the task description information or implicitly derived from task type or business rules. When the priority field is missing, a default priority attribute can be read from a pre-configured priority template based on the task type identifier. For example, for a task of type "detection", the default priority attribute can be "medium urgency, high importance"; for a task of type "cleaning", the default priority attribute can be "low urgency, low importance". This application embodiment obtains the urgency, importance level, and time constraints of each task to provide a basis for subsequent priority adjustment strategies, ensuring that urgent and important tasks receive priority execution.

[0090] Considering that task type identifiers, resource requirements, dependencies, and priority attributes are each represented in different data formats after extraction, the task type identifier can be mapped to a numerical value, such as mapping assembly to 1, handling to 2, and detection to 3. Resource requirements can be converted into a multi-dimensional numerical vector; for example, the required module type can be mapped to a numerical value, the amount of computing resources can be directly represented by a numerical value, and the number and type of sensors can be converted into a numerical sequence. Dependencies can be converted into a dependency matrix and flattened into a one-dimensional vector, or converted into an adjacency matrix of a directed graph and flattened. The urgency, importance level, and time window constraints in the priority attributes are each converted into numerical values.

[0091] In this embodiment, all converted numerical vectors are concatenated according to a preset dimensional order to form a fixed-length one-dimensional vector, which is the task feature vector corresponding to the original task. By integrating the task attribute information scattered in multiple places into a unified numerical representation, the integrity of information in each dimension is maintained, and it is also convenient for reinforcement learning models to perform efficient numerical calculations and feature extraction, providing standardized input data for intelligent scheduling decisions.

[0092] In some embodiments, a real-time state vector for each task execution module is generated based on the running status information of multiple task execution modules, task execution environment information, and task progress information, including:

[0093] For each task execution module, the running status information of the task execution module is determined based on at least one of the following: idle status, busy status, fault status, location information, current progress percentage of the task being executed, and estimated remaining completion time.

[0094] The task execution environment information is determined based on the environmental status parameters of the task execution environment of the task execution module and the health status information of the task execution module.

[0095] The task progress information is determined based on at least one of the following: the elapsed time of the task currently being executed by the task execution module, the estimated remaining completion time, and the percentage of tasks that have been completed.

[0096] The running status information, task execution environment information, and task progress information are aligned to obtain the real-time status vector of the task execution module.

[0097] The terms "Idle State" and "Busy State" refer to the state where the task execution module is not currently executing any tasks and is in a standby, schedulable state. "Busy State" refers to the state where the task execution module is currently executing a task and cannot accept new tasks temporarily. "Fault State" refers to the state where the task execution module has experienced a hardware or software malfunction and cannot execute tasks normally. "Location Information" refers to the spatial coordinates of the task execution module in the current workspace. For mobile devices such as AGVs and inspection robots, the location information changes dynamically over time; for fixed devices such as robotic arms, the location information is usually fixed installation coordinates. "Current Task Progress Percentage" refers to the percentage of the task being executed by the task execution module that has been completed, for example, an assembly task that is 60% complete. "Estimated Remaining Completion Time" refers to the estimated time required to complete the current task based on the current execution speed and the remaining task quantity.

[0098] Operational status information can be determined by selecting at least one of the above parameters. For example, for a stationary robotic arm, its availability can be determined solely by its idle or busy status, without needing location information; however, for an AGV, both idle status and location information are required to determine whether it can undertake a transport task. By acquiring the basic operational status of each task execution module, fundamental data is provided for determining module availability, estimating task start times, and allocating resources.

[0099] Environmental status parameters refer to the physical environment in which the task execution module is located, including at least one of the following: temperature, humidity, light intensity, air quality, and path congestion. Excessive temperature may affect the stability of electronic equipment; abnormal humidity may affect the accuracy of optical sensors; insufficient light intensity may affect the quality of visual inspection tasks; poor air quality may impact production tasks with high cleanliness requirements; and path congestion may affect the travel speed and path selection of mobile devices such as AGVs.

[0100] Health status information refers to the equipment condition of the task execution module itself, including at least one of the following: motor temperature, vibration amplitude, runtime, and fault codes. Excessive motor temperature may indicate overload or poor heat dissipation; abnormal vibration amplitude may indicate wear or loosening of mechanical parts; runtime can be used to determine whether the equipment requires regular maintenance; and fault codes directly indicate the specific type of equipment malfunction. Environmental parameters and health status information reflect the environmental conditions and health level of the task execution module, and are of great significance for assessing the feasibility and risks of task execution.

[0101] For example, suppose the environmental parameters of a robotic arm for product placement are: ambient temperature 28 degrees Celsius, light intensity 300 lux, good air quality, and low path congestion. Its health status information is: motor temperature 65 degrees Celsius, vibration amplitude 0.05 mm, operating time 800 hours, and no fault codes. Based on this information, the robotic arm's task execution environment is determined to be normal, the equipment is in good health, and it can perform tasks normally. If another AGV's health status information shows a motor temperature of 85 degrees Celsius and a vibration amplitude of 0.3 mm, it is determined to be in a sub-healthy state and should be given priority for light-load tasks or have a maintenance reminder triggered.

[0102] By comprehensively sensing the environmental conditions and equipment health status of the task execution module, environmental risk and equipment reliability information can be provided for scheduling decisions, avoiding the assignment of tasks to modules in poor environments or sub-healthy states.

[0103] Elapsed time refers to the length of time elapsed from the start of the currently executing task to the present moment. Estimated remaining completion time refers to the predicted remaining time based on the current execution speed and remaining workload. Completed task percentage refers to the proportion of completed work to the total workload of the entire task, which can be expressed as the ratio of subtasks by number or time. Task progress information can be determined by selecting at least one of the above parameters. For example, for a task, if the total workload and time used are known, the completed task percentage can be calculated; if the task execution speed and remaining workload are known, the estimated remaining completion time can be estimated. Task progress information plays a crucial role in predicting when the task execution module will release resources and determining whether the task will time out.

[0104] Considering that operational status information, task execution environment information, and task progress information come from different data sources and have different sampling frequencies and physical dimensions—operational status information may be reported dozens of times per second by the module controller, environmental status parameters may be reported once per second by environmental sensors, and task progress information may only be updated during task execution—alignment processing refers to synchronizing these data from different sources and with different timestamps according to the same time base, so that they reflect the system state at the same moment.

[0105] The aligned data still needs to undergo standardization to convert values ​​with different dimensions into a uniform range. For example, a temperature of 28 degrees Celsius might need to be converted to a normalized value of 0.6, a progress percentage of 60% might be directly represented by 0.6, and location coordinates might need to be normalized based on the site area. Standardization ensures that all features have similar numerical ranges, facilitating stable numerical calculations for reinforcement learning models.

[0106] After alignment and standardization, all values ​​in the operational status information, task execution environment information, and task progress information are concatenated according to a preset dimensional order to form a fixed-length one-dimensional vector, which is the real-time status vector of the task execution module. The real-time status vector fully represents the module's comprehensive state at a given moment, including its own operational status, environmental conditions, and task progress. For example, suppose the operational status information of a mounting robot arm is as follows: busy state is denoted as 1, position coordinates are normalized to 0.5 and 0.6, progress percentage is 60% (0.6), and estimated remaining completion time is 2 minutes (120 seconds). The task execution environment information is as follows: temperature 28 degrees Celsius (normalized to 0.7), light intensity 300 lux (normalized to 0.5), motor temperature in healthy state (normalized to 0.6), and vibration amplitude (normalized to 0.1). The task progress information is as follows: elapsed time 3 minutes (180 seconds), and task completion percentage 30% (0.3). By concatenating these values ​​in sequence, the real-time state vector of the robotic arm is obtained as [1, 0.5, 0.6, 0.6, 120, 0.7, 0.5, 0.6, 0.1, 180, 0.3]. By unifying multi-source, heterogeneous, and multi-frequency state data into a standardized vector representation, the reinforcement learning model can fully perceive the real-time integrated state of each task execution module, providing accurate and consistent state input for scheduling decisions.

[0107] In some embodiments, the reinforcement learning model is trained with the optimization objective of maximizing cumulative reward, which is determined based on at least one of task completion efficiency, execution latency, resource utilization, and task conflict. The task feature vector and real-time state vector are input into the reinforcement learning model to obtain a target scheduling strategy output by the reinforcement learning model for scheduling multiple original tasks among multiple task execution modules, including:

[0108] The task feature vectors of all original tasks are concatenated with the real-time state vectors to obtain the fused features;

[0109] The fused features are input into a reinforcement learning model, and the model performs forward propagation on the fused features to obtain the action probability distribution of candidate actions; the candidate actions include at least one of adjusting the task execution order, changing the resource allocation scheme, and raising or lowering the task priority;

[0110] The target scheduling strategy is determined based on the action probability distribution; wherein, the target scheduling strategy includes at least one of the following: the execution order adjustment decision of each original task relative to other original tasks, the resource allocation decision of the target execution module to which each original task is assigned, the priority adjustment strategy of each original task relative to the current priority, and the network resource allocation strategy corresponding to each original task.

[0111] Each task feature vector is a fixed-length one-dimensional numerical vector representing the task's type, resource requirements, dependencies, and priority attributes. Each real-time state vector is a fixed-length one-dimensional numerical vector representing the module's running status, environmental information, and task progress. The task feature vectors of all original tasks are concatenated sequentially according to their task IDs to form a long task feature combination vector. Simultaneously, the real-time state vectors of all task execution modules are concatenated sequentially according to their module IDs to form a long state feature combination vector. The task feature combination vector and the state feature combination vector are then concatenated again to obtain the fused feature. The fused feature is a one-dimensional numerical vector containing complete attribute information for all currently pending tasks and complete real-time state information for all task execution modules. Integrating task and state information into a unified feature representation allows the reinforcement learning model to simultaneously perceive task requirements and system resource status, providing complete input information for comprehensive decision-making.

[0112] Forward propagation refers to the computational process of starting input data from the input layer, passing it layer by layer through weighted summation and activation function transformations of neurons, and finally producing an output result at the output layer. In reinforcement learning models, fused features are fed into the input layer, undergo feature extraction and transformation in the hidden layers, and finally produce an output at the output layer.

[0113] Candidate actions are decision options that the reinforcement learning model can choose in the current state. In this embodiment, candidate actions include at least one of the following types: Adjusting the task execution order refers to changing the execution order of various tasks, such as moving a task that was originally second to the first position. Changing the resource allocation scheme refers to changing the allocation relationship between tasks and execution modules, such as reassigning a task originally assigned to robotic arm A to robotic arm B. Raising or lowering task priority refers to increasing or decreasing the priority level of a task, such as raising a task at risk of timeout from normal priority to high priority, allowing it to preempt resources.

[0114] After the reinforcement learning model performs forward propagation calculations on the fused features, the output layer generates a numerical value for each candidate action, commonly referred to as the Q-score or action score. By applying a normalized exponential function, the Softmax function, to the Q-scores of all candidate actions, these Q-scores can be converted into a probability distribution. Each value in the action probability distribution represents the probability of the corresponding candidate action being selected, and the sum of the probabilities of all candidate actions is 1. A higher probability action indicates that the model believes taking that action in the current state will yield a higher cumulative reward.

[0115] Based on the action probability distribution, a greedy strategy can be used to select the action with the highest probability as the execution action, or a random sampling strategy can be used to randomly select actions according to the probability distribution. The combination of one or more selected actions constitutes the target scheduling strategy. The target scheduling strategy is a complete decision-making scheme for scheduling multiple original tasks among multiple task execution modules.

[0116] Execution order adjustment decisions refer to determining the execution order of each original task relative to other original tasks. This decision can be represented as a task sequence, such as task A being executed first, task B next, and task C last, or as an execution order number for each task, such as task A being number 1, task B being number 2, and task C being number 3. Execution order adjustment decisions must satisfy dependency constraints between tasks; that is, if task B depends on task A, then the order number of task A must be less than the order number of task B.

[0117] Resource allocation decision-making refers to determining which target execution module to assign each original task to. This decision can be represented as a mapping relationship between tasks and modules; for example, task A is assigned to robotic arm 1, task B to robotic arm 2, and task C to AGV1. Resource allocation decisions need to consider the resource requirements of each task, ensuring that the target execution module can meet the module type, computing resource requirements, and sensor requirements of the task.

[0118] Priority adjustment strategies refer to determining the direction of change in the priority of each original task relative to its current priority. This decision can be expressed as increasing priority, decreasing priority, or keeping priority unchanged. For example, a task facing the risk of timeout can be increased from normal to high priority, allowing it to be processed first in resource contention. For a task that is already delayed but does not affect subsequent tasks, its priority can be appropriately decreased, freeing up resources for more urgent tasks.

[0119] Network resource allocation strategy refers to allocating communication bandwidth and transmission priority for data transmission corresponding to each original task. During multi-task collaborative execution, the task execution module and the scheduling controller need to frequently exchange instructions and status data. The network resource allocation strategy allocates higher communication bandwidth and transmission priority to high-priority tasks based on task priority and real-time requirements, ensuring that their instructions are delivered with low latency. For example, an emergency stop instruction can be allocated the highest priority, allowing for rapid transmission even during network congestion; a status query request can be allocated a lower priority to avoid consuming critical bandwidth.

[0120] This embodiment transforms the probability distribution output by the reinforcement learning model into a specific, executable, multi-dimensional scheduling strategy, including decisions on execution order, resource allocation, priority adjustment, and network resource allocation. This provides a complete instruction basis for subsequent task execution control, enabling optimized scheduling of multiple tasks across multiple modules.

[0121] In some embodiments, multiple task execution modules are controlled to execute multiple original tasks according to a target scheduling strategy, and task execution feedback information for the original tasks is obtained, including:

[0122] Adjust decisions based on the execution order to determine the target task that needs to be executed in the original task.

[0123] Based on resource allocation decisions, the target execution module for executing the target task is determined from multiple task execution modules;

[0124] Based on the priority adjustment strategy, determine the priority level of the target task during the execution process;

[0125] Based on the autonomous localization and obstacle avoidance capabilities of the target execution module, path planning instructions are generated for the target execution module; the path planning instructions include the target location coordinates, the sequence of movement path points, and obstacle avoidance strategy parameters;

[0126] Based on the adaptive operation capability of the target execution module, operation parameter adjustment instructions are generated for the target execution module; the operation parameter adjustment instructions include execution speed, action sequence, grasping force or rotation angle;

[0127] The path planning instructions and operation parameter adjustment instructions are sent to the target execution module, and the instruction identifier and timestamp are recorded to control the target execution module to execute the target task according to the execution order and priority level.

[0128] The execution order adjustment decision specifies the execution order of each original task relative to other tasks. The target task is the original task that, according to this execution order adjustment decision, should begin execution at the current moment. The task that is first in the execution order and has not yet begun execution is the target task. If multiple tasks can be executed in parallel, the execution order adjustment decision will identify which tasks can begin simultaneously; these tasks are all target tasks.

[0129] Decisions regarding adjusting the execution order must consider the dependencies between tasks. If a task's prerequisite tasks have not yet been completed, it cannot be designated as the target task, even if it is ranked high in the execution order. Therefore, determining the target task requires considering both the actual execution progress of the tasks and the fulfillment of their dependencies.

[0130] A target execution module refers to a specific task execution module designated to perform a target task based on the resource allocation decision. The resource allocation decision needs to consider the resource requirements of each task, as well as the current status and available capabilities of each task execution module. For example, if a placement task requires a placement robot arm, the resource allocation decision will assign it to an idle or soon-to-be-idle placement robot arm. If a task requires multiple resources, it may require coordinating multiple execution modules to complete the task; in this case, the target execution module can be a main module or a collection of multiple modules.

[0131] Priority level refers to the final priority assigned to a target task during execution, used to determine the weight of that task relative to other tasks in resource contention and scheduling decisions. Priority levels can include multiple levels such as high priority, normal priority, and low priority, and can also be represented numerically, with higher numbers indicating higher priority. When multiple tasks compete for the same resource simultaneously, the higher-priority task will gain priority access to the resource. For example, in communication bandwidth allocation, control commands from high-priority tasks will be transmitted first; in resource allocation, high-priority tasks will have priority access to idle execution modules.

[0132] Priority adjustment strategies can include raising a low priority task to a high priority task, such as for a task facing timeout risks; lowering a high priority task to a normal priority task, such as for a task that has completed a critical phase; or maintaining the original priority. By adjusting the priority based on the scheduling strategy, an importance level is assigned to the target task during its execution, ensuring that it can be rationally scheduled according to priority during resource contention and emergency handling.

[0133] Autonomous localization capability refers to the ability of a task execution module to determine its own position coordinates within the workspace, typically achieved through odometers, vision sensors, or LiDAR. Obstacle avoidance capability refers to the ability of a task execution module to perceive obstacles in its surrounding environment and actively avoid them. Different task execution modules possess varying degrees of autonomous localization and obstacle avoidance capabilities. For example, AGVs usually have complete localization and obstacle avoidance systems, while fixed robotic arms do not require movement and therefore lack these capabilities.

[0134] For task execution modules that need to move, such as AGVs and inspection robots, path planning instructions need to be generated to guide them from their current position to a target position. Path planning instructions consist of three components: target position coordinates (the spatial coordinates of the target task execution location within the workspace, such as the precise location of a mounting machine); a movement path point sequence (the coordinate sequence of a series of intermediate points along the route from the current position to the target position, used to plan the specific travel route); and obstacle avoidance strategy parameters (behavioral parameters used to handle obstacles encountered during movement, including obstacle avoidance trigger distance, maximum detour offset, and waiting timeout). It should be noted that if the target task execution module is a fixed device that does not need to move, the generation of path planning instructions can be omitted.

[0135] Adaptive operation capability refers to the ability of a task execution module to automatically adjust its operating parameters according to task requirements and environmental changes. For example, a robotic arm can adjust its gripping force based on the weight of the material, its movement speed based on the target position, and its rotation angle based on process requirements. Different task execution modules have different adaptive operation capabilities, and different operating parameters need to be set to complete different tasks.

[0136] Operation parameter adjustment commands are used to configure the specific operation parameters of the target execution module when performing the target task. Execution speed refers to the movement speed of the module when performing actions, such as the moving speed of a robotic arm or the travel speed of an AGV. Higher speeds result in higher efficiency but may reduce accuracy. Action sequence refers to the order in which a series of sub-actions are performed to complete the target task, such as descending, then gripping, and then lifting. Gripping force refers to the clamping force applied to the material by the robotic arm's grippers or suction cups, which needs to be adjusted according to the weight and fragility of the material. Rotation angle refers to the angle that the robotic arm's end effector or rotating platform needs to rotate.

[0137] The specific details of the operation parameter adjustment instructions depend on the type and requirements of the target task. For assembly tasks, it may be necessary to specify the execution speed, action sequence, and gripping force; for handling tasks, it may be necessary to specify the movement speed and gripping force; for inspection tasks, it may be necessary to specify the rotation angle to photograph the object being measured from different angles.

[0138] In this embodiment, after the path planning instruction and the operation parameter adjustment instruction are generated, they are sent to the target execution module via a communication network. Upon receiving the instructions, the target execution module parses the instruction content and configures its own parameters and execution path according to the instruction requirements. For modules that need to move, they first move to the target location according to the path planning instruction, and then execute the specific task operation according to the operation parameter adjustment instruction. For fixed modules, they directly begin executing the task operation according to the operation parameter adjustment instruction.

[0139] Each issued instruction is assigned a unique instruction identifier, used to identify the instruction's execution status in subsequent feedback information. A timestamp records the moment the instruction was issued, used to calculate instruction transmission latency and execution response time. Recording this information helps trace the instruction execution process and perform performance analysis.

[0140] If multiple tasks are executed concurrently, the modules compete for resources and cooperate according to their respective priorities. Higher-priority tasks can preferentially occupy shared resources during execution, such as communication bandwidth, computing resources, or public channels.

[0141] This application embodiment transforms the scheduling strategy into executable control instructions and accurately sends them to the target execution module. At the same time, it records the instruction information for correlation and comparison with subsequent feedback information, forming a closed-loop task execution control process.

[0142] In some embodiments, the feedback information includes at least one of the following: the actual completion time of the original task, the real-time status changes of the task execution module during execution, the amount of resources actually used during the execution of the original task, and abnormal events during the execution of the original task.

[0143] The process of updating the reinforcement learning model in real time based on task execution feedback includes:

[0144] The task completion efficiency of the original task is determined based on the actual completion time, the execution delay is determined based on real-time status changes, the resource utilization rate is determined based on the actual amount of resources used, and the task conflict situation is determined based on abnormal events.

[0145] The task completion efficiency, execution delay, resource utilization, and task conflict are weighted and summed as reward components to obtain a comprehensive reward signal.

[0146] The comprehensive reward signal, the fusion features that generated the comprehensive reward signal, and the selected action are used as training samples and stored in the experience replay buffer.

[0147] Randomly sample batch training samples from the experience replay buffer, use the temporal difference error method to determine the loss function of the reinforcement learning model, and update the network weights of the reinforcement learning model through the backpropagation algorithm, so that the updated reinforcement learning model tends to obtain higher cumulative rewards when encountering the same input state.

[0148] The actual completion time is the real duration of the task from start to finish. Task completion efficiency measures the speed of task execution and is calculated by comparing the actual completion time with the expected completion time. The shorter the actual completion time, the higher the task completion efficiency, and the greater the corresponding reward value.

[0149] Real-time state changes record the state transitions of the task execution module at various points in time, such as the time from idle to busy, and from busy to idle. Execution latency refers to the waiting time between a task being assigned by a scheduling decision and its actual commencement of execution. It can also include additional waiting time caused by resource contention or blocking during task execution. The longer the execution latency, the greater the impact on subsequent tasks, and the smaller the corresponding reward value or the larger the penalty value.

[0150] The actual amount of resources used refers to the computing resources, communication bandwidth, storage space, etc., actually consumed during task execution, provided by task execution feedback information. Resource utilization can be calculated as the ratio of actual resource usage to the total system resources, or it can be evaluated by comparing it with the expected resource usage. The higher the resource utilization, the more fully the system resources are utilized, the less idle waste, and the greater the corresponding reward value. However, excessively high resource utilization may lead to increased resource contention, so a proper balance is needed.

[0151] Abnormal events refer to unexpected situations that occur during task execution, such as resource contention, path interference, and equipment failure. These include path conflicts caused by two AGVs arriving at the same intersection point simultaneously, resource conflicts caused by multiple tasks simultaneously requesting the same scarce equipment, task interruption due to a malfunction in the task execution module, and task execution time exceeding the time window constraint. Task conflict status is used to measure the effectiveness of scheduling strategies in avoiding conflicts; the fewer the number of conflicts and the lower the severity of the conflicts, the greater the corresponding reward value. Conflicts are typically penalized with negative rewards.

[0152] For example, suppose a placement task actually takes 8 minutes to complete and is expected to take 10 minutes. The task completion efficiency is high, corresponding to a positive reward. The task waited 1 minute from scheduling to actual execution, which is a low to medium latency, corresponding to a slight positive reward. The task actually used 12 seconds of CPU time, while the system's total CPU resources were 100 seconds, resulting in a resource utilization rate of 12%, which is normal and corresponds to a neutral reward. No abnormal events occurred during task execution, and task conflicts were handled well, corresponding to a positive reward.

[0153] In this embodiment, task completion efficiency, execution latency, resource utilization, and task conflict each reflect the merits of the scheduling strategy from different dimensions. These factors need to be combined into a comprehensive reward signal to guide the optimization direction of the reinforcement learning model. Weighted summation involves assigning a weight coefficient to each reward component, multiplying each component by its corresponding weight, and then summing the results to obtain the total score.

[0154] The magnitude of the weighting coefficient reflects the importance of that dimension in the optimization objective. For example, in scenarios prioritizing production efficiency, a higher weight can be assigned to task completion efficiency; in resource-constrained scenarios, a higher weight can be assigned to resource utilization; and in scenarios with high security requirements, a higher weight can be assigned to task conflict. The weighting coefficients can be configured and adjusted according to the specific application scenario.

[0155] Optionally, the formula for calculating the overall reward signal can be expressed as: Overall reward equals task completion efficiency multiplied by its weight, plus execution latency multiplied by its weight, plus resource utilization multiplied by its weight, plus task conflict situation multiplied by its weight. Execution latency and task conflict situation are typically negative or penalized, so that longer latency and more conflicts lead to a lower overall reward.

[0156] The overall reward signal is a scalar value; a positive number indicates that the current scheduling policy is performing well, while a negative number indicates poor performance. The training objective of a reinforcement learning model is to maximize the cumulative reward, that is, to obtain the highest possible total overall reward over long-term operation.

[0157] In this embodiment, the training samples of the reinforcement learning model may include the following three elements: state, action, and reward. State refers to the environmental state corresponding to the generation of the reward, describing all current task attributes and the real-time state of the system. Action refers to the scheduling strategy selected by the reinforcement learning model in this state, i.e., the action combination corresponding to the target scheduling strategy. Reward refers to the comprehensive reward signal obtained after executing the action, reflecting the effect of the action in this execution. State, action, and reward constitute a complete transition experience. By learning a large amount of such experience, the reinforcement learning model gradually understands what reward can be obtained by taking what action in what state, thereby continuously optimizing its decision-making strategy.

[0158] An experience replay buffer is a storage structure used to store historical training samples, typically implemented as a fixed-size first-in-first-out queue. When the buffer is not full, new samples are stored directly; when the buffer is full, new samples overwrite the oldest samples. Experience replay technology can break the temporal correlation between consecutive samples, improving training stability and sample utilization efficiency.

[0159] After each task is completed, one or more training samples are generated. Each task can generate one sample, or multiple samples can be generated at fixed time intervals or in different execution phases. After the samples are stored in the experience replay buffer, they can be used for random sampling in subsequent model update steps.

[0160] A batch of training samples is randomly and uniformly drawn from the experience replay buffer. Each sample contains a state, action, and reward. The number of samples in a batch is usually called the batch size, such as 32 or 64 samples at a time. Random sampling can break the temporal correlation between samples and avoid the model overfitting to recent execution experience.

[0161] For each training sample, the difference between the target Q-value and the current Q-value is calculated as the error. The target Q-value equals the current reward plus a discount factor multiplied by the maximum estimated Q-value for the next state. The current Q-value is the model's output Q-value for the action selected in the current state. The difference between the two is the temporal difference error.

[0162] In this embodiment, the error can be propagated layer by layer from the output layer to the input layer by calculating the gradient of the loss function with respect to the weights of each layer of the network, and then the weight values ​​are adjusted according to the direction of gradient descent. Each update uses a small batch of samples to calculate the gradient and update the weights once. After multiple iterations, the output of the model gradually converges to the target value.

[0163] When the updated reinforcement learning model encounters an input state that is the same as or similar to that in the training samples, its output scheduling policy will tend to favor actions that yield higher cumulative rewards. This is because the model learns the mapping relationship between states, actions, and rewards through training, and can select actions that have historically yielded high rewards.

[0164] For example, 32 training samples are randomly sampled from the experience replay buffer. For each sample, the error is calculated using the temporal difference error method. The errors of all samples are summed to calculate the loss function, assuming a loss function value of 0.25. The gradient of the loss function with respect to the model network weights is calculated using the backpropagation algorithm, and then the weights are updated with a learning rate of 0.001. After one update, the model's predictions for similar states are closer to the actual rewards. After repeated updates, when encountering the same fusion features, the model tends to output scheduling strategies that yield higher overall rewards, such as prioritizing high-urgency tasks and allocating resources reasonably to avoid conflicts. This utilizes stored historical experience to train the reinforcement learning model online, enabling the model to learn from actual execution results and continuously improve its scheduling decision-making capabilities, achieving an adaptive optimization effect that becomes smarter with use.

[0165] In some embodiments, determining the task to be rescheduled from a plurality of original tasks based on task execution feedback information includes:

[0166] Based on the actual completion time, original tasks whose current execution progress has not reached the preset progress threshold are marked as incomplete tasks;

[0167] Based on the abnormal events and actual completion times in the feedback information, determine whether the dependencies between the original tasks have been broken, and mark the tasks to be executed with broken dependencies as tasks to be rescheduled.

[0168] During task execution, newly inserted high-priority tasks are marked as tasks to be rescheduled;

[0169] Incomplete tasks and tasks awaiting rescheduling are merged to obtain tasks awaiting rescheduling.

[0170] The preset progress threshold can be a pre-defined criterion used to determine whether the task's execution progress meets expectations. This threshold can be a time ratio, such as considering progress as unsatisfactory when the actual completion time exceeds 1.2 times the expected completion time; or it can be an absolute time value, such as considering a task expected to complete in 5 minutes as unsatisfactory if it takes more than 6 minutes to complete. The preset progress threshold can be flexibly configured according to task type, priority, and business requirements.

[0171] In this embodiment, the task execution feedback information includes the actual completion time of each original task. For completed tasks, the actual completion time is a fixed value; for incomplete tasks, the actual completion time is a continuously increasing process value. The current execution progress refers to the ratio of the time already consumed by the task to the expected completion time. If this ratio exceeds a preset progress threshold, it indicates that the task execution progress is lagging, and the task is marked as an incomplete task. Marked incomplete tasks will be added to the set of tasks awaiting rescheduling in order to re-evaluate their execution plans and resource allocation.

[0172] For example, suppose there are three original tasks: Task 1 has an expected completion time of 10 minutes, and a preset progress threshold of 1.2 times, or 12 minutes. When Task 1 has been running for 11 minutes and is still not completed, the current progress is 11 divided by 10, which equals 1.1, less than the threshold of 1.2, so it is not marked yet. When Task 1 has been running for 13 minutes and is still not completed, the current progress is 13 divided by 10, which equals 1.3, exceeding the threshold of 1.2, so Task 1 is marked as an incomplete task. Task 2 has an expected completion time of 5 minutes and has actually been completed within 3 minutes, so it does not need to be marked. Task 3 has not yet started and is not an incomplete task.

[0173] Dependencies refer to execution constraints between tasks, such as task B needing task A to complete before it can begin. A breach of a dependency occurs when an abnormal event causes previously satisfied dependency conditions to no longer hold, or the expected dependency sequence to fail. Specifically, this includes the following situations: a preceding task fails to complete on time due to a fault, preventing subsequent tasks from starting as planned; a preceding task is interrupted or canceled due to a conflict, causing the dependency conditions of subsequent tasks to become invalid; the actual completion time of a preceding task far exceeds expectations, causing the time window for subsequent tasks to be missed. Abnormal events include path conflicts, resource contention, device failure, and communication interruptions. When these events occur, the affected execution module may be unable to complete its ongoing task on time, thus affecting other tasks that depend on it. The abnormal events in the task execution feedback information record the occurrence time and scope of these situations.

[0174] A pending task is a task that has not yet started execution and is in a waiting state. When a prerequisite task that a pending task depends on encounters an abnormal event that prevents it from completing on time, the dependency relationship of the pending task is broken. These pending tasks with broken dependencies cannot continue to wait as originally planned and need to be rescheduled; therefore, they are marked as tasks awaiting rescheduling.

[0175] Considering that new task execution requests may be submitted at any time during task execution by the system front-end or external devices, these new tasks may have high priority, such as critical order processing tasks urgently submitted by users, repair tasks after equipment failure, or time-sensitive tasks with time windows about to close. Newly inserted high-priority tasks usually need to start execution as soon as possible to meet their time window constraints or business requirements.

[0176] When a new task is received, the system first parses its priority attribute. If the task's priority is higher than that of any currently executing or waiting tasks, it is marked as a task awaiting rescheduling. These high-priority new tasks are then added to the set of tasks awaiting rescheduling and participate in the next round of rescheduling. This marking mechanism allows high-priority new tasks to receive a faster response time without having to wait for the current scheduling cycle to end.

[0177] The merging operation refers to combining the three types of tasks mentioned above to form a complete set of tasks to be rescheduled. If a task satisfies multiple flag conditions simultaneously, such as an incomplete task that is also a prerequisite task whose dependency has been broken, it only needs to appear once in the set and does not need to be added repeatedly. By aggregating tasks to be rescheduled from different trigger conditions to form a complete task set, a unified input is provided for the rescheduling process, ensuring that the system can fully respond to various changing needs during execution.

[0178] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0179] Based on the same inventive concept, this application also provides a multi-task adaptive scheduling device based on reinforcement learning. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the multi-task adaptive scheduling device based on reinforcement learning provided below can be found in the limitations of the multi-task adaptive scheduling method based on reinforcement learning described above, and will not be repeated here.

[0180] like Figure 3 As shown, this application embodiment provides a multi-task adaptive scheduling device 300 based on reinforcement learning, including:

[0181] The response module 302 is used to respond to task execution requests, obtain task description information of multiple original tasks, extract task attributes of each original task based on the task description information, and determine the task feature vector corresponding to each original task based on the task attributes.

[0182] The generation module 304 is used to generate a real-time status vector for each task execution module based on the running status information, task execution environment information, and task progress information of multiple task execution modules.

[0183] Input module 306 is used to input the task feature vector and the real-time state vector into the reinforcement learning model to obtain the target scheduling strategy output by the reinforcement learning model for scheduling multiple original tasks among multiple task execution modules.

[0184] The execution module 308 is used to control multiple task execution modules to execute multiple original tasks according to the target scheduling strategy, and to obtain task execution feedback information for the original tasks in real time.

[0185] The update module 310 is used to update the reinforcement learning model in real time based on the task execution feedback information, and to determine the tasks to be rescheduled from multiple original tasks based on the task execution feedback information.

[0186] The fusion module 314 is used to fuse the current real-time state vector of the task execution module and the task feature vector of the task to be rescheduled to obtain the updated fused features.

[0187] The scheduling module 316 is used to input the updated fused features into the updated reinforcement learning model to obtain the updated scheduling policy, and to reschedule the rescheduled tasks according to the updated scheduling policy.

[0188] Each module in the reinforcement learning-based multi-task adaptive scheduling device 300 described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0189] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores multi-task scheduling-related data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the reinforcement learning-based multi-task adaptive scheduling method described above.

[0190] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the steps in the reinforcement learning-based multi-task adaptive scheduling method described above. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen; the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0191] Those skilled in the art will understand that Figure 4 or Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0192] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0193] In some embodiments, such as Figure 6 The diagram shows the internal structure of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method embodiments.

[0194] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0195] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0196] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0197] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0198] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-task adaptive scheduling method based on reinforcement learning, characterized in that, include: In response to a task execution request, the task description information of multiple original tasks is obtained, the task attributes of each original task are extracted based on the task description information, and the task feature vector corresponding to each original task is determined based on the task attributes. Based on the running status information, task execution environment information, and task progress information of multiple task execution modules, a real-time status vector for each task execution module is generated. The task feature vector and the real-time state vector are input into the reinforcement learning model to obtain the target scheduling strategy output by the reinforcement learning model for scheduling the multiple original tasks among the multiple task execution modules. According to the target scheduling strategy, the multiple task execution modules are controlled to execute the multiple original tasks, and task execution feedback information for the original tasks is obtained in real time. The reinforcement learning model is updated in real time based on the task execution feedback information, and the task to be rescheduled is determined from the plurality of original tasks based on the task execution feedback information. The updated fused features are obtained by fusing the current real-time state vector of the task execution module and the task feature vector of the task to be rescheduled. The updated fused features are input into the updated reinforcement learning model to obtain the updated scheduling policy, and the task to be rescheduled is rescheduled according to the updated scheduling policy.

2. The method according to claim 1, characterized in that, The response task execution request obtains task description information for multiple original tasks, extracts task attributes for each original task based on the task description information, and determines a task feature vector corresponding to each original task based on the task attributes, including: For each of the original tasks, the task type field of the original task is parsed from the task description information to obtain the task type identifier of the original task; The resource requirement field of the original task is parsed from the task description information to obtain the resource requirement of each task in the original task; The task description information is parsed to obtain a list of upstream and downstream tasks that depend on the original task. Based on the list of upstream and downstream tasks, the dependency relationship between the current original task and other original tasks is determined. The priority attributes of the original task are parsed from the task description information; the priority attributes include task urgency, task importance level, and time window constraints. The task feature vector of the original task is obtained by concatenating the task type identifier, resource requirements, dependencies, and priority attributes.

3. The method according to claim 1, characterized in that, The step of generating a real-time state vector for each task execution module based on the running status information, task execution environment information, and task progress information of multiple task execution modules includes: For each of the task execution modules, the running status information of the task execution module is determined based on at least one of the following: idle status, busy status, fault status, location information, current progress percentage of the task being executed, and estimated remaining completion time. The task execution environment information is determined based on the environmental status parameters of the task execution environment of the task execution module and the health status information of the task execution module; The task progress information is determined based on at least one of the following: the elapsed time of the task currently being executed by the task execution module, the estimated remaining completion time, and the percentage of completed tasks. The running status information, the task execution environment information, and the task progress information are aligned to obtain the real-time status vector of the task execution module.

4. The method according to claim 1, characterized in that, The reinforcement learning model is trained with the optimization objective of maximizing cumulative reward, which is determined based on at least one of task completion efficiency, execution latency, resource utilization, and task conflict. The step of inputting the task feature vector and the real-time state vector into the reinforcement learning model to obtain the target scheduling strategy output by the reinforcement learning model for scheduling the multiple original tasks among the multiple task execution modules includes: The task feature vectors of all original tasks are concatenated with the real-time state vector to obtain the fused features; The fused features are input into the reinforcement learning model to perform forward propagation on the fused features through the reinforcement learning model to obtain the action probability distribution of candidate actions; the candidate actions include at least one of adjusting the task execution order, changing the resource allocation scheme, and raising or lowering the task priority; The target scheduling strategy is determined based on the action probability distribution; wherein the target scheduling strategy includes at least one of the following: an execution order adjustment decision for each original task relative to other original tasks, a resource allocation decision for the target execution module to which each original task is assigned, a priority adjustment strategy for the priority of each original task relative to the current priority, and a network resource allocation strategy corresponding to each original task.

5. The method according to claim 4, characterized in that, The step of controlling the multiple task execution modules to execute the multiple original tasks according to the target scheduling strategy and obtaining task execution feedback information for the original tasks includes: Based on the execution order, adjust the decision to determine the target task that needs to be executed in the original task. Based on the resource allocation decision, a target execution module for executing the target task is determined from multiple task execution modules; Based on the priority adjustment strategy, the priority level of the target task during execution is determined; Based on the autonomous localization and obstacle avoidance capabilities of the target execution module, a path planning instruction is generated for the target execution module; the path planning instruction includes the target location coordinates, the sequence of movement path points, and obstacle avoidance strategy parameters; Based on the adaptive operation capability of the target execution module, operation parameter adjustment instructions are generated for the target execution module; the operation parameter adjustment instructions include execution speed, action sequence, grasping force or rotation angle; The path planning instruction and the operation parameter adjustment instruction are sent to the target execution module, and the instruction identifier and timestamp are recorded to control the target execution module to execute the target task according to the execution order and the priority level.

6. The method according to claim 1, characterized in that, The feedback information includes at least one of the following: the actual completion time of the original task, the real-time status changes of the task execution module during execution, the actual amount of resources occupied during the execution of the original task, and abnormal events during the execution of the original task. The process of updating the reinforcement learning model in real time based on the task execution feedback information includes: The task completion efficiency of the original task is determined based on the actual completion time, the execution delay is determined based on the real-time status changes, the resource utilization rate is determined based on the actual amount of resources used, and the task conflict situation is determined based on the abnormal events. The task completion efficiency, execution delay, resource utilization, and task conflict status are weighted and summed as reward components to obtain a comprehensive reward signal. The comprehensive reward signal, the fusion features that generated the comprehensive reward signal, and the selected action are used as training samples and stored in the experience replay buffer. Randomly sample a batch of training samples from the experience replay buffer, determine the loss function of the reinforcement learning model using the temporal difference error method, and update the network weights of the reinforcement learning model through the backpropagation algorithm, so that the updated reinforcement learning model tends to obtain higher cumulative rewards when encountering the same input state.

7. The method according to claim 6, characterized in that, The step of determining the task to be rescheduled from the plurality of original tasks based on the task execution feedback information includes: Based on the actual completion time, original tasks whose current execution progress has not reached the preset progress threshold are marked as incomplete tasks; Based on the abnormal events in the feedback information and the actual completion time, determine whether the dependencies between the original tasks have been broken, and obtain the tasks to be executed whose dependencies have been broken. Retrieve higher-priority tasks that are newly inserted during task execution; The incomplete tasks, tasks with broken dependencies, and higher-priority tasks are merged to obtain the tasks to be rescheduled.

8. A multi-task adaptive scheduling device based on reinforcement learning, characterized in that, include: The response module is used to respond to task execution requests, obtain task description information of multiple original tasks, extract task attributes of each original task based on the task description information, and determine the task feature vector corresponding to each original task based on the task attributes. The generation module is used to generate a real-time state vector for each of the task execution modules based on the running status information, task execution environment information, and task progress information of multiple task execution modules. The input module is used to input the task feature vector and the real-time state vector into the reinforcement learning model to obtain the target scheduling strategy output by the reinforcement learning model for scheduling the multiple original tasks among the multiple task execution modules. An execution module is used to control the multiple task execution modules to execute the multiple original tasks according to the target scheduling strategy, and to obtain task execution feedback information for the original tasks in real time. The update module is used to update the reinforcement learning model in real time based on the task execution feedback information, and to determine the tasks to be rescheduled from the plurality of original tasks based on the task execution feedback information. The fusion module is used to fuse the current real-time state vector of the task execution module and the task feature vector of the task to be rescheduled to obtain the updated fused features. The scheduling module is used to input the updated fused features into the updated reinforcement learning model to obtain the updated scheduling policy, and to reschedule the task to be rescheduled according to the updated scheduling policy.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images