Method and electronic device for task scheduling in a work environment

Abstract

Embodiments of the present application provide a method for task scheduling in a job environment, comprising: obtaining candidate scheduling actions in a determined state of the job environment, each candidate scheduling action representing the allocation of a task type to a focal job device in the job environment or representing keeping the focal job device idle; pruning the candidate scheduling actions by means of a preset rule model to obtain safe scheduling actions, with the delivery deadline of each task as a hard constraint; and determining a target scheduling action from the safe scheduling actions by means of an agent based on a machine learning model. Through the rule pruning process, the hard constraint condition of "zero delay" can be separated from the optimization target of the agent, fundamentally ensuring that the scheduling scheme does not violate the delivery deadline, and improving the training efficiency of the agent and the interpretability of the decision result.

Description

Technical Field

[0001] This application relates to the field of intelligent scheduling, and in particular to a method for task scheduling in a working environment, a method for training an intelligent agent, an electronic device, a computer program product, and a computer-readable storage medium. Background Technology

[0002] With the increasing level of automation and the rapid development of digital supply chain technology, on-time delivery has become a core objective for ensuring production efficiency and service quality in high-end manufacturing, intelligent logistics, and other operational fields. However, with the intensification of resource conflicts, the diversification of task types, and the increasing complexity of constraints in the operational environment, task scheduling faces increasingly severe challenges.

[0003] Currently, some typical scheduling methods rely heavily on human experience, making them prone to getting trapped in local optima, and their decision-making quality significantly declines as the complexity of the work environment increases. In recent years, although some machine learning-based solutions have emerged, these methods generally suffer from poor interpretability and unstable decision-making processes. Particularly in zero-latency task scenarios with strict time constraints, the time credit allocation problem becomes prominent, making it difficult for the model to converge to a feasible solution that meets the requirements.

[0004] Therefore, existing scheduling schemes still have significant limitations when dealing with complex operating environments. Summary of the Invention

[0005] The purpose of the embodiments of this application is to provide a method for task scheduling in a working environment, a method for training an intelligent agent, an electronic device, a computer program product, and a computer-readable storage medium, so as to at least solve some of the problems in the prior art.

[0006] According to a first aspect of this application, a method for task scheduling in a work environment is provided, comprising: Obtain candidate scheduling actions under a defined state of the work environment. Each candidate scheduling action represents assigning a task type to the focus work equipment in the work environment or keeping the focus work equipment idle. Using a pre-defined rule model and with the delivery deadlines of each task as hard constraints, the candidate scheduling actions are pruned to obtain safe scheduling actions; and The target scheduling action is determined from the security scheduling actions by using an intelligent agent based on a machine learning model.

[0007] This application specifically includes the following technical concept: Based on a scheduling framework constructed by combining rule-based models and machine learning models, the hard constraint of "zero latency" can be decomposed from the agent's optimization objective and transformed into feasibility rule filtering before decision-making, fundamentally ensuring that the scheduling scheme does not violate delivery deadlines. Simultaneously, by leveraging the machine learning model's ability to understand and estimate multi-dimensional, complex, coupled state information, the scheduling action with the best long-term overall performance is identified from the safe action subspace, thereby achieving a globally better scheduling decision while satisfying the hard constraint. Overall, by pruning and compressing the original action space into a high-confidence feasible subspace, the time credit allocation problem caused by latency penalties in subsequent decisions is effectively alleviated, significantly improving training efficiency and the interpretability of decision results.

[0008] According to an optional embodiment of this application, the candidate scheduling actions are pruned using a preset rule model with the delivery deadline of each task as a hard constraint. This includes: evaluating each candidate scheduling action according to a predetermined rule; if it is determined that a candidate scheduling action will cause any task to be unable to be completed within the delivery deadline in the future, then the candidate scheduling action is removed.

[0009] According to an optional embodiment of this application, using a preset rule model and with the delivery deadline of each task as a hard constraint, the candidate scheduling actions are pruned to obtain safe scheduling actions. This includes: mapping the candidate scheduling actions to a task to be processed in a task queue; particularly using heuristic rules to perform trial scheduling on the remaining tasks in the task queue other than the task to be processed, in order to obtain trial scheduling results. The trial scheduling includes: assigning tasks to corresponding work equipment in the work environment and determining the work order of the tasks; based on the trial scheduling results, evaluating whether there is a situation where any task cannot be completed within the delivery deadline.

[0010] Therefore, by mapping candidate actions to specific tasks and using predetermined rules to test the remaining tasks, the feasibility of candidate actions can be quickly and conservatively evaluated with low computational cost, laying a reliable foundation for a zero-latency pruning process.

[0011] According to an optional embodiment of this application, the trial scheduling includes: determining the job order of each remaining task in the task queue based on its urgency, wherein tasks with higher urgency are given priority for execution; specifically, the urgency is measured by the minimum slack time of each remaining task, the minimum slack time being calculated based on the delivery deadline of each remaining task and its expected job duration.

[0012] Therefore, by prioritizing urgent tasks based on their urgency, the trial scheduling results more closely reflect actual scheduling needs, improving the accuracy of pruning decisions. Calculating the minimum relaxation time more realistically reflects the actual urgency of tasks, further enhancing the alignment between trial scheduling results and real-world scenarios.

[0013] According to an optional embodiment of this application, the trial scheduling includes: allocating corresponding work equipment to the remaining tasks in the task queue according to at least one of the following preference rules: selecting preferred work equipment based on the work efficiency of each work equipment; selecting preferred work equipment based on the work load of each work equipment; and determining preferred work equipment based on fixed rules set based on historical experience.

[0014] Therefore, by using the above-mentioned task allocation preference equipment, the trial scheduling results are closer to the actual scheduling needs, thus improving the accuracy of pruning judgments.

[0015] According to an optional embodiment of this application, the candidate scheduling actions are pruned using a preset rule model with the delivery deadline of each task as a hard constraint. This includes: using lookahead simulation to pre-simulate the future global operation of all tasks in the work environment when each candidate scheduling action is selected; if, in the lookahead simulation results for a candidate scheduling action, the expected completion time of any task is later than its delivery deadline, then the corresponding candidate scheduling action is removed.

[0016] Therefore, compared to traditional simplified rules, forward-looking simulation, by explicitly extrapolating future operational scenarios, can more reliably simulate real operational processes and improve the accuracy of future overdue risk assessments.

[0017] According to an optional embodiment of this application, the look-ahead simulation is a one-time forward simulation, which generates only one trial arrangement result of the remaining tasks in the task queue in terms of job order and job equipment allocation for each candidate scheduling action.

[0018] Therefore, by employing a one-time forward simulation instead of iterative search, the computational cost of look-ahead simulation is significantly reduced, allowing the pruning process to meet the scheduling time requirements. This conservative strategy may sacrifice a small number of marginal feasible solutions, but it effectively ensures zero-latency hard constraints.

[0019] According to an optional embodiment of this application, in the look-ahead simulation, the estimated completion time of each task is calculated by: determining the estimated start time of each task based on the earliest available time of the work equipment allocated to each task; determining the estimated work duration of each task on the corresponding work equipment; and determining the estimated completion time based on the estimated start time and the estimated work duration.

[0020] Therefore, by calculating the expected completion time, look-ahead simulation can more accurately assess whether each task meets the time constraints, thereby improving the reliability of pruning decisions.

[0021] According to an optional embodiment of this application, the estimated completion time is further determined based on the task switching type of successive tasks on the same work equipment and / or the earliest available time of shared auxiliary resources in the work environment. For example, the task switching type includes: no switching required, material change only, work accessory change only, and simultaneous material and work accessory change.

[0022] Therefore, by taking into account task switching types and the availability of shared auxiliary resources, the calculation of the expected completion time is closer to the actual operation scenario, thus improving the reliability of pruning judgment.

[0023] According to an optional embodiment of this application, the look-ahead simulation further includes: using a simulator of the work environment to determine the subsequent state of the work environment after the candidate scheduling action is executed; and starting from the subsequent state, performing a look-ahead simulation of the future global work situation of all tasks in the work environment.

[0024] Therefore, by using a simulator to determine the successor state and perform forward simulation, look-ahead simulation can improve the accuracy and reliability of pruning decisions based on deterministic state transition rules.

[0025] According to an optional embodiment of this application, the method further includes: using a preset rule model, pruning the candidate scheduling actions based on immediate feasibility constraints, wherein the immediate feasibility constraints only test the feasibility of the candidate scheduling actions themselves.

[0026] Therefore, the immediate feasibility constraint is used to check whether the candidate scheduling action itself has the conditions for execution at the current moment. Together with the time constraint, it achieves a more comprehensive assessment of the risk of the action and improves the completeness of the pruning mechanism.

[0027] According to an optional embodiment of this application, a candidate scheduling action is determined to not meet the immediate feasibility constraint and is eliminated under at least one of the following circumstances: the task type involved in the candidate scheduling action has no pending operation requirements; the task type involved in the candidate scheduling action does not meet the compatibility constraint with the focus operation equipment; or the shared operation accessories required by the task type involved in the candidate scheduling action are occupied.

[0028] Thus, by identifying immediate infeasibility scenarios such as no demand, incompatibility, and resource occupation, physical screening can significantly reduce the candidate action space with extremely low computational cost.

[0029] According to an optional embodiment of this application, candidate scheduling actions are pruned in the following order using a preset rule model: a first pruning process is performed on the candidate scheduling actions based on immediate feasibility constraints; a second pruning process is performed on the candidate scheduling actions retained after the first pruning process based on hard constraints constructed with delivery deadlines; and the candidate scheduling actions retained after the first and second pruning processes are determined as the safe scheduling actions.

[0030] Therefore, by performing two layers of filtering sequentially, obviously inactive actions can be quickly eliminated first, reducing the computational burden of subsequent time-constrained pruning.

[0031] According to an optional embodiment of this application, determining a target scheduling action from safety scheduling actions using an agent based on a machine learning model includes: using the agent to determine a safety scheduling action from the safety scheduling actions that is expected to obtain the highest long-term benefit level of the working environment, as the target scheduling action.

[0032] Therefore, by taking the highest expected long-term return as the decision objective, it is possible to pursue optimal global performance while ensuring zero-latency security.

[0033] According to an optional embodiment of this application, the method includes: simulating the execution of each safety scheduling action using a simulator of the work environment to determine the subsequent state and immediate reward of the work environment after the execution of each safety scheduling action; and making a decision based on the subsequent state and immediate reward corresponding to each safety scheduling action using an agent based on a machine learning model to determine a target scheduling action from the safety scheduling actions.

[0034] Therefore, by using a simulator to simulate the execution of safe actions to obtain subsequent states and immediate rewards, the task of calculating the transition from action to state can be separated from the agent's decision-making process. This eliminates the need for the agent to implicitly infer the impact of each action on the environment from a large number of random trials, thus making its learning task more focused, requiring only a focus on making decisions based on states and immediate rewards.

[0035] According to an optional embodiment of this application, the immediate reward is used to characterize the single-step benefit obtained by performing a safe scheduling action. In particular, the simulator determines the immediate reward based on the task's gross profit, inventory holding costs, and task switching costs.

[0036] Therefore, the calculation of single-step revenue has a more practical economic meaning, which aligns the optimization goal of the agent with the actual business goal and improves the rationality of scheduling decisions.

[0037] According to an optional embodiment of this application, the operating environment is an industrial automation environment, such that the subsequent state determined by the simulator can reflect the true state of the operating environment after the actual execution of the safety scheduling action within a predetermined accuracy range.

[0038] Therefore, by introducing a deterministic simulator into an industrial automation environment, the simulation results are highly consistent with the actual execution results, ensuring the accuracy of state transitions and laying a reliable foundation for subsequent scheduling decisions of the intelligent agent.

[0039] According to an optional embodiment of this application, the intelligent agent includes: a trained machine learning model for generating a state value corresponding to a subsequent state based on the subsequent state of the working environment; and a rule decision module for determining the long-term benefit level corresponding to each safety scheduling action based on the state value and immediate reward of its corresponding subsequent state, and selecting the safety scheduling action with the highest long-term benefit level as the target scheduling action.

[0040] Therefore, by dividing the agent into a machine learning model and a rule-based decision-making module, the final decision incorporates both the single-step reward calculated by the rule and the future reward estimate predicted by the model, effectively improving the interpretability and reliability of the decision. Simultaneously, the learning task of the machine learning model is limited to state value estimation, reducing the difficulty of model training.

[0041] According to an optional embodiment of this application, determining a target scheduling action from the security scheduling actions using an agent based on a machine learning model includes: obtaining state information of the subsequent state of the work environment after the execution of the security scheduling action; representing the state information of the subsequent state as heterogeneous feature units, the heterogeneous feature units including at least: task type feature units, time slot feature units, and resource feature units; and determining the target scheduling action from the security scheduling actions based on the heterogeneous feature units using an agent based on a machine learning model.

[0042] Therefore, by representing subsequent states as heterogeneous feature units with different semantic categories, the model can more effectively parse and fuse multi-source heterogeneous information. For example, time slot feature units and task type feature units are complementary representations of the same set of demand data in the work environment across different dimensions (especially the vertical dimension): the former reflects the resource competition situation within each time slot from the time dimension, while the latter reflects the long-term demand trends of each type of task from the task type dimension. This complementary representation enables the model to simultaneously perceive the resource competition situation in the time dimension and the demand evolution in the task type dimension, thereby more accurately identifying time bottlenecks and resource conflicts.

[0043] According to an optional embodiment of this application, each time slot feature unit is used to characterize the workload distribution of each task type within the same time interval.

[0044] Therefore, by using time slot feature units to characterize the workload distribution of task type dimension, the model can better perceive the resource competition situation within the corresponding time period.

[0045] According to an optional embodiment of this application, each task type feature unit is used to characterize the workload distribution of the same task type in different time intervals.

[0046] Therefore, by characterizing the workload distribution over time using task type feature units, the model can perceive the demand change trend of each task type on the time axis.

[0047] According to an optional embodiment of this application, the method includes: dividing a future time window into multiple time slots; determining the time slot to which each task should belong based on the delivery deadline of each task; injecting the workload of each task into the corresponding time slot according to the task type corresponding to each task to form a demand matrix; generating the task type feature unit based on the rows of the demand matrix, and / or generating the time slot feature unit based on the columns of the demand matrix.

[0048] As a result, the demand information is explicitly organized into structured data, which facilitates the extraction of time slot feature units and task type feature units.

[0049] According to an optional embodiment of this application, the method further includes: superimposing position codes on the time slot feature units, the position codes being used to characterize the sequential order of each time slot in a future time window; and / or embedding superimposed types on the heterogeneous feature units to distinguish different categories of feature units.

[0050] Therefore, position encoding and type embedding enhance the model's ability to perceive spatiotemporal dimensions and semantic categories, thereby improving the accuracy of state value estimation.

[0051] According to an optional embodiment of this application, the resource feature unit includes a work equipment feature unit and a global feature unit, wherein: each work equipment feature unit corresponds to a work equipment and includes the earliest available time of the work equipment, the task type of the current work, and the currently assembled work accessories; the global feature unit includes the availability of shared auxiliary resources, the earliest available time, and global time information.

[0052] This allows the model to focus on local resource states while also being aware of global constraints.

[0053] According to an optional embodiment of this application, the machine learning model included in the intelligent agent is a trained artificial neural network, comprising a bipartite graph attention network, a Transformer network, and a task head. For example, the bipartite graph attention network is used to perform local information exchange between task type feature units and work equipment feature units, and output updated task type feature units and work equipment feature units; the Transformer network is used to globally fuse the updated task type feature units and work equipment feature units with time slot feature units and global feature units to output a fused global state representation; the task head is used to output the state value of the subsequent state corresponding to the safety scheduling action based on the fused global state representation.

[0054] Therefore, by using a bipartite graph attention network, targeted local information exchange can be carried out between compatible task types and operating devices, providing more accurate structured feature inputs for the subsequent global fusion of Transformers and improving the accuracy of state value estimation.

[0055] According to an optional embodiment of this application, the method further includes: obtaining the compatibility relationships between all task types and operating equipment related to the operating environment; and constructing the graph structure of the bipartite graph attention network based on the compatibility relationships, wherein edges are established only between compatible task type nodes and operating equipment nodes.

[0056] Therefore, the graph structure only contains physically meaningful information transmission paths, avoiding interference from invalid information and improving the computational efficiency of the network.

[0057] According to an optional embodiment of this application, the bipartite graph attention network employs a multi-head attention mechanism.

[0058] Therefore, different attention heads can learn the dependency relationship between the work equipment and the task type from different perspectives, which enhances the model's ability to perceive multi-dimensional information.

[0059] According to an optional embodiment of this application, the method is an event-driven dynamic rescheduling method, which is triggered to execute when at least one of the following triggering conditions is met: the task queue changes, a predetermined event occurs in the resource in the working environment, or a preset rescheduling time interval is reached.

[0060] Therefore, through the dynamic rescheduling mechanism, scheduling decisions can respond to changes in the working environment in real time, effectively adapting to complex and ever-changing working scenarios such as dynamic task arrival and resource status fluctuations.

[0061] According to an optional embodiment of this application, the method further includes: iteratively executing the method for each decision time step to obtain the target scheduling action corresponding to each decision time step; mapping each target scheduling action to the allocation relationship between tasks in the task queue and the working equipment; and summarizing the allocation relationship to generate the overall work plan for the working environment.

[0062] Therefore, by iteratively executing the scheduling decisions at each decision time step and summarizing them to generate the overall work plan, a complete and executable scheduling scheme can be output, covering the allocation and execution arrangements of all tasks on the time axis.

[0063] According to an optional embodiment of this application, the agent employs a reinforcement learning framework, particularly a deep reinforcement learning framework.

[0064] Therefore, by adopting a reinforcement learning framework, an agent can autonomously learn target strategies from its experience of interacting with the environment without relying on manually labeled data, making it particularly suitable for complex and dynamically changing work environments.

[0065] According to a second aspect of this application, a method is provided for training an agent, particularly a machine learning model included in the agent, wherein the agent is applied to the method according to the first aspect of this application, comprising: acquiring experience sample data, the experience sample data including at least: a determined state of a work environment, a scheduling action corresponding to the determined state, a subsequent state of the work environment after executing the scheduling action, and an immediate reward; generating corresponding state values ​​based on the determined state and the subsequent state, respectively, using the agent; calculating a training error based on the immediate reward, the state value of the determined state, and the state value of the subsequent state; and updating the internal parameters of the agent based on the training error.

[0066] Therefore, the model only needs to use state estimation as the learning objective, and the agent's scheduling decisions are generated based on this. This significantly reduces the number of parameters and training complexity of the machine learning model, while improving convergence speed and generalization ability. Simultaneously, expanding the action space does not affect the model structure, enhancing the scalability of the scheduling scheme. Furthermore, this training method utilizes a reinforcement learning framework, eliminating the need for manually labeled data; it learns solely from experience samples generated through interaction with the work environment, making it particularly suitable for real-world work scenarios with complex constraints, frequent resource conflicts, and difficulties in obtaining the true value of optimal decisions.

[0067] According to a third aspect of this application, an electronic device is provided, the electronic device including a memory and a processor, the memory storing computer program instructions, which, when executed by the processor, enable the processor to perform the method according to the first and / or second aspects of this application.

[0068] According to a fourth aspect of this application, a computer program product is provided, comprising computer program instructions, wherein, when executed by a processor, the computer program instructions enable the processor to perform the method according to the first and / or second aspects of this application.

[0069] According to a fifth aspect of this application, a computer-readable storage medium is provided that stores computer program instructions, which, when executed by a processor, cause the processor to perform the method described according to the first and / or second aspects of this application. Attached Figure Description

[0070] The principles, features, and advantages of this application will be better understood below with reference to the accompanying drawings. The drawings include: Figure 1 A flowchart is shown of a method for task scheduling in a job environment according to an exemplary embodiment of this application; Figure 2 A schematic diagram illustrating a task scheduling framework for a working environment according to an exemplary embodiment of this application is shown. Figure 3 A schematic diagram illustrating the principle of a preset rule model according to an exemplary embodiment of this application is shown; Figure 4 A schematic diagram illustrating the principle of a preset rule model according to another exemplary embodiment of this application is shown; Figure 5 A schematic diagram of a task scheduling framework for a working environment according to another exemplary embodiment of this application is shown; Figure 6 A schematic diagram illustrating the principle of generating target scheduling actions with the aid of an intelligent agent according to an exemplary embodiment of this application is shown. Figure 7 A schematic diagram illustrating the principle of the generation process of a time slot feature unit according to an exemplary embodiment of this application is shown; Figure 8 A schematic diagram of the network structure of a machine learning model included in an agent according to an exemplary embodiment of this application is shown. Figure 9 A flowchart of a method for training an agent according to an exemplary embodiment of this application is shown; Figure 10 A block diagram of an electronic device according to an exemplary embodiment of this application is shown. Detailed Implementation

[0071] To make the technical problems to be solved, the technical solutions, and the beneficial technical effects of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and several exemplary embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit the scope of protection of this application.

[0072] Before proceeding with the specific description, it should be noted that the terms "first," "second," etc., used in this document are only for distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as implicitly indicating the number of technical features indicated.

[0073] Figure 1 A flowchart of a method for task scheduling in a job environment according to an exemplary embodiment of this application is shown. In this embodiment, the method includes steps 101 to 103.

[0074] In step 101, candidate scheduling actions are obtained under a defined state of the working environment. Each candidate scheduling action represents assigning a task type to the focus working device or keeping the focus working device idle.

[0075] The work environment refers to systems or places that require task scheduling and resource allocation, such as industrial manufacturing environments (covering production, processing, assembly, etc., such as injection molding workshops, machining workshops, and electronic assembly lines), logistics and warehousing environments (such as automated storage and retrieval systems, AGV handling systems, and sorting centers), as well as food processing / distribution sites, pharmaceutical production workshops, etc.

[0076] In one embodiment, the operating environment is an industrial automation environment, meaning that each operational step related to production, processing, and assembly follows controllable and predictable operating parameters, and the state of the operating environment can be simulated within a predetermined accuracy range using a simulator. For example, in an injection molding workshop, the processing time, mold change time, and crane occupancy time of each piece of equipment are all known parameters, and the state of the operating environment (such as equipment idle time, occupancy status of work parts, and crane availability time) can be accurately calculated or predicted, thereby providing a reliable simulation basis for scheduling decisions.

[0077] Operating equipment refers to physical resources that perform tasks in a working environment, such as injection molding machines, CNC machine tools, machining centers, robotic arms, autonomous mobile robots, conveyor belts, testing equipment, and packaging equipment. Typically, operating equipment can only perform one task at a time and corresponds to one task type (e.g., product or part type). It can only start the next task after completing the previous one, and its idle or busy status and earliest available time can be obtained in real time.

[0078] A task refers to a work unit that needs to be performed by work equipment in a work environment; it can be understood as a customer order, picking order, or production order. Each task corresponds to a task type (e.g., product type, part type, or process type) and includes a corresponding delivery deadline (i.e., the latest time the task must be completed) and the estimated operation time on each work equipment. Taking an injection molding workshop as an example, a task is a production order, specifying that the part type to be processed is A, the quantity is 300 pieces, the delivery deadline is 9:00 AM on July 3rd, and the relevant process requirements (e.g., only compatible with machine M1 or machine M2, requiring the use of mold m2). Tasks are usually added to the task queue (or order pool) in the order of their arrival time, or they can be sorted according to preset priority rules (e.g., delivery deadline or urgency level) and wait to be assigned to work equipment for execution.

[0079] The defined state of the working environment refers to the specific state of the working environment at a decision time step, which includes at least the resource state, task state, and global state.

[0080] Task status includes, for example, task type (such as product type), delivery deadline, estimated operation time on each work equipment, compatibility constraints with work equipment and other shared auxiliary resources, outsourcing flag, and task switching type flag.

[0081] Resource status includes at least the relevant status of the work equipment, such as the earliest available time, the currently performing task, the currently assembled work parts, and efficiency information. Work equipment refers to physical resources that perform tasks in the work environment, such as injection molding machines, CNC machine tools, machining centers, robotic arms, autonomous mobile robots, conveyor belts, testing equipment, and packaging equipment. Typically, work equipment can only perform one task at a time, corresponding to one task type (e.g., product or part type). It can only begin performing the next task after completing the previous one. Its idle / busy status and earliest available time can be obtained in real time.

[0082] In addition, resource status may include information on shared auxiliary resources. Shared auxiliary resources refer to auxiliary resources that, while not directly used to actively perform operational tasks, may be needed in conjunction with the operational equipment when performing tasks. These resources are typically limited in number and may have conflicting uses. The status of shared auxiliary resources may further include the availability, assembly status, and earliest available time of shared operational parts, as well as the occupancy status and earliest available time of shared swapping resources (such as cranes).

[0083] Shared work accessories refer to process equipment that needs to be used in sync with the work equipment during operation, including but not limited to replaceable parts such as molds, fixtures, and cutting tools. These auxiliary resources are continuously occupied during task execution, and their availability and assembly status directly affect whether the task can be carried out normally.

[0084] Shared replacement resources refer to auxiliary equipment temporarily needed during task switching or the preparation phase before a task begins. Examples include cranes, preheating devices, and cleaning equipment used for changing work parts. These auxiliary resources are generally only used during the parts replacement operation and are released after the replacement is completed, without participating in the continuous processing of the task.

[0085] Taking an injection molding workshop as an example, shared operating parts include molds mounted on machines, while shared changeover resources include the single, non-seizing overhead crane required for mold replacement. The crane can only serve one piece of equipment at a time; other equipment must wait for it to be released before it can be used. The availability of shared auxiliary resources directly affects the estimated start time of the task, especially when changing operating parts.

[0086] The global state includes information such as the position of the current decision time step in the global time and the position of the time slot window, providing a complete context for scheduling decisions.

[0087] Each candidate scheduling action refers to a potential decision to assign a task type to the focus job device at the current decision time step. There are typically multiple candidate scheduling actions at each decision time step. The size of the action space is determined by the number of potentially executable task types in the job environment, and is independent of the number of specific task instances in the task queue. Candidate scheduling actions also include idle actions, where the current focus job device is temporarily not assigned any tasks and remains idle. The reason for introducing idle actions is that, in some situations (such as waiting for a more urgent task to arrive, or waiting for other devices to release resources for optimized allocation), keeping the focus job device idle may be more advantageous than immediately scheduling tasks.

[0088] For example, if there are four task types available for work in the work environment: P1, P2, P3, and P4, and the focus work device is M1 (i.e., in the current state, work device M1 is expected to be idle the earliest), then candidate scheduling actions may include: assigning task type P1 to M1, assigning task type P2 to M1, assigning task type P3 to M1, assigning task type P4 to M1, and may also include an idle action (i.e., not assigning any task type, keeping M1 idle). Candidate scheduling actions may also be a subset of the above set (e.g., only considering task types with job requirements in the current task queue), rather than necessarily including all task types.

[0089] A focal job is the earliest available job selected from all available jobs at the current decision time step. Specifically, it is determined by comparing the earliest available times of each job (i.e., the estimated completion time of the last task already scheduled on that job). If multiple jobs have the same earliest available time, one is selected as the focal job according to preset priority rules (e.g., equipment number order, work efficiency, historical workload, etc.). The focal job is the target of the current scheduling decision, and all allocation relationships in candidate scheduling actions are directed towards this job.

[0090] In one embodiment, once a candidate scheduling action specifies a task type (e.g., product type), it can be automatically mapped to the next pending task instance of that task type in the task queue according to a preset order rule (e.g., FIFO, delivery deadline priority, etc.). In this way, the scheduling decision only needs to focus on the allocation at the task type level, and which specific task instance to execute is automatically determined by the queue rules, thereby maintaining the compactness and stability of the action space.

[0091] In step 102, using a preset rule model and with the delivery deadline of each task as a hard constraint, candidate scheduling actions are pruned to obtain safe scheduling actions.

[0092] Preset rule models refer to online solvers based on deterministic rules, which are used to quickly determine the feasibility of candidate scheduling actions without involving model training or parameter learning.

[0093] Pruning, for example, refers to reducing the number of actions from multiple candidate scheduling actions according to preset criteria. Actions that do not meet the preset conditions are eliminated, and the remaining actions are considered safe scheduling actions, serving as the basis for subsequent scheduling decisions. Through pruning, the original action space is compressed into a high-confidence feasible subspace, providing a safe and reliable set of actions for subsequent decisions.

[0094] Hard constraints refer to conditions that are absolutely not allowed to be violated during the scheduling decision-making process. Once violated, the corresponding candidate scheduling action will be directly eliminated, and there is no possibility of tolerating the violation by adjusting weights, imposing penalties, or making compromises.

[0095] In contrast to hard constraints, soft constraints typically serve only as guiding conditions to direct the solution process, without imposing mandatory restrictions on the final feasible solution. For example, soft constraints can incorporate delivery deadlines into the objective function in the form of cost terms to guide scheduling decisions to prioritize tasks with tighter delivery deadlines, but do not require each task to be completed within its delivery deadline.

[0096] In one embodiment, in step 102 above, each candidate scheduling action can be evaluated according to predetermined rules. If it is determined that a candidate scheduling action will cause any task to fail to be completed within the delivery deadline in the future, then the candidate scheduling action is eliminated. The predetermined rules may include, for example, heuristic rules, metaheuristic rules (such as genetic algorithms, simulated annealing, etc.), planning-based solution rules, and fixed rules set based on experience or process knowledge.

[0097] In one embodiment, lookahead simulation can be used to pre-simulate the future global operational status of all tasks in the work environment under the selected candidate scheduling actions. Specifically, the estimated start and finish times of all tasks in the work environment after the execution of each candidate scheduling action can be simulated and calculated. Then, the estimated finish time of each task is compared with its respective delivery deadline. If the estimated finish time of any task in the lookahead simulation results is later than its delivery deadline, the candidate scheduling action is eliminated. This is particularly advantageous for scenarios where the work environment parameters are relatively certain and controllable, enabling a relatively accurate estimation of future operational status.

[0098] In another embodiment, instead of calculating the estimated completion time of each task through look-ahead simulation, the risk of task delay can be assessed, for example, using a probability-based or probability distribution-based approach. For instance, by utilizing the statistical distribution of task operation time and switchover time, the probability of each task being completed on time can be estimated. If the completion probability of any task is lower than a preset threshold, that candidate scheduling action is eliminated. This is particularly advantageous in scenarios where operational environment parameters are uncertain, enabling scheduling decisions with manageable risk even when deterministic information is insufficient.

[0099] The specific technical process of pruning will be discussed in the following text. Figure 3 and Figure 4 To elaborate further.

[0100] In step 103, a target scheduling action is determined from the security scheduling actions using an agent based on a machine learning model.

[0101] A machine learning model-based agent refers to a decision-making entity constructed using a machine learning framework, which may include at least a machine learning model (such as a trained artificial neural network, decision tree, random forest, or gradient boosting model). Additionally, the agent may optionally include a rule-based decision-making module (such as a comparator, maximum value selector, or other mathematical operation module) that works in conjunction with the machine learning model to perform decision-making tasks, as well as a preprocessing module for data preprocessing, feature cleaning, and feature extraction.

[0102] In one embodiment, the agent can be directly implemented as a trained machine learning model, such as a trained artificial neural network. This artificial neural network can be, for example, a deep neural network, including a multilayer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN), and / or graph neural network (GNN). In this implementation, the machine learning model, for example, directly receives all safety scheduling actions and the defined state information of the working environment as input, and directly outputs the target scheduling action in an end-to-end manner.

[0103] In one embodiment, the agent may employ a deep reinforcement learning framework, such as a deep Q-network (standard DQN) or a Q-learning algorithm. In this implementation, for example, the current working environment state may be input into a trained machine learning model to output value estimates (such as Q-values) for all actions, and based on these estimates, a target scheduling action may be selected.

[0104] In one embodiment, the agent can also employ a deep reinforcement learning framework modified from the standard DQN. Unlike the standard DQN, this implementation can, for example, accurately estimate the environmental state after an action is performed. Therefore, the machine learning model does not need to estimate the Q-values ​​of all actions, but only needs to output the state value. Its output dimension is fixed at 1 and is independent of the size of the action space. This framework can significantly improve the model's convergence speed and training efficiency.

[0105] In one embodiment, the agent can also work in conjunction with a deterministic simulator. For example, a deterministic simulator can be used to explicitly generate the successor state after the execution of candidate scheduling action hypotheses, thereby decoupling the computational tasks related to the dynamic environment from the network. Thus, the agent does not need to rely entirely on the network's random trial and error to implicitly learn the environment state transitions, significantly reducing the sample complexity and training instability of reinforcement learning. The specific structure and interaction mechanism of the deterministic simulator and agent will be discussed below. Figure 5 Further explanation.

[0106] In one embodiment, the working environment is an injection molding workshop, the focal operation equipment is one of multiple injection molding machines present in the working environment, and the task type is the type of product to be injection molded corresponding to the production order. The working environment also includes an overhead crane for performing mold lifting operations when the injection molding machine needs to change molds. The number of overhead cranes is limited (e.g., one unit), and when multiple injection molding machines simultaneously require mold changing, they must queue up and wait for the overhead crane.

[0107] In one embodiment, when pruning is performed in step 102 with delivery deadline as a hard constraint and / or when decision-making is made by an agent in step 103, the product switching type corresponding to two production orders processed consecutively on the same injection molding machine is also considered. This product switching type, for example, determines whether a mold needs to be changed for the injection molding machine and whether a crane needs to be occupied. Accordingly, mold changes and crane waiting both incur additional time and economic costs. These additional costs will affect the determination of whether the corresponding candidate scheduling action will cause the task to be overdue, and may also affect the calculation of the profit or cost corresponding to the candidate scheduling action.

[0108] For example, when a mold change is needed between two consecutive orders being processed on the same injection molding machine, a crane is required. This crane can only perform a mold change operation for one injection molding machine at a time. Therefore, if a candidate scheduling action involves a production order that requires a different mold from a currently executing production order on the same injection molding machine, the expected start time for that candidate scheduling action may be delayed until the crane becomes available before the mold change operation can be performed. This will be further described below with specific examples.

[0109] For example, when making decisions, the immediate reward considered by the agent also depends on the switching costs associated with the product switching type. For instance, the switching cost is lowest for not changing the mold or material, followed by changing only the material, then the mold, and finally, the combination of both. This reflects the impact of different switching types on economic benefits during the decision-making process. This will be further described below with specific examples.

[0110] A target scheduling action refers to a scheduling action ultimately selected from a set of safety scheduling actions, which can be applied to the actual job scheduling in the work environment. For example, if two safety scheduling actions are obtained after pruning, namely assigning task type P1 to work equipment M1 and assigning task type P2 to work equipment M1, then the target scheduling action is an action selected from these two, such as assigning task type P1 to work equipment M1. When the actual time reaches this decision time step (e.g., 9:00-10:00), the work environment will execute the actual task allocation and job execution according to the target scheduling action.

[0111] Depending on the specific type and requirements of the work environment, the selection criteria for target scheduling actions may differ. For example, the objective may be to maximize revenue, minimize total completion time, maximize equipment utilization, minimize task delays, or balance a comprehensive indicator of multiple optimization objectives. This application is not limited to a single selection criterion; different optimization objectives can be adopted for different work environments. In one embodiment, maximizing long-term revenue can be used as the selection criterion, where long-term revenue is understood as the total profit obtained by combining current and future revenue in the work environment.

[0112] In one embodiment, the task scheduling method described above is an event-driven dynamic rescheduling method. For example, when a preset trigger condition is detected, the method is executed starting from the current decision time step. The preset trigger condition includes at least one of the following: - Changes have occurred in the existing task queue. For example, a new task is released and added to the task queue, or an existing pending task is removed (e.g., task cancellation, change of delivery deadline). - A predetermined event occurs in the resources of the work environment. For example, a sudden failure, recovery from failure, or change in quantity of work equipment or shared auxiliary resources (such as cranes, molds); - Rescheduling is triggered at a preset time interval. For example, rescheduling is triggered at fixed time intervals (such as 1 hour, 1 day, 1 week) to adapt to dynamic changes in the working environment and maintain the timeliness of the scheduling strategy.

[0113] In one embodiment, steps 101 to 103 are executed iteratively. Specifically, the scheduling decisions at each decision time step are executed sequentially to obtain the target scheduling action sequence. For example, the scheduling method is executed at a decision time step t1 to determine a target scheduling action. Use this target to schedule actions. After updating the state of the working environment, proceed to the next decision time step t2, and execute the above scheduling method again to determine the target scheduling action. And so on. The decision at the next decision time step is based on the state of the working environment after the execution of the previous target scheduling action.

[0114] In one embodiment, the target scheduling actions derived at each decision time step can be sequentially mapped to pending tasks in the task queue, thereby obtaining the allocation relationship between pending tasks and work equipment at each decision time step. The results of all decision time steps are then summarized to generate the overall work plan. This iterative process continues until a termination condition is reached, such as all or more than a predetermined number of pending tasks in the task queue having been allocated.

[0115] In one embodiment, the overall work plan may include the following information: the allocation relationship between each task and the work equipment, the work sequence of each task, the estimated start time and estimated completion time of each task, whether a task is marked as outsourced, the estimated work duration of the task, required work accessories (such as mold numbers), crane occupancy status, inventory status changes, product type and quantity, etc. This information can be used to guide actual production execution, progress tracking, and resource scheduling optimization.

[0116] For example, an example of a total work schedule is shown below: 9:00-9:30: Task Job1 (Product type P1, quantity 300) is assigned to work equipment M1, using mold m1, expected to be completed by 9:30, inventory +300; 9:30-10:15: Task Job2 (Product type P2, quantity 200) is assigned to work equipment M2, using mold m2. Mold change is required, which will occupy the crane from 9:30 to 9:45. It is expected to be completed at 10:15. Inventory +200. 10:15-11:00: Task Job3 (Product type P1, quantity 250) is assigned to work equipment M1, using mold m1 (same mold as Job1, no mold change required), expected to be completed at 11:00, inventory +250; 11:00-11:40: Task Job4 (Product type P3, quantity 180) is assigned to work equipment M3, using mold m3. Mold change is required, which will occupy the crane from 11:00 to 11:15. It is expected to be completed at 11:40. Inventory +180.

[0117] It should be noted that the specific values ​​and information items such as quantity, time, and inventory status involved in the above overall work plan are exemplary. In practical applications, specific information content can be added, reduced, or adjusted according to the type of work environment and actual needs. In addition, the presentation format of the above overall work plan is not limited to structured text data; it can also use other meaningful forms of expression such as Excel forms and graphs.

[0118] Figure 2 A schematic diagram of a task scheduling framework for a working environment according to an exemplary embodiment of this application is shown.

[0119] exist Figure 2 The basic structure of a task scheduling framework for a work environment is shown, which includes, for example, at least a preset rule model 31 and a machine learning model-based agent 32.

[0120] The input to the preset rule model 31 is candidate scheduling actions 21. After pruning, the preset rule model 31 outputs a safe scheduling action 23. During the pruning process, the preset rule model 31 predicts the impact of each candidate scheduling action 21 on the future global operation, uses the delivery deadline of each task as a hard constraint, and eliminates candidate scheduling actions that may cause the task to be overdue, retaining the safe scheduling action 23. Subsequently, the safe scheduling action 23 is provided to the intelligent agent 32 based on the machine learning model, and the intelligent agent 32 determines the target scheduling action 24 from it.

[0121] like Figure 2 As shown in the example, the operating environment is first determined to be in a certain state. The candidate scheduling action 21 below, whose corresponding original action space is, for example, Using the preset rule model 31, the original action space Compressed into a filtered safe action subspace .

[0122] Unlike constructing a late payment penalty function or introducing related cost terms, this framework does not use penalty terms to allow the agent to learn slowly or merely guide the solver's search direction. Instead, it employs an active pruning strategy to directly eliminate all high-risk actions that could lead to future task delays. By embedding strict no-delay requirements directly into the pruning process, the zero-latency goal is transformed from a "soft constraint" into an inviolable "hard constraint."

[0123] By combining the pre-defined rule model 31 with the decision model 32, the hard constraint of "zero latency" can be extracted from the optimization objective of the decision model 32 before decision-making and transformed into a feasibility filter before decision-making, fundamentally ensuring that the scheduling scheme does not violate the delivery deadline. In addition, by compressing the original action space into a high-confidence feasible subspace, the time credit allocation problem caused by delay penalties in subsequent decision-making models is effectively alleviated, significantly improving the training efficiency of the decision model.

[0124] Subsequently, the pruned safety scheduling action 23 is passed to the machine learning model-based agent 32, which further selects the target scheduling action 24 from it. Figure 2 As shown, after the decision-making process of agent 32, only the safe action subspace is ultimately retained. One of the actions is used as the target scheduling action. .

[0125] In one embodiment, with the help of agent 32, for each security scheduling action... Determine the long-term benefit level of the working environment respectively. The target scheduling action is determined by identifying the safe scheduling action that corresponds to the highest long-term benefit level. Here, "long-term benefit level" not only considers the single-step benefit of the current decision time step, but also reflects a comprehensive measure of the potential chain reaction on the current operation and the future global operation if a certain scheduling action is selected.

[0126] In an industrial manufacturing setting, this long-term return level can be understood, for example, as the total manufacturing profit level. In other operating environments, the long-term return level can also be defined according to different optimization objectives.

[0127] Figure 3 A schematic diagram illustrating the principle of a preset rule model according to an exemplary embodiment of this application is shown.

[0128] In this embodiment, the preset rule model 31 includes a prediction screening engine 312. This prediction screening engine 312 uses look-ahead simulation to pre-simulate the future global operational status of all tasks in the operational environment after each candidate scheduling action 21 is selected. If, in the look-ahead simulation results for a candidate scheduling action, the expected completion time of any task is later than the delivery deadline, then that candidate scheduling action is eliminated.

[0129] like Figure 3 As exemplarily shown, the original action space of candidate scheduling action 21 It includes 6 actions, which are denoted as follows: to The predictive screening engine 312 sequentially assesses the overdue risk of each action to determine whether pruning is necessary. (Based on action...) For example, if the execution of this action is predicted through look-ahead simulation... If this action will cause at least one task in the future work environment to be overdue, then this action will be performed. From the original action space Remove from the middle. Similarly, for the original action space... The remaining candidate scheduling actions Forward-looking simulation and corresponding pruning were performed respectively.

[0130] like Figure 3 As shown, the prediction screening engine 312 includes, for example, a task mapping stage 410, a trial arrangement stage 420, a state transition stage 430, a simulation stage 440, and a rejection stage 450.

[0131] In the task mapping stage 410, each candidate scheduling action is mapped to a specific task to be processed in the task queue.

[0132] For example, a candidate scheduling action To "assign product type P1 to the focus job device M1", the next pending task (e.g., job1) corresponding to product type P1 is retrieved from the task queue and assigned to job device M1 for processing. After retrieving task job1, the remaining tasks job2...jobn in the task queue are obtained. Through this mapping process, each candidate scheduling action can be transformed into a specific task allocation instance, providing a clear set of tasks for subsequent trial scheduling and simulation.

[0133] In the trial scheduling stage 420, the remaining tasks in the task queue are trial-scheduled according to preset rules to determine the trial scheduling result. "Trial scheduling" refers, for example, assigning a group of tasks to be processed to corresponding work equipment according to preset rules and determining their work order; that is, the trial scheduling result includes both the work equipment allocation relationship and the work order determination. Specifically, the trial scheduling stage 420 can be further divided into a work order determination stage 421 and a work equipment allocation stage 422. For example, heuristic rules can be used to determine the trial scheduling result.

[0134] In one embodiment, heuristic rules can be used to determine the results of the trial arrangement.

[0135] In one embodiment, look-ahead simulation is a one-time forward simulation, that is, for each candidate scheduling action, only a unique trial scheduling result is generated for the remaining tasks in the task queue in terms of job order and job equipment allocation.

[0136] For example, a conservative strategy can be used to determine the unique trial scheduling result: prioritizing tasks with higher urgency and assigning each task to its preferred resources. This conservative strategy may lead to "over-pruning" of feasible solutions, that is, by employing strict verification criteria, actively sacrificing some complex or marginal feasible solutions to ensure that no actions with a risk of being overdue are approved. Although this may result in a few marginal feasible solutions being mistakenly pruned, under the hard constraint of zero-latency scheduling, safety is prioritized over optimality, thereby maximizing the guarantee of zero-latency scheduling.

[0137] For example, a candidate scheduling action a_1 corresponds to assigning task type P1 to job device M1, which is mapped to the specific task job1. The remaining tasks in the task queue are, for example, job2 to job5. The unique trial scheduling result determined for this purpose includes, for example, the job order as job2-job4-job5-job3, and the task allocation relationship to job device as job2-M2, job4-M2, job5-M3, job3-M1.

[0138] In the task order determination step 421, the task order can be determined based on the urgency of each remaining task in the task queue, with tasks with higher urgency being prioritized for execution.

[0139] In one embodiment, the urgency of a task is measured by its minimum slack time (MST). Specifically, the minimum slack time for each task is calculated based on its delivery deadline and its estimated operating time on the work equipment. MST, for example, reflects the remaining time for a given task at the current decision time step, after deducting necessary processing time (e.g., processing time, and optional changeover and waiting buffer times). A smaller MST indicates a more urgent task that needs to be scheduled as early as possible to avoid delays. Therefore, tasks with smaller MSTs are prioritized and processed in ascending order of minimum slack time to avoid delays. Tasks with larger MSTs are relatively more lenient and can be scheduled later.

[0140] For example, the minimum relaxation time can be calculated using the following formula: k represents the sequence number of the remaining tasks in the task queue; Let be the minimum relaxation time for task k; The delivery deadline for task k; The estimated operation time for task k on the corresponding work equipment. This estimated operation time can be determined, for example, based on the quantity of product types contained in each task and the unit processing time of that type on the corresponding equipment. If a preferred work equipment has been determined for task k according to preset rules, the estimated operation time can be further refined into the estimated operation time of the task on its preferred work equipment (i.e., taking into account the differences in work efficiency between different work equipment), thereby making the calculation of the minimum slack time more in line with the actual scheduling scenario.

[0141] In the equipment allocation stage 422, the corresponding equipment is determined for each remaining task in the task queue.

[0142] In one embodiment, the remaining tasks in the task queue may be assigned appropriate job equipment according to at least one of the following preference rules: - Select preferred work equipment based on the work efficiency of each work equipment. For example, for a certain task, compare its expected work time on each work equipment and select the work equipment with the shortest expected work time as its preferred work equipment; - Select preferred task equipment based on the workload of each task device. For example, the device with the lightest load and earliest availability can be selected as the preferred task equipment based on the current task queue length or the earliest available time. This rule helps balance the workload among devices, avoiding overload of some devices while others are idle, thereby improving the overall resource utilization of the system; - Preferred work equipment is determined by fixed rules based on historical experience. For example, default work equipment can be pre-assigned for each task type based on process requirements, equipment characteristics, or long-term production experience.

[0143] In addition, multiple rules can be combined to comprehensively determine resource allocation relationships. For example, when the efficiency of different operating equipment is similar, priority can be given to equipment with lighter loads to take into account the needs of multi-objective optimization.

[0144] It should be noted that the preferred job equipment allocation relationship determined for each remaining task according to the above preference rules is not the final scheduling execution order, but a conservative feasibility proxy used to quickly generate a trial scheduling scheme with low computational cost, in order to test whether the candidate scheduling actions under the current decision step may cause future task delays. This trial scheduling result is mainly used in the look-ahead simulation phase to estimate the expected completion time of each task under conservative conditions. In the actual scheduling of subsequent decision time steps, the system can select other job equipment for the remaining tasks, without being restricted by this preference allocation. The conservative strategy adopted in the above trial scheduling process ensures the efficiency of the look-ahead simulation, enabling the preset rule model 31 to provide a reliable safe action subspace for further filtering of the action space or the scheduling decision process without sacrificing scheduling flexibility.

[0145] In the state transition step 430, after mapping the candidate scheduling action to a specific task to be processed, the subsequent state of the job environment after the candidate scheduling action is executed can be determined first. and the subsequent state Provided to simulation stage 440. Simulation stage 440 from this subsequent state Starting from this point, and combining the established trial scheduling results, we conduct a forward simulation of the future global operational status of all tasks in the operational environment.

[0146] Specifically, in the state transition phase 430, the single-step successor state after executing the candidate scheduling action can be determined by using a simulator of the working environment. This simulator, for example, refers to a computational module that models the physical and logical operating rules of the work environment. Based on the deterministic state of the work environment and candidate scheduling actions, the simulator deduces the task execution process according to preset parameters and outputs the subsequent state after execution.

[0147] For example, the simulator will simulate the subsequent state of the working environment. The model is designed to include: task status, resource status, and global status within the work environment. Specifically, by calculating the estimated work duration and estimated completion time of the task to be processed on the focal work equipment, the earliest available time of the work equipment, the currently executing task type, the task queue status, and global time information are updated accordingly. If the task involves shared auxiliary resources (e.g., requiring replacement of work parts or occupying a crane), the availability and earliest available time of that resource are also updated accordingly.

[0148] The above information constitutes a snapshot of the working environment one step forward from the current decision-making time step, providing initial conditions for the look-ahead simulation of the remaining tasks. This design explicitly reflects the impact of the current decision action in the subsequent state to reflect changes in the real working environment, enabling look-ahead simulation to more accurately calculate the estimated completion time of each task, thereby improving the reliability of pruning decisions.

[0149] In simulation phase 440, a prospective simulation is performed on all tasks based on the trial run results to calculate the estimated completion time of each task.

[0150] In one embodiment, the estimated start time of each task and its estimated operation time on the corresponding work equipment can be determined first, and then the estimated completion time can be calculated based on both. For example, the estimated completion time can be calculated using the following formula: in, Let be the estimated completion time for task k. Let be the estimated start time of task k. Let k be the estimated duration of operation on its operating equipment.

[0151] For example, if the estimated start time of task k on the work equipment M1 is 10:00 and the estimated work duration is 30 minutes, then the estimated completion time of the task is 10:30.

[0152] In one embodiment, the estimated start time of each task can be determined based on the earliest available time of the work equipment assigned to each task. Specifically, according to the allocation relationship and job order in the trial scheduling results, the remaining tasks are sequentially assigned to each work equipment, and an independent task queue is maintained on each equipment. The earliest available time of a work equipment is the estimated completion time of its previously assigned task. For example, if work equipment M1 is currently processing task job1 with an estimated completion time of 10:00, the estimated start time of the next task job2 assigned to work equipment M1 is also 10:00. And so on. Through independent task queues, the task order and timeline on each equipment can be clearly recorded, ensuring the correct simulation of the switching logic and resource usage.

[0153] In one embodiment, the estimated job duration depends, for example, on the number and type of products included in the task, and can be calculated in advance using a mathematical model or by looking up a table. Furthermore, the estimated job duration may also be affected by the equipment allocated to the task, and therefore needs to be determined in conjunction with the equipment allocation relationships from the trial scheduling.

[0154] In one embodiment, the estimated completion time is further determined based on the task switching type of successive tasks on the same work equipment and / or the earliest available time of shared auxiliary resources in the work environment. The task switching type includes at least: no switching required (i.e., continuous production of the same task type), material-only switching (e.g., same mold, different material), mold-only switching (different mold, same material), and material and mold switching simultaneously.

[0155] Specifically, the switching type can be determined by looking up a table or reading the value of a flag field. A switching type matrix can be pre-constructed for all combinations of task types. This matrix records the switching type, switching time, and whether shared auxiliary resources are required to switch from one task type to another. During lookup simulation, the relevant information can be quickly obtained by looking up the table based on the combination of the current task and the previous task on the same workstation, thereby improving computational efficiency.

[0156] For example, the estimated completion time can be calculated using the following formula: Where j is the sequence number of the previous task and k is the sequence number of the current task. Let be the estimated completion time for task k. Let be the estimated start time of task k. Let k be the estimated duration of operation on its operating equipment. This is the sequence-dependent switching time from task j to task k, and its value depends on the switching type between task j and k (e.g., mold change time, material change and cleaning time, etc.).

[0157] In one embodiment, the estimated start time may also be calculated based on the switching type between two successive tasks on the same work equipment. .

[0158] If the switch type is "no mold change required" (i.e., use the same mold), the expected start time is equal to the earliest available time of the working equipment: If the switch type requires mold replacement, the earliest available time of shared auxiliary resources (such as molds and cranes) must also be considered, and the later one should be taken: in, The earliest available time for the operating equipment. The earliest available time for the crane. This is the earliest available time for the mold.

[0159] After changing the mold, the available time for the machine needs to be updated: in, The fixed time required for a single overhead crane to perform one mold change.

[0160] In the elimination phase 450, based on the look-ahead simulation results from the simulation phase 440, it is determined whether any task's estimated completion time is later than its delivery deadline. If so, the candidate scheduling action is eliminated from the candidate set; otherwise, if the estimated completion time of all tasks is no later than their respective delivery deadlines, the candidate scheduling action is retained and determined as a safe scheduling action.

[0161] Through this conservative strategy, the retained candidate scheduling actions have a high degree of feasibility confidence, while the actions that are eliminated may not be absolutely infeasible in theory. However, this conservative strategy lays a solid foundation for the zero-delay execution of the overall scheduling scheme by setting a relatively strict zero-delay test standard and actively sacrificing some marginal feasible solutions.

[0162] In one embodiment, the estimated completion time of each task calculated in the look-ahead simulation can be used as the basis for the calculation. Its delivery period Compare the results; if any task satisfies the relation... If so, the candidate scheduling action is determined to be unsafe.

[0163] For example, suppose we are considering a candidate scheduling action After performing a prospective simulation, in the future global job scenario, one of the tasks, job1, has an estimated completion time of 10:30 and a delivery deadline of 10:00. Even if other tasks, job2, job3, etc., are not overdue, this candidate scheduling action... It will still be deemed unsafe and removed.

[0164] exist Figure 3 In the embodiment shown, the aforementioned links 410, 420, 430, 440, and 450 in the preset rule model 31 work together to complete the prospective simulation evaluation and pruning of candidate scheduling actions.

[0165] It should be noted that in the trial scheduling phase 420, the order of the work sequence determination phase 421 and the work equipment allocation phase 422 is not fixed or mandatory. In actual implementation, these two phases 421 and 422 can be interchanged according to the specific application scenario or preset strategy, or they can be performed in parallel. For example, the work sequence can be determined first and then the work equipment can be allocated, or the work equipment can be allocated first and then the work sequence can be determined, or the two can be determined alternately or simultaneously.

[0166] It should also be noted that although the state transition step 430 is shown in the figure to occur after the trial scheduling step 420, this application is not limited to this order. The state transition step 430 may also occur before the trial scheduling step 420, or be executed synchronously or alternately with the trial scheduling step 420.

[0167] It should also be noted that although heuristic rules were used to determine the trial arrangement results in this embodiment, this is not the only option. In practical applications, other solution rules can also be used according to specific needs, such as metaheuristic algorithms, planning algorithms, random rules (such as the Monte Carlo method), priority-based rules, or experience-based rules.

[0168] It should also be noted that Figure 3 This only corresponds to one possible pruning scheme with delivery deadline as a hard constraint, namely pruning based on look-ahead simulation. However, in actual implementation, look-ahead simulation can also be omitted. For example, the judgment can be made directly based on whether the minimum relaxation time is less than a preset threshold, or other rule-based or statistical fast pruning methods can be used.

[0169] Figure 4 A schematic diagram illustrating the principle of a preset rule model according to another exemplary embodiment of this application is shown.

[0170] exist Figure 4 In the preset rule model 31, in addition to the prediction screening engine 312, there is also a physical screening engine 311. The physical screening engine 311 prunes the candidate scheduling action 21 based on the immediate feasibility constraint. This immediate feasibility constraint only checks the feasibility of the candidate scheduling action 21 itself and does not involve the inference of future states. This layer of rule engine 311 checks the candidate scheduling actions that cannot be executed at the current decision time step. It has low computational cost and can significantly reduce the space of actions that need to be processed later.

[0171] In one embodiment, the physical screening engine 311 determines that the candidate scheduling action 21 does not meet the immediate feasibility constraint and discards it under at least one of the following conditions: - The task type involved in candidate scheduling action 21 has no pending job requirements. For example, if the task queue corresponding to this task type is empty and there are no pending tasks, then this candidate scheduling action does not need to be considered further and can be directly eliminated.

[0172] - The task type involved in candidate scheduling action 21 does not meet compatibility constraints with the focus operation equipment. For example, the process required by the task type does not match the capabilities of the focus operation equipment, or the equipment does not have the mold assembly conditions, precision, or qualifications required to process this type of task, making the action infeasible.

[0173] - The shared work parts required for the task type involved in candidate scheduling action 21 are occupied. For example, the task type requires a specific mold, which is currently being used by other work equipment and cannot be shared.

[0174] In one embodiment, when pruning using immediate feasibility constraints, a qualification matrix between task types and operating equipment is pre-constructed based on static configuration information of the operating environment. Specifically, the qualification matrix... For a pre-configured static matrix, where The number of task types, For the number of working devices, matrix elements This indicates that the i-th task type can be performed on the j-th work equipment. This indicates that the task is not processable. During the pruning process, for the candidate scheduling action "assign task type i to focus job device j", the system can query the corresponding element in the eligibility matrix. The value, if If the candidate scheduling action does not meet the immediate feasibility constraint, it will be removed.

[0175] exist Figure 4 In the illustrated embodiment, the predictive screening engine 312 and the physical screening engine 311 can perform pruning in a certain order. First, a first pruning process is performed on the candidate scheduling actions 21 based on immediate feasibility constraints, eliminating actions that cannot be executed at the current moment. Then, for the candidate scheduling actions 22 retained after the first pruning process, a second pruning process is performed based on hard constraints constructed with delivery deadlines, eliminating candidate scheduling actions that may cause future tasks to be overdue. The candidate scheduling actions retained after the first and second pruning processes are determined as safe scheduling actions 23.

[0176] The advantage of this hierarchical pruning process is that, through the physical sieve engine 311, obviously infeasible candidate scheduling actions can be quickly eliminated without performing complex look-ahead simulations. Therefore, if a candidate scheduling action violates the immediate feasibility constraint from the outset, it does not need to undergo subsequent look-ahead simulation evaluations that require certain computational overhead, thereby reducing the number of candidate scheduling actions that require look-ahead simulations and effectively narrowing the action space.

[0177] After the physical screening engine 311 completes the initial screening, the remaining candidate scheduling actions 22 then enter the predictive screening engine 312 for further feasibility verification through look-ahead simulation. This not only reduces the overall computational overhead and improves the real-time performance of scheduling decisions, but also allows the second-layer pruning to focus on fewer, more promising candidate scheduling actions, thus improving pruning efficiency. The sequential cooperation of the two filtering engines 311 and 312 together constitutes an efficient and reliable rule-based pruning mechanism.

[0178] like Figure 4 As exemplified, for a given state of the working environment at a certain decision time step, the original action space... For example, it includes multiple candidate scheduling actions. In addition, it includes an idle action. The idle action indicates a decision to temporarily suspend any tasks assigned to the currently focused work device and maintain an idle / standby state.

[0179] After processing by the first-stage physical screening engine 311, the action... Removed due to failure to meet immediate feasibility constraint; remaining actions include... Original action space Compressed to a subset After processing by the second-layer predictive screening engine 312, the action... Because it was determined in the prospective simulation that it would cause some future tasks to be overdue, it was further eliminated, and only the actions were ultimately retained. and idle actions As safety scheduling action 23, the action space of safety actions is obtained. .

[0180] As can be seen, the two pruning processes filter the candidate scheduling actions in turn, gradually narrowing down the feasible subspace, and finally outputting a set of safe scheduling actions that satisfy the hard constraints.

[0181] In one embodiment, all outsourceable tasks are assumed to be outsourced by default, meaning they do not participate in the pruning process using a pre-defined rule model. During actual production scheduling, outsourceable tasks can be dynamically assigned to internal resources based on the availability of resources within the operational environment to increase profits. This avoids outsourceable tasks excessively consuming internal resources during the pruning phase, helps expand the search space for feasible solutions, and improves the feasibility of the scheduling scheme.

[0182] It should be noted that, although Figure 4 The diagram shows that the physical sieve engine 311 first performs immediate feasibility pruning, followed by time-constrained pruning using the predictive sieve engine 312. However, this application is not limited to this order. In actual implementation, depending on specific needs, physical sieve pruning and time-constrained pruning can also be performed in parallel, alternately, or in other desired order.

[0183] It should also be noted that Figure 4 The number and number of actions shown in the candidate scheduling actions and safety scheduling actions are exemplary; in actual applications, there may be more or fewer actions.

[0184] Figure 5 A schematic diagram of a task scheduling framework for a working environment according to another exemplary embodiment of this application is shown.

[0185] Compared to Figure 2 The frame shown Figure 5 In the framework shown, a simulator 33 for the work environment is additionally introduced. The inputs to the simulator 33 are the pruned safety scheduling action 23 and the defined state of the work environment. Safety dispatch action 23 includes, exemplarily, [the following]. ,in The action space representing the safety scheduling action, compared to the original action space of the candidate scheduling actions. The pruning process has significantly reduced the size of the branches.

[0186] Furthermore, agent 32 is shown as comprising a trained machine learning model 321 and a rule-based decision model 322. Therefore, the final decision not only considers the machine learning model's predictions of subsequent states but also incorporates the single-step value (instant reward) of the action calculated based on purely deterministic rules, thus balancing the predictive power of machine learning with the determinism of rule-based computation, improving the interpretability and reliability of the decision. Compared to purely end-to-end machine learning schemes, this design makes the decision-making process more interpretable, allowing for the traceability of which parts of the final evaluation value come from rule-based computation and which from model predictions.

[0187] Using simulator 33, various safety scheduling actions are simulated and executed. To obtain the subsequent state of the work environment after the corresponding safety scheduling action is executed. and instant rewards Subsequently, the subsequent state and immediate reward corresponding to each safety scheduling action are provided to the agent in sequence. Agent 32 then uses these subsequent states... and instant rewards Decision-making is conducted, and the target scheduling action 24 is ultimately determined, exemplarily as follows: .

[0188] By introducing this deterministic simulator 33, the calculation of the action-to-state transition can be separated from the decision-making calculation of agent 32, reducing the learning burden of machine learning model 321. Therefore, agent 32 does not need to implicitly learn the influence of each action on the environment through a large number of trials and errors, but only needs to focus on making subsequent decisions based on state information, thus making its learning task more focused and improving the stability and interpretability of the decisions.

[0189] In one embodiment, agent 32 employs a reinforcement learning framework (e.g., a deep reinforcement learning framework). This framework can significantly reduce training complexity and sample requirements, improve training efficiency and convergence stability, and enable it to have stronger adaptability and decision-making performance in complex, high-dimensional scheduling scenarios.

[0190] In one embodiment, the simulator 33 models the state space of the working environment as follows: for each decision time step t, the state of the working environment at that decision time step t. This includes task status, resource status, and global status.

[0191] Task status includes, for example, a list of remaining tasks in the task queue. Each task may contain attributes such as delivery deadline, compatibility constraints, task type, workload, estimated job duration, efficiency information, and task switching type.

[0192] Resource status includes, for example, the status of operating equipment and the status of shared auxiliary resources. The status of operating equipment includes, for example, the earliest available time of each operating device, the type of task currently being processed, and the currently assembled operating accessories; the status of shared auxiliary resources includes, for example, the occupancy status and earliest available time of the unique non-preemptive crane and / or operating accessories (such as molds).

[0193] Global state includes information such as the position of the current decision time step within global time and the position of the time slot window. In one embodiment, the simulator 33 performs a deterministic forward inference for each safety scheduling action. First, the safety scheduling action is mapped to the next pending task of the corresponding task type in the task queue, and the estimated completion time of the pending task on the focal work equipment is calculated. Then, starting from the decision time step corresponding to the deterministic state, up to the estimated completion time, the deterministic state of the work environment is updated to the subsequent state based on a pre-modeled work environment state space model. This update may involve, for example, updating the work equipment occupancy status and earliest available time, changing the currently executed task type and assembly work parts, removing the corresponding task from the task queue, updating the shared auxiliary resource occupancy status and earliest available time, and advancing global time. In this way, the simulator 33 outputs the deterministic subsequent state and immediate reward after executing the safety scheduling action, providing a reliable basis for the agent 32's subsequent decisions.

[0194] For example, simulator 33 can be combined with the foregoing description. Figure 3 The method for determining the subsequent state, as described, calculates the estimated operation time and estimated completion time of the task on the focal work equipment, and updates the earliest available time of the work equipment, the task queue status, the occupancy status and earliest available time of shared auxiliary resources, and global time information accordingly. If the task involves shared auxiliary resources (e.g., requiring replacement of work parts or occupying the crane), the availability and earliest available time of those resources are updated accordingly. Further details are omitted here.

[0195] In one embodiment, the immediate reward reflects the immediate, one-step economic benefit of the task itself during the execution of the scheduled action (i.e., the entire period from start to finish of the task on the focus device). For example, the immediate reward can be determined based on the task's gross profit, inventory holding costs, and task switching costs.

[0196] For example, the instant reward is calculated using the following formula: in: The gross profit of the task processed by the current scheduling action, which is the revenue obtained by the task through internal production minus the direct production costs. The time required to complete the task ahead of schedule. Cost of holding inventory per unit time This refers to inventory holding costs. Inventory holding costs represent the expenses incurred due to storage and capital tied up while waiting for delivery after a task is completed ahead of schedule. A higher value indicates that the earlier the task is completed, the longer the inventory is held, and the higher the warehousing costs. By introducing this cost term into the training objective, the model can be guided to complete tasks closer to the delivery deadline, rather than completing them too early, thereby reducing inventory backlog and optimizing capital tied up and warehousing efficiency. Let be the sequence-dependent switching cost from the previous task j to the current task k. This cost depends on the switching type; for example, the cost is zero when no switching is required; the cost is low when only materials are changed (e.g., cleaning costs); the cost is high when only work accessories are changed (e.g., molds); and the cost is highest when both materials and work accessories are changed. By introducing switching costs into the training objective, the model can be guided to reduce unnecessary switching in scheduling decisions, thereby reducing job transition overhead and improving overall job efficiency.

[0197] Through the design of the above-mentioned immediate reward, agent 32 can take into account both task benefits and related costs in single-step decision-making, thereby gradually learning and forming a better global decision-making strategy in the long-term scheduling process.

[0198] In one embodiment, the operating environment is an industrial automation environment, enabling the subsequent state determined by the simulator 33 to reflect the true state of the operating environment after the execution of the safety scheduling action within a predetermined accuracy range. This is because in an industrial automation environment, key parameters such as processing time, changeover time, equipment idle time, and crane occupancy time all have definite values ​​or stable statistical distributions, and the operation process follows predictable physical laws rather than uncontrollable random disturbances. In such an environment, the simulator 33 can perform high-precision simulation of the task execution process and resource status changes based on these deterministic or predictable parameters, thereby providing a reliable basis for scheduling decisions.

[0199] The trained machine learning model 321 is used to determine the subsequent state of the job environment after the execution of a safety scheduling action. Generate the successor state Corresponding state value The value of this state. For example, it reflects the subsequent state. Starting point: the cumulative revenue level that the operating environment can achieve in the future.

[0200] In one embodiment, the machine learning model 321 is a trained artificial neural network, which can be implemented using various network structures, such as multilayer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN), graph neural network (GNN), or Transformer network.

[0201] In one embodiment, the machine learning model 321 is a reinforcement learning model, particularly a deep reinforcement learning model.

[0202] Rule-based decision module 322 is based on successor states State value With instant rewards Calculate each safety scheduling action Long-term return level and from all security scheduling actions The action with the highest long-term return is selected as the target scheduling action. This shows that the final decision-making process simultaneously incorporates the single-step returns calculated by rule determinism and the predictions of long-term future returns by machine learning, making full use of the predictive power of machine learning models and the accurate calculation capabilities of rule models based on physical / economic modeling.

[0203] In one embodiment, machine learning model 321 targets each security scheduling action. Generate its successor state one by one. Corresponding state value Its output dimension is fixed at 1 and is independent of the size of the action space. Specifically, each time only the successor state corresponding to a safe scheduling action is input into the machine learning model 321 to obtain the state value estimate corresponding to the successor state. After the above processing is completed for all safe scheduling actions, the rule decision module 322 summarizes the long-term benefits of each action and makes the final selection.

[0204] Therefore, the task of action selection is handled by the rule decision module 322, while the calculation of state-action transitions is undertaken by the front-end environment simulator 33. The output dimension of the machine learning model 321 does not expand with the size of the action space, avoiding the problem of network output layer expansion caused by the increase in the action space, significantly reducing the difficulty of model training, and especially helping to achieve stable convergence in complex and multi-constrained working environments. Thus, while ensuring decision quality, it also has flexibility and interpretability.

[0205] For example, the rule decision module 322 calculates the long-term benefit level corresponding to each safety scheduling action according to the following formula. : in: This is the immediate reward corresponding to the execution of a safety scheduling action, used to measure the single-step benefit brought by the action; This is the discount factor, and its value range is usually [range missing]. This is used to balance the importance of immediate rewards with the estimation of the state value of subsequent states. For example, The possible value is 1; For successor state The corresponding state value is used to measure the future long-term benefits derived from that subsequent state.

[0206] For example, the rule decision module 322 selects target scheduling actions according to the following rules. : in, For the action space of safe scheduling actions, Schedule actions for the target.

[0207] exist Figure 5 In the illustrated embodiment, by using long-term profit level as the optimization objective for scheduling decisions, agent 32 considers not only the merits of the current action itself but also the overall impact of the current action on long-term profit potential. The resulting target scheduling action maximizes the global profit of the operational environment while meeting the hard constraint of delivery deadlines.

[0208] Figure 6 A schematic diagram illustrating the principle of generating target scheduling actions with the aid of an intelligent agent according to an exemplary embodiment of this application is shown.

[0209] exist Figure 6 In the middle, firstly with Figure 5 In a similar manner, the subsequent state of the work environment after the execution of safety scheduling actions is determined. Furthermore, the subsequent state is first processed by a preprocessing module 34. These are represented as heterogeneous feature units 61, 62, 63, and 64. Then, with the help of an agent 32 based on a machine learning model, the target scheduling action 24 is determined from the security scheduling actions based on these heterogeneous feature units 61, 62, 63, and 64.

[0210] In this context, a feature unit refers to encoding scattered information from a specified state of the operational environment into a unified vector representation. Its specific form can be a feature vector, an embedded representation, or other forms of numerical encoding. Multiple feature units arranged sequentially can form a token sequence.

[0211] "Heterogeneous" refers to the fact that the types of feature units are heterogeneous, meaning that different categories of entities (such as task types and operating equipment) correspond to different types of feature units, which are at least semantically distinct. Each feature unit, for example, corresponds to task type, time slot, operating equipment, and global state, respectively. Compared to directly inputting the raw state information of the operating environment into the network, the representation of heterogeneous feature units 61, 62, 63, and 64 provides the machine learning model with a more compact and interpretable state representation, helping the model to better understand the relationships and differences between information of different semantic categories.

[0212] In one embodiment, the preprocessing module 34 can be implemented as a machine learning encoder, for example. For each entity type in the state information, the preprocessing module 34 extracts the corresponding raw feature information and then maps it to a fixed dimension, for example, through a learnable linear projection layer, to obtain the corresponding feature units. Different entity types can use independent or shared projection parameters. This process transforms the unstructured raw state information into a structured sequence of fixed-dimensional vectors, providing a standardized input form for subsequent machine learning models.

[0213] In another embodiment, the preprocessing module 34 may also be a rule-based encoder that extracts and maps structured state data (e.g., a demand matrix) into corresponding feature units by row or column through rule-based mathematical transformations.

[0214] In one embodiment, the heterogeneous feature units 61, 62, 63, and 64 include at least: a task type feature unit 61, a time slot feature unit 62, and a resource feature unit. The resource feature unit may further include a job device feature unit 63 and a global feature unit 64.

[0215] In one embodiment, there are typically multiple feature units of each type, and therefore it can also be understood as a sequence of feature units. For example, a task-type feature unit sequence can be represented as follows: Each task type feature unit (such as Each time slot corresponds to a task type (e.g., product types A, B, C). The sequence of time slot feature units can be represented as... Each time slot feature unit (e.g.) Each slot corresponds to a future time slot (e.g., slot1, slot2, slot3). The sequence of characteristic units of the operating equipment can be represented as... Each feature unit corresponds to a specific work device. A single global feature unit, denoted as G, is typically used to represent global state information in the work environment, such as the availability of shared auxiliary resources, the current time window position, and global timing information. By encoding different categories of entities into independent sequences of feature units, the structured semantics of the state information can be preserved, providing a clear and standardized input representation for subsequent machine learning models.

[0216] In one embodiment, each task type feature unit 61 corresponds to a task type, used to characterize the workload distribution (i.e., time dimension distribution) of the same task type across different time intervals. The workload can be expressed in terms of expected work duration (e.g., in minutes), reflecting the time requirements of all tasks of that task type in different time periods. Furthermore, in some cases, the workload can also be expressed in terms of the number of products to be processed.

[0217] For example, assuming there are three task types A, B, and C in the task queue, and the future time window is divided into three time slots, slot1, slot2, and slot3, the corresponding task type feature unit sequence can be represented as: in, This indicates that the workload of task type A in time slots 1, 2, and 3 is 200 minutes, 60 minutes, and 100 minutes, respectively. and And so on.

[0218] By extracting the raw state information of the work environment into task type feature units, machine learning models can obtain semantic information about the future demand distribution related to task types. This allows them to anticipate the resource demand intensity of various types of tasks at different time periods, which helps to identify time bottlenecks, rationally arrange production sequences, and improve adaptability to dynamic demand changes.

[0219] In one embodiment, each time slot feature unit 62 corresponds to a future time interval, used to characterize the workload distribution (i.e., type dimension distribution) of each task type within the same time interval.

[0220] For example, assuming there are three task types A, B, and C in the task queue, and the future time window is divided into three time slots, slot1, slot2, and slot3, the corresponding time slot feature unit sequence can be represented as: in, This indicates that within slot 1 of the time slot, the workloads for task types A, B, and C are 200 minutes, 150 minutes, and 50 minutes, respectively. and And so on.

[0221] In one embodiment, task type feature units and time slot feature units represent the same set of demand information from different dimensions. Specifically, time slot feature units and task type feature units are, for example, perpendicular to each other and complementary in terms of workload distribution: the former is organized according to the time slot dimension, and the latter is organized according to the task type dimension. They describe the same set of demand data from different perspectives. This complementary representation enables the model to observe complete demand information from two vertical dimensions, simultaneously perceiving the resource competition situation in the time dimension and the demand trend in the task type dimension. This allows for more accurate identification of time bottlenecks and resource conflicts, effectively avoiding short-term local optimal decisions, and improving the long-term global performance of decisions.

[0222] In one embodiment, each work equipment feature unit 63 corresponds to a work equipment, for example, used to characterize the earliest available time of the work equipment, the task type of the current work, and the work accessories currently assembled.

[0223] For example, for the working device M1, its corresponding working device feature unit 63 can be represented as follows: , This is the earliest available time for the equipment. Indicates the type of task currently being executed. This indicates the currently assembled work accessory (e.g., a mold). The same logic applies to other work equipment in the work environment, such as M2, M3, etc.

[0224] For example, the global feature unit 64 can be represented as .in Indicates the availability of shared auxiliary resources (e.g., whether cranes are available). Indicates the earliest available time of the shared auxiliary resource. This indicates global time information.

[0225] Through the equipment feature unit 63 and the global feature unit 64, the machine learning model 321 can perceive the occupancy status of each piece of equipment, the assembly status of equipment parts, the competition and occupancy of shared auxiliary resources, and the position of the current decision time step within the time window. This information provides crucial basis for avoiding resource conflicts and rationally arranging task order in scheduling decisions.

[0226] In one embodiment, before inputting the heterogeneous feature units 61, 62, 63, and 64 into the machine learning model, type embeddings can be superimposed on them to enable the network to distinguish between different categories of feature units (e.g., task type feature units, time slot feature units, job equipment feature units, and global feature units). The type embeddings can use fixed orthogonal vectors and are not updated during training.

[0227] For example, task type feature units can be overlaid. , is the superposition of time slot feature units , for superimposed feature units of operating equipment , is a superposition of global feature units The type embeddings are orthogonal to each other to ensure that the model can clearly distinguish inputs with different semantics.

[0228] In one embodiment, before inputting the heterogeneous feature units 61, 62, 63, and 64 into the machine learning model, a positional encoding may be superimposed on the time slot feature unit 62, which may be used, for example, to characterize the chronological order of the time slots in a future time window.

[0229] For example, linear positional encoding can be used, assigning a learnable linear position vector to each time slot and adding it element-wise to the feature units of each time slot. This allows the model to perceive the sequence of time slots, thereby more accurately understanding the distribution of workload on the time axis. Furthermore, in some cases, sine / cosine positional encoding or relative positional encoding can also be used.

[0230] and Figure 5 Similarly, in the illustrated embodiment, the heterogeneous feature units 61, 62, 63, and 64 corresponding to the subsequent states are subsequently input into the intelligent agent 32 based on the machine learning model. The intelligent agent 32 then uses these heterogeneous feature units to determine the target scheduling action from the security scheduling actions. .

[0231] It should be noted that Figure 6 The number and sequence length of the heterogeneous feature units shown are merely illustrative; in actual applications, the number of various feature units may be more or less.

[0232] Figure 7 A schematic diagram illustrating the principle of the generation process of a time slot feature unit according to an exemplary embodiment of this application is shown.

[0233] like Figure 7As shown, a demand matrix 70 is first constructed based on the task information in the subsequent states. The rows of the demand matrix 70 correspond to task types P1, P2, P3, and P4 (e.g., product types), and the columns correspond to time slots T1, T2, T3, and T4. The demand matrix 70 is used to structure the workload of each task type (P1, P2, P3, and P4) in different time slots T1, T2, T3, and T4.

[0234] To construct the demand matrix 70, the future time window first needs to be divided into multiple time slots T1, T2, T3, and T4. These future time windows can be dynamically updated using a sliding window mechanism, with window lengths determined in days, weeks, or months. Each time, only tasks within that time window from the current moment are released from the task queue / task pool and injected into the demand matrix 70; tasks exceeding the time window are not included. As time progresses, expired time slots are removed, and corresponding future time slots are added. Figure 7 As shown in the example, the future time window, which is based on days, is divided into four time slots, each corresponding to a 6-hour time interval, such as 0-6, 6-12, 12-18, and 18-24.

[0235] During the injection process, the time slot to which each task should belong is determined based on its delivery deadline. Then, for example, the workload can be calculated according to the task type corresponding to each task and injected into the corresponding task type of the corresponding time slot, thereby constructing a complete demand matrix 70. For different tasks of the same task type within the same time slot, their workloads are aggregated and accumulated, no longer distinguishing between specific task instances, thus integrating discrete task information into the total workload of each task type within each time slot.

[0236] In one embodiment, assume a task of type P1 has a delivery deadline of 9:00 AM and an estimated job duration of 1 hour (i.e., a workload of 60 minutes). Since 9:00 AM falls within the 6:00-12:00 time slot T2, and the task needs to be completed within the delivery deadline, the 60-minute workload of this task is injected into time slot T2. After all tasks have been injected, the workloads of all tasks of type P1 within time slot T2 are aggregated, resulting in a total workload of 200 minutes, indicating that task type P1 within time slot T2 requires a total of 200 minutes of work time.

[0237] In one embodiment, tasks can be directly injected into the time slots where their delivery deadlines are located, meaning each task enters only one time slot. In this case, the workload within each time slot can be characterized, for example, by the number of tasks falling into it.

[0238] In another embodiment, when injecting tasks into time slots, the estimated operation time of the task or the corresponding number of products to be processed can also be considered. Therefore, a task may fall into exactly one time slot or it may span multiple adjacent time slots. For example, if the delivery deadline for a task is 7:00 AM and the estimated operation time is 180 minutes, then this task may need to continuously occupy the operation equipment resources in two adjacent time slots T1 and T2. For such tasks that span time slots, the workload can be split according to the estimated operation time in each time slot T1 and T2, and injected into the corresponding time slots respectively.

[0239] Based on the constructed demand matrix 70, the original feature representation of the time slot feature unit can be generated by taking each column of the matrix. This means that within time slot T1, the workloads of task types P1, P2, P3, and P4 are 200, 0, 100, and 30 (in minutes, for example). If necessary, the extracted original feature representations can be further transformed through a learnable linear projection layer to map them to a fixed dimension, thereby unifying the vector dimensions of feature units from different sources.

[0240] Accordingly, the original feature representation of the task type feature unit can be generated by extracting each row of the demand matrix 70, for example... This indicates that for task type P1, the workload in time slots T1, T2, T3, and T4 is 200, 220, 100, and 150 respectively (in minutes, for example).

[0241] In the above manner, the task type feature unit and the time slot feature unit encode the demand information from the time dimension and the type dimension, respectively, and together provide the machine learning model with comprehensive spatiotemporal demand distribution information.

[0242] In one embodiment, the granularity of the time slots corresponding to the time slot feature units can be dynamically adjusted. For example, the length or number of time slots can be dynamically adjusted according to the demand distribution characteristics of the work environment. For instance, when future demand distribution is relatively concentrated, a finer time slot granularity (such as one time slot per hour) can be used to improve the accuracy of demand description. In addition, the number of time slots can be adaptively adjusted according to the density of task arrivals, increasing the number of time slots during high-load periods to finely characterize resource competition, and reducing the number of time slots during low-load periods to compress the state space. This dynamic adjustment mechanism enables the method of this application to flexibly adapt to the scheduling requirements of different work environments, achieving a balance between computational accuracy and state complexity.

[0243] Figure 8 A schematic diagram of the network structure of a machine learning model included in an intelligent agent according to an exemplary embodiment of this application is shown. Figure 8 In the illustrated embodiment, Figure 5 and Figure 6 The machine learning model shown is, for example, a trained artificial neural network, including, for example, a bipartite graph attention network 71, a Transformer network 72, and a task head 73.

[0244] The input to the bipartite graph attention network 71 is the task type feature unit 61 and the work equipment feature unit 63. The network 71 is used to perform local information exchange between these two types of feature units 61 and 63 to output updated task type feature units and work equipment feature units 61' and 63'.

[0245] In one embodiment, the graph structure of the bipartite graph attention network 71 is constructed as follows: each task type and each operating device are treated as two types of nodes, the compatibility relationships between each task type and each operating device in the operating environment are obtained, and edges are established based on these compatibility relationships. For example, edges are only built between compatible task type nodes and operating device nodes, thereby forming the graph structure of the bipartite graph attention network.

[0246] like Figure 8 As illustrated, the graph contains four task type nodes P1, P2, P3, and P4, and three job device nodes M1, M2, and M3. Edges are established only between compatible task type-device pairs, rather than full connections. For example, task type node P1 is only connected to job device nodes M1 and M2, indicating that task type P1 can only be performed on job devices M1 and M2, and not on M3. This graph structure design based on compatibility constraints ensures that attention computation is performed only between physically feasible node pairs, avoiding the transmission of invalid information and improving the computational efficiency of the network.

[0247] In the bipartite graph attention network 71, a multi-head attention mechanism is used, where each node aggregates the features of its neighboring nodes and updates them using attention weights. These attention weights are dynamically calculated based on the feature similarity and compatibility between nodes, enabling the network to automatically learn the importance of different task types and work equipment nodes. Through this layer, task type nodes can perceive the load status and processing capacity of compatible work equipment; work equipment nodes can also perceive the urgency and workload of the tasks to be processed.

[0248] For example, the bipartite graph attention network 71 uses the following node update formula: in, For nodes The original feature vector; For nodes Updated feature vector; It is a learnable linear transformation weight matrix used to transform the input features; The attention weights represent the distances from node j to node j. The degree of importance or strength of the relationship; For nodes The set of neighboring nodes (i.e., nodes with the node) (Nodes with edges) It is a non-linear activation function (such as LeakyReLU or ELU).

[0249] Through this residual connection, nodes retain their own characteristics while absorbing relevant information from their compatible neighboring nodes, thus achieving effective exchange of local information.

[0250] The aforementioned attention mechanism enables the model to assign different levels of importance to different neighboring nodes, thereby selectively focusing on neighboring nodes that are more important to the current node during the information aggregation process.

[0251] In one embodiment, the bipartite graph attention network 71 may employ a multi-head attention mechanism, concatenating or averaging the outputs of multiple attention heads as the final node update features. Through training, different attention heads can learn the importance relationships between nodes from different feature subspaces or relational perspectives. For example, one head may focus on the idle time of the operating equipment, another head may focus on the cost of changing task types, and yet another head may focus on operational efficiency, etc.

[0252] Thus, the model can autonomously identify which neighboring nodes are more critical to the current node's value estimation (e.g., idle machines are more important than busy machines, and low-cost matches are more important than high-cost matches), thereby dynamically adjusting the allocation of attention weights to achieve selective information aggregation. This mechanism enables the network to adaptively learn the importance relationships between nodes from the data without relying on manual annotation.

[0253] In one embodiment, after the bipartite graph attention network 71 completes the modeling of the relationship between task type and operating equipment, it forms a splicing sequence 65 with the updated task type feature unit 61' and operating equipment feature unit 63', together with the time slot feature unit 62 and the global feature unit 64, and inputs it into the Transformer network 72 for global context fusion.

[0254] Through the self-attention and multi-head attention mechanisms of the Transformer network 72, the feature units in the concatenated sequence 65 can interact with each other to obtain the fused global state representation 66. This global state representation 66 is then output to the task head 73, which is, for example, a feedforward neural network (such as a multilayer perceptron), to map the high-dimensional vector form of the global state representation 66 to the final task output, i.e., the state value of the successor state. .

[0255] In one embodiment, a learnable CLS feature unit can also be added at the beginning of the splicing sequence 65. This CLS feature unit interacts with all other feature units in the sequence through a self-attention mechanism, gradually aggregating the global state information of the entire working environment. Its corresponding output vector can be used as the global state representation 66 and output to the task head 73 for state value generation.

[0256] It should be noted that, in combination Figure 8 The number of nodes, connections, network layers, and task header structure described are all exemplary and can be adjusted according to the specific working environment in actual applications.

[0257] Figure 9 A flowchart of a method for training an agent according to an exemplary embodiment of this application is shown. The method shown includes steps 901 to 904.

[0258] In this embodiment, the agent is applied, for example, to the task scheduling method described above, and the machine learning model included in the agent is a reinforcement learning model, such as a deep reinforcement learning model.

[0259] In step 901, experience sample data is obtained, which includes at least: a defined state of the work environment, a scheduling action corresponding to the defined state, the subsequent state of the work environment after the scheduling action is executed, and the immediate reward obtained in the work environment after the scheduling action is executed.

[0260] The determined state and the scheduling action, for example, correspond to the same decision time step t. That is, the scheduling action is one of the candidate decisions generated in this determined state, representing either assigning a task type to the focus job equipment or keeping the focus job equipment idle. Accordingly, the subsequent state can be understood as the state of the work environment after the scheduling action is executed.

[0261] Immediate reward can be understood as a feedback signal that the work environment returns immediately after a scheduling action is executed, used to evaluate the single-step benefit brought about by the execution of that scheduling action. Immediate reward is usually determined based on factors such as task gross profit, inventory holding costs, and task switching costs, reflecting the instantaneous impact of the scheduling action on the economic indicators of the work environment.

[0262] In one embodiment, empirical sample data can be represented as a quadruple. ,in, The working environment is in a deterministic state at decision time step t. This is a scheduling action generated under this defined state. The immediate reward returned by the job environment after this scheduling action is executed. This refers to the subsequent state of the working environment.

[0263] In one embodiment, experience sample data can be obtained either through simulation using a work environment simulator or through direct interaction with the real work environment. In simulation mode, the simulator can quickly calculate subsequent states and immediate rewards based on deterministic rules according to deterministic states and scheduling actions, thereby efficiently generating a large number of samples for training. In real interaction mode, scheduling actions can also be sent to the actual work environment for execution, collecting real subsequent states and calculating immediate rewards.

[0264] In step 902, with the help of the intelligent agent, and especially with the machine learning model it includes, based on the determined state... Generate the corresponding state value And based on the successor state Generate the corresponding state value .

[0265] Reflecting from a definite state Starting point: the cumulative revenue level that the operating environment can achieve in the future. Reflecting from the successor state Starting point: the cumulative revenue level that the operating environment can achieve in the future. Ideally, the difference between the two is precisely the execution scheduling action. The single-step revenue brought That is, satisfying In the context of scheduling decisions, the state value of a given state can also be understood as the reflection of the long-term return level corresponding to a certain scheduling action being selected. It constitutes an assessment measure of the economic value that a scheduling action can create.

[0266] In one embodiment, the determined state and subsequent state of the working environment can be represented as heterogeneous feature units (e.g., task type feature units, time slot feature units, resource feature units, and global feature units), and then these heterogeneous feature units can be input into a reinforcement learning model to output the corresponding state value estimate with the help of the reinforcement learning model.

[0267] In step 903, the training error is calculated based on the immediate reward, the state value of the determined state, and the state value of the subsequent state.

[0268] In one embodiment, the timing difference error can be calculated. And based on this time-series difference error Constructing the loss function According to the loss function Update the agent's parameters. For example, the timing difference error... Defined as: in, This is the discount factor, and its value range is usually [range missing]. This is used to balance the importance of immediate rewards and future rewards. For example, The possible value is 1.

[0269] This timing difference error This reflects the state value estimation of a given state by a machine learning model. The target value integrates environmental feedback and subsequent state value estimation. The deviation between them. The value estimate of the determined state is iteratively updated by utilizing the value estimate of the successor state. This allows for single-step learning of the model.

[0270] For example, loss function Mean square error can be used: By minimizing this loss The output of the machine learning model gradually approximates the true expected long-term return. With each update of the model parameters, the value estimate of the current state is adjusted one step towards a more accurate target value.

[0271] In step 904, the agent's intrinsic parameters are updated based on the training error. These intrinsic parameters may include, for example, the weight parameters of the reinforcement learning model, and may also include parameters of other learnable layers or modules jointly trained with the reinforcement learning model. Using temporal difference learning, the reinforcement learning model can update its parameters online and incrementally. With iterative training on a large number of empirical samples, the reinforcement learning model gradually converges, thereby accurately estimating the long-term reward level of the working environment under any state.

[0272] In one embodiment, a semi-gradient temporal difference approach is used to optimize the agent's internal parameters. Specifically, when calculating the gradient of the loss function, the target value can be... As a fixed optimization objective, the state value is determined solely by updating the state through gradient descent. The corresponding model parameters. By repeatedly executing the above steps, the agent gradually learns the mapping from state to value estimation, thereby providing accurate state value assessment for scheduling decisions.

[0273] In one embodiment, training of the agent is stopped when a preset convergence condition is met. The preset convergence condition may include, for example, the change in the loss function value over several consecutive iterations being lower than a preset threshold, or reaching a preset maximum number of training steps, or the absolute value of the temporal difference error remaining within a small range for an extended period.

[0274] In one embodiment, the training sample data should cover different months and combinations of different task types (e.g., the demand structure of different product combinations) to improve the network's generalization ability under different working environments. By introducing diverse demand distributions during the training phase, the agent can learn more robust state-value mapping relationships, avoiding overfitting to specific months or specific product combinations. For example, actual data from multiple production months (such as August, October, and November) can be collected, each month having different distributions of order types, demand quantities, and delivery deadlines, and these can be mixed together as a training set.

[0275] It should be noted that, although in Figure 9 The training process of the agent is mainly described using a reinforcement learning model as an example. However, in actual implementation, the agent may also include other modules or learnable components that are jointly trained with the reinforcement learning model (such as a learnable projection layer jointly trained with the feature encoder). Therefore, when optimizing the agent's internal parameters, the parameters of the aforementioned jointly trained modules should also be optimized accordingly.

[0276] Figure 10 A block diagram of an electronic device according to an exemplary embodiment of this application is shown.

[0277] Electronic device 10 includes processor 11 and memory 12. Memory 12 (such as a computer-readable storage medium like a hard disk, RAM, or flash memory) stores computer program instructions. When processor 11 (e.g., a central processing unit / CPU, microprocessor, digital signal processor / DSP, or other general-purpose processor) executes these instructions, it can implement methods for task scheduling in an operating environment and / or methods for training intelligent agents. The specific implementation details of these methods have been described in detail above and will not be repeated here.

[0278] The electronic device 10 can be deployed on a local terminal device or on a server (e.g., a cloud platform). The machine learning model used in the aforementioned method can be stored in the local memory of the electronic device 10 or in other locations, such as by calling a model service deployed in the cloud through an application programming interface.

[0279] The electronic device 10 may also include an input / output interface to visualize relevant scheduling actions, decision results, and overall work plans when necessary. For example, such visualizations may be output in the form of charts, tables, or text.

[0280] It should be understood that the methods of the various embodiments of this application can be implemented by computer programs / software. This software can be loaded into the processor's memory and, when run, is used to execute the methods according to the various embodiments of this application.

[0281] It should be understood that the same or similar parts between the various embodiments in this specification can be referred to each other, and each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus embodiments, since their control logic basically corresponds to that of the method embodiments, the description is relatively brief, and relevant parts can be referred to the description of the method embodiments.

[0282] According to another embodiment of this application, a computer program product including computer program instructions is provided, the computer program instructions being configured to perform the methods according to various embodiments of this application when the computer program product is run on a computer or stored on a computer-readable storage medium (such as a CD-ROM). The machine-readable storage medium is, for example, an optical storage medium or a solid-state medium supplied together with or as part of other hardware.

[0283] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of this application, even when only a single embodiment is described with respect to a particular feature. The feature examples provided in this application are intended to be illustrative and not limiting, unless otherwise stated. In practice, multiple features may be combined with each other as needed and where technically feasible. In particular, features from different embodiments may also be combined with each other. Various substitutions, modifications, and alterations are conceived without departing from the spirit and scope of this application.

Claims

1. A method for task scheduling in a work environment, comprising: Obtain candidate scheduling actions under a defined state of the work environment. Each candidate scheduling action represents assigning a task type to the focus work equipment in the work environment or keeping the focus work equipment idle. Using a pre-defined rule model and with the delivery deadline of each task as a hard constraint, the candidate scheduling actions are pruned to obtain safe scheduling actions. as well as The target scheduling action is determined from the security scheduling actions by using an intelligent agent based on a machine learning model.

2. The method according to claim 1, wherein, Using a pre-defined rule model and with the delivery deadline of each task as a hard constraint, the candidate scheduling actions are pruned, including: Each candidate scheduling action is evaluated according to predetermined rules. If it is determined that a candidate scheduling action will cause any task to fail to be completed within the delivery deadline in the future, the candidate scheduling action is removed.

3. The method according to claim 1 or 2, wherein, Using a pre-defined rule model and with the delivery deadline of each task as a hard constraint, the candidate scheduling actions are pruned to obtain safe scheduling actions, including: Map the candidate scheduling action to a task to be processed in the task queue; In particular, heuristic rules are used to perform trial scheduling on the remaining tasks in the task queue other than the task to be processed, so as to obtain the trial scheduling results. The trial scheduling includes: assigning tasks to the corresponding work equipment in the work environment, and determining the work order of the tasks. Based on the results of the trial scheduling, assess whether there are any situations where any task cannot be completed within the delivery deadline.

4. The method according to claim 3, wherein, The trial arrangement includes: The order of tasks is determined based on the urgency of each remaining task in the task queue, with tasks of higher urgency being scheduled for execution first. Specifically, the urgency level is measured by the minimum slack time of each remaining task, which is calculated based on the delivery deadline and the estimated operation time of each remaining task.

5. The method according to claim 3 or 4, wherein, The trial arrangement includes: Assign appropriate job equipment to the remaining tasks in the task queue according to at least one of the following preference rules: Select preferred operating equipment based on the operating efficiency of each type of equipment; Select the preferred operating equipment based on the operating load of each operating device; Preferred operating equipment is determined by fixed rules based on historical experience.

6. The method according to any one of claims 1 to 5, wherein, Using a pre-defined rule model and with the delivery deadline of each task as a hard constraint, the candidate scheduling actions are pruned, including: By using prospective simulation, we can pre-simulate the future global operation of all tasks in the work environment under the condition that each candidate scheduling action is selected; If, in the prospective simulation results for a candidate scheduling action, there exists any task whose estimated completion time is later than its delivery deadline, then the corresponding candidate scheduling action is removed.

7. The method according to claim 6, wherein, The prospective simulation is a one-time forward simulation, which generates only one trial arrangement result for the remaining tasks in the task queue in terms of job order and job equipment allocation for each candidate scheduling action.

8. The method according to claim 6 or 7, wherein, In the prospective simulation, the estimated completion time of each task is calculated in the following manner: The estimated start time for each task is determined based on the earliest available time of the working equipment allocated to each task. Determine the estimated operation time for each task on the corresponding operating equipment; The estimated completion time is determined based on the estimated start time and the estimated operation duration.

9. The method according to any one of claims 6 to 8, wherein, The estimated completion time is also determined based on the task switching type of successive tasks on the same work equipment and / or the earliest available time of shared auxiliary resources in the work environment; Specifically, the task switching types include: no switching required, material switching only, operation accessory switching only, and both material and operation accessory switching.

10. The method according to any one of claims 6 to 9, wherein, The prospective simulation also includes: Using the simulator of the work environment, the subsequent state of the work environment after the candidate scheduling action is executed is determined; Starting from the subsequent state, perform a forward simulation of the future global operation of all tasks in the operating environment.

11. The method according to any one of claims 1 to 10, wherein, The method further includes: Using a preset rule model and based on immediate feasibility constraints, the candidate scheduling actions are pruned. In particular, the immediate feasibility constraints only check the feasibility of the candidate scheduling actions themselves.

12. The method according to claim 11, wherein, Candidate scheduling actions are determined to not meet the immediate feasibility constraint and are eliminated under at least one of the following conditions: The task types involved in the candidate scheduling actions have no pending job requirements; The task types involved in the candidate scheduling actions do not meet the compatibility constraints with the focus operation equipment; The shared job components required for the task types involved in the candidate scheduling actions are occupied.

13. The method according to any one of claims 1 to 12, wherein, Using a pre-defined rule model, candidate scheduling actions are pruned in the following order: The first pruning process is performed on the candidate scheduling actions based on the immediate feasibility constraint. For the candidate scheduling actions retained after the first pruning process, a second pruning process is performed based on the hard constraints constructed with the delivery deadline; The candidate scheduling actions retained after the first and second pruning processes are determined as the safe scheduling actions.

14. The method according to any one of claims 1 to 13, wherein, Determining target scheduling actions from safety scheduling actions using an agent based on a machine learning model, including: Using the intelligent agent, the safe scheduling action that is expected to obtain the highest long-term benefit level of the working environment is determined from the safe scheduling actions, and is taken as the target scheduling action.

15. The method according to any one of claims 1 to 14, wherein, The method includes: Using a simulator of the aforementioned work environment, various safety scheduling actions are simulated to determine the subsequent state and immediate reward of the work environment after each safety scheduling action is executed; and By leveraging a machine learning-based agent, decisions are made based on the subsequent states and immediate rewards corresponding to each safety scheduling action, in order to determine the target scheduling action from the safety scheduling actions.

16. The method according to claim 15, wherein, The instant reward is used to characterize the single-step benefit obtained from performing a safe scheduling action. In particular, the simulator determines the instant reward based on the task's gross profit, inventory holding costs, and task switching costs.

17. The method according to claim 15 or 16, wherein, The operating environment is an industrial automation environment, which allows the subsequent state determined by the simulator to reflect the true state of the operating environment after the actual execution of the safety scheduling action within a predetermined accuracy range.

18. The method according to any one of claims 15 to 17, wherein, The intelligent agent includes: A trained machine learning model is used to generate the state value corresponding to the subsequent state based on the subsequent state of the working environment. The rule decision module is used to determine the long-term benefit level of each safe scheduling action based on the state value and immediate reward of its corresponding subsequent state, and select the safe scheduling action with the highest long-term benefit level as the target scheduling action.

19. The method according to any one of claims 1 to 18, wherein, Using a machine learning-based agent, the target scheduling action is determined from the security scheduling actions, including: Obtain the status information of the subsequent state of the working environment after the execution of the safety scheduling action; The state information of the subsequent state is represented as a heterogeneous feature unit, which includes at least: a task type feature unit, a time slot feature unit, and a resource feature unit; Using a machine learning-based agent, the target scheduling action is determined from the security scheduling actions based on the heterogeneous feature units.

20. The method according to claim 19, wherein, Each time slot feature unit is used to characterize the workload distribution of each task type within the same time interval.

21. The method according to claim 19 or 20, wherein, Each task type feature unit is used to characterize the workload distribution of the same task type in different time intervals.

22. The method according to any one of claims 19 to 21, wherein, The method includes: Divide the future time window into multiple time slots; Based on the delivery deadline of each task, determine the time slot to which each task should belong; Based on the task type, the workload of each task is injected into the corresponding time slot, forming a demand matrix; and The task type feature unit is generated based on the rows of the demand matrix, and / or the time slot feature unit is generated based on the columns of the demand matrix.

23. The method according to any one of claims 19 to 22, wherein, The method further includes: The time slot feature units are superimposed with position codes, which are used to characterize the chronological order of each time slot in a future time window; and / or The heterogeneous feature unit superposition type is embedded to distinguish different categories of feature units.

24. The method according to any one of claims 19 to 23, wherein, The resource feature unit includes an operating equipment feature unit and a global feature unit, wherein: Each work equipment feature unit corresponds to one work equipment, including the earliest available time of the work equipment, the current task type, and the currently assembled work accessories; The global feature unit includes the availability of shared auxiliary resources, the earliest available time, and global time information.

25. The method according to any one of claims 1 to 24, wherein, The machine learning model included in the intelligent agent is a trained artificial neural network, including a bipartite graph attention network, a Transformer network, and a task head, specifically: The bipartite graph attention network is used to perform local information exchange between task type feature units and work equipment feature units, and output updated task type feature units and work equipment feature units. The Transformer network is used to globally fuse the updated task type feature unit and job device feature unit with the time slot feature unit and global feature unit to output the fused global state representation. The task header is used to output the state value of the successor state corresponding to the security scheduling action based on the fused global state representation.

26. The method of claim 25, wherein, The method further includes: Obtain the compatibility relationships between all task types and operating equipment related to the operating environment; and The graph structure of the bipartite graph attention network is constructed based on the compatibility relationship, wherein edges are established only between compatible task type nodes and job device nodes.

27. The method according to claim 25 or 26, wherein, The bipartite graph attention network employs a multi-head attention mechanism.

28. The method according to any one of claims 1 to 27, wherein, The method is an event-driven dynamic rescheduling method, which is triggered when at least one of the following conditions is met: the task queue changes, a predetermined event occurs in the resource of the job environment, or a preset rescheduling time interval is reached.

29. The method according to any one of claims 1 to 28, wherein, The method further includes: For each decision time step, the method is executed iteratively to obtain the target scheduling action corresponding to each decision time step; Each target scheduling action is mapped to the allocation relationship between tasks in the task queue and the working equipment. The allocation relationships are summarized to generate the overall work plan for the work environment.

30. The method according to any one of claims 1 to 29, wherein, The agent employs a reinforcement learning framework, particularly a deep reinforcement learning framework.

31. The method according to any one of claims 1 to 30, wherein, The operating environment is an injection molding workshop, the focal operation equipment is one of the multiple injection molding machines present in the operating environment, and the task type is the type of product to be injection molded corresponding to the production order.

32. The method according to claim 31, wherein, When pruning is performed with delivery deadlines as a hard constraint and / or when decisions are made with the aid of the agent, the product switching type corresponding to two production orders processed consecutively on the same injection molding machine is considered. The product switching type determines whether it is necessary to change the mold for the injection molding machine and whether it is necessary to occupy the crane.

33. A method for training an agent, particularly a machine learning model included in the agent, wherein the agent is applied to the method according to any one of claims 1 to 32, comprising: Acquire experience sample data, which includes at least: a defined state of the work environment, a scheduling action corresponding to the defined state, the subsequent state of the work environment after the scheduling action is executed, and an immediate reward. With the help of the intelligent agent, corresponding state values ​​are generated based on the determined state and the subsequent state, respectively; The training error is calculated based on the immediate reward, the state value of the determined state, and the state value of the subsequent state; and The internal parameters of the agent are updated based on the training error.

34. An electronic device comprising a memory and a processor, the memory storing computer program instructions which, when executed by the processor, enable the processor to perform the method according to any one of claims 1 to 33.

35. A computer program product comprising computer program instructions, wherein, When executed by a processor, the computer program instructions enable the processor to perform the method according to any one of claims 1 to 33.

36. A computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 33.

Images