A human-robot collaborative flexible job shop scheduling method based on offline reinforcement learning

Abstract

The present application relates to the technical field of intelligent manufacturing and industrial artificial intelligence, in particular to a man-machine cooperation flexible job shop scheduling method based on offline reinforcement learning, which firstly standardizes mathematical modeling of man-machine cooperation double-resource flexible job shop scheduling problem; constructs a heterogeneous scheduling graph containing three types of heterogeneous nodes of process, machine and worker; designs a multi-attention feature extractor; constructs a quantile alignment Actor-Critic offline reinforcement learning framework based on uncertainty control to complete robust offline policy training; and finally realizes end-to-end real-time scheduling decision through the trained model. The present application can complete model training without online environment interaction, has high solution accuracy, strong generalization ability, low deployment cost and high decision robustness, and is suitable for man-machine cooperation flexible production scene in the industrial 5.0 environment.

Description

Technical Field

[0001] This invention relates to the fields of intelligent manufacturing and industrial artificial intelligence technology, specifically to a human-machine collaborative flexible workshop scheduling method based on offline reinforcement learning. Background Technology

[0002] In the Industry 5.0 paradigm, manufacturing systems have evolved from simply pursuing automation to a human-machine collaborative architecture with deep coupling of processes, machines, and workers. With increasing demands for production flexibility, the traditional job shop scheduling problem has expanded into the more challenging dual-resource flexible job shop scheduling problem (DFJSP-W) with worker constraints. This problem requires simultaneous joint optimization of process sequence, machine allocation, and worker allocation within a highly coupled state space. This dual-resource constraint not only exponentially increases the difficulty of combinatorial optimization but also places stringent demands on the real-time response capability and global search accuracy of the scheduling algorithm.

[0003] Existing solutions for DFJSP-W still face multiple bottlenecks in practical applications: Traditional exact algorithms, such as mixed-integer linear programming, can find the optimal solution, but in large-scale industrial scenarios, the computational load increases exponentially with the problem size, resulting in a serious "curse of dimensionality" that cannot meet the requirements of real-time scheduling. While heuristic rules are computationally efficient, they struggle to capture the deep coupling logic between processes and human-machine resources, resulting in poor scheduling quality. Meta-heuristic strategies such as genetic algorithms, while possessing global search capabilities, require time-consuming iterative searches when handling new tasks, exhibiting poor real-time response capabilities and a tendency to get trapped in local optima. Online deep reinforcement learning, as an end-to-end decision-making paradigm, has shown great potential in dynamic scheduling. However, it relies heavily on agents to update strategies through massive "trial and error" processes, which can lead to high interaction costs and potential security risks in physical production lines, making it difficult to deploy directly. While existing offline reinforcement learning methods can extract decision knowledge from static historical data without online interaction, they are limited in their ability to effectively describe the complex topological constraints between processes, machines, and workers when dealing with the high-dimensional, discrete scheduling space of DFJSP-W. On the one hand, the graph representation methods are too simplistic to adequately describe the complex topological constraints between processes, machines, and workers. On the other hand, the standard algorithms are limited by the decision space coverage of the historical dataset, and when dealing with unseen scheduling actions, they are prone to serious overestimation of value, leading to the collapse of the scheduling strategy or a significant performance decline in practical applications.

[0004] In summary, existing technologies struggle to simultaneously address the issues of flexible scheduling in human-machine collaboration, including solution quality, real-time response capabilities, and deployment security in industrial settings. The key technical challenge remaining to be solved in the field of industrial intelligent decision-making is how to fully utilize historical production data to construct an intelligent scheduling system that can deeply characterize the coupling relationships of multiple resources and effectively suppress decision-making biases, without requiring online interaction. Summary of the Invention

[0005] The purpose of this invention is to provide a human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning, which can simultaneously take into account the scheduling solution quality, real-time response capability, deployment security and decision robustness, and realize efficient, robust and intelligent scheduling of dual-resource flexible job shops under human-machine collaboration.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning, comprising the following steps: S1. Construct a scheduling model for a flexible workshop with dual resources in a human-machine collaborative environment. The scheduling objective is to determine the optimal start time, processing machine, and operator for all processes to minimize the maximum completion time and satisfy process pre-sequence constraints, resource uniqueness constraints, and resource compatibility constraints. S2. Define the instantaneous state of the scheduling system as a heterogeneous scheduling graph. The heterogeneous scheduling graph includes three types of heterogeneous nodes: process nodes, machine nodes, and worker nodes, as well as edges used to describe the relationships between nodes. At the same time, construct corresponding high-dimensional feature tensors for each type of node and edge of the heterogeneous scheduling graph as the underlying input for feature extraction. S3. Construct a multi-attention feature extractor based on heterogeneous scheduling graphs to perform deep feature extraction on the heterogeneous scheduling graphs of the current scheduling state and generate a global state representation vector. S4. Construct an offline reinforcement learning algorithm framework for quantile alignment Actor-Critic based on uncertainty control. Use historical static datasets to train the scheduling strategy. In this framework, quantile regression is introduced to represent the reward distribution, and the variance of the value distribution is used to construct uncertainty weights to suppress the overestimation of the value of out-of-distribution actions in offline learning. S5. Apply the trained strategy model to actual production scheduling to make real-time decisions on new tasks.

[0007] Preferably, the specific construction method of the dual-resource flexible job shop scheduling problem model in step S1 is as follows: Define the set of jobs to be processed Machine resource collection A group of workers with different skills Each assignment It consists of a series of processes with strict sequential constraints. composition, For homework The first in One process; for the process It needs to be from a compatible machine set Select a machine and from skill-matched worker sets The middleman assigned a worker Collaborative operations; Define process Total occupied time ,in For machine processing time, Set the time for workers on the corresponding machines; The scheduling objective is to minimize the maximum completion time. ,in For process The completion time.

[0008] Preferably, in step S1: The pre-sequence constraint is: the start time of the subsequent process in the same operation. , This is the start time of the subsequent process; The resource uniqueness constraint is: at any time t, each machine can only process one process, and each worker can only operate one machine; The resource compatibility constraint is: the machine assigned to a process belongs to its compatible machine set. The assigned workers belong to its skill-matched worker set. .

[0009] Preferably, in step S2: Heterogeneous scheduling graph is represented as heterogeneous node set , For process node set, For machine node set, For worker node set; Edge set It includes three types of semantic edges: process-process edges, process-machine edges, and machine-worker edges; The process-process boundary describes the technological topology logic of processes within the same operation; the process-machine boundary defines the processing feasibility of the machine for the process; and the machine-worker boundary describes the matching relationship between the worker's operating authority and skills.

[0010] Preferably, the high-dimensional feature tensor in step S2 specifically includes: 10-dimensional process node features Includes indicators such as task urgency and remaining workload; 8-dimensional machine node features This includes real-time equipment load, availability status, and processing efficiency characteristics; 6-dimensional worker node features Includes the number of available candidate operations, skill coverage, average setup time, and real-time idle time; 4D Machine-Worker Features This includes relative set time, percentage of worker-specific remaining tasks, and worker execution efficiency; The multidimensional feature tensors together constitute the underlying input for the scheduling agent's decision-making.

[0011] Preferably, in step S3, the multi-attention feature extractor integrates the operation message attention block, the machine message attention block, and the worker message attention block; The operation message attention block is used to model the technical dependencies between processes within the same job; the machine message attention block is used to capture the dynamic competitive intensity of each device for the current task to be processed; and the worker message attention block is used to quantify the collaborative potential and task competition relationship among workers. Each attention block adopts a multi-head attention mechanism. After multiple layers of information propagation and updating, the embedding vectors of process nodes, machine nodes, and worker nodes are averaged and concatenated to generate a global state representation vector.

[0012] Preferably, in the worker message attention block, for any two workers... and Define competitive feature vectors Determine the degree of overlap between the two in the current set of candidate processes, combined with the worker's own hidden characteristics. and Calculate attention coefficient The calculation formula is: ; in , and These are all learnable parameters of the model. This is a vector concatenation operation; the attention coefficients are Softmax normalized and weighted to obtain worker embedding vectors that contain labor competition dynamics. Global state representation vector The calculation logic is as follows: ; Where L is the number of multi-attention layers. , , These are the embedding vectors of the process node, machine node, and worker node, updated after L-layer information propagation. , and These represent the number of processes, machines, and workers currently in operation.

[0013] Preferably, in step S4, the quantile alignment of the Critic network within the Actor-Critic framework uses a set of discrete quantiles. Approximating the return distribution, action value The mean of all quantiles is: ; The Critic network, combined with the Dueling architecture, embeds the global state into the input state value stream. After fusing local and global features, the input action advantage stream is obtained. The estimated values ​​of each quantile are output through mean-centering aggregation operation, thereby achieving decoupled representation of state baseline and action gain. The construction and application of uncertainty weights are as follows: calculate the variance of the value distribution. Constructing uncertainty weighting coefficients The calculation formula is: ; ; in The modulation intensity coefficient, For numerical smoothing terms, , Trim the upper and lower limits of the weights; During the model update phase, weights are applied to the quantile Huber loss term, combined with a conservative Q-learning CQL regularization term, to form the final Critic loss function: ; in For time-series difference loss, This is the CQL regularization loss; During the Actor policy improvement phase, the weights are used to adjust the policy gradient, so that the agent prioritizes updating actions with high confidence in value estimation.

[0014] Preferably, in step S4, a dense differential reward function that integrates domain knowledge is used to guide the training strategy, with a single-step reward... Defined as the difference between the lower bounds of the maximum completion time estimates between adjacent decision steps, i.e.: ; in The current scheduling state is as follows. This is the lower bound of the estimated maximum completion time under the current conditions; The core parameter configuration for offline training of the scheduling strategy is as follows: the embedding dimension of the feature extractor is set to 32, and the multi-attention layer is configured to have 2 layers; the multilayer perceptron (MLP) uses 2 hidden layers with 64 hidden units and a Dropout layer with a probability of 0.2; the Adam optimizer is used, the training batch size is 128, and the learning rate of the Actor network is 3×10⁻⁶. -5 The Critic network has a learning rate of 3×10⁻⁶. -4 The total number of training steps is 100,000; the number of quantiles Set to 32, CQL regularization coefficient to 1.0, temperature coefficient to 0.001, discount factor to 0.99, and uncertainty scaling factor to... The weight is set to 1.2, and the weight pruning range is limited to [0.1, 5.0]. The target network adopts a soft update mechanism with an update coefficient of 0.005.

[0015] Preferably, the real-time decision-making process in step S5 is as follows: Real-time collection of work order information, equipment status, worker on-duty and skill information from the production site to construct corresponding heterogeneous scheduling diagrams; Input the heterogeneous scheduling graph into the multi-attention feature extractor to generate a global state representation vector; Input the global state representation vector into the trained policy model, and output the current optimal scheduling action end-to-end, including the selection of the process to be processed, the allocation of processing machines and the assignment of operators. After executing the scheduling action, update the production status and repeat the above process until all work orders are completed.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a heterogeneous scheduling graph representation system, achieving refined and structured modeling of three types of heterogeneous production entities—processes, machines, and workers—and their complex topological constraints. The accompanying multi-attention feature extraction architecture can deeply capture the coupled interaction logic between multiple resources, solving the problem that traditional methods struggle to accurately depict the deep relationships in human-machine collaboration. Employing an offline reinforcement learning paradigm, it directly mines scheduling decision knowledge from historical static data. The training process does not require online interaction with the physical production environment, effectively avoiding the high costs and production safety risks associated with online trial and error, and significantly lowering the industrial implementation threshold of intelligent scheduling systems. By combining a quantile regression-based uncertainty modulation mechanism with a conservative learning strategy, it effectively suppresses the overestimation of the value of out-of-distribution actions in offline learning, significantly improving the decision robustness and cross-scenario generalization ability of the scheduling strategy. Ultimately, it achieves a comprehensive balance of scheduling solution quality, real-time response capability, deployment security, and scenario adaptability, effectively addressing the core pain point that existing technologies cannot simultaneously meet the multi-dimensional scheduling needs of flexible production. This provides stable and efficient core technical support for the intelligent and flexible transformation of human-machine collaborative manufacturing models in the Industry 5.0 environment. Attached Figure Description

[0017] Figure 1 This invention presents a heterogeneous diagram of a human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning. Figure 2 This is a diagram of the MAFE architecture, which is a human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning, according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1-2 This invention provides a technical solution: a human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning, comprising the following steps: S1. Modeling the scheduling problem of flexible workshop with human-machine collaboration and dual resources A model for scheduling in a flexible workshop with dual resources under a human-machine collaborative environment is constructed. The scheduling objective is to determine the optimal start time, processing machine, and operator for all processes to minimize the maximum completion time and satisfy process pre-sequence constraints, resource uniqueness constraints, and resource compatibility constraints. This embodiment provides a standardized mathematical model for the dual-resource flexible job shop scheduling problem (DFJSP-W) in a human-machine collaborative environment, specifically defined as follows: The specific construction method of the dual-resource flexible job shop scheduling problem model is as follows: Production resource definition: Given a set of jobs to be processed A set of machine resources and a group of workers with different skills. Each assignment It consists of a series of processes with strict sequential constraints. composition, For homework The first in One process; Resource allocation rules: For each process The system must be from a compatible machine set Select a machine and from skill-matched worker sets The middleman assigned a worker Collaborative operations. Define the process. Total occupied time Processing time of the machine The setup time required for a worker to perform a task on a specific machine. Together constitute, that is ; Scheduling objective: Determine the optimal start time for all processes. Processing machines and operators To minimize the maximum completion time Its mathematical expression is ,in Representative process The completion time; Constraints: The pre-sequence constraint is: for the same operation, the start time of the subsequent operation must meet the following conditions. , This is the start time of the subsequent process; The resource uniqueness constraint is: to ensure that at any time t, each machine Each worker can only handle one process. Only one machine can be operated at a time; The resource compatibility constraint is: the machine assigned to a process belongs to its compatible machine set. The assigned workers belong to its skill-matched worker set. ; Through the above modeling, the complex production scheduling process is transformed into a combined optimization problem subject to multiple constraints such as process sequence, machine load and human resources.

[0020] S2. Heterogeneous Scheduling Graph Representation and State Definition To achieve structured modeling of complex constraint relationships among multiple entities in human-machine collaborative scheduling scenarios, the instantaneous state of the scheduling system is defined as a heterogeneous scheduling graph. The heterogeneous scheduling graph includes three types of heterogeneous nodes: process nodes, machine nodes, and worker nodes, as well as edges used to describe the relationships between nodes. Corresponding high-dimensional feature tensors are constructed for each type of node and edge of the heterogeneous scheduling graph, which serve as the underlying input for feature extraction. The specific structure and features are defined as follows: Heterogeneous node set: It fully covers the three core production factors: processes, machines, and workers. For process node set, For machine node set, For worker node set; Semantic edge set: It includes three types of semantic edges, which accurately describe the physical and logical relationships between production factors, namely: Process-process edge: Used to depict the process topology logic between processes within the same operation; Process-Machine Edge: By defining the feasibility of a machine in processing a specific process, the requirements of production tasks for equipment resources are made explicit; Machine-worker edge: Used to describe the matching relationship between workers' operating permissions and skills, and to establish the assignment logic between personnel and equipment; High-dimensional feature tensors specifically include: 10-dimensional process node features It includes key indicators reflecting the urgency of the task and the remaining workload, specifically including: 1. Index of the process in the operation; 2. The number of remaining processes in the current operation; 3. The urgency of the process (e.g., latest start time - current time); 4. Average processing time of the process on all compatible machines; 5. The shortest processing time for each process; 6. The longest processing time for a process; 7. Are the procedures ready? 8. Waiting time; 9. The remaining workload of the assigned task; 10. Number of candidate machines for each process; 8-dimensional machine node features It consists of features that reflect the real-time load, current availability, and processing efficiency of the equipment, specifically including: 1. Is the machine currently idle? 2. The machine's estimated idle time; 3. Queue length on the machine; 4. The machine's processing efficiency coefficient; 5. Machine failure rate; 6. The number of process types that the machine can process; 7. The machine's current load rate; 8. Machine compatibility indicator with the currently scheduled process; 6-dimensional worker node features It records key workforce statuses such as the number of available candidate operations, skill coverage, average setup time, and real-time idle time for workers, specifically including: 1. Are the workers available? 2. Expected idle time for workers; 3. The number of skills possessed by workers; 4. Average setup time for workers; 5. The number of work processes that a worker can currently choose from; 6. Worker skill level coefficient; 4D Machine-Worker Features By calculating ratios such as relative setup time, the percentage of remaining tasks for workers, and worker execution efficiency, a refined characterization of human-machine collaboration efficiency is achieved, specifically including: 1. Relative setup time for a specific worker-machine combination; 2. The percentage of the worker's remaining tasks; 3. The worker's execution efficiency (1 / average setup time); 4. Historical success rate / frequency of use of this combination; The above heterogeneous scheduling graph structure is as follows: Figure 1 As shown, this graph representation method can explicitly encode complex resource constraints, providing rich structured state representations for subsequent reinforcement learning-based intelligent decision-making. These multi-level and multi-dimensional state representations together constitute the underlying input for the scheduling agent to make sequential decisions, laying a data foundation for achieving robust offline policy learning.

[0021] This invention achieves accurate modeling of three heterogeneous entities—processes, machines, and workers—and their complex constraint relationships by constructing a heterogeneous scheduling graph and designing a multi-attention feature extractor. This architecture can capture high-dimensional interaction features between multi-dimensional resources in parallel. In particular, it effectively quantifies the competition and collaboration dynamics of labor through worker message attention blocks, solving the problem that traditional methods struggle to finely describe the deep human-machine coupling logic.

[0022] S3, Multi-Attention Feature Extraction and Global State Aggregation Construct a multi-attention feature extractor based on heterogeneous scheduling graphs to perform deep feature extraction on the heterogeneous scheduling graphs of the current scheduling state and generate a global state representation vector; This embodiment designs a multi-attention feature extractor (MAFE) based on a heterogeneous scheduling graph to parallelize the high-dimensional information interaction structure between different resource dimensions. The architecture is as follows: Figure 2 As shown. This architecture integrates operation message attention blocks, machine message attention blocks, and worker message attention blocks to achieve deep feature extraction of multiple constraint relationships in a human-machine collaborative production environment. The multi-attention feature extractor integrates operation message attention blocks, machine message attention blocks, and worker message attention blocks. The operation message attention block is used to model the technical dependencies between processes within the same job. The machine message attention block is used to capture the dynamic competition intensity of each device for the current task to be processed; The worker message attention block is specifically designed to quantify the collaborative potential and task competition relationships among workers. It addresses the unique labor resource constraints inherent in the DFJSP-W problem. Within the worker message attention block, for any two workers… and Define competitive feature vectors Determine the degree of overlap between the two in the current set of candidate processes, combined with the worker's own hidden characteristics. and Calculate attention coefficient The calculation formula is: ; in , and These are all learnable parameters of the model. This is a vector concatenation operation; the attention coefficients are Softmax normalized and weighted to obtain worker embedding vectors that contain labor competition dynamics. Each attention block employs a multi-head attention mechanism to enhance the model's expressive power. After the layer information is propagated and updated, the system will then process the process nodes. machine nodes and worker nodes The embedding vectors are subjected to mean pooling and then concatenated to form a global state representation vector. ; Global state representation vector The calculation logic is as follows: ; Where L is the number of multi-attention layers. , , These are the embedding vectors of the process node, machine node, and worker node, updated after L-layer information propagation. , and This vector represents the number of processes, machines, and workers currently in operation. It can highly condense macro-level information such as the urgency of tasks, equipment load status, and real-time labor saturation in the production environment, providing compact and complete feature inputs for subsequent strategy generation and value assessment.

[0023] S4. Construction and Training of Offline Reinforcement Learning Model Based on Uncertainty Control To address the challenges of high-dimensional discrete action space and distribution offset in human-machine collaborative scheduling in offline environments, a quantile-aligned Actor-Critic offline reinforcement learning algorithm framework (DQUAC) based on uncertainty control is constructed. This framework extracts scheduling strategies with strong generalization capabilities from a limited static historical dataset D. The specific implementation process is as follows: Use a set of discrete quantiles Approximating the return distribution, action value The mean of all quantiles is: ; The Critic network, combined with the Dueling architecture, embeds the global state into the input state value stream. After fusing local and global features, the input action advantage stream is obtained. The estimated values ​​of each quantile are output through mean-centering aggregation operation, thereby achieving decoupled representation of state baseline and action gain. The construction and application of uncertainty weights are as follows: a lightweight uncertainty modulation mechanism is built based on the shape of the quantile distribution. Since the quantile Critic directly outputs multiple quantiles of the return distribution, its dispersion between different quantiles naturally contains uncertainty information. The variance of the value distribution is then calculated. Constructing uncertainty weighting coefficients The calculation formula is: ; ; in The modulation intensity coefficient, For numerical smoothing terms, , Trim the upper and lower limits of the weights; During the model update phase, weights are applied to the quantile Huber loss term, and a Conservative Q-Learning (CQL) regularization term is introduced into the loss function. This forces the policy to make decisions within the data support domain by suppressing the Q-value of unseen actions and increasing the value of real actions in the dataset. Combined with uncertainty weights, the final Critic loss function is formed: ; in For time-series difference loss, This is the CQL regularization loss; During the Actor policy improvement phase, the weights are used to adjust the policy gradient, so that the agent prioritizes updating actions with high confidence in value estimation, thereby avoiding the impact of erroneous high variance estimation on policy convergence.

[0024] A dense differential reward function that integrates domain knowledge, by monitoring the current scheduling state. Lower bound of the estimated maximum completion time Changes provide immediate incentives, one-step rewards. Defined as the difference between the lower bounds of the maximum completion time estimates between adjacent decision steps, i.e.: ; in The current scheduling state is as follows. This is the lower bound of the estimated maximum completion time under the current conditions; This reward design can accurately quantify the marginal contribution of each "process-machine-worker" allocation action to shortening the overall production cycle, guiding the agent to identify and learn efficient scheduling logic in complex offline data.

[0025] This method employs an offline reinforcement learning paradigm, directly mining scheduling knowledge from historical static data without requiring real-time interaction with the physical production environment during training. This effectively avoids the high interaction costs and potential production safety risks associated with online reinforcement learning, making intelligent scheduling strategies easier to deploy in human-centered Industry 5.0 environments. Furthermore, it introduces quantile regression and lightweight uncertainty modulation mechanisms. By dynamically adjusting the learning weights through calculating the variance of the value distribution, this method significantly suppresses overestimation of the value of unseen scheduling actions, ensuring that the system can make robust and reliable scheduling decisions even under complex and variable real-world conditions.

[0026] In this embodiment, the core parameters for offline training of the scheduling strategy are configured as follows: the embedding dimension of the feature extractor is set to 32, and the multi-attention layer is configured to have 2 layers; the multilayer perceptron (MLP) uses 2 hidden layers with 64 hidden units, and a Dropout layer with a probability of 0.2 is set; the Adam optimizer is used, the training batch size is 128, and the learning rate of the Actor network is 3×10.-5 The Critic network has a learning rate of 3×10⁻⁶. -4 The total number of training steps is 100,000; the number of quantiles Set to 32, CQL regularization coefficient to 1.0, temperature coefficient to 0.001, discount factor to 0.99, and uncertainty scaling factor to... The weight is set to 1.2, and the weight pruning range is limited to [0.1, 5.0]. The target network adopts a soft update mechanism with an update coefficient of 0.005.

[0027] The experimental environment was based on a computing platform equipped with an Intel Xeon Platinum 8370C CPU and an NVIDIA RTX A6000 GPU. To verify the stability of the algorithm, all evaluation metrics were based on the average results of five different random seeds. Experimental analysis shows that the MA-DQUAC algorithm proposed in this invention... It performs exceptionally well in large-scale scenarios, achieving a gap value of -16.66% under the sampling strategy, significantly outperforming traditional benchmark algorithms such as OR-Tools and DANIEL. Furthermore, in... In the large-scale generalization experiment, our method still achieved an excellent performance of -24.83%, fully demonstrating its strong generalization ability and sample utilization efficiency in complex, high-dimensional scheduling tasks. The comparison results of different algorithms are shown in Table I, and the generalization experiment results are shown in Table II.

[0028] S5, Real-time Scheduling Decision Reasoning The trained strategy model is applied to actual production scheduling to make real-time decisions on new tasks. The real-time decision-making process is as follows: After the model training is completed, the process for making real-time scheduling decisions is as follows: Collect real-time production data for production scenarios to be scheduled, including the process flow of the operations to be processed, equipment availability, worker on-duty status and skill information; Transform real-time production data into standardized heterogeneous scheduling graph state inputs; The trained multi-attention feature extractor is used to perform feature extraction and global state aggregation, generating a global state representation vector. The global state representation vector is input into the trained offline reinforcement learning model, and the optimal scheduling action is output through the Actor network, which is the joint decision result of process sequencing, machine allocation and worker assignment. Based on the decision results, a complete workshop scheduling plan is generated, outputting the start time of each process, processing equipment and operator information, to complete end-to-end intelligent scheduling.

[0029] Table I: Comparison of Benchmark Algorithms Note: Bold text represents the best result for each evaluation mode under this size instance, and red text represents the best performance result under this instance.

[0030] Table II. Generalization Experiments of Small-Scale Training Models on Large-Scale Test Instances Note: Bold text represents the best result for each evaluation mode under this size instance, and red text represents the best performance result under this instance.

[0031] This invention adopts an end-to-end deep decision-making paradigm, achieving accurate reasoning for scheduling decisions through structured graph representation and deep feature extraction. Experimental results show that in a large-scale industrial scenario of 20×10×5, the method of this invention achieves a gap value of -16.66% under the sampling strategy, significantly outperforming commercial OR-Tools solvers, the DANIEL benchmark algorithm, as well as genetic algorithms (GA) and various priority scheduling rules (PDRs). Simultaneously, the single-step decision-making time is only about 1.4 seconds, on par with traditional heuristic rules, and far lower than the hundreds of seconds required by genetic algorithms. This meets the real-time scheduling needs under dynamic production conditions, completely solving the core defects of traditional algorithms: "accurate calculations are slow, and fast calculations are inaccurate." This invention comprises a heterogeneous scheduling graph with three types of heterogeneous nodes and three types of semantic edges. It is complemented by a multi-attention feature extractor integrating worker message attention blocks, enabling parallel capture of high-dimensional interactive features related to process constraints, equipment competition, labor skill matching, and collaborative competition dynamics. Experimental results show that the representation system of this invention can effectively adapt to complex scheduling scenarios with strong human-machine coupling. In a large-scale scenario of 20×10×5 with even stronger human-machine resource constraints, the gap value optimization exceeds 218% compared to the DANIEL benchmark algorithm; in an ultra-large-scale scenario of 40×10×8, the gap value further reaches -24.83%, proving that this invention can accurately characterize the deep coupling logic between multiple resources, output a globally superior scheduling scheme, and effectively shorten the entire production cycle. This invention constructs an Actor-Critic framework based on quantile alignment. It characterizes the stochastic risk of the reward distribution through quantile regression, constructs uncertainty weights using the variance of the value distribution, and combines conservative Q-learning regularization to dynamically suppress the overestimation of the value of out-of-distribution actions. Generalization experiments show that the method of this invention has extremely strong cross-scale generalization performance. A model trained on a small 10×5×3 dataset can be directly applied to a massive 40×10×8 unseen scheduling scenario, with a gap value of -24.83%, far superior to the -4.29% of the DANIEL algorithm under the same conditions, while maintaining a real-time inference speed of 3 seconds. In contrast, traditional metaheuristic algorithms require time-consuming iterative searches for massive scenarios, and online reinforcement learning algorithms are prone to a sharp decline in performance in unseen scenarios. The method of this invention can maintain robust and high-quality scheduling decisions under complex and ever-changing real-world conditions, perfectly adapting to the core requirements of flexible production with multiple varieties and variable batches.

[0032] Meanwhile, this invention employs an offline reinforcement learning paradigm, directly mining decision-making knowledge from historical static scheduling data in the workshop. The training process requires no real-time interaction with the physical production environment, completely avoiding the safety hazards and economic losses such as production line downtime and product scrap caused by online trial and error. Furthermore, the method of this invention has extremely high sample utilization efficiency, requiring only a small number of historical scheduling instances to achieve stable convergence. It eliminates the need to build a costly digital twin simulation environment, significantly reducing the deployment threshold and implementation cost of intelligent scheduling systems, perfectly adapting to the human-centric production scenarios of Industry 5.0.

[0033] This invention forms a complete technical closed loop, from historical data acquisition, heterogeneous state representation, and deep feature extraction to offline policy learning and real-time inference decision-making. It eliminates the need for manual scheduling rule design and can be directly adapted to flexible human-machine collaborative workshop scenarios of different scales and industries. Experimental verification shows that the method of this invention maintains leading solution performance and real-time performance across a full range of scenarios, from small-scale (10×5×3) to ultra-large-scale (40×10×8). Compared with existing technologies, it achieves comprehensive breakthroughs in four core dimensions: scheduling accuracy, generalization ability, deployment cost, and response speed. It can provide core intelligent decision-making technology support for the transformation of modern manufacturing towards intelligence and flexibility.

[0034] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0035] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning, characterized in that: Includes the following steps: S1. Construct a scheduling model for a flexible workshop with dual resources in a human-machine collaborative environment. The scheduling objective is to determine the optimal start time, processing machine, and operator for all processes to minimize the maximum completion time and satisfy process pre-sequence constraints, resource uniqueness constraints, and resource compatibility constraints. S2. Define the instantaneous state of the scheduling system as a heterogeneous scheduling graph. The heterogeneous scheduling graph includes three types of heterogeneous nodes: process nodes, machine nodes, and worker nodes, as well as edges used to describe the relationships between nodes. At the same time, construct corresponding high-dimensional feature tensors for each type of node and edge of the heterogeneous scheduling graph as the underlying input for feature extraction. S3. Construct a multi-attention feature extractor based on heterogeneous scheduling graphs to perform deep feature extraction on the heterogeneous scheduling graphs of the current scheduling state and generate a global state representation vector. S4. Construct an offline reinforcement learning algorithm framework for quantile alignment Actor-Critic based on uncertainty control. Use historical static datasets to train the scheduling strategy. In this framework, quantile regression is introduced to represent the reward distribution, and the variance of the value distribution is used to construct uncertainty weights to suppress the overestimation of the value of out-of-distribution actions in offline learning. S5. Apply the trained strategy model to actual production scheduling to make real-time decisions on new tasks.

2. The human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning according to claim 1, characterized in that: The specific construction method of the dual-resource flexible job shop scheduling problem model in step S1 is as follows: Define the set of jobs to be processed Machine resource collection A group of workers with different skills Each assignment It consists of a series of processes with strict sequential constraints. composition; Define process Total occupied time ,in For machine processing time, Set the time for workers on the corresponding machines; The scheduling objective is to minimize the maximum completion time. ,in For process The completion time.

3. The human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning according to claim 1, characterized in that: In step S1: The pre-sequence constraint is: the start time of the subsequent process in the same operation. , This is the start time of the subsequent process; The resource uniqueness constraint is: at any time t, each machine can only process one process, and each worker can only operate one machine; The resource compatibility constraint is: the machine assigned to a process belongs to its compatible machine set. The assigned workers belong to its skill-matched worker set. .

4. The human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning according to claim 1, characterized in that: In step S2: Heterogeneous scheduling graph is represented as heterogeneous node set , For process node set, For machine node set, For worker node set; Edge set It includes three types of semantic edges: process-process edges, process-machine edges, and machine-worker edges; The process-process boundary describes the technological topology logic of processes within the same operation; the process-machine boundary defines the processing feasibility of the machine for the process; and the machine-worker boundary describes the matching relationship between the worker's operating authority and skills.

5. The human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning according to claim 4, characterized in that: The high-dimensional feature tensor in step S2 specifically includes: 10-dimensional process node features Includes indicators such as task urgency and remaining workload; 8-dimensional machine node features This includes real-time equipment load, availability status, and processing efficiency characteristics; 6-dimensional worker node features Includes the number of available candidate operations, skill coverage, average setup time, and real-time idle time; 4D Machine-Worker Features This includes relative set time, percentage of worker-specific remaining tasks, and worker execution efficiency; The multidimensional feature tensors together constitute the underlying input for the scheduling agent's decision-making.

6. The human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning according to claim 1, characterized in that: In step S3, the multi-attention feature extractor integrates the operation message attention block, the machine message attention block, and the worker message attention block. The operation message attention block is used to model the technical dependencies between processes within the same job; the machine message attention block is used to capture the dynamic competitive intensity of each device for the current task to be processed; and the worker message attention block is used to quantify the collaborative potential and task competition relationship among workers. Each attention block adopts a multi-head attention mechanism. After multiple layers of information propagation and updating, the embedding vectors of process nodes, machine nodes, and worker nodes are averaged and concatenated to generate a global state representation vector.

7. The human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning according to claim 6, characterized in that: In the worker message attention block, for any two workers and Define competitive feature vectors Determine the degree of overlap between the two in the current set of candidate processes, combined with the worker's own hidden characteristics. and Calculate attention coefficient The calculation formula is: ； in , and These are all learnable parameters of the model. This is a vector concatenation operation; the attention coefficients are Softmax normalized and weighted to obtain worker embedding vectors that contain labor competition dynamics. Global state representation vector The calculation logic is as follows: ； Where L is the number of multi-attention layers. , , These are the embedding vectors of the process node, machine node, and worker node, updated after L-layer information propagation. , and These represent the number of processes, machines, and workers currently in operation.

8. The human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning according to claim 1, characterized in that: In step S4, the quantile alignment of the Critic network within the Actor-Critic framework uses a set of discrete quantiles. Approximating the return distribution, action value The mean of all quantiles is: ； The Critic network, combined with the Dueling architecture, embeds the global state into the input state value stream. After fusing local and global features, the input action advantage stream is obtained. The estimated values ​​of each quantile are output through mean-centering aggregation operation, thereby achieving decoupled representation of state baseline and action gain. The construction and application of uncertainty weights are as follows: calculate the variance of the value distribution. Constructing uncertainty weighting coefficients The calculation formula is: ； ； in The modulation intensity coefficient, For numerical smoothing terms, , Trim the upper and lower limits of the weights; During the model update phase, weights are applied to the quantile Huber loss term, combined with a conservative Q-learning CQL regularization term, to form the final Critic loss function: ； in For time-series difference loss, This is the CQL regularization loss; During the Actor policy improvement phase, the weights are used to adjust the policy gradient, so that the agent prioritizes updating actions with high confidence in value estimation.

9. A human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning according to claim 1, characterized in that: In step S4, a dense differential reward function that integrates domain knowledge is used to guide the training strategy, with a single-step reward. Defined as the difference between the lower bounds of the maximum completion time estimates between adjacent decision steps, i.e.: ； in The current scheduling state is as follows. This is the lower bound of the estimated maximum completion time under the current conditions.

10. The human-machine collaborative flexible job shop scheduling method based on offline reinforcement learning according to claim 1, characterized in that: The real-time decision-making process in step S5 is as follows: Real-time collection of work order information, equipment status, worker on-duty and skill information from the production site to construct corresponding heterogeneous scheduling diagrams; Input the heterogeneous scheduling graph into the multi-attention feature extractor to generate a global state representation vector; Input the global state representation vector into the trained policy model, and output the current optimal scheduling action end-to-end, including the selection of the process to be processed, the allocation of processing machines and the assignment of operators. After executing the scheduling action, update the production status and repeat the above process until all work orders are completed.

Images