A resource state prediction and deep reinforcement learning scheduling fusion workshop active scheduling method, device and readable storage medium
By integrating resource state prediction with deep reinforcement learning scheduling, this method uses cyclic graph convolutional neural networks and Transformer models to predict machine failures, and combines Markov decision models for workshop load balancing and job allocation. This solves the problem of untimely disturbance response in dynamic scheduling and achieves efficient proactive scheduling and fault avoidance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2025-06-30
- Publication Date
- 2026-06-09
AI Technical Summary
Existing dynamic scheduling technologies are slow to respond to disturbances and are passive, failing to effectively avoid the impact of uncertain disturbances in production systems.
A method combining resource state prediction and deep reinforcement learning scheduling is adopted. Machine failure time is predicted by recurrent graph convolutional neural network and Transformer model, and Markov decision model is combined for workshop load balancing and workpiece allocation. A dynamic splicing and parsing graph model is designed to achieve active scheduling.
It improves the real-time nature and accuracy of scheduling decisions, effectively avoids machine failure disturbances, enhances production efficiency and equipment utilization, and reduces downtime and production costs.
Smart Images
Figure CN120764935B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent manufacturing and dynamic workshop scheduling, and more specifically, relates to a workshop active scheduling method, device and readable storage medium that integrates resource state prediction and deep reinforcement learning scheduling. Background Technology
[0002] Production scheduling is a crucial aspect of manufacturing systems, and efficient scheduling optimization techniques are key to improving the optimality of production systems, directly impacting enterprise efficiency and competitiveness. In the context of economic globalization, distributed manufacturing has become a mainstream manufacturing model, with significant applications in industries such as aerospace, consumer electronics, and automotive manufacturing. Distributed job shops are a typical distributed manufacturing model, widely present in various discrete manufacturing industries, requiring consideration of production collaboration among multiple job shops. Efficiently solving the Distributed Job-shop Scheduling Problem (DSP) can significantly improve the production efficiency of distributed job shops.
[0003] A production workshop is essentially a complex manufacturing system. Workshop manufacturing resources are the core elements supporting production activities, encompassing five key elements: materials, machinery, personnel, methods, and environment. Workshop resource status analysis involves real-time monitoring of the operational data of equipment, materials, and personnel to dynamically optimize production scheduling, thereby improving equipment utilization, reducing inventory costs, and accelerating anomaly response, supporting flexible production and intelligent decision-making. Among these, processing resources (machines and equipment) are fundamental elements. Through real-time monitoring and algorithmic prediction of equipment health status, a shift from "passive maintenance" to "proactive prevention" is achieved, effectively reducing downtime due to malfunctions and the resulting decrease in production efficiency, thus ensuring product quality.
[0004] In the context of smart manufacturing, the Internet of Things (IoT) covers the entire production workshop, and sensor technology enables real-time monitoring of the production process, allowing workshop resource data to be acquired, transmitted, and stored. Analyzing historical and real-time workshop data to uncover production information characteristics, predicting upcoming anomalies, and proactively adjusting scheduling plans to avoid these anomalies is a crucial approach to mitigating the impact of uncertain disturbances in the production system. Proactive scheduling technology is a significant technological requirement in the context of smart manufacturing.
[0005] Therefore, there is an urgent need to design a resource state prediction technology based on deep learning for processing resources such as machines, in order to proactively avoid the impact of disturbances. Summary of the Invention
[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a workshop active scheduling method, apparatus, and readable storage medium that integrates resource state prediction and deep reinforcement learning scheduling. Its purpose is to solve the problems of untimely disturbance response and passive disturbance response in existing dynamic scheduling technologies.
[0007] To achieve the above objectives, according to one aspect of the present invention, a workshop active scheduling method integrating resource state prediction and deep reinforcement learning scheduling is provided. Based on the current dynamic splicing analysis graph and information on unscheduled workpieces in the production system, the method completes the workshop allocation of newly arriving workpieces and the selection of workpieces on idle machines. The dynamic splicing analysis graph is dynamically improved according to the following process: for each arriving workpiece, determining which workshop it is allocated to; for each machine, if the current workpiece has been processed, immediately selecting the next workpiece to be processed from its buffer; and gradually improving the dynamic splicing analysis graph based on the workpiece workshop allocation and the workpiece processing order on each machine; including:
[0008] S1 uses a workpiece workshop allocation rule based on workshop load balancing to allocate newly arriving workpieces to the corresponding workshops;
[0009] S2 Based on the process characteristics of the workpiece, place it into the corresponding machine buffer zone of the selected workshop;
[0010] S3 uses a recurrent graph convolutional neural network to extract and fuse features in the time and spatial domains of the dynamically stitched parsed graph at the current time, and inputs the obtained information into a scheduling decision network based on a multilayer perceptron.
[0011] S4 Calculate the probability of a workpiece being selected in the buffer of each machine using the scheduling decision network. The workpiece with the highest probability is the workpiece being processed on the current machine at the current moment.
[0012] S5 uses a resource state prediction method based on convolutional neural networks and Transformers to predict the failure time of the machine corresponding to the currently selected workpiece.
[0013] S6 adopts a shop floor proactive scheduling strategy that integrates resource state prediction and deep reinforcement learning scheduling. Specifically, it compares the estimated completion time of the selected workpiece on the current machine with the predicted machine failure time. If the estimated completion time of the workpiece is greater than the predicted machine failure time, machine maintenance is performed. After the machine maintenance is completed, the selected workpiece is processed again. Otherwise, the selected workpiece is processed immediately.
[0014] S7 Updates the dynamic splicing and parsing diagram based on the result of step S6, completing the workpiece selection and processing decision for this time;
[0015] S8 Repeat steps S1 to S7 until all production tasks are completed and a complete scheduling plan is obtained.
[0016] Furthermore, in step S1, the workpiece is allocated to the workshop using the workpiece workshop allocation rule based on workshop load balancing: when a workpiece arrives at the production system, it is immediately allocated to the workshop with the smallest number of workpieces at the current moment for processing.
[0017] Furthermore, in step S4, the decision-making process of the scheduling decision network is based on a Markov decision model of the problem to be solved, including a state space, an action space, a reward function, and a state transition process.
[0018] Furthermore, the state space includes the process nodes in each machine's buffer, the process nodes that each machine is currently processing, and the process nodes that each machine has recently completed.
[0019] The action space includes all operations within the time window, that is, all operations that have been completed on all machines, operations that are being processed, and all operations within the machine buffer.
[0020] The reward function is as follows:
[0021]
[0022] in, It's a reward value. The current machine Cumulative running time, It is the current time;
[0023] The state transition process is the transfer of actions and the change of workshop state.
[0024] Further, in step S2, the recurrent graph convolutional neural network extracts the workshop state features in the following manner:
[0025] For each node in the dynamically stitched parsing graph:
[0026]
[0027] in, It is a graph convolution operator. Indicates to Time characteristics Performing graph convolution operations and extracting features at both the temporal and spatial thresholds yields the temporal and spatial features of the workshop state; the convolution kernel is a graph Laplacian operator. The function has Chebyshev coefficients as follows: ;symbol This represents the Hadamard product. , and These represent the input gate, forget gate, and output gate, respectively. , , , These are weighting coefficients. and It is the bias, where the subscript is... express , and ; sigmoid functions and The function is an activation function.
[0028] Furthermore, in step S4, at the decision point At any given time, the scheduling decision network will use the output of the cyclic graph convolutional neural network. As a basis for decision-making, the workpieces in the action space are scored, and the unselectable workpieces are blocked by setting a mask.
[0029] Furthermore, in step S5, the machine's remaining lifespan is predicted using CNN and Transformer to calculate the machine's failure point. The machine's failure point is when its remaining lifespan is 0.
[0030] Furthermore, in step S6, the estimated completion time of the selected workpiece on the current machine refers to the current time plus the processing time of the selected workpiece on the current machine, which is determined by the workpiece's technological characteristics.
[0031] According to another aspect of the invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the workshop active scheduling method as described in any of the preceding claims.
[0032] According to another aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the workshop active scheduling method as described in any of the preceding claims.
[0033] In summary, the technical solutions conceived in this invention, compared with the prior art, can achieve the following beneficial effects:
[0034] 1. Based on the distributed and dynamic characteristics of distributed workshops under the constraints of random workpiece arrival and machine failure, this invention designs a dynamic splicing and parsing graph model, which effectively adapts to the real-time and variability of the workshop scheduling problem through dynamic updates.
[0035] 2. Based on the distributed characteristics of workshops, this invention designs a workpiece workshop allocation rule based on workshop load balancing. This allocation rule is simple, has a small computational load, and can maintain a high decision-making speed while ensuring load balancing, which is conducive to realizing a real-time and efficient scheduling process.
[0036] 3. Based on the real-time and efficiency requirements of actual production for scheduling decisions, this invention constructs a workshop state feature extraction method based on a cyclic graph convolutional neural network. This method not only integrates the spatial features of each process node of the decision point, but also establishes the temporal correlation between different decision points, providing a comprehensive and profound decision basis for the scheduling decision-making agent.
[0037] 4. The resource status prediction method based on CNN and Transformer designed in this invention can predict the remaining lifespan of a machine based on machine lifecycle data (real-time machine status), thereby determining the machine fault point. It effectively processes multi-dimensional and complex machine status data, realizes the effective fusion of local and global features of machine status, improves the prediction accuracy of the model, and provides a reliable decision basis for proactive scheduling decisions.
[0038] 5. The workshop active scheduling strategy that integrates resource state prediction and deep reinforcement learning scheduling constructed in this invention makes full use of the prediction results of the resource state prediction model and the workpiece selection results of the scheduling decision model, and is the core link to actively avoid machine failure disturbances.
[0039] 6. To further improve the intelligence and automation of scheduling decisions, this invention uses deep reinforcement learning algorithms for scheduling decisions and establishes a Markov decision model for the dynamic DJSP to be solved, modeling the problem to be solved as a sequence decision problem. Attached Figure Description
[0040] Figure 1 This is a preferred embodiment of the present invention, which constructs a workshop active scheduling method and system that integrates resource state prediction and deep reinforcement learning scheduling.
[0041] Figure 2 This is a dynamically spliced parsing diagram with a variable structure in a preferred embodiment of the present invention. The example in the diagram includes two workshops, and each workpiece has four processes. At any given time, the production system contains only one workpiece; At any given time, the production system contains two workpieces, and workpiece 1 is assigned to workshop 1; At any given time, the production system contains 3 workpieces. Workpieces 1 and 3 are processed in workshop 1, and workpiece 2 is processed in workshop 2.
[0042] Figure 3 This is a schematic diagram of the key steps of the resource state prediction algorithm based on CNN and Transformer in a preferred embodiment of the present invention, which includes three important components: data preprocessing, feature extraction network, and linear regression network.
[0043] Figure 4This is a diagram of the resource state prediction network structure based on CNN and Transformer in a preferred embodiment of the present invention;
[0044] Figure 5 This is a comparison diagram of the workshop active scheduling strategy and the online scheduling strategy that integrates resource state prediction and deep reinforcement learning scheduling in a preferred embodiment of the present invention.
[0045] Figure 6 This is an experimental result graph verifying the effectiveness of the proactive scheduling strategy in a distributed production scenario with different numbers of workshops in the preferred embodiment. It shows the production indicators of different workshops, such as total delay time. Maximum completion time and the number of workpieces whose processing was interrupted Box plots and win rate pie charts of the relative percentage increase (RPI) of the experimental results of the scheduling decision-making agent described in this invention under active scheduling mode and online scheduling mode, under different scenarios with different numbers of workshops. Detailed Implementation
[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0047] This invention preferably proposes an active job shop scheduling method that integrates resource state prediction and deep reinforcement learning scheduling. Based on the actual production scenario and the technological characteristics of the processing tasks, a dynamic mosaic disjunctive graph is drawn to solve the dynamic distributed job shop scheduling problem. When a processing task (i.e., a workpiece) arrives at the production shop, it determines which shop the workpiece will be assigned to. Each machine, upon completing the processing of its current workpiece, immediately selects the next workpiece from its buffer. That is, based on the current dynamic mosaic disjunctive graph and information on unscheduled workpieces in the production system, the shop assignment of newly arrived workpieces and the selection of workpieces on idle machines are completed. Preferably, this active scheduling method includes the following steps:
[0048] S1 uses a workpiece workshop allocation rule based on workshop load balancing to allocate newly arriving workpieces to appropriate workshops;
[0049] S2 Based on the process characteristics of the workpiece, place it in the buffer zone of the corresponding machine in the selected workshop;
[0050] S3 uses a recurrent graph convolutional neural network to extract features in the temporal and spatial domains of the structurally variable splicing image at the current time, and inputs the obtained fused information into a scheduling decision network based on a multilayer perceptron (MLP).
[0051] S4 Calculate the probability of a workpiece being selected in the buffer of each machine using the scheduling decision network, and select the workpiece with the highest probability as the workpiece to be processed by the current machine.
[0052] S5 uses a resource status prediction method based on CNN and Transformer to predict the failure time of each machine;
[0053] S6 adopts a shop floor proactive scheduling strategy that integrates resource state prediction and deep reinforcement learning scheduling. Specifically, it compares the estimated completion time of the selected workpiece on each machine (the current time plus the processing time of the selected workpiece on the current machine, which is determined by the workpiece's technological characteristics) with the predicted machine failure time. If the estimated completion time of the workpiece is greater than the predicted machine failure time, machine maintenance is performed proactively. The selected workpiece is then processed after the machine maintenance is completed. Otherwise, the selected workpiece is processed immediately.
[0054] S7 Updates the dynamically stitched parsing image based on the result of step S6;
[0055] S8 Repeat steps S1 to S7 until all production tasks are completed and a complete scheduling plan is obtained.
[0056] Preferably, the updating of the dynamic splicing parsing graph includes determining which workshop each arriving workpiece is assigned to; on each machine, if the current workpiece is completed, the next workpiece to be processed is immediately selected from its buffer; and gradually forming a complete scheduling scheme dynamic splicing parsing graph based on the workshop allocation of the workpieces and the workpiece processing order on each machine.
[0057] Preferably, step S1 designs a workpiece workshop allocation rule based on workshop load balancing for workpiece workshop allocation. When a workpiece arrives at the production system, it is immediately allocated to the workshop with the fewest workpieces at the current moment for processing. This allocation rule is simple, has low computational load, and can maintain a high decision-making speed while ensuring load balancing, which is conducive to achieving a real-time and efficient scheduling process.
[0058] Preferably, in subsequent steps, step S3 uses a recurrent graph convolutional neural network to mine the temporal and spatial features of the workshop state at the scheduling decision point; then, an MLP-based scheduling decision model is used for workpiece selection to achieve real-time and efficient scheduling decisions; step S4 establishes a Markov decision model for the problem to be solved, including time window-based workshop state observation, actions, a reward function based on machine utilization, and real-time state transition processes, modeling the problem to be solved as a sequential decision problem; step S5 designs a resource state prediction model based on CNN and Transformer to effectively predict the failure time of each machine in the distributed workshop; simultaneously, machine failure prediction is performed in conjunction with the resource state prediction method. Finally, step S6 combines the machine failure prediction model and the scheduling decision model to construct a workshop proactive scheduling method that integrates resource state prediction and deep reinforcement learning scheduling.
[0059] More preferably, in step S1, the workpiece workshop allocation rule based on workshop load balancing is described as follows: when a workpiece arrives at the production system, it is immediately allocated to the workshop with the fewest workpieces at the current moment. This allocation rule is simple, has low computational load, and can maintain a high decision-making speed while ensuring load balancing, which is conducive to achieving real-time and efficient scheduling.
[0060] More preferably, in step S3, the distributed workshop state feature extraction method based on recurrent graph convolutional neural networks is described as follows:
[0061] For each node (Each node corresponds to a workpiece operation), and it contains 5 features, namely:
[0062] (1) Node corresponding to process Processing time .
[0063] (2) Node corresponding to process Processing machines.
[0064] (3) Node corresponding to process Estimated completion time At the current moment, if the process... If processing has been completed, then This is equal to its actual completion time. If the process... If it has not been processed, its estimated completion time is equal to the estimated completion time of its predecessor process plus its own processing time. If It is a workpiece The first step, namely Then its estimated completion time is equal to the current time. Add process time .
[0065] (4) Node corresponding to process Estimated start time of processing At the current moment, if the process... If the processing has been completed, then This equals the actual start time of processing. If the process... If it has not yet been processed, then It is a workpiece The first step, namely The estimated start time of processing is the estimated completion time of the previous process.
[0066] (5) Node corresponding to process Estimated waiting time It is equal to the estimated start time of the process and the time of the workpiece. The difference in arrival time.
[0067] Regarding the aforementioned node characteristics,
[0068]
[0069] in, It is a graph convolution operator. Indicates to Time characteristics Graph convolution operations are performed, with the convolution kernel being the graph Laplacian operator. The function has Chebyshev coefficients as follows: .symbol This represents the Hadamard product. , and These represent the input gate, forget gate, and output gate, respectively. , , , These are weighting coefficients. and It is the bias, where, "express" " "and" ". sigmoid functions and The function is a commonly used activation function.
[0070] More preferably, in one specific embodiment, the MLP-based scheduling decision network comprises six fully connected layers (excluding the input layer), with each hidden layer having 32 neurons. Decision point At any moment, the network The output of the feature extraction recurrent graph convolutional neural network As a decision-making basis, workpieces in the motion space are scored, and unselectable processes are masked by setting a minimum value. The function normalizes the selection probability of all actions to between 0 and 1, so that unselectable workpieces receive a score of 0 and are therefore not selected. It should be understood that the number and structure of the fully connected layers and hidden layers can be adjusted according to the actual situation.
[0071] More preferably, in step S5, the resource state prediction method based on CNN and Transformer calculates the machine failure point by predicting the remaining machine lifetime; the machine failure point is defined as the remaining machine lifetime being 0. The resource state prediction method comprises three parts: data preprocessing, a machine state feature extraction network based on CNN and Transformer, and a linear regression prediction network.
[0072] More preferably, in step S6, the active scheduling strategy for the workshop, which integrates resource state prediction and deep reinforcement learning scheduling, is a key step in the active scheduling algorithm and is crucial for avoiding machine failure disturbances. Its simplest operation is to directly compare the estimated completion time and predicted machine failure time of the selected workpieces on each machine and manually select maintenance and processing strategies based on the comparison results, or to preset corresponding rules for automatic execution by the machine.
[0073] More preferably, in step S4, the scheduling decision network agent based on MLP includes the design of the workshop state, action space, reward function, and state transition process.
[0074] The invention will be further explained below with reference to a specific application scenario:
[0075] The dynamic stitching parsing model of this invention is a dynamic stitching parsing model for DJSP under the constraints of random workpiece arrival and machine faults. Preferably, this invention employs a more flexible dynamic stitching parsing model. :
[0076]
[0077] in, This represents a set of nodes dynamically assembled from the parsed graph, where each node represents a process step. ~ Indicates workshop 1~ f The connecting arcs, each representing a process feature of the workpiece. ~ Indicates workshop 1~ f The processing sequence of workpieces on the same machine. f Indicates the number of workshops.
[0078] Its topology changes dynamically as the workpiece arrives. Unlike traditional disjunctive map models, this embodiment preferably uses a dynamically stitched disjunctive map model. It does not include virtual nodes that mark the start and end of the process, i.e. , m This refers to the number of processes involved in each workpiece. n This refers to the total number of parts in the order. Taking a problem with 2 workshops and 4 processes per part as an example, such as... Figure 2 As shown, the processing steps in workshop 1 are filled with diagonal lines, the processing steps in workshop 2 are filled with grids, and the processing steps on the same machine are marked by the same rings. For example, dashed rings mark the processing steps on machine 1, solid rings mark the processing steps on machine 2, single-dot dashed rings mark the processing steps on machine 3, and double-dot dashed rings mark the processing steps on machine 3. At time 1, workpiece 1 is in a state where it has arrived at the production system but has not yet been assigned. Since the decision time for workpiece assignment is not considered, then... This is the arrival time of workpiece 1. At that moment, workpiece 2 arrived, and workpiece 1 had already been assigned to workshop 1. At any given time, workpieces 1 and 3 are being processed in workshop 1. The processing sequence of each process on the corresponding machine is marked by a dashed arrow of the corresponding color. For example, on machine 1 in workshop 1, the process... and They are processed sequentially; workpiece 2 is processed in workshop 2, and its first process begins on machine 3.
[0079] Then, the following method is used to obtain a complete workshop active scheduling scheme by dynamically updating and improving the dynamically stitched parsing map:
[0080] S1 adopts a workpiece workshop allocation rule based on workshop load balancing to allocate newly arrived workpieces to suitable workshops. The workpiece workshop allocation rule based on workshop load balancing is that when a workpiece arrives at the production system, it is immediately allocated to the workshop with the smallest number of workpieces at the current moment.
[0081] S2. Based on the workpiece's technological characteristics, the workpiece is placed in the corresponding machine buffer zone of the selected workshop;
[0082] S3 employs a recurrent graph convolutional neural network to extract features in the time and spatial domains from the dynamically stitched parsed graph at the current moment. The resulting fused information is then input into a scheduling decision network based on a multilayer perceptron. The scheduling decision process is based on the constructed Markov decision model of DJSP under the constraints of random job arrival and machine failure, as described in detail below:
[0083] Design an algorithm based on recurrent graph convolutional neural networks and MLP to solve the problem:
[0084] First, a Markov decision model based on the proposed dynamic splicing disjunctive graph is established, which typically consists of a state space, an action space, a reward function, a state transition process, and a discount factor.
[0085] 1) State:
[0086] Workshop condition observation based on time windows.
[0087] The problem to be solved is a real-time scheduling decision problem, which is typically short-sighted. To ensure production continuity, it requires real-time decision-making, thus focusing more on information close to the decision point. Production information from earlier moments may have very little impact on the current decision. Based on this, this invention designs a feature observation method based on a time window. In a distributed workshop, when a machine completes its current process, it needs to select the next process from its buffer. Obviously, the process information in the machine buffer is crucial for scheduling decisions. Therefore, the agent's observation scope includes the process nodes in each machine's buffer, the process nodes currently being processed by each machine, and the most recently completed process node for each machine. For the "most recently" completed process of each machine, a time window with a length of [missing information] is introduced. The time window is used to adjust the number of processes that each machine has recently completed, as observed by the intelligent body.
[0088] At the scheduling decision point Time, State It contains all information about the dynamically stitched parsed graph within the time window, i.e., the set of connected arcs. Workpiece process path, breakout arc The processing sequence of workpieces on each machine. Any node within the time window. It includes 5 features, described in detail below:
[0089] (a) Node corresponding to process Processing time, ; i For process numbering, j Number the workpiece.
[0090] (b) Node corresponding to process Processing machines.
[0091] (c) Node corresponding to process Estimated completion time At the current moment, if the process... If processing has been completed, then This is equal to its actual completion time. If the process... If it has not been processed, its estimated completion time is equal to the estimated completion time of its predecessor process plus its own processing time. If It is a workpiece The first step, namely Then its estimated completion time is equal to the current time. Add process time .
[0092] (d) Node corresponding to process Estimated start time of processing At the current moment, if the process... If the processing has been completed, then This equals the actual start time of processing. If the process... If it has not yet been processed, then It is a workpiece The first step, namely The estimated start time of processing is the estimated completion time of the previous process.
[0093] (e) Node corresponding to process Estimated waiting time It is equal to the estimated start time of the process and the workpiece. The difference in arrival time.
[0094] 2) Action:
[0095] The scheduling decision-making agent selects actions based on the observed workshop state and executes the selected action to enter the next state. In this invention, the action space is closely related to the time window, which includes all processes within the time window, namely, all completed processes, processes currently being processed, and all processes within the machine buffer on all machines. When a machine is idle, it selects a process from its buffer for processing. At this time, all completed processes, processes currently being processed, and processes within the buffers of other machines are unselectable processes; therefore, a mask is used to set the selection probability of these processes to 0.
[0096] 3) Reward function:
[0097] Reward Value This refers to the evaluation of the scheduling decision-making agent's choice of the current action in the current state. A well-designed reward mechanism can guide the agent to make decisions that are beneficial to goal optimization. This invention designs a reward function based on machine utilization, which is closely related to minimizing total time delay. As shown in the following formula,
[0098]
[0099] in, The current machine Cumulative running time, This is the current time. Higher machine utilization indicates less machine downtime, which helps to complete processing tasks more efficiently and reduce total lead time.
[0100] 4) State transition process ( State Transition This refers to the transfer of actions and changes in the workshop state:
[0101] The scheduling decision-making agent selects an action from the action space, and the corresponding process immediately begins processing. After the process is completed, the workpiece is moved to the next machine buffer to await processing according to the corresponding workpiece's process path, and the workshop state is updated. This process continues until all processes for the current workpiece are completed.
[0102] S4 Calculate the probability of a workpiece being selected in the buffer of each machine using the scheduling decision network. The workpiece with the highest probability is the workpiece being processed on the current machine at the current moment.
[0103] S5 designs a resource state prediction model based on CNN and Transformer to predict the current machine failure time. The core of this step is to train a deep learning model based on a dataset showing the correspondence between bearing vibration signals and bearing lifespan, resulting in an agent capable of predicting bearing lifespan based on bearing vibration signals. As a preferred example, the resource state prediction model based on CNN and Transformer in this embodiment is described as follows:
[0104] This embodiment analyzes the machine bearing status in the classic PHM 2012 dataset and designs a resource status prediction model based on CNN and Transformer, such as... Figure 3 As shown, the prediction model mainly includes three important steps: data preprocessing, data feature extraction, and regression prediction.
[0105] Data preprocessing:
[0106] This invention selects 6 sets of data from the 7 sets of bearing data in the PHM 2012 dataset as training data for the resource status prediction model, and uses the remaining set as test data for model performance. The horizontal and vertical vibration signals of the bearing data are both one-dimensional time-domain signals, but CNN networks are better suited to processing two-dimensional image data. Therefore, this invention uses Continuous Wavelet Transform (CWT) to transform the time-domain signals into a time-frequency graph as input to the CNN network. CWT is a method for simultaneously analyzing signals in the time and frequency domains. It transforms the signal using wavelet functions at different scales and locations to obtain the local characteristics of the signal, as expressed by the formula:
[0107]
[0108] in, This indicates that the original signal, after being stretched by a variable of... The translation variable is The signal obtained after transformation, It's about time. t The input signal, Indicates the mother wavelet The scaling and translation transformations are performed on the top, where, It is a scaling parameter that is non-zero, controlling the scaling of the wavelet. These are translation parameters that control the wavelet's position on the time axis. This chapter uses the Morlet wavelet as the mother wavelet. It combines cosine waves and Gaussian functions, making it suitable for frequency analysis. The expression is:
[0109]
[0110] in, Indicates the center frequency. Represents an imaginary number.
[0111] The input to CWT is the average of 20 consecutive data points of the horizontal or vertical vibration signal for each bearing. The CWT result is scaled to the range [0, 1] using the "Min-Max" normalization method shown in the following formula.
[0112]
[0113] in, Scaled two-dimensional time-domain and frequency-domain signals, This represents the two-dimensional time-domain and frequency-domain signal before scaling. This represents the maximum value of the two-dimensional time-domain and frequency-domain signal before scaling. This represents the minimum value of the two-dimensional time-domain and frequency-domain signal before scaling.
[0114] The vibration signal of rolling bearings in the PHM 2012 dataset is a long-term time series signal. It's difficult to directly use the entire time series signal during model training. This increases the computational load during training and the long-distance signal may not significantly impact the current prediction result, hindering model optimization. Therefore, this invention employs the sliding window method, a common data processing method in predicting the remaining life of rolling bearings. The two-dimensional time-frequency domain signal is then processed... The model divides the data into fixed-length subsequences (samples) and predicts the remaining machine lifetime for the next time step from these subsequences, moving one time step at a time. The sliding window length is obtained through model parameter tuning experiments.
[0115] The PHM 2012 dataset contains full lifecycle data for seven bearings, with varying lifespans for different bearings. If the actual time steps of the bearings are used directly... Modeling can lead to inconsistencies in the time scale of the data. To facilitate analysis and comparison, the lifespan of different bearings needs to be normalized to between 0 and 1. This invention uses the following formula to calculate the percentage of bearing lifespan.
[0116]
[0117] in This is the time step (or data acquisition sequence number) of the current bearing data sample. This represents the current total lifespan of the bearing, i.e., the time step before failure. This represents the cumulative percentage of lifespan consumed. Using the calculated percentage as the label (target value) of the training set standardizes the bearing degradation process, facilitating comparison and modeling of the lifespans of different bearings on the same scale. In the model prediction process, as shown in the following formula, [the formula is used]. Prediction results Calculate the predicted remaining bearing life This allows us to predict the bearing failure point.
[0118]
[0119] Resource state prediction model based on CNN and Transformer (CNN-Trans):
[0120] Depend on Figure 3 It is understood that the prediction network of the present invention includes a local feature extraction network based on CNN, a global feature extraction network based on Transformer to mine time dependencies, and a linear regression prediction network.
[0121] like Figure 4 As shown, the CNN-based local feature extraction network comprises three two-dimensional convolutional layers. The first convolutional layer has 2 input channels (in_channels) and 16 output channels (out_channels); the second convolutional layer has 16 input channels and 32 output channels; and the third convolutional layer has 32 input channels and 64 output channels. Each convolutional layer has a kernel size of 3, a kernel stride of 2, and image padding of 1. The activation function for each convolutional layer is ReLU. The Flatten function is used to flatten the multidimensional image data output from the convolutional layers into a format suitable for the input of the fully connected layers.
[0122] The global feature extraction network based on Transformer consists of four Transformer encoding layers. Each Transformer encoding layer has the same structure, containing two sub-layers: a multi-head self-attention layer and a feedforward neural network. Each sub-layer is followed by a residual connection layer and normalization. The multi-head self-attention mechanism performs multiple independent queries on the input. Q ), key K ) and value (Value, V The transformation of ) forms multiple subspace representations:
[0123]
[0124] in For the number of attention heads. , and These are all learnable parameter matrices. Next, the parameters for each head are calculated separately. The attention output, that is:
[0125]
[0126] in This is the dimension of each attention head, and its calculation formula is as follows:
[0127]
[0128] This refers to the feature dimensions of the input data to the Transformer network. Concatenating the outputs of multiple attention heads yields the outputs of all heads, i.e.:
[0129]
[0130] in, Concat The parentheses () function combines multiple strings together to form a single larger string. These are learnable parameters used to integrate the outputs of all heads. Multi-head attention mechanisms can simultaneously focus on different features in sequence data, model the relationships between different positions in the input sequence, capture global dependencies, and enhance model performance. Feedforward networks are used for feature transformation and to enhance the model's expressive power, while residual connections and normalization make the gradient descent process more stable and model training more efficient.
[0131] The linear network for regression prediction is a 256×1 fully connected layer that maps the Transformer network output to the cumulative lifetime percentage, enabling the prediction of the machine's remaining lifetime.
[0132] S6 employs a proactive shop floor scheduling strategy that integrates resource state prediction and deep reinforcement learning scheduling. Specifically, it compares the estimated completion time of the selected workpiece on the current machine with the predicted machine failure time. If the estimated completion time of the workpiece is greater than the predicted machine failure time, machine maintenance is performed. The selected workpiece is only processed after maintenance is complete; otherwise, processing begins immediately. The proactive shop floor scheduling strategy integrating resource state prediction and deep reinforcement learning scheduling is described below:
[0133] In existing online scheduling processes, once any machine completes its current processing task, the scheduling decision-making agent immediately selects the best-scoring process from that machine's buffer for processing. If a machine malfunctions during a process, the processing is interrupted, and machine repair begins. After repair, the interrupted process resumes. In online scheduling, machine malfunctions leading to machine repairs and process interruptions significantly impact production stability, processing efficiency, and product quality. This invention designs a resource state prediction model based on CNN and Transformer, enabling effective prediction of machine failure points. Based on this, an active scheduling strategy based on resource state prediction is designed.
[0134] In online scheduling problems, decision points At any given moment, the agent makes a scheduling decision, and the selected process is... Start processing time for ,Right now In the active scheduling problem, scheduling decision-making... At any given moment, the decision-making agent selects the optimal processing step. Simultaneously, the resource status prediction model described in this invention predicts machine failure points based on the current machine status. The selected processing step is considered collaboratively. Processing time and predicted current machine downtime Determine the selected process The start time of processing is determined to effectively avoid interruptions in the processing due to machine malfunctions. (Comparison process) The estimated completion time and machine-predicted failure points, if the process Able to predict fault points If the previous processing was completed, then the start time of its processing is... Otherwise, initiate machine maintenance. After machine maintenance is completed, the process... Processing has just begun. Assume machine maintenance time is... Then the process The start processing time is Process The start time of processing is shown in the formula below. Figure 5This invention demonstrates the active scheduling strategy described in the present invention, by comparing the scheduling on a certain machine. The real-time online scheduling process further illustrates the necessity of proactive scheduling.
[0135]
[0136] The present invention will be further described below with reference to specific embodiments.
[0137] To verify the effectiveness of the CNN-Trans resource state prediction model described in this invention, it was compared with classic machine fault prediction methods such as RNN, MLP, CNN, LSTM, CNN-LSTM, and Transformer. The prediction accuracy of the CNN-Trans model was verified by solving the state of bearing 1 throughout its entire lifespan in the classic PHM2012 dataset. Table 1 shows the comparison results of the CNN-Trans algorithm with other algorithms for remaining lifetime prediction. All compared algorithms used the same training process as CNN-Trans, and the evaluation metrics were Score (a specific evaluation metric for the PHM 2012 dataset), Mean Absolute Error (MAE), and R². As shown in Table 1, CNN-Trans obtained the largest average Score and the smallest MAE among all compared algorithms, indicating that the proposed algorithm achieves the best prediction performance compared to the compared algorithms, with the highest good fit between the predicted and true values. In terms of model interpretability, the R² value of CNN-Trans is second only to the compared algorithm CNN, with an average R² value of 0.5212. The experimental results in Table 1 show that the proposed algorithm has the best prediction performance compared to the comparison algorithms, and its interpretability is second only to the CNN algorithm.
[0138]
[0139] To verify the effectiveness of the active scheduling strategy described in this invention, a DJSP simulation environment based on the Simpy discrete-time simulation library under the constraints of random job arrival and machine faults was established. The environment parameter settings are shown in Table 2. From the data in Table 2, 243 ( ( ) Groups of different environmental parameter configurations. Because it is necessary to predict the remaining lifespan of the machines, the common time interval between adjacent machine failures in shop floor scheduling problem simulations follows a parameter of... The exponential distribution of machine condition simulation models is not applicable because it cannot establish a connection between machine condition and machine faults, nor can it generate machine fault labels, thus making it difficult for machine fault prediction models to function effectively in simulation environments. Therefore, this invention employs machine condition and fault labels based on bearing data collected from the classic PHM 2012 dataset in the simulation environment.
[0140]
[0141] In active scheduling mode, the accuracy of machine fault prediction is crucial to whether active scheduling can exert its due advantages. Based on this, this invention first verifies the performance of the CNN-Trans machine fault prediction model on seven sets of bearing data in the PHM 2012 dataset, using the bearing data with the highest prediction accuracy as the machine state data in the active scheduling simulation environment. As shown in Table 3, the CNN-Trans model performs best on bearing 1, with the lowest Score and MAE values, indicating that the CNN-Trans model can effectively predict the machine state of bearing 1 throughout its entire lifecycle. The CNN-Trans model has the highest R² value, indicating that the model described in this invention has the highest interpretability when processing bearing 1 data. Based on the experimental results presented in Table 3, the implementation state data of each machine in the active scheduling simulation environment is generated using the bearing 1 data. At the initial moment of the simulation, different initial machine states are randomly generated based on the entire lifecycle data of bearing 1.
[0142]
[0143] To verify the effectiveness of the active scheduling algorithm based on machine fault prediction described in this invention, in an active scheduling simulation environment with 243 different parameter settings, the DJSP solution under the constraints of random workpiece arrival and machine fault disturbance was performed using both an active scheduling mode including a CNN-Trans perturbation prediction model and a passive scheduling mode without perturbation prediction. The total delay time was recorded for each mode. Maximum completion time and the number of workpieces whose processing was interrupted. Production indicators, etc. Figure 6 This is a comparison of proactive and reactive scheduling strategies in production scenarios with varying numbers of workshops. For example... Figure 6 As shown in (a.1), when the number of workshops When the value is 2, the median and mean of the total delay time RPI under the active scheduling mode are both smaller than those obtained under the passive scheduling mode. Meanwhile, as... Figure 6 As shown in (a.2), the passive scheduling mode achieved the minimum total delay time for 44 simulation environments, resulting in a higher win rate of 54.32 (44 / 81). Figure 6 As shown in (b.1) and (c.1), when the number of workshops is 3, the mean RPI of total delay time in the active scheduling mode is greater than that in the passive scheduling mode; when the number of workshops is 4, the mean RPI of the active scheduling mode is less than that in the passive scheduling mode. In terms of the median, the median values in the two scenarios are similar.
[0144] For the maximum completion time The scheduling decision-making agent described in this invention, under active scheduling mode, achieves better results in terms of mean RPI, median, and win rate compared to passive scheduling. For example... Figure 6 As shown in (d.1), when the number of workshops is 2, active scheduling is performed in... In terms of both mean and median RPI, the active scheduling strategy achieved a certain advantage. The active scheduling strategy achieved a win rate of 53.09% (43 / 81), which is higher than the passive scheduling strategy (46.91%, 38 / 81). When the number of workshops is 3, such as... Figure 6 As shown in (e.1), although the mean values of the results obtained from active and passive scheduling are comparable, the median of active scheduling is significantly smaller than that of passive scheduling. Figure 6 As shown in (e.2), proactive scheduling has a higher win rate of 51.85 (42 / 81). When the number of workshops is 4, as... Figure 6 As shown in (f), active scheduling in The mean, median, and win rate of the RPI have all achieved significant advantages.
[0145] In production scenarios with varying numbers of workshops, the number of workpieces interrupted due to machine malfunctions is significantly reduced in the active scheduling mode compared to the passive scheduling mode. Figure 6 (g) to Figure 6 (i) It can be seen that in any scenario with 2, 3, and 4 workshops, the proactive scheduling mode achieved a significant advantage, obtaining better mean, median, and win rate in all scenarios. The win rates in different scenarios were 62.96% (51 / 81), 69.14% (56 / 81), and 66.67% (54 / 81), respectively. Therefore, proactive scheduling is of great significance for improving product quality.
[0146] S7 Repeat the above steps until all production tasks are completed and a complete scheduling plan is obtained.
[0147] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A workshop proactive scheduling method integrating resource state prediction and deep reinforcement learning scheduling, characterized in that, Based on the current dynamic splicing and analysis map and the information of unscheduled workpieces in the production system, the workshop allocation of newly arrived workpieces and the selection of workpieces on idle machines are completed; the dynamic splicing and analysis map is dynamically improved according to the following process: for each arriving workpiece, determine which workshop it is assigned to; for each machine, if the current workpiece has been processed, immediately select the next workpiece to be processed from its buffer. Based on the workshop allocation of workpieces and the processing sequence of workpieces on each machine, the dynamic splicing and analysis diagram is gradually improved; including: S1 uses a workpiece workshop allocation rule based on workshop load balancing to allocate newly arriving workpieces to the corresponding workshops; S2 Based on the process characteristics of the workpiece, place it into the corresponding machine buffer zone of the selected workshop; S3 uses a recurrent graph convolutional neural network to extract and fuse features in the time and spatial domains of the dynamically stitched parsed graph at the current time, and inputs the obtained information into a scheduling decision network based on a multilayer perceptron. S4 Calculate the probability of a workpiece being selected in the buffer of each machine using the scheduling decision network. The workpiece with the highest probability is the workpiece being processed on the current machine at the current moment. S5 uses a resource state prediction method based on convolutional neural networks (CNN) and Transformer to predict the failure time of the machine corresponding to the currently selected workpiece. The CNN-based local feature extraction network comprises three two-dimensional convolutional layers. The first convolutional layer has 2 input channels and 16 output channels; the second convolutional layer has 16 input channels and 32 output channels; and the third convolutional layer has 32 input channels and 64 output channels. Each convolutional layer has a kernel size of 3, a stride of 2, and an image boundary padding value of 1. The activation function for each convolutional layer is ReLU. The Flatten function is used to flatten the multidimensional image data output from the convolutional layers into an input format suitable for fully connected layers. The global feature extraction network based on Transformer consists of four Transformer encoding layers. Each Transformer encoding layer has the same structure, containing two sub-layers: a multi-head self-attention layer and a feedforward neural network. Each sub-layer is followed by a residual connection layer and a normalization layer. The multi-head self-attention mechanism performs multiple independent queries on the input X. Q ,key K Sum V The transformation forms multiple subspace representations: Among them, subscript For the number of attention heads; , and All of these are learnable parameter matrices; next, each head is calculated separately. The attention output, that is: in This is the dimension of each attention head, and its calculation formula is as follows: This refers to the feature dimensions of the input data to the Transformer network; concatenating the outputs of multiple attention heads yields the outputs of all heads, i.e.: in, Concat The parentheses () function combines multiple strings together to form a single larger string. These are learnable parameters used to integrate the outputs of all heads; S6 adopts a shop floor proactive scheduling strategy that integrates resource state prediction and deep reinforcement learning scheduling. Specifically, it compares the estimated completion time of the selected workpiece on the current machine with the predicted machine failure time. If the estimated completion time of the workpiece is greater than the predicted machine failure time, machine maintenance is performed. After the machine maintenance is completed, the selected workpiece is processed again. Otherwise, the selected workpiece is processed immediately. S7 Updates the dynamic splicing and parsing diagram based on the result of step S6, completing the workpiece selection and processing decision for this time; S8 Repeat steps S1 to S7 until all production tasks are completed and a complete scheduling plan is obtained.
2. The workshop active scheduling method integrating resource state prediction and deep reinforcement learning scheduling as described in claim 1, characterized in that, In step S1, the workpiece is allocated to the workshop using the workpiece workshop allocation rule based on workshop load balancing: when a workpiece arrives at the production system, it is immediately allocated to the workshop with the smallest number of workpieces at the current moment for processing.
3. The workshop active scheduling method integrating resource state prediction and deep reinforcement learning scheduling as described in claim 1, characterized in that, In step S4, the scheduling decision network is based on a Markov decision model of the problem to be solved, including a state space, an action space, a reward function, and a state transition process.
4. The workshop active scheduling method integrating resource state prediction and deep reinforcement learning scheduling as described in claim 3, characterized in that, The state space includes the process nodes in each machine's buffer, the process nodes that each machine is currently processing, and the process nodes that each machine has recently completed. The action space includes all operations within the time window, that is, all operations that have been completed on all machines, operations that are being processed, and all operations within the machine buffer. The reward function is as follows: in, It's a reward value. The current machine Cumulative running time, It is the current time; The state transition process is the transfer of actions and the change of workshop state.
5. The workshop active scheduling method integrating resource state prediction and deep reinforcement learning scheduling as described in claim 1, characterized in that, In step S2, the recurrent graph convolutional neural network extracts the workshop state features in the following manner: For each node in the dynamically stitched parsing graph: in, It is a graph convolution operator. Indicates to Time characteristics Performing graph convolution operations and extracting features at both the temporal and spatial thresholds yields the temporal and spatial features of the workshop state; the convolution kernel is a graph Laplacian operator. The function has Chebyshev coefficients as follows: ;symbol This represents the Hadamard product. , and These represent the input gate, forget gate, and output gate, respectively. , , , These are weighting coefficients. and It is the bias, where the subscript is... express , and ; sigmoid functions and The function is an activation function.
6. The workshop active scheduling method integrating resource state prediction and deep reinforcement learning scheduling as described in claim 4, characterized in that, In step S4, at the decision point At any given time, the scheduling decision network will use the output of the cyclic graph convolutional neural network. As a basis for decision-making, the workpieces in the action space are scored, and the unselectable workpieces are blocked by setting a mask.
7. The workshop active scheduling method integrating resource state prediction and deep reinforcement learning scheduling as described in claim 1, characterized in that, In step S5, the remaining lifespan of the machine is predicted by the convolutional neural network and the Transformer, and the machine failure point is calculated. When the remaining lifespan of the machine is 0, it is the moment of machine failure.
8. The workshop active scheduling method integrating resource state prediction and deep reinforcement learning scheduling as described in any one of claims 1 to 7, characterized in that, In step S6, the estimated completion time of the selected workpiece on the current machine is the current time plus the processing time of the selected workpiece on the current machine, which is determined by the workpiece's technological characteristics.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the workshop active scheduling method according to any one of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the workshop active scheduling method according to any one of claims 1 to 8.