Method and system for optimizing workshop production scheduling based on deep reinforcement learning

Abstract

This invention provides a workshop production scheduling optimization method and system based on deep reinforcement learning. It obtains a set of garment workshop production task units, including task identifiers, task process sequences, task delivery time window boundaries, and task process dependency descriptions. These are then input into a deep reinforcement learning scheduling network, which outputs a sequence of process allocation decision actions, specifying the processing equipment and start time for each process. The network is then invoked to predict the cumulative revenue of the process allocation decision action sequence, generating a state-action value assessment vector. The set of workshop production task units, the process allocation decision action sequence, and the state-action value assessment vector are combined and stored as a scheduling experience transfer unit. Based on this scheduling experience transfer unit, the network is driven to update the strategy output probability distribution, generating subsequent optimized process allocation decision action sequences. This invention comprehensively considers multi-dimensional constraints in garment production, achieving global optimized scheduling, improving production efficiency, and reducing production costs.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method and system for optimizing workshop production scheduling based on deep reinforcement learning. Background Technology

[0002] In the garment manufacturing industry, workshop production scheduling is a crucial link in ensuring the efficient and orderly operation of the production process. Traditional workshop production scheduling methods often rely on manual experience or algorithms based on simple rules. These methods have many limitations when faced with large-scale, complex, and ever-changing garment production tasks. For example, manual scheduling is easily influenced by the scheduler's subjective factors, making it difficult to guarantee the scientific nature and consistency of the scheduling. Algorithms based on simple rules often can only handle single constraints and cannot comprehensively consider the complex dependencies between tasks and processes in garment production, competition for equipment resources, and multi-dimensional constraints such as delivery time windows. This results in scheduling results that are difficult to optimize, leading to low production efficiency and failing to meet the garment industry's requirements for rapid response to market demands and improved production efficiency. Summary of the Invention

[0003] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a workshop production scheduling optimization method based on deep reinforcement learning, the method comprising:

[0004] Obtain a set of workshop production task units, which includes task identifiers, task process sequences, task delivery time window boundaries, and a set of task process dependency descriptions.

[0005] The set of workshop production task units is input into a deep reinforcement learning scheduling network. The deep reinforcement learning scheduling network outputs a process allocation decision action sequence based on the set of workshop production task units. The process allocation decision action sequence includes the processing equipment identifier and process processing start time parameter corresponding to each process in each production task unit.

[0006] The deep reinforcement learning scheduling network is invoked to perform cumulative revenue prediction processing on the process allocation decision action sequence, generating a state action value evaluation vector corresponding to the process allocation decision action sequence.

[0007] The deep reinforcement learning scheduling network combines and stores the set of workshop production task units, the sequence of process allocation decision actions, and the state action value evaluation vector into a scheduling experience transfer unit.

[0008] The deep reinforcement learning scheduling network is driven by the scheduling experience transfer unit to update the policy output probability distribution, and the policy output probability distribution generates the optimized process allocation decision action sequence in subsequent scheduling cycles.

[0009] In another aspect, embodiments of the present invention also provide a workshop production scheduling optimization system based on deep reinforcement learning, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0010] Based on the above, this embodiment of the invention, by acquiring a set of workshop production task units containing rich task information and inputting it into a deep reinforcement learning scheduling network, can comprehensively consider multi-dimensional constraints such as the dependencies between task processes, equipment resource allocation, and delivery time windows in garment production. The process allocation decision action sequence output by the deep reinforcement learning scheduling network accurately allocates processing equipment and determines the processing start time for each production task unit, effectively avoiding equipment resource conflicts and production delays. By performing cumulative benefit prediction processing on the process allocation decision action sequence, the generated state action value evaluation vector can accurately evaluate the long-term benefits of different scheduling decisions, providing a scientific basis for scheduling decisions. The set of workshop production task units, the process allocation decision action sequence, and the state action value evaluation vector are combined and stored as scheduling experience transfer units, and based on these experiences, the deep reinforcement learning scheduling network is driven to update the strategy output probability distribution, enabling the scheduling network to continuously learn and optimize, adapt to different production scenarios and task requirements, achieve global optimization of garment workshop production scheduling, and significantly improve production efficiency and reduce production costs. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the execution flow of the workshop production scheduling optimization method based on deep reinforcement learning provided in an embodiment of the present invention.

[0012] Figure 2 This is a schematic diagram of exemplary hardware and software components of a workshop production scheduling optimization system based on deep reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0013] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a workshop production scheduling optimization method based on deep reinforcement learning, provided in one embodiment of the present invention. The following is a detailed description of this workshop production scheduling optimization method based on deep reinforcement learning.

[0014] Step S110: Obtain a set of workshop production task units, which includes task identifiers, task process sequences, task delivery time window boundaries, and task process dependency description sets.

[0015] This embodiment uses the custom suit production scenario in a garment manufacturing workshop as an example. First, the current unprocessed custom suit work orders are extracted from the workshop manufacturing execution system. Each work order corresponds to an independent production task unit. A unique task identifier is assigned to each task unit. This identifier is composed of the suit style code, customer order number, and production batch number, such as "Business Suit-XZ202305001-B01". This identifier uniquely identifies the specific production task. Next, the process route document for each task unit is parsed. Taking a standard business suit as an example, its process sequence includes fifteen steps: fabric inspection, pattern layout, fabric cutting, front panel sewing, back panel sewing, sleeve sewing, collar sewing, body assembly, armhole splicing, collar sewing, placket sewing, buttonhole making, button attaching, pressing and shaping, and quality inspection and packaging. These steps are arranged in the order required by the process requirements to form the task process sequence. Extract delivery time constraints from customer order information. For example, an order may require an earliest start date of a certain year, month, and day, and a latest completion date of the same year, month, and day. Combine these two time points to form the task delivery time window boundary. Finally, analyze the dependencies between processes. For example, the "body assembly" process can only begin after the "front piece sewing" and "back piece sewing" processes are completed, and the "collar sewing" process needs to be started after the "collar piece sewing" and "body assembly" processes are completed. Record these dependencies in the form of a "process identifier - preceding process identifier list" to form a task process dependency description set.

[0016] Step S111: Extract all production work order records currently to be scheduled from the workshop manufacturing execution system. Each production work order record corresponds to a workshop production task unit. Assign a unique task identifier to each workshop production task unit. The task identifier distinguishes the identity of different workshop production task units.

[0017] The Manufacturing Execution System (MES) in a garment production workshop stores production order information for various garments. When production scheduling is required, all production order records currently awaiting scheduling are retrieved from this system. Each order record represents a specific garment production task unit, such as a men's casual suit production task or a women's dress production task. To ensure that each production task unit can be accurately identified and distinguished, a unique task identifier is assigned to it. This identifier can be composed of the garment type, order-related information, etc. For example, for a men's casual suit production task unit, its task identifier can be set in the form of "Men's Casual Suit - Order Number - Production Batch". This identifier clearly identifies the specific identity of the task unit and avoids confusion during scheduling.

[0018] Step S112: Parse the process route document of each workshop production task unit, extract multiple process description fields from the process route document according to the processing order, and combine the multiple process description fields into a task process sequence according to the extraction order. The order of the process description fields in the task process sequence represents the processing sequence constraint that the workshop production task unit must follow.

[0019] Each production task unit in the workshop has a corresponding process route document, which details the various processes involved in garment production and their sequence. Taking the men's casual suit production task unit as an example, its process route document clearly lists each process from fabric processing to final product inspection. Following the processing sequence, the process description fields for fabric inspection, cutting, sewing the front piece, sewing the back piece, sewing the sleeves, assembly, ironing, and quality inspection are extracted from the process route document. Combining these extracted process description fields according to their order in the process route document forms the task process sequence for this men's casual suit production task unit. The order of the process description fields in this task process sequence strictly reflects the processing sequence constraints that the production task unit must adhere to. For example, the cutting process must be completed before the front piece sewing process can begin, and the assembly process can only begin after the front, back, and sleeve sewing processes are completed.

[0020] Step S113: Read the customer order information corresponding to each workshop production task unit from the workshop manufacturing execution system, extract the earliest allowed start time and the latest allowed completion time from the customer order information, combine the earliest allowed start time and the latest allowed completion time as the task delivery time window boundary, and determine whether the process allocation decision action sequence meets the delivery constraints based on the task delivery time window boundary.

[0021] Customer order information is crucial for determining production task time requirements. The system retrieves customer order information for each production task unit from the workshop manufacturing execution system. This information clearly records the earliest allowed start time and the latest allowed completion time for each production task. For example, a customer orders a batch of men's casual suits, requiring production to begin on a certain date and be completed and delivered before another date. These two dates are extracted and combined to form the task delivery time window boundary for that men's casual suit production task unit. When subsequently generating the process allocation decision sequence, it's necessary to ensure that all process arrangements occur within this time window boundary to determine if the decision sequence meets the delivery constraints. If the process completion time exceeds the latest allowed completion time, the decision sequence does not meet the requirements and needs to be readjusted and optimized.

[0022] Step S114: Analyze the pre- and post-process dependencies between processes in the task process sequence of each workshop production task unit. For each process, determine the process identifiers of all direct preceding processes, associate and store the process identifiers of all direct preceding processes with the process identifiers of the process, and generate a task process dependency description set. Each dependency record in the task process dependency description set indicates that a process can only start processing after another process is completed.

[0023] For each production task unit in the workshop, the sequence of tasks needs careful analysis of the dependencies between each task. Taking the task sequence of men's casual suits as an example, the completion of the cutting process is a prerequisite for starting the sewing of the front, back, and sleeves processes, which in turn must be completed before the assembly process begins. For each task, the task identifiers of all its direct predecessor processes are determined. For example, the direct predecessor process for sewing the front piece is the cutting process, and the direct predecessor processes for assembly are sewing the front, back, and sleeves processes. The task identifiers of these direct predecessor processes are then associated with the task identifier of the task itself. For instance, the "sewing the front piece task identifier" is associated with the "cutting task identifier," and the "assembly task identifier" is associated with the "sewing the front piece task identifier," "sewing the back piece task identifier," and "sewing the sleeves task identifier," respectively. Through this method, a task process dependency description set is generated, where each record clearly indicates that a task must be completed before processing can begin.

[0024] Step S120: Input the set of workshop production task units into the deep reinforcement learning scheduling network. The deep reinforcement learning scheduling network outputs a process allocation decision action sequence based on the set of workshop production task units. The process allocation decision action sequence includes the processing equipment identifier and process processing start time parameter corresponding to each process in each production task unit.

[0025] In a garment production workshop, the organized set of workshop production task units is input into a deep reinforcement learning scheduling network. This network comprehensively analyzes and processes the input task unit set, taking into account factors such as the process sequence of each task, delivery time window boundaries, and process dependencies, ultimately outputting a sequence of action decisions for process allocation. Taking the production task unit of a men's casual suit as an example, this sequence will clearly indicate which cutting equipment will perform the cutting process and the specific time when the process begins; which sewing equipment will be responsible for the front panel sewing process and at what time it will begin after the cutting process is completed, etc. This information is crucial for the rational allocation of production equipment and time, ensuring that production tasks are completed on time.

[0026] Step S121: Encode and integrate the task identifiers, task process sequences, task delivery time window boundaries, and task process dependency description sets in the set of production task units in the workshop to generate an initial scheduling state feature tensor containing a task dependency constraint graph structure. Each channel in the initial scheduling state feature tensor corresponds to the resource requirement attributes and time constraint attributes of different task process nodes.

[0027] For each piece of information in the set of production task units in the workshop, encoding and integration processing is required to generate an initial scheduling state feature tensor. Task identifiers are converted into corresponding vectors using a specific encoding method to distinguish different production tasks. Each process description field in the task process sequence is also encoded into a vector form, reflecting the type and characteristics of the process. The earliest allowed start time and latest allowed finish time in the task delivery time window boundaries are processed and converted into time feature vectors. The task process dependency description set is represented by constructing a graph structure, forming a task dependency constraint graph. The encoded vectors and graph structure are integrated together to generate the initial scheduling state feature tensor. Each channel of this feature tensor corresponds to the resource requirement attributes and time constraint attributes of different task process nodes. For example, one channel might correspond to the fabric resource attributes required for the cutting process, while another channel might correspond to the time constraint attributes of the sewing process, thus comprehensively reflecting the state information of the production tasks.

[0028] Step S122: Input the initial scheduling state feature tensor into the deep reinforcement learning scheduling network, and extract spatial structure features from the initial scheduling state feature tensor through the deep reinforcement learning scheduling network to generate a task process association feature map. The task process association feature map encodes the dependency constraint strength and resource competition relationship between different task process nodes.

[0029] After the initial scheduling state feature tensor is input into the deep reinforcement learning scheduling network, the network begins to extract its spatial structure features. This process aims to capture the spatial relationships and structural information between different task process nodes in the feature tensor. In garment production, there are complex dependencies and resource competition relationships between different process nodes. For example, the cutting and sewing processes may compete for certain fabric resources, while the sewing process depends on the completion of the cutting process. Through the network's spatial structure feature extraction, these relationships and their strengths can be encoded into the generated task process association feature map. Each element in this feature map represents the association between the corresponding task process nodes, enabling the network to better understand the structure and constraints of the entire production task.

[0030] Step S1221: Input the initial scheduling state feature tensor into the first convolutional kernel group of the deep reinforcement learning scheduling network. The first convolutional kernel group contains multiple convolutional kernels with different receptive field sizes. The initial scheduling state feature tensor is convolved by the first convolutional kernel group to generate a multi-scale task process local feature set. Each scale feature map in the multi-scale task process local feature set retains the neighborhood structure information of the task process node within the corresponding receptive field.

[0031] The initial scheduling state feature tensor first enters the first convolutional kernel group of the deep reinforcement learning scheduling network. This convolutional kernel group contains multiple convolutional kernels with different receptive field sizes, which can scan and extract features from the feature tensor from different spatial ranges. In garment production scheduling, different process nodes have different neighborhood ranges and degrees of influence. Using convolutional kernels with different receptive fields can extract local features more comprehensively. For example, convolutional kernels with smaller receptive fields can capture the direct dependencies between adjacent processes, while convolutional kernels with larger receptive fields can capture the indirect relationships between processes at greater distances. After convolutional operations are performed on the initial scheduling state feature tensor by the first convolutional kernel group, multiple feature maps of different scales are generated. These feature maps together constitute a multi-scale task process local feature set, and each scale feature map retains the neighborhood structure information of the task process nodes within its corresponding receptive field.

[0032] Step S1222: Perform feature stitching processing on the local feature set of the multi-scale task process along the channel dimension to obtain a cross-scale fused feature tensor. The number of channels of the cross-scale fused feature tensor is equal to the sum of the number of channels of each feature map in the local feature set of the multi-scale task process.

[0033] After obtaining the local feature sets of the multi-scale task processes, it is necessary to fuse the feature maps of different scales. This step uses feature concatenation along the channel dimension to achieve fusion. The feature maps of each scale are concatenated along the channel dimension so that the number of channels in the concatenated cross-scale fused feature tensor is the sum of the number of channels in each feature map. This concatenation method can integrate feature information from different scales, so that the cross-scale fused feature tensor contains both fine local features under small receptive fields and global structural features under large receptive fields. In garment production scheduling, this cross-scale feature fusion helps the network to fully understand the complex relationships between various process nodes.

[0034] Step S1223: Input the cross-scale fusion feature tensor into the second convolutional kernel group of the deep reinforcement learning scheduling network. The second convolutional kernel group includes a one-dimensional dilated convolutional kernel. The one-dimensional dilated convolutional kernel is used to expand the receptive field of the features in the cross-scale fusion feature tensor along the path of task process dependency relationship to extract long-range dependency relationship features between task process nodes and generate process dependency relationship enhancement feature map.

[0035] The cross-scale fusion feature tensor is input into the second convolutional kernel group of the deep reinforcement learning scheduling network, which contains one-dimensional dilated convolutional kernels. One-dimensional dilated convolutional kernels, by setting intervals between kernel elements, can expand the receptive field without increasing the number of parameters. In garment production scheduling, long-range dependencies exist between task processes; for example, a subsequent process may depend on multiple preceding processes. By scanning the features along the task process dependency path in the cross-scale fusion feature tensor using one-dimensional dilated convolutional kernels, long-range dependency features between task process nodes can be effectively extracted. After the above processing, an enhanced process dependency feature map is generated, which more clearly reflects the long-range dependencies between different process nodes, further enhancing the expressive power of the feature map.

[0036] Step S1224: Calculate the global average pooling value of each channel of the process dependency enhancement feature map through the deep reinforcement learning scheduling network, generate the attention weight coefficient of each channel based on the global average pooling value, and perform channel weight adjustment on the process dependency enhancement feature map through the attention weight coefficient to obtain the channel recalibrated feature map.

[0037] After obtaining the enhanced feature map of process dependencies, the deep reinforcement learning scheduling network calculates the global average pooling value for each channel of the feature map. The global average pooling value reflects the overall strength of the feature in each channel. Based on these global average pooling values, attention weight coefficients are generated for each channel, and the magnitude of the weight coefficients indicates the importance of the corresponding channel feature. In garment production scheduling, the features of different channels have different importance to scheduling decisions. By adjusting the channel weights of the enhanced feature map of process dependencies through attention weight coefficients, the features of important channels can be highlighted and the features of unimportant channels can be suppressed, thus obtaining a channel recalibrated feature map. This feature map can more effectively reflect the feature information that has an important impact on scheduling decisions.

[0038] Step S1225: Calculate the maximum and average values ​​of the channel recalibration feature map in the spatial dimension through the deep reinforcement learning scheduling network, generate a spatial attention mask, and perform element-wise multiplication of the spatial attention mask with the channel recalibration feature map to obtain a task process association feature map. The feature value of each spatial position in the task process association feature map encodes the dependency constraint strength and resource competition relationship between the task process node and other task process nodes.

[0039] The deep reinforcement learning scheduling network computes the maximum and average values ​​of the channel recalibration feature map in the spatial dimension. These two values ​​reflect the spatial importance of the feature map. The maximum and average values ​​are combined to generate a spatial attention mask. Elements in the spatial attention mask represent the importance of their corresponding spatial locations. This mask is then multiplied element-wise with the channel recalibration feature map, enhancing features at important spatial locations and weakening features at less important locations, thus obtaining a task-process association feature map. In garment production scheduling, the feature value of each spatial location in this feature map encodes the strength of dependencies and resource competition relationships between task-process nodes and other task-process nodes, helping the network make more accurate scheduling decisions.

[0040] Step S123: Calculate the attention weight distribution between each task process node and other task process nodes in the task process association feature map through the deep reinforcement learning scheduling network, and perform weighted aggregation processing on the task process association feature map according to the attention weight distribution to generate a process node context feature set with global dependency awareness.

[0041] Deep reinforcement learning scheduling networks process task-process association feature maps, calculating the attention weight distribution between each task-process node and other task-process nodes. This attention weight distribution reflects the degree of association and importance between different task-process nodes. Based on these weight distributions, the task-process association feature maps are weighted and aggregated, integrating the feature information of other nodes into the features of the current node according to their weights, generating a set of contextual features for process nodes with global dependency awareness. In garment production scheduling, this feature set enables each process node to contain information about its related nodes globally, thus providing a better understanding of the global dependencies of the entire production task.

[0042] Step S1231: The deep reinforcement learning scheduling network performs matrix multiplication operations on the task process association feature map with the query weight matrix, key weight matrix and value weight matrix respectively to generate the query feature matrix, key feature matrix and value feature matrix. The number of rows of the query feature matrix, key feature matrix and value feature matrix is ​​equal to the number of task process nodes, and the number of columns is equal to the attention feature dimension.

[0043] The deep reinforcement learning scheduling network performs matrix multiplication operations on the task-step association feature map with the query weight matrix, key weight matrix, and value weight matrix, respectively. These three weight matrices are parameters learned by the network during training. Through matrix multiplication, the task-step association feature map is transformed into query feature matrix, key feature matrix, and value feature matrix. The query feature matrix represents the feature of each task-step node as a query, the key feature matrix represents the feature of each node as a key, and the value feature matrix represents the feature of each node as a value. The number of rows in these three matrices is the same as the number of task-step nodes, and the number of columns is equal to the attention feature dimension; together, they form the basis for calculating the attention weight distribution.

[0044] Step S1232: Calculate the dot product between the query feature matrix and the transpose of the key feature matrix through the deep reinforcement learning scheduling network to obtain the original attention score matrix. Divide each element in the original attention score matrix by the square root of the attention feature dimension to obtain the scaled attention score matrix.

[0045] The deep reinforcement learning scheduling network calculates the dot product between the query feature matrix and the transpose of the key feature matrix to obtain the original attention score matrix. The elements in this matrix represent the original association scores between query nodes and key nodes. To avoid gradient problems caused by excessively large dot product results when the attention feature dimension is large, each element in the original attention score matrix is ​​divided by the square root of the attention feature dimension to obtain a scaled attention score matrix. This scaling process makes the attention scores more stable, which is beneficial for network training and optimization.

[0046] Step S1233: Obtain the preceding process constraints recorded in the task process dependency description set through the deep reinforcement learning scheduling network, generate a mask matrix according to the preceding process constraints, and the mask value corresponding to the prohibited access position in the mask matrix is ​​negative infinity. Obtain the constrained attention score matrix by adding the scaled attention score matrix to the mask matrix element by element.

[0047] The deep reinforcement learning scheduling network extracts the constraints of preceding processes from the task process dependency description set, i.e., which processes must be completed before starting. A mask matrix is ​​generated based on these constraints. For positions where association is not allowed in the process dependencies, a mask value of negative infinity is set in the mask matrix. The scaled attention score matrix is ​​added element-wise to the mask matrix, making the attention score of the masked positions negative infinity. In subsequent normalization, the attention weights of these positions will approach zero. Through this method, a constrained attention score matrix is ​​obtained, ensuring that the calculation of attention weights conforms to the dependency constraints between processes.

[0048] Step S1234: Perform exponential normalization operation on each row of the constrained attention score matrix through the deep reinforcement learning scheduling network to generate an attention weight matrix. Each element in the attention weight matrix represents the attention weight value between the corresponding row task process node and the corresponding column task process node.

[0049] The deep reinforcement learning scheduling network performs exponential normalization on each row of the constrained attention score matrix. This involves performing an exponential operation on each element of the row and then dividing by the sum of the exponential results of all elements in that row. This process transforms the constrained attention score matrix into an attention weight matrix. Each element in the matrix represents the attention weight between a task node in the corresponding row and a task node in the corresponding column. These weights reflect the strength of the association between different nodes; a larger weight indicates a stronger association between the two nodes.

[0050] Step S1235: Perform matrix multiplication on the attention weight matrix and the value feature matrix to obtain a weighted aggregated feature matrix. Map the weighted aggregated feature matrix back to the original feature dimension through the deep reinforcement learning scheduling network to generate a process node context feature set with global dependency awareness.

[0051] The attention weight matrix and the value feature matrix are multiplied to obtain a weighted aggregated feature matrix. Each element in this matrix is ​​the weighted sum of the corresponding element in the value feature matrix and the attention weight, reflecting the aggregated information of features from different nodes. The deep reinforcement learning scheduling network maps the weighted aggregated feature matrix back to the original feature dimension through a certain mapping operation, generating a set of contextual features for process nodes with global dependency awareness. Each feature vector in this feature set contains the dependency information of the corresponding process node in a global scope.

[0052] Step S124: Output the action probability distribution corresponding to each schedulable process node through the deep reinforcement learning scheduling network. The action probability distribution includes the processing equipment selection probability component and the processing start time selection probability component.

[0053] The deep reinforcement learning scheduling network analyzes and processes each schedulable process node based on a generated set of contextual features of process nodes with global dependency awareness, outputting a corresponding action probability distribution. The action probability distribution includes a processing equipment selection probability component and a processing start time selection probability component. The processing equipment selection probability component represents the probability that the process node will be assigned to different processing equipment, while the processing start time selection probability component represents the probability that the process will start at different times.

[0054] Step S125: Generate a process allocation decision action sequence based on the action probability distribution. The process allocation decision action sequence includes the processing equipment identifier and process processing start time parameter corresponding to each process in each production task unit, wherein the processing start time parameter is not earlier than the processing completion time of all preceding processes of the process.

[0055] Sampling is performed based on the probability distribution of actions, selecting the processing equipment and processing start time with the highest probability to generate a sequence of actions for process allocation decisions. This sequence clearly defines the processing equipment identifier and processing start time parameter for each process in each production task unit. When determining the processing start time parameter, the dependencies between processes are strictly followed to ensure that the processing start time is not earlier than the completion time of all preceding processes. For example, in the production of men's casual suits, the processing start time for the sewing of the front panel must be after the cutting process is completed to ensure smooth production. The process allocation decision sequence generated in this way can rationally allocate production resources and time, improving production efficiency.

[0056] Step S130: Invoke the deep reinforcement learning scheduling network to perform cumulative revenue prediction processing on the process allocation decision action sequence, and generate the state action value evaluation vector corresponding to the process allocation decision action sequence.

[0057] After obtaining the sequence of actions for work process allocation decisions, a deep reinforcement learning scheduling network is invoked to predict the cumulative benefit of this sequence. The network comprehensively considers various benefits and costs associated with the sequence, such as the benefit of on-time delivery, the benefit of equipment utilization, and the expenditure of processing costs. Through complex calculations and analysis, it generates a state-action value assessment vector corresponding to the work process allocation decision sequence. Each element in this vector represents the expected cumulative benefit obtained by executing the action sequence in the corresponding state. In garment production scheduling, the state-action value assessment vector helps decision-makers determine whether the current work process allocation scheme can bring the maximum cumulative benefit.

[0058] Step S131: Input the set of workshop production task units and the sequence of process allocation decision actions into the deep reinforcement learning scheduling network. Through the deep reinforcement learning scheduling network, perform joint feature extraction on the task identifier, task process sequence, task delivery time window boundary and task process dependency description set in the set of workshop production task units and the processing equipment identifier and process processing start time parameter in the sequence of process allocation decision actions to generate a state-action joint feature tensor.

[0059] The set of workshop production task units and the sequence of process allocation decision actions are input into a deep reinforcement learning scheduling network. The network performs joint feature extraction on these two pieces of information, fusing and processing information such as task identifiers, task process sequences, task delivery time window boundaries, task process dependency description sets, processing equipment identifiers, and process processing start time parameters to generate a state-action joint feature tensor. This feature tensor integrates the state information and decision action information of the production task, and can comprehensively reflect the current production scheduling situation.

[0060] Step S132: Input the state-action joint feature tensor into the temporal convolutional layer of the deep reinforcement learning scheduling network. The temporal convolutional layer contains multiple dilated convolutional kernels. The temporal convolutional layer performs temporal feature scanning on the state-action joint feature tensor along the time dimension of the process sequence, extracts the temporal dependency pattern in the process allocation decision action sequence, and generates a temporal augmentation feature tensor.

[0061] The joint state-action feature tensor is input into the temporal convolutional layer of the deep reinforcement learning scheduling network, which contains multiple dilated convolutional kernels. These dilated kernels expand the receptive field in the temporal dimension, capturing the temporal dependencies of long sequences. The joint state-action feature tensor is scanned along the time dimension of the process sequence by the temporal convolutional layer to extract temporal dependency patterns in the process allocation decision action sequence, such as the order of processes and the mutual influence of processing times. After this processing, a temporally enhanced feature tensor is generated, which better reflects the temporal characteristics of the process allocation decision action sequence.

[0062] Step S133: Calculate the average value of each channel of the temporal augmentation feature tensor through the deep reinforcement learning scheduling network to obtain the global average pooling feature; calculate the maximum value of each channel of the temporal augmentation feature tensor through the deep reinforcement learning scheduling network to obtain the global max pooling feature; concatenate the global average pooling feature and the global max pooling feature along the channel dimension to obtain the global statistical feature vector.

[0063] The deep reinforcement learning scheduling network calculates the average value of the temporal augmentation feature tensor across all channels to obtain the global average pooling feature, which reflects the average level of the temporal augmentation feature tensor across all channels. Simultaneously, it calculates the maximum value of the temporal augmentation feature tensor across all channels to obtain the global max pooling feature, which reflects the strongest response across each channel. The global average pooling feature and the global max pooling feature are concatenated along the channel dimension to obtain the global statistical feature vector, which integrates the overall statistical information of the temporal augmentation feature tensor.

[0064] Step S134: Input the global statistical feature vector into the value regression network of the deep reinforcement learning scheduling network. The value regression network contains multiple fully connected layers and nonlinear activation units located between the fully connected layers. The global statistical feature vector is regressed layer by layer through the value regression network to generate a state-action value scalar. The state-action value scalar represents the cumulative revenue prediction value that can be obtained after executing the process allocation decision action sequence from the current scheduling state.

[0065] The global statistical feature vector is input into the value regression network of the deep reinforcement learning scheduling network. This network contains multiple fully connected layers and non-linear activation units. The fully connected layers perform linear transformations on the feature vector, while the non-linear activation units increase the network's non-linear expressive power. The value regression network performs layer-by-layer regression mapping on the global statistical feature vector, mapping the high-dimensional feature vector to a scalar value, namely the state-action value scalar. This scalar represents the predicted cumulative reward value that can be obtained from the subsequent execution of the process allocation decision action sequence starting from the current scheduling state, and is an important indicator for evaluating the quality of the decision action sequence.

[0066] For example, in step S1341: the global statistical feature vector is input into the first fully connected layer of the value regression network, and the global statistical feature vector is linearly transformed through the first fully connected layer to generate a first-layer hidden feature vector. The dimension of the first-layer hidden feature vector is lower than the dimension of the global statistical feature vector.

[0067] The global statistical feature vector first enters the first fully connected layer of the value regression network. The first fully connected layer performs a linear transformation on the global statistical feature vector through its weight matrix, mapping the high-dimensional feature vector to a lower-dimensional space, generating the first layer of hidden feature vectors. This dimensionality reduction process reduces the feature dimension, improves the network's computational efficiency, and extracts more crucial feature information.

[0068] Step S1342: Input the first layer hidden feature vector into the first nonlinear activation unit of the value regression network, and perform nonlinear mapping processing on each element in the first layer hidden feature vector through the first nonlinear activation unit to obtain the first layer activated feature vector.

[0069] The first-layer hidden feature vector is input into the first non-linear activation unit, which performs a non-linear mapping on each element of the vector. This non-linear mapping enhances the non-linear expressive power of the features, enabling the network to better capture complex feature relationships. After processing, the first-layer activated feature vector is obtained, which possesses a stronger non-linear feature expressive power.

[0070] Step S1343: Input the first layer activation feature vector into the second fully connected layer of the value regression network, and perform linear transformation on the first layer activation feature vector through the second fully connected layer to generate the second layer hidden feature vector. The dimension of the second layer hidden feature vector is lower than that of the first layer hidden feature vector.

[0071] The first layer's activated feature vectors are fed into the second fully connected layer, where they undergo another linear transformation to further reduce the dimensionality of the feature vectors, generating the second layer's hidden feature vectors. Through multiple dimensionality reduction and feature extraction processes, the most critical feature information for value regression can be gradually extracted.

[0072] Step S1344: Input the second layer hidden feature vector into the second nonlinear activation unit of the value regression network, and perform nonlinear mapping processing on each element in the second layer hidden feature vector through the second nonlinear activation unit to obtain the second layer activated feature vector.

[0073] The second-layer hidden feature vector is processed by the nonlinear mapping of the second nonlinear activation unit to obtain the second-layer activated feature vector. This further nonlinear processing enhances the expressive power of the features, enabling the network to better handle complex nonlinear relationships.

[0074] Step S1345: Input the second layer activation feature vector into the output fully connected layer of the value regression network, and map the second layer activation feature vector into a scalar value through the output fully connected layer to obtain the state action value scalar.

[0075] The second activation feature vector is input to the output fully connected layer, which maps the two-dimensional feature vector to a scalar value, namely the state-action value scalar. This scalar value comprehensively reflects the cumulative profit prediction that can be obtained by executing the sequence of action decisions for task allocation starting from the current scheduling state.

[0076] Step S135: Copy and expand the state action value scalar into a state action value evaluation vector with the same length as the process allocation decision action sequence, wherein the component at each position in the state action value evaluation vector is equal to the state action value scalar.

[0077] To maintain consistency with the length of the decision action sequence for process allocation, the state action value scalar is replicated and expanded to generate a state action value assessment vector. Each component in this vector is equal to the state action value scalar, thus easily linking each decision action to its corresponding value assessment.

[0078] Step S140: The set of workshop production task units, the sequence of process allocation decision actions, and the state action value evaluation vector are combined and stored as scheduling experience transfer units through the deep reinforcement learning scheduling network.

[0079] Deep reinforcement learning scheduling networks combine three pieces of information: a set of workshop production task units, a sequence of process allocation decision actions, and a state-action value assessment vector. These combined information forms scheduling experience transfer units, which are then stored. These experience transfer units record the value assessment obtained from executing specific decision actions under specific production task states, serving as a crucial data foundation for the network's learning and optimization. By accumulating a large number of scheduling experience transfer units, the network can continuously learn from experience and improve its scheduling strategies.

[0080] Step S141: The task identifiers, task process sequences, task delivery time window boundaries, and task process dependency description sets in the workshop production task unit set are converted into fixed-dimensional task embedding vectors through the deep reinforcement learning scheduling network to obtain task embedding feature vectors.

[0081] The deep reinforcement learning scheduling network embeds various information in the set of production task units in the workshop, converting task identifiers, task process sequences, task delivery time window boundaries, and task process dependency description sets into fixed-dimensional task embedding vectors. These embedding vectors can compress complex task information into a fixed-dimensional vector space, which is convenient for the network to process and calculate, while retaining the key feature information of the task.

[0082] Step S142: The deep reinforcement learning scheduling network is used to convert the processing equipment identifier and the processing start time parameter in the process allocation decision action sequence into a fixed-dimensional action embedding vector to obtain the action embedding feature vector.

[0083] For the processing equipment identifier and process start time parameter in the process allocation decision action sequence, the deep reinforcement learning scheduling network converts them into fixed-dimensional action embedding vectors. Action embedding vectors represent the information of decision actions in vector form, facilitating combination and processing with task embedding vectors to form a unified feature representation.

[0084] Step S143: Normalize and compress each component in the state-action value evaluation vector through the deep reinforcement learning scheduling network to generate a value embedding feature vector. The dimension of the value embedding feature vector is consistent with the dimension of the task embedding feature vector and the action embedding feature vector.

[0085] The deep reinforcement learning scheduling network normalizes and compresses each component in the state-action value evaluation vector, mapping it to a certain numerical range to generate a value embedding feature vector. To facilitate subsequent feature concatenation and processing, the dimension of the value embedding feature vector is kept consistent with the dimensions of the task embedding feature vector and the action embedding feature vector.

[0086] Step S144: The task embedding feature vector, action embedding feature vector, and value embedding feature vector are concatenated along the channel dimension to obtain a combined experience feature tensor. The combined experience feature tensor is stored in the circular buffer queue of the deep reinforcement learning scheduling network. The circular buffer queue manages multiple scheduling experience transfer units in a first-in-first-out order.

[0087] The task embedding feature vector, action embedding feature vector, and value embedding feature vector are concatenated along the channel dimension to form a combined experience feature tensor. This tensor integrates the embedding information of task, action, and value, and fully represents a scheduling experience transfer unit. The combined experience feature tensor is stored in a circular buffer queue of the deep reinforcement learning scheduling network. This queue manages multiple scheduling experience transfer units in a first-in-first-out order, ensuring that the network can efficiently access and utilize historical experience data.

[0088] Step S145: Add a timestamp tag and a priority weight coefficient to each scheduling experience transfer unit in the circular buffer queue. The priority weight coefficient is dynamically adjusted according to the absolute value of the state action value evaluation vector. The priority weight coefficient controls the sampling probability distribution when sampling scheduling experience transfer units from the circular buffer queue.

[0089] In the circular buffer queue, each scheduled experience transfer unit is appended with a timestamp label and a priority weight coefficient. The timestamp label records the time when the experience was generated, facilitating time-related learning by the network. The priority weight coefficient is dynamically adjusted based on the absolute value of the state-action value evaluation vector. Experiences with larger absolute value evaluations are generally considered to have higher learning value and are therefore assigned a higher priority weight coefficient. When sampling scheduled experience transfer units from the circular buffer queue, the priority weight coefficient controls the sampling probability distribution, ensuring that high-priority experiences have a higher probability of being sampled, thereby improving the network's learning efficiency and effectiveness.

[0090] Step S150: Based on the scheduling experience transfer unit, drive the deep reinforcement learning scheduling network to update the policy output probability distribution, and the policy output probability distribution generates the optimized process allocation decision action sequence in the subsequent scheduling cycle.

[0091] By utilizing scheduling experience transfer units stored in a circular buffer queue, a deep reinforcement learning scheduling network is driven to update its policy output probability distribution. Through learning from these experiences, the network continuously adjusts its parameters and optimizes its decision-making strategy, ensuring that the output action probability distribution more accurately reflects the optimal work process allocation scheme. The updated policy output probability distribution is then used to generate optimized work process allocation decision action sequences for subsequent scheduling cycles, thereby achieving continuous optimization of production scheduling.

[0092] Step S151: Sample multiple scheduling experience transfer units from the circular buffer queue of the deep reinforcement learning scheduling network according to priority weight coefficients to obtain a sampled experience batch set. The sampled experience batch set contains multiple combined experience feature tensors and the state-action value evaluation vector corresponding to each combined experience feature tensor.

[0093] From the circular buffer queue of the deep reinforcement learning scheduling network, samples are taken according to the priority weight coefficients of each scheduled experience transfer unit, and multiple experience transfer units are selected to form a sampled experience batch set. This set contains multiple combined experience feature tensors and their corresponding state-action value evaluation vectors. Through this priority-based sampling method, the network can learn more high-value experiences, improving learning efficiency.

[0094] Step S152: Calculate the policy gradient loss value between the policy output probability distribution and the process allocation decision action sequence recorded in the sampled experience batch set through the deep reinforcement learning scheduling network. At the same time, calculate the temporal difference loss value between the state action value evaluation vector and the target value vector output by the target value network. Weight the policy gradient loss value and the temporal difference loss value to obtain the joint optimization loss scalar.

[0095] The deep reinforcement learning scheduling network computes the policy gradient loss and the temporal difference loss. The policy gradient loss reflects the difference between the current policy output probability distribution and the actual decision actions in the sampling experience, while the temporal difference loss reflects the difference between the state-action value assessment vector and the target value vector output by the target value network. These two loss values ​​are weighted and summed to obtain the joint optimization loss scalar, which integrates the loss information from policy optimization and value assessment, and is used to guide the updating of network parameters.

[0096] Step S153: Calculate the gradient value of the joint optimization loss scalar with respect to the parameter weights of the deep reinforcement learning scheduling network through the deep reinforcement learning scheduling network, and perform gradient pruning on the gradient value to obtain a set of pruned gradient values.

[0097] Deep reinforcement learning scheduling networks compute the gradient values ​​of the joint optimization loss scalar with respect to the network parameter weights. These gradient values ​​represent the degree to which parameter changes affect the loss value. To avoid the gradient explosion problem, gradient clipping is performed to restrict the gradient values ​​to a certain range, resulting in a clipped set of gradient values. This process ensures stable network training.

[0098] Step S154: Update the parameter weights of the deep reinforcement learning scheduling network according to the pruned gradient value set to obtain the updated deep reinforcement learning scheduling network. At the same time, update the parameter weights of the target value network using a soft update method. The soft update method uses interpolation coefficients to weight and fuse the parameter weights of the deep reinforcement learning scheduling network and the parameter weights of the target value network.

[0099] Based on the pruned gradient value set, the parameter weights of the deep reinforcement learning scheduling network are updated using optimization algorithms such as gradient descent, resulting in the updated network. Simultaneously, a soft update method is used to update the parameter weights of the target value network, employing an interpolation coefficient to weightedly fuse the parameter weights of the deep reinforcement learning scheduling network and the target value network. This soft update method allows the parameters of the target value network to change slowly, improving training stability.

[0100] Step S155: Add probability-decreasing exploration noise to the action probability distribution output by the updated deep reinforcement learning scheduling network to obtain the optimized process allocation decision action sequence.

[0101] To ensure optimal decision-making while retaining some exploratory capability, probability-decreasing exploration noise is added to the action probability distribution output by the updated deep reinforcement learning scheduling network. As training progresses, the intensity of the exploration noise gradually decreases, causing the network to converge to the optimal policy. The action probability distribution with added noise is used to generate an optimized sequence of action decisions for process allocation. This sequence, while maintaining a certain level of optimization, can explore new scheduling schemes, helping to discover better decisions.

[0102] Step S156: Obtain the set of feedback workshop production task units collected after executing the optimized process allocation decision action sequence within the current scheduling cycle. The set of feedback workshop production task units includes the actual processing time parameters of completed processes and the equipment idle time window offset parameters.

[0103] After executing the optimized process allocation decision sequence within the current scheduling cycle, actual production data is collected to form a feedback workshop production task unit set. This set includes the actual processing time parameters of completed processes, i.e., the actual processing time spent by each process, and the equipment idle time window offset parameters, i.e., the deviation between the actual idle time and the planned idle time of the equipment. The above feedback data reflects the difference between the actual production situation and the decision expectations, and is an important basis for further network optimization.

[0104] Step S157: Input the set of feedback workshop production task units and the optimized process allocation decision action sequence into the deep reinforcement learning scheduling network. The deep reinforcement learning scheduling network calculates the time deviation penalty value based on the actual processing time parameter and the preset rated processing time parameter in the task process sequence, calculates the equipment utilization reward value based on the equipment idle time window offset parameter, and calculates the delivery delay penalty value based on the task delivery time window boundary and the expected completion time. The time deviation penalty value, equipment utilization reward value and delivery delay penalty value are weighted and summed to generate a feedback reward signal scalar.

[0105] The set of feedback workshop production task units and the sequence of optimized process allocation decisions are input into a deep reinforcement learning scheduling network. The network calculates a time deviation penalty based on the actual processing time parameter and the preset rated processing time parameter; if the actual processing time exceeds the rated time, a penalty is imposed. It calculates a equipment utilization reward based on the equipment idle time window offset parameter; if the equipment idle time is short and the utilization rate is high, a reward is imposed. It calculates a delivery delay penalty based on the task delivery time window boundary and the expected completion time; if the expected completion time exceeds the delivery time window boundary, a penalty is imposed. These three values ​​are weighted and summed to generate a feedback reward signal scalar, which reflects the reward and penalty situation after the actual execution of the decision action sequence.

[0106] Step S158: Superimpose the feedback reward signal scalar onto each component of the state-action value evaluation vector to obtain the corrected state-action value evaluation vector.

[0107] The feedback reward signal scalar is superimposed on each component of the state-action value evaluation vector to correct the value evaluation, resulting in the corrected state-action value evaluation vector. This correction allows the value evaluation to more accurately reflect the actual decision-making effect.

[0108] Step S159: Store the set of feedback workshop production task units, the sequence of optimized process allocation decision actions, and the modified state action value evaluation vector combination as a feedback scheduling experience transfer unit, and add the feedback scheduling experience transfer unit to the circular buffer queue.

[0109] By combining the set of feedback workshop production task units, the sequence of optimized process allocation decision actions, and the value evaluation vector of the corrected state actions, a feedback scheduling experience transfer unit is formed and added to the circular buffer queue. This continuously enriches the network's experience base, enabling the network to learn from actual feedback and continuously improve its scheduling strategy.

[0110] Step S1510: Based on the feedback scheduling experience transfer unit, drive the deep reinforcement learning scheduling network again to update the policy output probability distribution, forming a continuous optimization loop.

[0111] The feedback scheduling experience transfer unit is used to drive the deep reinforcement learning scheduling network to update the policy output probability distribution. By continuously acquiring feedback, correcting value assessments, storing experience, and updating the network, the network's scheduling policy is continuously improved and perfected, thereby achieving continuous optimization of the scheduling in the garment production workshop.

[0112] Figure 2 The illustration shows exemplary hardware and software components of a deep reinforcement learning-based workshop production scheduling optimization system 100 that can implement the ideas of this application, according to some embodiments of this application. For example, a processor 120 can be used in the deep reinforcement learning-based workshop production scheduling optimization system 100 and to perform the functions in this application.

[0113] The deep reinforcement learning-based workshop production scheduling optimization system 100 can be a general-purpose server or a special-purpose server; both can be used to implement the deep reinforcement learning-based workshop production scheduling optimization method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0114] For example, a deep reinforcement learning-based shop floor production scheduling optimization system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the deep reinforcement learning-based shop floor production scheduling optimization system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The deep reinforcement learning-based shop floor production scheduling optimization system 100 also includes an I / O interface 150 between the computer and other input / output devices.

[0115] For ease of explanation, only one processor is described in the deep reinforcement learning-based shop floor scheduling optimization system 100. However, it should be noted that the deep reinforcement learning-based shop floor scheduling optimization system 100 of this application may also include multiple processors. Therefore, the steps executed by one processor as described in this application may also be executed jointly by multiple processors or individually. For example, if the processor of the deep reinforcement learning-based shop floor scheduling optimization system 100 executes steps A and B, it should be understood that steps A and B may also be executed jointly by two different processors or individually by one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor jointly execute steps A and B.

[0116] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned workshop production scheduling optimization method based on deep reinforcement learning is implemented.

[0117] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A workshop production scheduling optimization method based on deep reinforcement learning, characterized in that, The method includes: Obtain a set of workshop production task units, which includes task identifiers, task process sequences, task delivery time window boundaries, and a set of task process dependency descriptions. The set of workshop production task units is input into a deep reinforcement learning scheduling network. The deep reinforcement learning scheduling network outputs a process allocation decision action sequence based on the set of workshop production task units. The process allocation decision action sequence includes the processing equipment identifier and process processing start time parameter for each process in each production task unit. The deep reinforcement learning scheduling network is invoked to perform cumulative revenue prediction processing on the process allocation decision action sequence, generating a state action value evaluation vector corresponding to the process allocation decision action sequence. The deep reinforcement learning scheduling network combines and stores the set of workshop production task units, the sequence of process allocation decision actions, and the state action value evaluation vector into a scheduling experience transfer unit. The deep reinforcement learning scheduling network is driven by the scheduling experience transfer unit to update the policy output probability distribution, and the policy output probability distribution generates the sequence of optimized process allocation decision actions in subsequent scheduling cycles.

2. The workshop production scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process involves inputting the set of workshop production task units into a deep reinforcement learning scheduling network. The deep reinforcement learning scheduling network outputs a sequence of process allocation decisions based on the set of workshop production task units. This sequence includes the processing equipment identifier and process start time parameters corresponding to each process within each production task unit, including: The task identifiers, task process sequences, task delivery time window boundaries, and task process dependency description sets in the set of workshop production task units are encoded and integrated to generate an initial scheduling state feature tensor containing a task dependency constraint graph structure. Each channel in the initial scheduling state feature tensor corresponds to the resource requirement attributes and time constraint attributes of different task process nodes. The initial scheduling state feature tensor is input into the deep reinforcement learning scheduling network. The deep reinforcement learning scheduling network extracts spatial structure features from the initial scheduling state feature tensor to generate a task process association feature map. The task process association feature map encodes the dependency constraint strength and resource competition relationship between different task process nodes. The deep reinforcement learning scheduling network is used to calculate the attention weight distribution between each task process node and other task process nodes in the task process association feature map. The task process association feature map is then weighted and aggregated according to the attention weight distribution to generate a set of process node context features with global dependency awareness. The deep reinforcement learning scheduling network outputs the action probability distribution corresponding to each schedulable process node, and the action probability distribution includes the processing equipment selection probability component and the processing start time selection probability component. The process allocation decision action sequence is generated by sampling the action probability distribution. The process allocation decision action sequence includes the processing equipment identifier and process processing start time parameter corresponding to each process in each production task unit, wherein the processing start time parameter is not earlier than the processing completion time of all the preceding processes of the process.

3. The workshop production scheduling optimization method based on deep reinforcement learning according to claim 2, characterized in that, The initial scheduling state feature tensor is input into the deep reinforcement learning scheduling network. The deep reinforcement learning scheduling network extracts spatial structure features from the initial scheduling state feature tensor to generate a task-process association feature map. The task-process association feature map encodes the dependency constraint strength and resource competition relationship between different task-process nodes, including: The initial scheduling state feature tensor is input into the first convolutional kernel group of the deep reinforcement learning scheduling network. The first convolutional kernel group contains multiple convolutional kernels with different receptive field sizes. The initial scheduling state feature tensor is convolved by the first convolutional kernel group to generate a multi-scale task process local feature set. Each scale feature map in the multi-scale task process local feature set retains the neighborhood structure information of the task process node within the corresponding receptive field. The local feature set of the multi-scale task process is spliced ​​along the channel dimension to obtain a cross-scale fused feature tensor. The number of channels of the cross-scale fused feature tensor is equal to the sum of the number of channels of each feature map in the local feature set of the multi-scale task process. The cross-scale fusion feature tensor is input into the second convolutional kernel group of the deep reinforcement learning scheduling network. The second convolutional kernel group includes a one-dimensional dilated convolutional kernel. The one-dimensional dilated convolutional kernel is used to expand the receptive field of the features in the cross-scale fusion feature tensor along the path of task process dependency relationship, extract the long-range dependency relationship features between task process nodes, and generate a process dependency relationship enhancement feature map. The deep reinforcement learning scheduling network calculates the global average pooling value of each channel of the process dependency enhancement feature map, generates the attention weight coefficient of each channel based on the global average pooling value, and performs channel weight adjustment on the process dependency enhancement feature map using the attention weight coefficient to obtain the channel recalibrated feature map. The deep reinforcement learning scheduling network calculates the maximum and average values ​​of the channel recalibration feature map in the spatial dimension to generate a spatial attention mask. The spatial attention mask is then multiplied element-wise with the channel recalibration feature map to obtain a task process association feature map. The feature value of each spatial location in the task process association feature map encodes the dependency constraint strength and resource competition relationship between the task process node and other task process nodes.

4. The workshop production scheduling optimization method based on deep reinforcement learning according to claim 2, characterized in that, The deep reinforcement learning scheduling network calculates the attention weight distribution between each task process node and other task process nodes in the task process association feature map. Based on this attention weight distribution, the task process association feature map is weighted and aggregated to generate a set of process node context features with global dependency awareness, including: The deep reinforcement learning scheduling network performs matrix multiplication operations on the task process association feature map with the query weight matrix, key weight matrix and value weight matrix respectively to generate the query feature matrix, key feature matrix and value feature matrix. The number of rows of the query feature matrix, key feature matrix and value feature matrix is ​​equal to the number of task process nodes, and the number of columns is equal to the attention feature dimension. The deep reinforcement learning scheduling network calculates the dot product between the query feature matrix and the transpose of the key feature matrix to obtain the original attention score matrix. Each element in the original attention score matrix is ​​divided by the square root of the attention feature dimension to obtain the scaled attention score matrix. The deep reinforcement learning scheduling network obtains the preceding process constraints recorded in the task process dependency description set, generates a mask matrix based on the preceding process constraints, and the mask value corresponding to the prohibited access position in the mask matrix is ​​negative infinity. By adding the scaled attention score matrix to the mask matrix element by element, the constrained attention score matrix is ​​obtained. The deep reinforcement learning scheduling network performs exponential normalization on each row of the constrained attention score matrix to generate an attention weight matrix. Each element of the attention weight matrix represents the attention weight value between the task process node in the corresponding row and the task process node in the corresponding column. The attention weight matrix and the value feature matrix are multiplied to obtain a weighted aggregated feature matrix. The weighted aggregated feature matrix is ​​then mapped back to the original feature dimension through the deep reinforcement learning scheduling network to generate a process node context feature set with global dependency awareness.

5. The workshop production scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The step of invoking the deep reinforcement learning scheduling network to perform cumulative revenue prediction processing on the sequence of work process allocation decision actions, generating a state-action value evaluation vector corresponding to the sequence of work process allocation decision actions, includes: The set of workshop production task units and the sequence of process allocation decision actions are input into the deep reinforcement learning scheduling network. The deep reinforcement learning scheduling network performs joint feature extraction on the task identifiers, task process sequences, task delivery time window boundaries and task process dependency description sets in the set of workshop production task units and the processing equipment identifiers and process processing start time parameters in the sequence of process allocation decision actions, generating a state-action joint feature tensor. The state-action joint feature tensor is input into the temporal convolutional layer of the deep reinforcement learning scheduling network. The temporal convolutional layer contains multiple dilated convolutional kernels. The temporal convolutional layer performs temporal feature scanning on the state-action joint feature tensor along the time dimension of the process sequence, extracts the temporal dependency pattern in the process allocation decision action sequence, and generates a temporal augmentation feature tensor. The deep reinforcement learning scheduling network calculates the average value of each channel of the temporal augmentation feature tensor to obtain the global average pooling feature. The deep reinforcement learning scheduling network calculates the maximum value of each channel of the temporal augmentation feature tensor to obtain the global max pooling feature. The global average pooling feature and the global max pooling feature are concatenated along the channel dimension to obtain the global statistical feature vector. The global statistical feature vector is input into the value regression network of the deep reinforcement learning scheduling network. The value regression network contains multiple fully connected layers and non-linear activation units located between the fully connected layers. The global statistical feature vector is regressed and mapped layer by layer through the value regression network to generate a state-action value scalar. The state-action value scalar represents the cumulative revenue prediction value that can be obtained after executing the process allocation decision action sequence from the current scheduling state. The state action value scalar is copied and expanded into a state action value evaluation vector with the same length as the process allocation decision action sequence, wherein the component at each position in the state action value evaluation vector is equal to the state action value scalar.

6. The workshop production scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of storing the set of workshop production task units, the sequence of process allocation decision actions, and the state action value evaluation vectors as a scheduling experience transfer unit through the deep reinforcement learning scheduling network includes: The deep reinforcement learning scheduling network is used to convert the task identifiers, task process sequences, task delivery time window boundaries and task process dependency description sets in the workshop production task unit set into fixed-dimensional task embedding vectors to obtain task embedding feature vectors. The deep reinforcement learning scheduling network converts the processing equipment identifier and process start time parameter in the process allocation decision action sequence into a fixed-dimensional action embedding vector to obtain the action embedding feature vector. The deep reinforcement learning scheduling network normalizes and compresses each component of the state-action value evaluation vector to generate a value embedding feature vector. The dimension of the value embedding feature vector is consistent with the dimension of the task embedding feature vector and the action embedding feature vector. The task embedding feature vector, action embedding feature vector, and value embedding feature vector are concatenated along the channel dimension to obtain a combined experience feature tensor. The combined experience feature tensor is stored in the circular buffer queue of the deep reinforcement learning scheduling network. The circular buffer queue manages multiple scheduling experience transfer units in a first-in-first-out order. In the circular buffer queue, a timestamp tag and a priority weight coefficient are attached to each scheduling experience transfer unit. The priority weight coefficient is dynamically adjusted according to the absolute value of the state action value evaluation vector. The priority weight coefficient controls the sampling probability distribution when sampling scheduling experience transfer units from the circular buffer queue.

7. The workshop production scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning scheduling network is driven by the scheduling experience transfer unit to update the policy output probability distribution. The policy output probability distribution generates an optimal process allocation decision sequence for subsequent scheduling cycles, including: Multiple scheduling experience transfer units are sampled from the circular buffer queue of the deep reinforcement learning scheduling network according to priority weight coefficients to obtain a sampled experience batch set. The sampled experience batch set contains multiple combined experience feature tensors and state-action value evaluation vectors corresponding to each combined experience feature tensor. The deep reinforcement learning scheduling network calculates the policy gradient loss value between the policy output probability distribution and the process allocation decision action sequence recorded in the sampled experience batch set. At the same time, it calculates the temporal difference loss value between the state action value evaluation vector and the target value vector output by the target value network. The policy gradient loss value and the temporal difference loss value are weighted and summed to obtain the joint optimization loss scalar. The gradient values ​​of the joint optimization loss scalar with respect to the parameter weights of the deep reinforcement learning scheduling network are calculated through the deep reinforcement learning scheduling network, and the gradient values ​​are pruned to obtain a set of pruned gradient values. The parameter weights of the deep reinforcement learning scheduling network are updated based on the pruned gradient value set to obtain the updated deep reinforcement learning scheduling network. At the same time, the parameter weights of the target value network are updated using a soft update method, which uses interpolation coefficients to weight and fuse the parameter weights of the deep reinforcement learning scheduling network and the target value network. By adding probability-decreasing exploration noise to the action probability distribution output by the updated deep reinforcement learning scheduling network, the optimal process allocation decision action sequence is obtained.

8. The workshop production scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The step of driving the deep reinforcement learning scheduling network to update the policy output probability distribution based on the scheduling experience transfer unit, wherein the policy output probability distribution generates the sequence of optimization process allocation decision actions in subsequent scheduling cycles, further includes: Obtain the set of feedback workshop production task units collected after executing the sequence of optimized process allocation decision actions within the current scheduling cycle. The set of feedback workshop production task units includes the actual processing time parameters of completed processes and the equipment idle time window offset parameters. The set of feedback workshop production task units and the sequence of optimized process allocation decision actions are input into the deep reinforcement learning scheduling network. The deep reinforcement learning scheduling network calculates the time deviation penalty value based on the actual processing time parameter and the preset rated processing time parameter in the task process sequence, calculates the equipment utilization reward value based on the equipment idle time window offset parameter, and calculates the delivery delay penalty value based on the task delivery time window boundary and the expected completion time. The time deviation penalty value, equipment utilization reward value and delivery delay penalty value are weighted and summed to generate a feedback reward signal scalar. The feedback reward signal scalar is superimposed onto each component of the state-action value evaluation vector to obtain the modified state-action value evaluation vector. The set of feedback workshop production task units, the sequence of optimized process allocation decision actions, and the modified state action value evaluation vector are combined and stored as a feedback scheduling experience transfer unit, and the feedback scheduling experience transfer unit is added to the circular buffer queue. The feedback scheduling experience transfer unit drives the deep reinforcement learning scheduling network to update the policy output probability distribution again, forming a continuous optimization loop.

9. The workshop production scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of obtaining the set of workshop production task units includes task identifiers, task process sequences, task delivery time window boundaries, and a set of task process dependency descriptions, including: Extract all production work order records currently awaiting scheduling from the workshop manufacturing execution system. Each production work order record corresponds to a workshop production task unit. Assign a unique task identifier to each workshop production task unit. The task identifier distinguishes the identity of different workshop production task units. The process route document of each workshop production task unit is parsed, and multiple process description fields are extracted from the process route document according to the processing order. The multiple process description fields are combined into a task process sequence according to the extraction order. The order of the process description fields in the task process sequence represents the processing sequence constraint that the workshop production task unit must follow. Read customer order information corresponding to each workshop production task unit from the workshop manufacturing execution system, extract the earliest allowed start time and the latest allowed completion time from the customer order information, combine the earliest allowed start time and the latest allowed completion time as the task delivery time window boundary, and use the task delivery time window boundary to determine whether the process allocation decision action sequence meets the delivery constraints. Analyze the pre- and post-process dependencies between processes in the task process sequence of each workshop production task unit. For each process, determine the process identifiers of all direct preceding processes. Associate and store the process identifiers of all direct preceding processes with the process identifier of the process to generate a task process dependency description set. Each dependency record in the task process dependency description set indicates that a process can only start processing after another process is completed.

10. A workshop production scheduling optimization system based on deep reinforcement learning, characterized in that, The deep reinforcement learning-based workshop production scheduling optimization system includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to implement the deep reinforcement learning-based workshop production scheduling optimization method according to any one of claims 1-9.

Images