Chip production logistics optimization scheduling method and system based on deep reinforcement learning

Abstract

The present invention provides a method and system for optimizing and scheduling chip production logistics based on deep reinforcement learning, which relates to the technical field of deep reinforcement learning, including: collecting real-time data of chip production lines, using improved bidirectional long short-term memory networks and graph neural networks to extract material flow timing laws, equipment operation cycle characteristics and logistics network topology characteristics, generating fusion state representation through contrast learning networks, building a hierarchical and collaborative deep reinforcement learning architecture, using an improved encoder-decoder structure to generate strategy vectors, and performing hierarchical joint training through a multi-agent execution network, inputting real-time states and strategy vectors into the trained deep reinforcement learning architecture, generating scheduling instructions, and combining a Bayesian neural network ensemble model and a recurrent neural network prediction model to perform hierarchical model predictive control to achieve optimized scheduling of chip production logistics.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning technology, and in particular to a chip production logistics optimization scheduling method and system based on deep reinforcement learning. Background Art

[0002] Chip manufacturing is an extremely complex process involving hundreds of production steps and the coordination of a large number of materials, equipment, and personnel. Efficient production logistics is crucial to ensuring on-time chip delivery, reducing production costs, and improving product quality. Traditional chip production logistics scheduling methods often have difficulty coping with the dynamic changes and complex constraints of the production environment.

[0003] With the continuous advancement of chip manufacturing technology and the expansion of production scale, the demand for more intelligent and adaptive logistics scheduling methods is growing. The existing chip production logistics optimization scheduling methods mainly have the problems of being difficult to effectively deal with the dynamics and uncertainty of the chip production environment, lacking in-depth mining of complex material flow rules and equipment operation cycle characteristics, and being difficult to achieve multi-objective collaborative optimization.

[0004] Therefore, a solution is urgently needed to solve the problems existing in the prior art. Summary of the invention

[0005] The embodiments of the present invention provide a chip production logistics optimization scheduling method and system based on deep reinforcement learning, which can at least solve some of the problems existing in the prior art.

[0006] A first aspect of an embodiment of the present invention provides a chip production logistics optimization scheduling method based on deep reinforcement learning, comprising:

[0007] Collect real-time data from the chip production line and input it into the hybrid feature extraction model. Use the improved bidirectional long short-term memory network module to extract the material flow timing rules and equipment operation cycle characteristics. Set residual connections and multi-head self-attention layers in the module, sparsely process the attention scores and generate discrete attention weights, which are weighted and combined with the timing features to generate a timing representation vector. Use the graph neural network module to extract the topological features of the logistics network, combine the dynamic adjacency matrix with the edge weights and node attribute information to construct a graph convolution operator, introduce jump connections and gating mechanisms to adjust the feature transfer strength, generate a spatial representation vector, input the timing and spatial representation vectors into the contrastive learning network, construct a multi-view contrast loss function, use the momentum encoder to update the negative sample queue, and output the fusion state representation.

[0008] Construct a hierarchical collaborative deep reinforcement learning architecture, build an improved encoder-decoder structure in the policy generation network, implant causal convolution modules and multi-scale feature pyramids, combine multi-scale features with fusion state representations and input them into the decoder, embed adaptive sampling layers and policy heads, generate policy vectors based on timing rules and device features, implant trust region constraint modules in the pre-set multi-agent execution network, construct hierarchical operation spaces and dual valuation networks, perform hierarchical joint training, transfer policy knowledge in combination with online distillation mechanisms, store experience samples in hierarchical experience pools, train recurrent neural network prediction models and dynamically adjust exploration strategies to determine real-time states;

[0009] The real-time state and the strategy vector are input into the trained deep reinforcement learning architecture to generate scheduling instructions, a Bayesian neural network ensemble model is constructed to model the logistics state, the uncertainty of the logistics state is determined by combining Monte Carlo random sampling, and hierarchical model predictive control is performed in combination with the output of the recurrent neural network prediction model. The step size and constraint boundaries are adjusted according to the material flow law and equipment cycle characteristics, the optimization problem is solved and the scheduling instructions are output, a recursive Bayesian estimator is constructed to calculate the confidence of the scheduling instructions, and if the confidence is lower than a preset confidence threshold, re-optimization is initiated, new samples are stored in a hierarchical experience pool, and the sampling probability is dynamically adjusted, the network parameters are continuously optimized, and an optimized scheduling plan is obtained.

[0010] In an optional embodiment,

[0011] Collect real-time data of the chip production line and input it into the hybrid feature extraction model. Use the improved bidirectional long short-term memory network module to extract the material flow timing law and equipment operation cycle characteristics. Set residual connections and multi-head self-attention layers in the module, sparsely process the attention scores and generate discrete attention weights, and combine them with the timing features to generate a timing representation vector. Use the graph neural network module to extract the topological features of the logistics network, combine the dynamic adjacency matrix with the edge weights and node attribute information to construct a graph convolution operator, introduce jump connections and gating mechanisms to adjust the feature transfer strength, generate a spatial representation vector, input the timing and spatial representation vectors into the contrast learning network, construct a multi-view contrast loss function, use the momentum encoder to update the negative sample queue, and output the fusion state representation including:

[0012] Collecting real-time data of the chip production line, the real-time data includes equipment process parameters, material attributes, work order information and logistics sensor data, wherein the equipment process parameters include temperature, pressure, power, speed and equipment switch status, operation mode, the material attributes include size specifications, material type, processing procedures, the work order information includes batch number, planned output, delivery time, and the logistics sensor data includes material location, transmission speed, and backlog status;

[0013] Preprocessing the real-time data, filling missing values ​​by moving average method and removing outliers by box plot method, mapping continuous features to the range of zero to one by maximum and minimum value normalization method, converting discrete features into binary vector representation by one-hot encoding method, and generating a mixed feature matrix;

[0014] Inputting the mixed feature matrix into an improved bidirectional long short-term memory network module, the improved bidirectional long short-term memory network module extracts temporal features in both forward and backward directions, and setting a residual connection in the improved bidirectional long short-term memory network module, passing the input features to the output end through a bypass to perform element-wise addition with the forward and backward hidden states;

[0015] A multi-head self-attention layer is stacked on the hidden layer of the improved bidirectional long short-term memory network module, each attention head projects the hidden state into three feature spaces of query, key, and value, an attention score matrix is ​​obtained by calculating the dot product of the query vector and the key vector, an adaptive threshold is set for the attention score matrix for sparse processing, a re-parameterization technique is used to discretize and sample the sparse attention scores to generate discrete attention weights, a weighted sum of the discrete attention weights and the value vector is performed to obtain an attention head output feature, and the output features of multiple attention heads are spliced ​​and linearly transformed and then residually connected with the original hidden state to obtain a time series representation vector;

[0016] The chip production line is constructed as an undirected graph model, with production equipment as nodes and material flow paths as edges. The adjacency matrix corresponding to each node is updated based on real-time monitoring data to generate a dynamic adjacency matrix. An attribute feature vector containing process parameters, operating status, and equipment type is constructed for each equipment node, and an edge feature vector containing transmission time, material quantity, and queue length is constructed for each flow path.

[0017] The dynamic adjacency matrix, the attribute feature vector of each node and the edge feature are input into the improved graph convolution operator, the spatial dependency of the logistics network is learned through the message passing mechanism, a skip connection is introduced in the graph convolution process to pass the attribute feature vector of each node in the previous layer to the current layer and add it to the convolution result, a gating mechanism is introduced after the graph convolution layer, forgetting gating and updating gating are generated by calculating feature similarity, and the feature transfer result after gate adjustment is spliced ​​with the initial feature of the node to obtain a spatial representation vector;

[0018] The temporal representation vector and the spatial representation vector are input into a contrastive learning network, positive sample pairs are generated through a data enhancement method, a momentum encoder is used to maintain a negative sample queue, a multi-view contrast loss function is constructed based on mutual information maximization, and the feature representation obtained by contrastive learning is concatenated with the original temporal representation vector and the spatial representation vector to obtain a fusion state representation vector.

[0019] In an optional embodiment,

[0020] The multi-view contrast loss function based on mutual information maximization is constructed as shown in the following formula:

[0021] ;

[0022] Among them, LD represents the loss value of contrastive learning, h i represents the representation vector of sample i, represents the positive sample representation of sample i, τ represents the temperature hyperparameter, sim(h i , ) represents the similarity between sample i and sample j, exp represents the exponential function, K represents the total number of negative samples, represents the kth negative sample representation of sample i.

[0023] In an optional embodiment,

[0024] Construct a layered collaborative deep reinforcement learning architecture, build an improved encoder-decoder structure in the strategy generation network, implant causal convolution modules and multi-scale feature pyramids, combine multi-scale features with fusion state representations and input them into the decoder, embed adaptive sampling layers and strategy heads, and combine timing rules and device features to generate strategy vectors, including:

[0025] Constructing a hierarchical collaborative deep reinforcement learning architecture, wherein the hierarchical collaborative deep reinforcement learning architecture includes an upper-layer strategy generation network and a lower-layer multi-agent execution network, wherein the upper-layer strategy generation network and the lower-layer multi-agent execution network interact with each other through a strategy transmission mechanism and a feedback mechanism;

[0026] Constructing an encoder structure of the upper-layer strategy generation network, the encoder structure includes a trunk feature extraction branch and an auxiliary information processing branch, wherein multiple layers of causal convolution are set in the trunk feature extraction branch, the convolution kernel size of the causal convolution increases layer by layer from 3 to 7, the number of channels of the causal convolution doubles layer by layer from 64 to 256, a batch normalization layer and a ReLU activation function are connected after each layer of the causal convolution, the trunk feature extraction branch outputs time series features, a fully connected layer is set in the auxiliary information processing branch, and the fully connected layer processes the equipment status information and the material information to obtain non-time series features;

[0027] A multi-scale feature pyramid module is implanted, and the temporal features and the non-temporal features are downsampled through the top-down feature transfer path of the multi-scale feature pyramid module to obtain feature maps of different resolutions, and the feature map of each resolution level is stored as a pre-order fusion state representation of the current level, and the feature map is upsampled and fused with the corresponding layer features through the bottom-up feature transfer path of the multi-scale feature pyramid module, and the fused features are concatenated with the fusion state representation of the previous level after adjusting the number of channels through convolution to obtain multi-scale features corresponding to the current level;

[0028] Constructing a decoder structure, inputting the multi-scale features into a gated recurrent unit in the decoder structure, wherein the hidden state dimension of the gated recurrent unit is 512, calculating the correlation between the current hidden state and the historical state through the attention mechanism in the decoder structure to generate an attention weight, and fusing the features according to the attention weight to obtain a fused feature;

[0029] An adaptive sampling layer is implanted, and the sampling frequency is dynamically adjusted based on the uncertainty of the fused features to obtain sampling features. The sampling features are input into the strategy head network, and the action distribution is predicted by multivariate Gaussian distribution modeling through the action distribution prediction branch of the strategy head network. The state value is estimated through the dual network structure of the state value estimation branch of the strategy head network.

[0030] In an optional embodiment,

[0031] The trust region constraint module is implanted in the pre-set multi-agent execution network, a hierarchical operation space and a dual valuation network are constructed, hierarchical joint training is performed, and the strategy knowledge is transferred by combining the online distillation mechanism. The experience samples are stored in the hierarchical experience pool, the recurrent neural network prediction model is trained and the exploration strategy is dynamically adjusted. The real-time status is determined, including:

[0032] A trust region constraint module is implanted in the preset lower-layer multi-agent execution network, and the initial value of the trust region radius of the trust region constraint module is set to 0.01 and linearly decays with the training process. If the policy update of the pre-acquired predicted action distribution exceeds the trust region, the learning rate is adjusted through binary search;

[0033] Construct a hierarchical operation space, map the predicted action distribution to the high-level operation space to obtain the process path planning strategy and the capacity allocation strategy, map to the middle-level operation space to obtain the equipment scheduling strategy and the material distribution strategy, and map to the low-level operation space to obtain the parameter adjustment strategy and the execution instruction strategy;

[0034] Constructing a dual valuation network, wherein the feature extraction layer of the dual valuation network uses a residual network structure to extract the features of the process path planning strategy, the capacity allocation strategy, the equipment scheduling strategy, the material distribution strategy, the parameter adjustment strategy, and the execution instruction strategy, and predicting the strategy value through the three-layer fully connected value prediction layer of the dual valuation network;

[0035] Perform layered pre-training, use expert data to pre-train each layer of the network independently, and then perform end-to-end fine-tuning, adopt an online distillation mechanism, set the initial value of the distillation temperature parameter of the online distillation mechanism to 20 and gradually reduce it during the training process, and pass the strategy knowledge of the teacher network to the student network;

[0036] A hierarchical experience pool is constructed, and experience samples containing state information, action information, reward information and next state information are stored in the hierarchical experience pool, where the capacity of each layer of the hierarchical experience pool is 1 million samples. A recurrent neural network prediction model is trained, and the experience samples are input into a long short-term memory network structure with a hidden layer dimension of 256 to predict the state and reward at the next moment. The exploration strategy is dynamically adjusted based on the uncertainty of the prediction and the real-time state is modified.

[0037] In an optional embodiment,

[0038] The real-time state and the strategy vector are input into the trained deep reinforcement learning framework to generate scheduling instructions, a Bayesian neural network ensemble model is constructed to model the logistics state, the uncertainty of the logistics state is determined by combining Monte Carlo random sampling, and hierarchical model predictive control is performed in combination with the output of the recurrent neural network prediction model. The step size and constraint boundaries are adjusted according to the material flow law and equipment cycle characteristics, the optimization problem is solved and the scheduling instructions are output, a recursive Bayesian estimator is constructed to calculate the confidence of the scheduling instructions. If the confidence is lower than a preset confidence threshold, re-optimization is started, new samples are stored in a hierarchical experience pool, and the sampling probability is dynamically adjusted. The network parameters are continuously optimized to obtain an optimized scheduling solution including:

[0039] Input the real-time state and the strategy vector into the trained deep reinforcement learning architecture to generate scheduling instructions, build a Bayesian neural network ensemble model with a mixed Gaussian variational autoencoder structure, perform feature extraction based on the work order information and logistics sensor data in the real-time data, build a material flow diagram based on the pre-acquired logistics network topology structure, add the material flow diagram to the Bayesian neural network ensemble model, and obtain the material flow law;

[0040] Based on the Monte Carlo random sampling idea, the Bayesian neural network ensemble model is sampled by a nested sampling method. The outer layer of the nested sampling method adopts importance sampling to construct a proposed distribution according to the gradient information of the logistics state. The inner layer adopts Hamiltonian Monte Carlo sampling, combined with an adaptive step size mechanism, and dynamically adjusts the sampling step size according to the material flow law. Repeated sampling determines the uncertainty of the logistics state.

[0041] Based on the uncertainty of the logistics state, a distributed model predictive control framework is constructed, the logistics state prediction problem is decomposed into sub-problems corresponding to multiple workstations, the augmented Lagrangian algorithm is used to coordinate the relationship between the sub-problems, and the adaptive barrier method is used to adjust the step size and constraint boundary according to the pre-acquired material flow law and the equipment cycle characteristics pre-extracted from the production equipment, a nonlinear constraint function considering the dynamic characteristics of the equipment is constructed, and the barrier parameter update step size is dynamically adjusted according to the degree of constraint violation;

[0042] The improved alternating direction multiplier method is used to solve the optimization problem and output the scheduling instruction. The optimization problem is decomposed into a state update subproblem and a control generation subproblem. The state update subproblem adopts the forward Euler method, and the control generation subproblem adopts the projected gradient method and introduces the momentum term to accelerate convergence. A recursive Bayesian estimator of a deep integrated learning structure is constructed to calculate the confidence of the scheduling instruction. The basic learner adopts an eight-layer deep residual network, each layer contains two residual blocks, and a soft voting mechanism is used for integration. When the confidence output by the recursive Bayesian estimator is lower than a preset confidence threshold, reoptimization is triggered and an improved neighbor retrieval algorithm is used for sample retrieval. The new sample is stored in a hierarchical experience pool of a hierarchical priority experience playback structure. A priority queue is set and the time decay factor is considered. The softmax function is used to calculate the sampling probability. The experience pool is divided into an online pool and an offline pool. The multi-agent collaborative learning framework is used to continuously optimize the network parameters of the deep residual network. The graph attention network is combined for agent interaction. The local value network adopts a double Q structure, and the global value network adopts a hierarchical attention mechanism to obtain an optimized scheduling solution.

[0043] In an optional embodiment,

[0044] The control generation subproblem adopts the projected gradient method and introduces the momentum term to accelerate convergence as shown in the following formula:

[0045] ;

[0046] Among them, θ (t+1) represents the new parameter value obtained by gradient update and projection operation, represents the projection operation, θ (t) represents the parameter value corresponding to the current time step t, η (t)represents the step size, β represents the momentum coefficient, is the momentum variable, which represents the weighted average of the gradient before the current time step, Represents the loss function with respect to the current parameter θ (t) The gradient of is used to indicate the adjustment direction of the parameters in the current iteration. Represents a set of constraints.

[0047] A second aspect of an embodiment of the present invention provides a chip production logistics optimization scheduling system based on deep reinforcement learning, including:

[0048] The first unit is used to collect real-time data of the chip production line and input it into the hybrid feature extraction model, use the improved bidirectional long short-term memory network module to extract the material flow timing law and equipment operation cycle characteristics, set residual connections and multi-head self-attention layers in the module, sparsely process the attention scores and generate discrete attention weights, and combine them with the time series features to generate a time series representation vector, use the graph neural network module to extract the topological features of the flow network, combine the dynamic adjacency matrix with the edge weights and node attribute information to construct a graph convolution operator, introduce jump connections and gating mechanisms to adjust the feature transfer strength, generate a spatial representation vector, input the time series and spatial representation vectors into the contrast learning network, construct a multi-view contrast loss function, use the momentum encoder to update the negative sample queue, and output the fusion state representation;

[0049] Unit 2 is used to build a hierarchical collaborative deep reinforcement learning architecture. It builds an improved encoder-decoder structure in the policy generation network, implants causal convolution modules and multi-scale feature pyramids, combines multi-scale features with fusion state representations into the decoder, embeds adaptive sampling layers and policy heads, generates policy vectors based on timing rules and device features, implants trust region constraint modules in the pre-set multi-agent execution network, builds hierarchical operation spaces and dual valuation networks, performs hierarchical joint training, transfers policy knowledge in combination with online distillation mechanisms, stores experience samples in hierarchical experience pools, trains recurrent neural network prediction models and dynamically adjusts exploration strategies, and determines real-time states.

[0050] The third unit is used to input the real-time state and the strategy vector into the trained deep reinforcement learning architecture, generate scheduling instructions, build a Bayesian neural network ensemble model to model the logistics state, combine Monte Carlo random sampling to determine the uncertainty of the logistics state, combine the output of the recurrent neural network prediction model to perform hierarchical model predictive control, adjust the step size and constraint boundaries according to the material flow law and equipment cycle characteristics, solve the optimization problem and output scheduling instructions, build a recursive Bayesian estimator to calculate the confidence of the scheduling instruction, if the confidence is lower than a preset confidence threshold, start re-optimization, store the new samples in the hierarchical experience pool and dynamically adjust the sampling probability, continuously optimize the network parameters, and obtain an optimized scheduling plan.

[0051] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including: a processor;

[0052] a memory for storing processor-executable instructions;

[0053] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0054] A fourth aspect of the embodiments of the present invention is:

[0055] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0056] In the present invention, by combining deep reinforcement learning, hybrid feature extraction model (including improved bidirectional long short-term memory network and graph neural network) and contrastive learning, it is possible to more accurately capture the complex material flow timing rules, equipment operation cycle characteristics and logistics network topology characteristics of the chip production line, thereby generating better scheduling instructions and effectively improving the efficiency of chip production logistics. The layered collaborative deep reinforcement learning architecture, combined with causal convolution, multi-scale feature pyramid, adaptive sampling and trust region constraints, can make the generated strategy more robust and adapt to complex production environment changes. The application of online distillation mechanism and dynamic exploration strategy further improves the adaptability and learning efficiency of the scheduling scheme. The Bayesian neural network ensemble model and Monte Carlo random sampling are used to evaluate the uncertainty of the logistics state, and the hierarchical model predictive control is combined with the recurrent neural network prediction model to control the logistics process more accurately. The introduction of recursive Bayesian estimator and reoptimization mechanism effectively improves the reliability of scheduling instructions and continuously optimizes the scheduling scheme through continuous learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of a process flow of a chip production logistics optimization scheduling method based on deep reinforcement learning according to an embodiment of the present invention;

[0058] Figure 2 This is a schematic diagram of the structure of a chip production logistics optimization scheduling system based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0061] Figure 1 FIG. 1 is a flow chart of a chip production logistics optimization scheduling method based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0062] S1. Collect real-time data of the chip production line and input it into the hybrid feature extraction model. Use the improved bidirectional long short-term memory network module to extract the material flow timing law and equipment operation cycle characteristics. Set residual connections and multi-head self-attention layers in the module, sparsely process the attention scores and generate discrete attention weights, and combine them with the timing features to generate a timing representation vector. Use the graph neural network module to extract the topological features of the logistics network, combine the dynamic adjacency matrix with the edge weights and node attribute information to construct a graph convolution operator, introduce jump connections and gating mechanisms to adjust the feature transfer strength, generate a spatial representation vector, input the timing and spatial representation vectors into the contrast learning network, construct a multi-view contrast loss function, use the momentum encoder to update the negative sample queue, and output the fusion state representation;

[0063] The hybrid feature extraction model is a model that combines multiple feature extraction methods. It improves the representation ability of the model by integrating different types of features (for example, image features, text features, and time series features), and aims to extract more comprehensive and discriminative feature representations, thereby enhancing the performance of the model in different tasks. The time series representation vector refers to the vector form representation obtained after encoding the time series data, which is usually used to capture the evolution law of data in the time dimension. The dynamic adjacency matrix is ​​a matrix that represents the relationship between nodes in the graph, which changes with time or conditions. The graph convolution operator is a convolution operation applied to graph structured data, which aims to aggregate information for each node through the features of adjacent nodes, thereby learning the relationship between nodes and the structural features of the graph. The multi-view contrast loss function is a loss function based on contrastive learning, which aims to learn the representation of data under multiple perspectives or modalities so that similar samples from different perspectives are close in the feature space and different samples from different perspectives are far away. The momentum encoder is a neural network encoder used to accelerate the training process. The update speed of the parameters is adjusted by using the momentum mechanism (i.e., weighting the gradient update), thereby avoiding excessive oscillation, promoting more stable convergence, and accelerating the training process of the model.

[0064] In an optional embodiment,

[0065] Collect real-time data of the chip production line and input it into the hybrid feature extraction model. Use the improved bidirectional long short-term memory network module to extract the material flow timing law and equipment operation cycle characteristics. Set residual connections and multi-head self-attention layers in the module, sparsely process the attention scores and generate discrete attention weights, and combine them with the timing features to generate a timing representation vector. Use the graph neural network module to extract the topological features of the logistics network, combine the dynamic adjacency matrix with the edge weights and node attribute information to construct a graph convolution operator, introduce jump connections and gating mechanisms to adjust the feature transfer strength, generate a spatial representation vector, input the timing and spatial representation vectors into the contrast learning network, construct a multi-view contrast loss function, use the momentum encoder to update the negative sample queue, and output the fusion state representation including:

[0066] Collecting real-time data of the chip production line, the real-time data includes equipment process parameters, material attributes, work order information and logistics sensor data, wherein the equipment process parameters include temperature, pressure, power, speed and equipment switch status, operation mode, the material attributes include size specifications, material type, processing procedures, the work order information includes batch number, planned output, delivery time, and the logistics sensor data includes material location, transmission speed, and backlog status;

[0067] Preprocessing the real-time data, filling missing values ​​by moving average method and removing outliers by box plot method, mapping continuous features to the range of zero to one by maximum and minimum value normalization method, converting discrete features into binary vector representation by one-hot encoding method, and generating a mixed feature matrix;

[0068] Inputting the mixed feature matrix into an improved bidirectional long short-term memory network module, the improved bidirectional long short-term memory network module extracts temporal features in both forward and backward directions, and setting a residual connection in the improved bidirectional long short-term memory network module, passing the input features to the output end through a bypass to perform element-wise addition with the forward and backward hidden states;

[0069] A multi-head self-attention layer is stacked on the hidden layer of the improved bidirectional long short-term memory network module, each attention head projects the hidden state into three feature spaces of query, key, and value, an attention score matrix is ​​obtained by calculating the dot product of the query vector and the key vector, an adaptive threshold is set for the attention score matrix for sparse processing, a re-parameterization technique is used to discretize and sample the sparse attention scores to generate discrete attention weights, a weighted sum of the discrete attention weights and the value vector is performed to obtain an attention head output feature, and the output features of multiple attention heads are spliced ​​and linearly transformed and then residually connected with the original hidden state to obtain a time series representation vector;

[0070] The chip production line is constructed as an undirected graph model, with production equipment as nodes and material flow paths as edges. The adjacency matrix corresponding to each node is updated based on real-time monitoring data to generate a dynamic adjacency matrix. An attribute feature vector containing process parameters, operating status, and equipment type is constructed for each equipment node, and an edge feature vector containing transmission time, material quantity, and queue length is constructed for each flow path.

[0071] The dynamic adjacency matrix, the attribute feature vector of each node and the edge feature are input into the improved graph convolution operator, the spatial dependency of the logistics network is learned through the message passing mechanism, a skip connection is introduced in the graph convolution process to pass the attribute feature vector of each node in the previous layer to the current layer and add it to the convolution result, a gating mechanism is introduced after the graph convolution layer, forgetting gating and updating gating are generated by calculating feature similarity, and the feature transfer result after gate adjustment is spliced ​​with the initial feature of the node to obtain a spatial representation vector;

[0072] The temporal representation vector and the spatial representation vector are input into a contrastive learning network, positive sample pairs are generated through a data enhancement method, a momentum encoder is used to maintain a negative sample queue, a multi-view contrast loss function is constructed based on mutual information maximization, and the feature representation obtained by contrastive learning is concatenated with the original temporal representation vector and the spatial representation vector to obtain a fusion state representation vector.

[0073] The box plot method is a commonly used statistical data visualization method for displaying the distribution of data sets, especially the median, quartiles and outliers of the data. The box plot helps to observe the central tendency and dispersion of the data by drawing the quartiles and extreme values ​​of the data. The maximum and minimum normalization method is a common data preprocessing method that linearly maps the data to a specified interval (usually [0, 1]). The reparameterization technique is a technique for training generative models (such as variational autoencoders). By introducing separable noise terms, the randomness of the model is stripped out of the parameter space, so that the gradient can be directly propagated, thereby supporting efficient back propagation and optimization. The discrete attention weight refers to the importance of different input positions determined by calculating the attention weight in discrete input data (such as sequence or graph data). The flow path refers to the transmission path of data or information in a dynamic system or graph model, which describes how information flows or propagates in the system. It is usually used in scenarios such as transportation networks, information dissemination, and social networks, and reflects the dependencies or interactions between nodes.

[0074] Real-time data is collected from various sensors and databases in the chip production line, including: equipment process parameters that reflect the working status of the equipment, such as temperature, pressure, power, speed, equipment switch status and operating mode; material attributes that describe material characteristics, such as size specifications, material type and processing procedures; work order information that records production plans, such as batch number, planned output and delivery time; and logistics sensor data that reflects material transportation conditions, such as material location, transmission speed and backlog status. For example, the data collected at a certain moment may include: the temperature of etcher No. 1 is 25 degrees Celsius, the pressure is 1.5 standard atmospheres, the power is 10 kilowatts, the speed is 5000 rpm, it is in the on state, and the operating mode is automatic mode; a batch of silicon wafers is 12 inches in size, the material is single crystal silicon, and the current process is photolithography; the work order number of this batch of silicon wafers is 20231026001, the planned output is 1,000 pieces, and the delivery time is November 1, 2023; the logistics sensor shows that this batch of silicon wafers is on the conveyor belt of the photolithography process, with a transmission speed of 0.5 meters per second, and there is no backlog.

[0075] Preprocess the collected real-time data. For the missing values ​​that may exist in the data, the moving average method is used to fill them, for example, the missing values ​​are replaced by the average value of several time points before and after the data point. In order to eliminate the influence of abnormal data on the model, the box plot method is used to eliminate outliers, that is, set upper and lower thresholds, and treat data points that exceed the threshold range as outliers and eliminate them. In order to unify the dimensions of different features, continuous features, such as temperature, pressure, power, etc., are mapped to the range of zero to one using the maximum and minimum value normalization method. For discrete features, such as equipment switch status, operating mode, material type, etc., the unique hot encoding method is used to convert them into binary vector representation. For example, the equipment switch status has only two states, "on" and "off", which can be represented by two binary vectors [1, 0] and [0, 1] respectively. After preprocessing, all features are combined into a mixed feature matrix.

[0076] The mixed feature matrix is ​​input into the improved bidirectional long short-term memory network module, and the time series features are extracted in the forward and backward directions respectively, so as to capture the information of the past and the future. In order to avoid the problems of gradient vanishing and gradient exploding, a residual connection is set in the module, and the input features are passed to the output end through a bypass, and are element-wise added to the forward and backward hidden states. For example, the input features at time t are added to the outputs of the forward LSTM and backward LSTM at time t, and a multi-head self-attention layer is stacked on top of the hidden layer. Each attention head projects the hidden state into three feature spaces: query, key, and value, and the attention score matrix is ​​obtained by calculating the dot product of the query vector and the key vector. In order to reduce the computational complexity, an adaptive threshold is set for the attention score matrix for sparse processing, and only important attention scores are retained. The Gumbel-Softmax reparameterization technique is used to discretize the sparse attention scores and generate discrete attention weights. The discrete attention weights and the value vector are weighted summed to obtain the attention head output features. The output features of multiple attention heads are concatenated and linearly transformed, and then residually connected with the original hidden state to obtain the final time series representation vector.

[0077] The chip production line is constructed as an undirected graph model, in which the production equipment is used as the nodes of the graph and the material flow path is used as the edge of the graph. The adjacency matrix is ​​dynamically updated based on real-time monitoring data to represent the connection relationship between the equipment. For each equipment node, an attribute feature vector containing process parameters, operating status, and equipment type is constructed, and for each flow path, an edge feature vector containing transmission time, material quantity, and queue length is constructed. For example, if etching machine No. 1 can transport materials to photolithography machine No. 2, the element at the corresponding position in the adjacency matrix is ​​1, otherwise it is 0. The node feature vector of etching machine No. 1 may include information such as its temperature, pressure, and operating status, and the feature vector of the edge connecting etching machine No. 1 and photolithography machine No. 2 may include information such as transmission time and material quantity.

[0078] The dynamic adjacency matrix, node features, and edge features are input into the improved graph convolution operator, which learns the spatial dependencies of the logistics network through the message passing mechanism. In order to fuse feature information at different levels, skip connections are introduced in the graph convolution process to pass the node features of the previous layer to the current layer and add them to the convolution results. In order to control the strength of feature transfer, a gating mechanism is introduced after the graph convolution layer. The forget gate and update gate are generated by calculating the feature similarity, and the feature transfer result after gate adjustment is spliced ​​with the initial node feature to obtain the spatial representation vector.

[0079] The temporal representation vector and the spatial representation vector are input into the contrastive learning network. Positive sample pairs are generated through data augmentation methods, such as randomly blocking some features. The momentum encoder is used to maintain the negative sample queue and store the representation vectors of historical samples. A multi-view contrast loss function is constructed based on mutual information maximization to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs. The feature representation obtained by contrastive learning is concatenated with the original temporal representation vector and the spatial representation vector to obtain the final fusion state representation vector.

[0080] In this embodiment, by fusing temporal and spatial features and combining the contrastive learning optimization model, the dynamic information of the chip production line can be captured more comprehensively, thereby improving the prediction accuracy. The application of technologies such as residual connection, attention mechanism sparsification, and gating mechanism can effectively alleviate the overfitting problem and enhance the robustness of the model. The sparse processing of the attention mechanism can reduce the amount of calculation and improve the operating efficiency of the model.

[0081] In an optional embodiment,

[0082] The multi-view contrast loss function based on mutual information maximization is constructed as shown in the following formula:

[0083] ;

[0084] Among them, LD represents the loss value of contrastive learning, h i represents the representation vector of sample i, represents the positive sample representation of sample i, τ represents the temperature hyperparameter, sim(h i , ) represents the similarity between sample i and sample j, exp represents the exponential function, K represents the total number of negative samples, represents the kth negative sample representation of sample i.

[0085] In this embodiment, through contrastive learning, the model can learn feature representations that are more invariant to changes in different perspectives (such as lighting, angle, occlusion, etc.), thereby improving the robustness of the model in complex scenes. By learning the common information between different perspectives, the model can better capture the essential characteristics of the sample, thereby improving the generalization ability of the model on unseen data. By learning feature representations that are more robust and generalizable, the performance of downstream tasks (such as image classification, object detection, etc.) can be significantly improved.

[0086] S2. Build a hierarchical collaborative deep reinforcement learning architecture, build an improved encoder-decoder structure in the policy generation network, implant causal convolution modules and multi-scale feature pyramids, combine multi-scale features with fusion state representations into the decoder, embed adaptive sampling layers and policy heads, generate policy vectors based on timing rules and device features, implant trust region constraint modules in the pre-set multi-agent execution network, build hierarchical operation spaces and dual valuation networks, perform hierarchical joint training, transfer policy knowledge in combination with online distillation mechanisms, store experience samples in hierarchical experience pools, train recurrent neural network prediction models and dynamically adjust exploration strategies, and determine real-time states;

[0087] The causal convolution module is a special convolution module designed to process time series data or causal relationship data. The strategy head is a part of the network used to generate the final decision in reinforcement learning, usually located in the output layer of the neural network, and generates the output of the strategy, such as action selection and probability distribution, based on the features extracted from the front-end network. The strategy vector refers to the vector used to represent the current strategy state in reinforcement learning, and extracts features from the environment state through the neural network to output the strategy or action. The trust region constraint module is a method used in strategy optimization to limit the amplitude of strategy updates and prevent drastic changes in the strategy during training. The hierarchical operation space is introduced in multi-task learning or multi-agent systems. The concept of input divides the entire action space into multiple levels, each level is responsible for different tasks or subtasks, the dual valuation network is an architecture in reinforcement learning, used to simultaneously estimate state values ​​and action values, usually composed of two neural networks, one for estimating the state value of the current strategy, and the other for estimating the value of each action, the hierarchical joint training is a training strategy, usually used in multi-task learning or multi-agent systems, the hierarchical experience pool is an experience replay technology, often used in reinforcement learning, especially in the environment of multi-task learning, by storing the experience of different tasks in different pools, allowing agents to share experience between different tasks or give priority to the experience of specific tasks, so as to improve learning efficiency.

[0088] In an optional embodiment,

[0089] Construct a layered collaborative deep reinforcement learning architecture, build an improved encoder-decoder structure in the strategy generation network, implant causal convolution modules and multi-scale feature pyramids, combine multi-scale features with fusion state representations and input them into the decoder, embed adaptive sampling layers and strategy heads, and combine timing rules and device features to generate strategy vectors, including:

[0090] Constructing a hierarchical collaborative deep reinforcement learning architecture, wherein the hierarchical collaborative deep reinforcement learning architecture includes an upper-layer strategy generation network and a lower-layer multi-agent execution network, wherein the upper-layer strategy generation network and the lower-layer multi-agent execution network interact with each other through a strategy transmission mechanism and a feedback mechanism;

[0091] Constructing an encoder structure of the upper-layer strategy generation network, the encoder structure includes a trunk feature extraction branch and an auxiliary information processing branch, wherein multiple layers of causal convolution are set in the trunk feature extraction branch, the convolution kernel size of the causal convolution increases layer by layer from 3 to 7, the number of channels of the causal convolution doubles layer by layer from 64 to 256, a batch normalization layer and a ReLU activation function are connected after each layer of the causal convolution, the trunk feature extraction branch outputs time series features, a fully connected layer is set in the auxiliary information processing branch, and the fully connected layer processes the equipment status information and the material information to obtain non-time series features;

[0092] A multi-scale feature pyramid module is implanted, and the temporal features and the non-temporal features are downsampled through the top-down feature transfer path of the multi-scale feature pyramid module to obtain feature maps of different resolutions, and the feature map of each resolution level is stored as a pre-order fusion state representation of the current level, and the feature map is upsampled and fused with the corresponding layer features through the bottom-up feature transfer path of the multi-scale feature pyramid module, and the fused features are concatenated with the fusion state representation of the previous level after adjusting the number of channels through convolution to obtain multi-scale features corresponding to the current level;

[0093] Constructing a decoder structure, inputting the multi-scale features into a gated recurrent unit in the decoder structure, wherein the hidden state dimension of the gated recurrent unit is 512, calculating the correlation between the current hidden state and the historical state through the attention mechanism in the decoder structure to generate an attention weight, and fusing the features according to the attention weight to obtain a fused feature;

[0094] An adaptive sampling layer is implanted, and the sampling frequency is dynamically adjusted based on the uncertainty of the fused features to obtain sampling features. The sampling features are input into the strategy head network, and the action distribution is predicted by multivariate Gaussian distribution modeling through the action distribution prediction branch of the strategy head network. The state value is estimated through the dual network structure of the state value estimation branch of the strategy head network.

[0095] The backbone feature extraction branch refers to the part of the neural network architecture used to extract the main features, usually the front end of the network. The auxiliary information processing branch is the part of the neural network used to process additional information or auxiliary data. The batch normalization layer is a common layer in a deep learning network, which is used to standardize the input of each layer, usually normalizing the data to a mean of 0 and a variance of 1 during the training process of each batch. The strategy head network is a reinforcement learning network structure specifically used to generate strategies based on input states. The multivariate Gaussian distribution is a common probability distribution, which is usually used to describe the joint distribution of multidimensional random variables.

[0096] Construct a hierarchical collaborative deep reinforcement learning architecture, including an upper-layer policy generation network and a lower-layer multi-agent execution network. The upper-layer policy generation network is responsible for generating macro-strategies based on global environmental information, while the lower-layer multi-agent execution network is responsible for generating specific actions based on the macro-strategies and its own local observations. The policy transfer mechanism transfers the macro-strategies generated by the upper layer to the lower-layer execution network, and the feedback mechanism transfers the execution results and environmental feedback information of the lower-layer execution network to the upper-layer policy generation network, realizing information interaction and policy adjustment between the upper and lower layers. For example, the upper-layer policy generation network outputs "occupy a favorable position", and the lower-layer multi-agent execution network plans a specific path based on its own position and target position. The feedback mechanism can be in the form of a reward function, such as a reward of 1 for completing the target and -1 for not completing it.

[0097] Construct the encoder structure of the upper-layer policy generation network, which includes a trunk feature extraction branch and an auxiliary information processing branch. The trunk feature extraction branch uses multiple layers of causal convolution to process time series information, such as historical trajectory data. The convolution kernel size of the causal convolution increases layer by layer, from 3 to 7, and the number of channels doubles layer by layer, from 64 to 256. Each layer of causal convolution is connected to a batch normalization layer and a ReLU activation function to enhance the expressive power of the network. The auxiliary information processing branch uses a fully connected layer to process non-time series features such as device status information and material information, such as the power of the device, the type and quantity of the material, etc. Assuming that the dimension of the device status information is 10 and the dimension of the material information is 20, the fully connected layer can map this information to a 64-dimensional vector.

[0098] A multi-scale feature pyramid module is implanted to perform multi-scale processing on the temporal features output by the trunk feature extraction branch and the non-temporal features output by the auxiliary information processing branch. Feature maps of different resolutions are generated and fused through top-down and bottom-up feature transfer paths. The feature map of each resolution level is stored as the previous fusion state representation of the current level. In the bottom-up process, the feature map is upsampled and fused with the features of the corresponding layer. The fused features are convolved to adjust the number of channels and then spliced ​​with the fusion state representation of the previous level to obtain the multi-scale features corresponding to the current level. For example, the original feature map can be downsampled to twice the resolution and four times the resolution, the feature map with four times the resolution can be upsampled to twice the resolution and fused with the feature map with two times the resolution, and then the fused feature map can be upsampled to the original resolution and fused with the original feature map.

[0099] Construct a decoder structure, input the multi-scale features into the gated recurrent unit in the decoder structure, and set the hidden state dimension of the gated recurrent unit to 512. The attention mechanism in the decoder structure calculates the correlation between the current hidden state and the historical state to generate the attention weight, and fuses the features according to the attention weight to obtain the fused features. For example, the current hidden state can be correlated with the hidden state of the past 10 time steps.

[0100] An adaptive sampling layer is implanted to dynamically adjust the sampling frequency based on the uncertainty of the fused features to obtain the sampling features. For example, when the uncertainty of the fused features is high, the sampling frequency is increased; when the uncertainty is low, the sampling frequency is reduced.

[0101] The sampled features are input into the policy head network, which contains an action distribution prediction branch and a state value estimation branch. The action distribution prediction branch uses multivariate Gaussian distribution modeling to predict action distribution, and the state value estimation branch uses a dual network structure to estimate state value. For example, the action can be a two-dimensional continuous variable, then the action distribution prediction branch outputs the mean and variance of the multivariate Gaussian distribution. The state value estimation branch outputs the value of the current state, such as the expected cumulative reward in the future.

[0102] In this embodiment, through the hierarchical architecture and multi-scale feature pyramid, information at different levels and scales can be effectively extracted and fused to improve the efficiency of strategy generation. The causal convolution module can capture the causal relationship of temporal information. The adaptive sampling layer can dynamically adjust the sampling frequency according to uncertainty, thereby enhancing the robustness of the strategy. Through the strategy transmission mechanism and feedback mechanism, effective information interaction between the upper-level strategy generation network and the lower-level multi-agent execution network can be achieved, thereby improving the collaborative performance of the multi-agent.

[0103] In an optional embodiment,

[0104] The trust region constraint module is implanted in the pre-set multi-agent execution network, a hierarchical operation space and a dual valuation network are constructed, hierarchical joint training is performed, and the strategy knowledge is transferred by combining the online distillation mechanism. The experience samples are stored in the hierarchical experience pool, the recurrent neural network prediction model is trained and the exploration strategy is dynamically adjusted. The real-time status is determined, including:

[0105] A trust region constraint module is implanted in the preset lower-layer multi-agent execution network, and the initial value of the trust region radius of the trust region constraint module is set to 0.01 and linearly decays with the training process. If the policy update of the pre-acquired predicted action distribution exceeds the trust region, the learning rate is adjusted through binary search;

[0106] Construct a hierarchical operation space, map the predicted action distribution to the high-level operation space to obtain the process path planning strategy and the capacity allocation strategy, map to the middle-level operation space to obtain the equipment scheduling strategy and the material distribution strategy, and map to the low-level operation space to obtain the parameter adjustment strategy and the execution instruction strategy;

[0107] Constructing a dual valuation network, wherein the feature extraction layer of the dual valuation network uses a residual network structure to extract the features of the process path planning strategy, the capacity allocation strategy, the equipment scheduling strategy, the material distribution strategy, the parameter adjustment strategy, and the execution instruction strategy, and predicting the strategy value through the three-layer fully connected value prediction layer of the dual valuation network;

[0108] Perform layered pre-training, use expert data to pre-train each layer of the network independently, and then perform end-to-end fine-tuning, adopt an online distillation mechanism, set the initial value of the distillation temperature parameter of the online distillation mechanism to 20 and gradually reduce it during the training process, and pass the strategy knowledge of the teacher network to the student network;

[0109] A hierarchical experience pool is constructed, and experience samples containing state information, action information, reward information and next state information are stored in the hierarchical experience pool, where the capacity of each layer of the hierarchical experience pool is 1 million samples. A recurrent neural network prediction model is trained, and the experience samples are input into a long short-term memory network structure with a hidden layer dimension of 256 to predict the state and reward at the next moment. The exploration strategy is dynamically adjusted based on the uncertainty of the prediction and the real-time state is modified.

[0110] The process path planning strategy refers to a method or algorithm adopted in the production or manufacturing process to optimize production efficiency, reduce costs, and reduce resource consumption, which is used to plan and optimize the path selection of each production link.

[0111] A lower-level multi-agent execution network is pre-set and a trust region constraint module is implanted. The initial trust region radius of this module is set to 0.01 and decays linearly as training progresses. For example, at the 1000th round of training, the trust region radius decays to 0.005. If the policy update of the predicted action distribution exceeds the trust region, the learning rate is adjusted through binary search to ensure that the policy update is within the trust region. For example, if the policy update exceeds the trust region when the learning rate is 0.01, the learning rate is adjusted to 0.005 and it is judged again whether it exceeds the trust region until a suitable learning rate is found.

[0112] Construct a hierarchical operation space and map the predicted action distribution to different operation space levels. For example, the highest level is the process path planning strategy and capacity allocation strategy. The middle level is the equipment scheduling strategy and material distribution strategy. The lowest level is the parameter adjustment strategy and execution instruction strategy. Assuming that the predicted action distribution is a 10-dimensional vector, its first two digits can be mapped to the process path planning strategy, the next two digits to the capacity allocation strategy, and so on.

[0113] A dual valuation network is constructed, and the feature extraction layer adopts a residual network structure to extract the features of strategies at each level. For example, the starting point, end point, path length and other features of the process path planning strategy can be extracted. The extracted features are then input into the three-layer fully connected value prediction layer to predict the strategy value. Assuming that 5 features are extracted for each strategy, a total of 30 features, the first fully connected layer can map 30 features to 128 dimensions, the second layer maps to 64 dimensions, and the third layer outputs the strategy value.

[0114] Perform layered pre-training and use expert data to pre-train each layer of the network independently. For example, you can use historical production data to train the process path planning strategy network, and fine-tune the pre-trained layers of the network end-to-end. During the fine-tuning process, an online distillation mechanism is used to transfer the policy knowledge of the teacher network to the student network. The distillation temperature parameter of the online distillation mechanism is initially set to 20 and is gradually reduced during the training process. For example, in the 1000th round of training, the distillation temperature is reduced to 10.

[0115] Construct a hierarchical experience pool and store experience samples containing state information, action information, reward information, and next state information into the hierarchical experience pool. The capacity of each level of the experience pool is 1 million samples. For example, an experience sample can include the current production state, the action taken, the reward obtained, and the production state at the next moment.

[0116] Train the recurrent neural network prediction model. Input the experience samples into the long short-term memory network structure with a hidden layer dimension of 256 to predict the state and reward at the next moment. For example, the state and action of the past 10 moments can be used as input to predict the state and reward at the next moment.

[0117] Dynamically adjust the exploration strategy based on the predicted uncertainty and modify the real-time state. For example, if the predicted uncertainty is high, increase the exploration ratio and try some new actions. If the predicted uncertainty is low, reduce the exploration ratio and mainly execute known effective actions.

[0118] In this embodiment, through hierarchical control and trust region constraints, each link of the production process can be controlled more finely, the control accuracy can be improved, and the errors in the production process can be reduced. Through the dual valuation network and online distillation mechanism, the stability and generalization ability of the strategy can be improved, the adaptability of the system to environmental changes can be enhanced, and the robustness can be improved. Through the recurrent neural network prediction model and dynamic exploration strategy, the production status can be predicted in advance, and the control strategy can be dynamically adjusted according to the prediction results, thereby improving production efficiency and reducing production costs.

[0119] S3. Input the real-time state and the strategy vector into the trained deep reinforcement learning architecture to generate scheduling instructions, build a Bayesian neural network ensemble model to model the logistics state, combine Monte Carlo random sampling to determine the uncertainty of the logistics state, combine the output of the recurrent neural network prediction model to perform hierarchical model predictive control, adjust the step size and constraint boundaries according to the material flow law and equipment cycle characteristics, solve the optimization problem and output scheduling instructions, build a recursive Bayesian estimator to calculate the confidence of the scheduling instructions, if the confidence is lower than the preset confidence threshold, start re-optimization, store the new samples in the hierarchical experience pool and dynamically adjust the sampling probability, continuously optimize the network parameters, and obtain the optimized scheduling plan.

[0120] The Bayesian neural network ensemble model is a neural network structure based on Bayesian theory. The uncertainty of the model is estimated by treating the parameters in the network as random variables and probabilistically modeling the parameters. The Monte Carlo random sampling is a statistical method that is widely used in the probabilistic solution of high-dimensional complex problems. The constraint boundary is a restriction defined in the optimization problem, which stipulates the specific conditions that the optimization variables must meet. The normalized Bayesian estimator is an estimation method based on Bayesian statistics, which estimates unknown parameters by calculating the posterior distribution. The reoptimization is a process of further optimizing or refining the solution based on the preliminary optimization results.

[0121] In an optional embodiment,

[0122] The real-time state and the strategy vector are input into the trained deep reinforcement learning framework to generate scheduling instructions, a Bayesian neural network ensemble model is constructed to model the logistics state, the uncertainty of the logistics state is determined by combining Monte Carlo random sampling, and hierarchical model predictive control is performed in combination with the output of the recurrent neural network prediction model. The step size and constraint boundaries are adjusted according to the material flow law and equipment cycle characteristics, the optimization problem is solved and the scheduling instructions are output, a recursive Bayesian estimator is constructed to calculate the confidence of the scheduling instructions. If the confidence is lower than a preset confidence threshold, re-optimization is started, new samples are stored in a hierarchical experience pool, and the sampling probability is dynamically adjusted. The network parameters are continuously optimized to obtain an optimized scheduling solution including:

[0123] Input the real-time state and the strategy vector into the trained deep reinforcement learning architecture to generate scheduling instructions, build a Bayesian neural network ensemble model with a mixed Gaussian variational autoencoder structure, perform feature extraction based on the work order information and logistics sensor data in the real-time data, build a material flow diagram based on the pre-acquired logistics network topology structure, add the material flow diagram to the Bayesian neural network ensemble model, and obtain the material flow law;

[0124] Based on the Monte Carlo random sampling idea, the Bayesian neural network ensemble model is sampled by a nested sampling method. The outer layer of the nested sampling method adopts importance sampling to construct a proposed distribution according to the gradient information of the logistics state. The inner layer adopts Hamiltonian Monte Carlo sampling, combined with an adaptive step size mechanism, and dynamically adjusts the sampling step size according to the material flow law. Repeated sampling determines the uncertainty of the logistics state.

[0125] Based on the uncertainty of the logistics state, a distributed model predictive control framework is constructed, the logistics state prediction problem is decomposed into sub-problems corresponding to multiple workstations, the augmented Lagrangian algorithm is used to coordinate the relationship between the sub-problems, and the adaptive barrier method is used to adjust the step size and constraint boundary according to the pre-acquired material flow law and the equipment cycle characteristics pre-extracted from the production equipment, a nonlinear constraint function considering the dynamic characteristics of the equipment is constructed, and the barrier parameter update step size is dynamically adjusted according to the degree of constraint violation;

[0126] The improved alternating direction multiplier method is used to solve the optimization problem and output the scheduling instruction. The optimization problem is decomposed into a state update subproblem and a control generation subproblem. The state update subproblem adopts the forward Euler method, and the control generation subproblem adopts the projected gradient method and introduces the momentum term to accelerate convergence. A recursive Bayesian estimator of a deep integrated learning structure is constructed to calculate the confidence of the scheduling instruction. The basic learner adopts an eight-layer deep residual network, each layer contains two residual blocks, and a soft voting mechanism is used for integration. When the confidence output by the recursive Bayesian estimator is lower than a preset confidence threshold, reoptimization is triggered and an improved neighbor retrieval algorithm is used for sample retrieval. The new sample is stored in a hierarchical experience pool of a hierarchical priority experience playback structure. A priority queue is set and the time decay factor is considered. The softmax function is used to calculate the sampling probability. The experience pool is divided into an online pool and an offline pool. The multi-agent collaborative learning framework is used to continuously optimize the network parameters of the deep residual network. The graph attention network is combined for agent interaction. The local value network adopts a double Q structure, and the global value network adopts a hierarchical attention mechanism to obtain an optimized scheduling solution.

[0127] The mixed Gaussian variational autoencoder structure is a generative model based on variational inference, combining the ideas of Gaussian mixture model and variational autoencoder (VAE). The material flow diagram is a tool that graphically represents the path and state of material flow during production, processing or transportation. The Hamiltonian Monte Carlo sampling is a Monte Carlo method based on the principle of Hamiltonian mechanics, which generates samples by simulating the movement of particles in the Hamiltonian force field. The augmented Lagrangian algorithm is an algorithm for solving constrained optimization problems. The constraints are converted into part of the optimization target by introducing Lagrangian multipliers, and the penalty for the constraints is increased by augmentation methods. The improved alternating direction multiplier method is an optimization algorithm. , which is particularly suitable for large-scale and distributed optimization problems. It decomposes the original problem into multiple sub-problems and solves these sub-problems alternately to obtain the optimal solution to the entire problem. The forward Euler method is a numerical integration method used to solve ordinary differential equations. The projected gradient method is an algorithm for constrained optimization problems. It combines gradient descent and projection operations. It uses gradient descent to perform preliminary updates, and then projects the updated solution into the constraint space to ensure that the final solution satisfies the constraints. The soft voting mechanism is an integrated learning method that makes a final decision by combining the prediction results of multiple models. In soft voting, each model outputs a probability distribution, and the final prediction result is obtained by the weighted average of the probability distributions.

[0128] In order to accurately model the state of complex logistics systems, a Bayesian neural network ensemble model is constructed, and a mixed Gaussian variational autoencoder structure is used to handle the high dimensionality and complexity of logistics data. Key features such as the priority of the work order, the type and location of the material, the operating status of the equipment, etc. are extracted from real-time work order information and logistics sensor data. The material flow diagram is constructed using the pre-acquired logistics network topology, such as the flow direction of the material, the connection relationship between the workstations, etc., and the diagram is integrated into the Bayesian neural network ensemble model to capture the law of material flow. For example, material A needs to be processed in workstations 1, 2, and 3 in sequence. In the material flow diagram, workstations 1, 2, and 3 are connected, and the flow direction of material A is marked.

[0129] In order to quantify the uncertainty of logistics state, the Monte Carlo random sampling method is used to sample the Bayesian neural network ensemble model. Specifically, a nested sampling method is adopted, the outer layer uses importance sampling, the proposed distribution is constructed according to the gradient information of the logistics state, and the inner layer uses Hamiltonian Monte Carlo sampling. At the same time, combined with the adaptive step size mechanism, the sampling step size is dynamically adjusted according to the learned material flow law. For example, if the material flow diagram shows that the material flow speed of a certain station is faster, a smaller step size is used when sampling the station to capture its state changes more finely. By repeated sampling, a series of possible logistics state samples are obtained to determine the uncertainty of the logistics state. For example, by sampling 1000 times, 1000 possible logistics state samples are obtained, and the mean and variance of each state variable can be calculated to quantify its uncertainty.

[0130] Based on the obtained logistics state uncertainty, a distributed model predictive control framework is constructed to decompose the overall logistics state prediction problem into multiple sub-problems corresponding to each workstation, and the augmented Lagrangian algorithm is used to coordinate the relationship between the sub-problems to ensure global optimality. According to the pre-acquired material flow law and the equipment cycle characteristics extracted from the production equipment, the adaptive barrier method is used to dynamically adjust the step size and constraint boundaries. For example, if the cycle of a certain equipment is long, a larger step size is used when controlling it to reduce the amount of calculation. In addition, a nonlinear constraint function considering the dynamic characteristics of the equipment is constructed, and the barrier parameters are dynamically adjusted according to the degree of constraint violation to ensure the feasibility of the control strategy. For example, if the production capacity of a certain equipment is limited, its maximum production capacity limit needs to be considered when constructing the constraint function.

[0131] The improved alternating direction multiplier method is used to solve the optimization problem and output the scheduling instructions, and the optimization problem is decomposed into a state update subproblem and a control generation subproblem. The state update subproblem is solved by the forward Euler method, and the control generation subproblem is solved by the projected gradient method and the momentum term is introduced to accelerate convergence. For example, in the control generation subproblem, the momentum term can be set to 0.9 to speed up the convergence.

[0132] In order to evaluate the reliability of scheduling instructions, a recursive Bayesian estimator with a deep integrated learning structure is constructed to calculate the confidence of scheduling instructions. An eight-layer deep residual network is used as the basic learner, each layer contains two residual blocks, and a soft voting mechanism is used for integration. If the confidence of the estimator output is lower than the preset confidence threshold, the reoptimization process is triggered.

[0133] During the re-optimization process, an improved nearest neighbor retrieval algorithm is used to retrieve similar samples from the experience pool, and the new samples are stored in the hierarchical experience pool of the hierarchical priority experience playback structure. The priority queue is set and the time decay factor is considered. The softmax function is used to calculate the sampling probability. The experience pool is divided into an online pool and an offline pool to balance the use of new and old data. For example, the size of the online pool can be set to 1000, the size of the offline pool can be set to 10000, and the time decay factor can be set to 0.9.

[0134] The multi-agent collaborative learning framework is used to continuously optimize the network parameters of the deep residual network. The graph attention network is used for agent interaction, the local value network adopts a double Q structure, and the global value network adopts a hierarchical attention mechanism. For example, the number of agents can be set to 10 and the learning rate can be set to 0.001. Through continuous optimization, the performance of the scheduling solution is continuously improved, and finally an optimized scheduling solution is obtained.

[0135] In this embodiment, through the combination of deep reinforcement learning and model predictive control, high-quality scheduling instructions can be quickly generated, and the scheduling efficiency can be significantly improved. The uncertainty of the logistics state is taken into account, and the Bayesian neural network ensemble model is used for modeling. It can effectively respond to various disturbances in the logistics system and enhance scheduling robustness. Through the distributed model predictive control framework and the adaptive barrier method, resource allocation can be optimized and resource utilization can be improved.

[0136] In an optional embodiment,

[0137] The control generation subproblem adopts the projected gradient method and introduces the momentum term to accelerate convergence as shown in the following formula:

[0138] ;

[0139] Among them, θ (t+1) represents the new parameter value obtained by gradient update and projection operation, represents the projection operation, θ (t) represents the parameter value corresponding to the current time step t, η (t) represents the step size, β represents the momentum coefficient, is the momentum variable, which represents the weighted average of the gradient before the current time step, Represents the loss function with respect to the current parameter θ (t) The gradient of is used to indicate the adjustment direction of the parameters in the current iteration. Represents a set of constraints.

[0140] In this embodiment, the projection operation is used to ensure that the parameters are always within the constraint set and can meet the limitations of the actual problem. The introduction of the momentum term can accelerate the convergence speed and reduce the training time. The momentum term helps to escape from the local optimal solution and find a better solution.

[0141] Figure 2 FIG. 1 is a schematic diagram of the structure of a chip production logistics optimization scheduling system based on deep reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0142] The first unit is used to collect real-time data of the chip production line and input it into the hybrid feature extraction model, use the improved bidirectional long short-term memory network module to extract the material flow timing law and equipment operation cycle characteristics, set residual connections and multi-head self-attention layers in the module, sparsely process the attention scores and generate discrete attention weights, and combine them with the time series features to generate a time series representation vector, use the graph neural network module to extract the topological features of the flow network, combine the dynamic adjacency matrix with the edge weights and node attribute information to construct a graph convolution operator, introduce jump connections and gating mechanisms to adjust the feature transfer strength, generate a spatial representation vector, input the time series and spatial representation vectors into the contrast learning network, construct a multi-view contrast loss function, use the momentum encoder to update the negative sample queue, and output the fusion state representation;

[0143] Unit 2 is used to build a hierarchical collaborative deep reinforcement learning architecture. It builds an improved encoder-decoder structure in the policy generation network, implants causal convolution modules and multi-scale feature pyramids, combines multi-scale features with fusion state representations into the decoder, embeds adaptive sampling layers and policy heads, generates policy vectors based on timing rules and device features, implants trust region constraint modules in the pre-set multi-agent execution network, builds hierarchical operation spaces and dual valuation networks, performs hierarchical joint training, transfers policy knowledge in combination with online distillation mechanisms, stores experience samples in hierarchical experience pools, trains recurrent neural network prediction models and dynamically adjusts exploration strategies, and determines real-time states.

[0144] The third unit is used to input the real-time state and the strategy vector into the trained deep reinforcement learning architecture, generate scheduling instructions, build a Bayesian neural network ensemble model to model the logistics state, combine Monte Carlo random sampling to determine the uncertainty of the logistics state, combine the output of the recurrent neural network prediction model to perform hierarchical model predictive control, adjust the step size and constraint boundaries according to the material flow law and equipment cycle characteristics, solve the optimization problem and output scheduling instructions, build a recursive Bayesian estimator to calculate the confidence of the scheduling instruction, if the confidence is lower than a preset confidence threshold, start re-optimization, store the new samples in the hierarchical experience pool and dynamically adjust the sampling probability, continuously optimize the network parameters, and obtain an optimized scheduling plan.

[0145] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including: a processor;

[0146] a memory for storing processor-executable instructions;

[0147] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0148] A fourth aspect of the embodiments of the present invention is:

[0149] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0150] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A chip production logistics optimization scheduling method based on deep reinforcement learning, characterized in that: include: Collect real-time data from the chip production line and input it into the hybrid feature extraction model. Use the improved bidirectional long short-term memory network module to extract the material flow timing rules and equipment operation cycle characteristics. Set residual connections and multi-head self-attention layers in the module, sparsely process the attention scores and generate discrete attention weights, which are weighted and combined with the timing features to generate a timing representation vector. Use the graph neural network module to extract the topological features of the logistics network, combine the dynamic adjacency matrix with the edge weights and node attribute information to construct a graph convolution operator, introduce jump connections and gating mechanisms to adjust the feature transfer strength, generate a spatial representation vector, input the timing and spatial representation vectors into the contrastive learning network, construct a multi-view contrast loss function, use the momentum encoder to update the negative sample queue, and output the fusion state representation. Construct a hierarchical collaborative deep reinforcement learning architecture, build an improved encoder-decoder structure in the policy generation network, implant causal convolution modules and multi-scale feature pyramids, combine multi-scale features with fusion state representations and input them into the decoder, embed adaptive sampling layers and policy heads, generate policy vectors based on timing rules and device features, implant trust region constraint modules in the pre-set multi-agent execution network, construct hierarchical operation spaces and dual valuation networks, perform hierarchical joint training, transfer policy knowledge in combination with online distillation mechanisms, store experience samples in hierarchical experience pools, train recurrent neural network prediction models and dynamically adjust exploration strategies to determine real-time states; The real-time state and the strategy vector are input into the trained deep reinforcement learning architecture to generate scheduling instructions, a Bayesian neural network ensemble model is constructed to model the logistics state, the uncertainty of the logistics state is determined by combining Monte Carlo random sampling, and hierarchical model predictive control is performed in combination with the output of the recurrent neural network prediction model. The step size and constraint boundaries are adjusted according to the material flow law and equipment cycle characteristics, the optimization problem is solved and the scheduling instructions are output, a recursive Bayesian estimator is constructed to calculate the confidence of the scheduling instructions, and if the confidence is lower than a preset confidence threshold, re-optimization is initiated, new samples are stored in a hierarchical experience pool, and the sampling probability is dynamically adjusted, the network parameters are continuously optimized, and an optimized scheduling plan is obtained.

2. The method according to claim 1, characterized in that Collect real-time data of the chip production line and input it into the hybrid feature extraction model. Use the improved bidirectional long short-term memory network module to extract the material flow timing law and equipment operation cycle characteristics. Set residual connections and multi-head self-attention layers in the module, sparsely process the attention scores and generate discrete attention weights, and combine them with the timing features to generate a timing representation vector. Use the graph neural network module to extract the topological features of the logistics network, combine the dynamic adjacency matrix with the edge weights and node attribute information to construct a graph convolution operator, introduce jump connections and gating mechanisms to adjust the feature transfer strength, generate a spatial representation vector, input the timing and spatial representation vectors into the contrast learning network, construct a multi-view contrast loss function, use the momentum encoder to update the negative sample queue, and output the fusion state representation including: Collecting real-time data of the chip production line, the real-time data includes equipment process parameters, material attributes, work order information and logistics sensor data, wherein the equipment process parameters include temperature, pressure, power, speed and equipment switch status, operation mode, the material attributes include size specifications, material type, processing procedures, the work order information includes batch number, planned output, delivery time, and the logistics sensor data includes material location, transmission speed, and backlog status; Preprocessing the real-time data, filling missing values ​​by moving average method and removing outliers by box plot method, mapping continuous features to the range of zero to one by maximum and minimum value normalization method, converting discrete features into binary vector representation by one-hot encoding method, and generating a mixed feature matrix; Inputting the mixed feature matrix into an improved bidirectional long short-term memory network module, the improved bidirectional long short-term memory network module extracts temporal features in both forward and backward directions, and setting a residual connection in the improved bidirectional long short-term memory network module, passing the input features to the output end through a bypass to perform element-wise addition with the forward and backward hidden states; A multi-head self-attention layer is stacked on the hidden layer of the improved bidirectional long short-term memory network module, each attention head projects the hidden state into three feature spaces of query, key, and value, an attention score matrix is ​​obtained by calculating the dot product of the query vector and the key vector, an adaptive threshold is set for the attention score matrix for sparse processing, a re-parameterization technique is used to discretize and sample the sparse attention scores to generate discrete attention weights, a weighted sum of the discrete attention weights and the value vector is performed to obtain an attention head output feature, and the output features of multiple attention heads are spliced ​​and linearly transformed and then residually connected with the original hidden state to obtain a time series representation vector; The chip production line is constructed as an undirected graph model, with production equipment as nodes and material flow paths as edges. The adjacency matrix corresponding to each node is updated based on real-time monitoring data to generate a dynamic adjacency matrix. An attribute feature vector containing process parameters, operating status, and equipment type is constructed for each equipment node, and an edge feature vector containing transmission time, material quantity, and queue length is constructed for each flow path. The dynamic adjacency matrix, the attribute feature vector of each node and the edge feature are input into the improved graph convolution operator, the spatial dependency of the logistics network is learned through the message passing mechanism, a skip connection is introduced in the graph convolution process to pass the attribute feature vector of each node in the previous layer to the current layer and add it to the convolution result, a gating mechanism is introduced after the graph convolution layer, forgetting gating and updating gating are generated by calculating feature similarity, and the feature transfer result after gate adjustment is spliced ​​with the initial feature of the node to obtain a spatial representation vector; The temporal representation vector and the spatial representation vector are input into a contrastive learning network, positive sample pairs are generated through a data enhancement method, a momentum encoder is used to maintain a negative sample queue, a multi-view contrast loss function is constructed based on mutual information maximization, and the feature representation obtained by contrastive learning is concatenated with the original temporal representation vector and the spatial representation vector to obtain a fusion state representation vector.

3. The method according to claim 2, characterized in that The multi-view contrast loss function based on mutual information maximization is constructed as shown in the following formula: ； Among them, LD represents the loss value of contrastive learning, h i represents the representation vector of sample i, represents the positive sample representation of sample i, τ represents the temperature hyperparameter, sim(h i , ) represents the similarity between sample i and sample j, exp represents the exponential function, K represents the total number of negative samples, represents the kth negative sample of sample i.

4. The method according to claim 1, characterized in that: Construct a layered collaborative deep reinforcement learning architecture, build an improved encoder-decoder structure in the strategy generation network, implant causal convolution modules and multi-scale feature pyramids, combine multi-scale features with fusion state representations and input them into the decoder, embed adaptive sampling layers and strategy heads, and combine timing rules and device features to generate strategy vectors, including: Constructing a hierarchical collaborative deep reinforcement learning architecture, wherein the hierarchical collaborative deep reinforcement learning architecture includes an upper-layer strategy generation network and a lower-layer multi-agent execution network, wherein the upper-layer strategy generation network and the lower-layer multi-agent execution network interact with each other through a strategy transmission mechanism and a feedback mechanism; Constructing an encoder structure of the upper-layer strategy generation network, the encoder structure includes a trunk feature extraction branch and an auxiliary information processing branch, wherein multiple layers of causal convolution are set in the trunk feature extraction branch, the convolution kernel size of the causal convolution increases layer by layer from 3 to 7, the number of channels of the causal convolution doubles layer by layer from 64 to 256, a batch normalization layer and a ReLU activation function are connected after each layer of the causal convolution, the trunk feature extraction branch outputs time series features, a fully connected layer is set in the auxiliary information processing branch, and the fully connected layer processes the equipment status information and the material information to obtain non-time series features; A multi-scale feature pyramid module is implanted, and the temporal features and the non-temporal features are downsampled through the top-down feature transfer path of the multi-scale feature pyramid module to obtain feature maps of different resolutions, and the feature map of each resolution level is stored as a pre-order fusion state representation of the current level, and the feature map is upsampled and fused with the corresponding layer features through the bottom-up feature transfer path of the multi-scale feature pyramid module, and the fused features are concatenated with the fusion state representation of the previous level after adjusting the number of channels through convolution to obtain multi-scale features corresponding to the current level; Constructing a decoder structure, inputting the multi-scale features into a gated recurrent unit in the decoder structure, wherein the hidden state dimension of the gated recurrent unit is 512, calculating the correlation between the current hidden state and the historical state through the attention mechanism in the decoder structure to generate an attention weight, and fusing the features according to the attention weight to obtain a fused feature; An adaptive sampling layer is implanted, and the sampling frequency is dynamically adjusted based on the uncertainty of the fused features to obtain sampling features. The sampling features are input into the strategy head network, and the action distribution is predicted by multivariate Gaussian distribution modeling through the action distribution prediction branch of the strategy head network. The state value is estimated through the dual network structure of the state value estimation branch of the strategy head network.

5. The method according to claim 1, characterized in that The trust region constraint module is implanted in the pre-set multi-agent execution network, a hierarchical operation space and a dual valuation network are constructed, hierarchical joint training is performed, and the strategy knowledge is transferred by combining the online distillation mechanism. The experience samples are stored in the hierarchical experience pool, the recurrent neural network prediction model is trained and the exploration strategy is dynamically adjusted. The real-time status is determined, including: A trust region constraint module is implanted in the preset lower-layer multi-agent execution network, and the initial value of the trust region radius of the trust region constraint module is set to 0.01 and linearly decays with the training process. If the policy update of the pre-acquired predicted action distribution exceeds the trust region, the learning rate is adjusted through binary search; Construct a hierarchical operation space, map the predicted action distribution to the high-level operation space to obtain the process path planning strategy and the capacity allocation strategy, map to the middle-level operation space to obtain the equipment scheduling strategy and the material distribution strategy, and map to the low-level operation space to obtain the parameter adjustment strategy and the execution instruction strategy; Constructing a dual valuation network, wherein the feature extraction layer of the dual valuation network uses a residual network structure to extract the features of the process path planning strategy, the capacity allocation strategy, the equipment scheduling strategy, the material distribution strategy, the parameter adjustment strategy, and the execution instruction strategy, and predicting the strategy value through the three-layer fully connected value prediction layer of the dual valuation network; Perform layered pre-training, use expert data to pre-train each layer of the network independently, and then perform end-to-end fine-tuning, adopt an online distillation mechanism, set the initial value of the distillation temperature parameter of the online distillation mechanism to 20 and gradually reduce it during the training process, and pass the strategy knowledge of the teacher network to the student network; A hierarchical experience pool is constructed, and experience samples containing state information, action information, reward information and next state information are stored in the hierarchical experience pool, where the capacity of each layer of the hierarchical experience pool is 1 million samples. A recurrent neural network prediction model is trained, and the experience samples are input into a long short-term memory network structure with a hidden layer dimension of 256 to predict the state and reward at the next moment. The exploration strategy is dynamically adjusted based on the uncertainty of the prediction and the real-time state is modified.

6. The method according to claim 1, characterized in that The real-time state and the strategy vector are input into the trained deep reinforcement learning framework to generate scheduling instructions, a Bayesian neural network ensemble model is constructed to model the logistics state, the uncertainty of the logistics state is determined by combining Monte Carlo random sampling, and hierarchical model predictive control is performed in combination with the output of the recurrent neural network prediction model. The step size and constraint boundaries are adjusted according to the material flow law and equipment cycle characteristics, the optimization problem is solved and the scheduling instructions are output, a recursive Bayesian estimator is constructed to calculate the confidence of the scheduling instructions. If the confidence is lower than a preset confidence threshold, re-optimization is started, new samples are stored in a hierarchical experience pool, and the sampling probability is dynamically adjusted. The network parameters are continuously optimized to obtain an optimized scheduling solution including: Input the real-time state and the strategy vector into the trained deep reinforcement learning architecture to generate scheduling instructions, build a Bayesian neural network ensemble model with a mixed Gaussian variational autoencoder structure, perform feature extraction based on the work order information and logistics sensor data in the real-time data, build a material flow diagram based on the pre-acquired logistics network topology structure, add the material flow diagram to the Bayesian neural network ensemble model, and obtain the material flow law; Based on the Monte Carlo random sampling idea, the Bayesian neural network ensemble model is sampled by a nested sampling method. The outer layer of the nested sampling method adopts importance sampling to construct a proposed distribution according to the gradient information of the logistics state. The inner layer adopts Hamiltonian Monte Carlo sampling, combined with an adaptive step size mechanism, and dynamically adjusts the sampling step size according to the material flow law. Repeated sampling determines the uncertainty of the logistics state. Based on the uncertainty of the logistics state, a distributed model predictive control framework is constructed, the logistics state prediction problem is decomposed into sub-problems corresponding to multiple workstations, the augmented Lagrangian algorithm is used to coordinate the relationship between the sub-problems, and the adaptive barrier method is used to adjust the step size and constraint boundary according to the pre-acquired material flow law and the equipment cycle characteristics pre-extracted from the production equipment, a nonlinear constraint function considering the dynamic characteristics of the equipment is constructed, and the barrier parameter update step size is dynamically adjusted according to the degree of constraint violation; The improved alternating direction multiplier method is used to solve the optimization problem and output the scheduling instruction. The optimization problem is decomposed into a state update subproblem and a control generation subproblem. The state update subproblem adopts the forward Euler method, and the control generation subproblem adopts the projected gradient method and introduces the momentum term to accelerate convergence. A recursive Bayesian estimator of a deep integrated learning structure is constructed to calculate the confidence of the scheduling instruction. The basic learner adopts an eight-layer deep residual network, each layer contains two residual blocks, and a soft voting mechanism is used for integration. When the confidence output by the recursive Bayesian estimator is lower than a preset confidence threshold, reoptimization is triggered and an improved neighbor retrieval algorithm is used for sample retrieval. The new sample is stored in a hierarchical experience pool of a hierarchical priority experience playback structure. A priority queue is set and the time decay factor is considered. The softmax function is used to calculate the sampling probability. The experience pool is divided into an online pool and an offline pool. The multi-agent collaborative learning framework is used to continuously optimize the network parameters of the deep residual network. The graph attention network is combined for agent interaction. The local value network adopts a double Q structure, and the global value network adopts a hierarchical attention mechanism to obtain an optimized scheduling solution.

7. The method according to claim 6, characterized in that The control generation subproblem adopts the projected gradient method and introduces the momentum term to accelerate convergence as shown in the following formula: ； Among them, θ (t+1) represents the new parameter value obtained by gradient update and projection operation, represents the projection operation, θ (t) represents the parameter value corresponding to the current time step t, η (t) represents the step size, β represents the momentum coefficient, is the momentum variable, which represents the weighted average of the gradient before the current time step, Represents the loss function with respect to the current parameter θ (t) The gradient of is used to indicate the adjustment direction of the parameters in the current iteration. Represents a set of constraints.

8. A chip production logistics optimization scheduling system based on deep reinforcement learning, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect real-time data of the chip production line and input it into the hybrid feature extraction model, use the improved bidirectional long short-term memory network module to extract the material flow timing law and equipment operation cycle characteristics, set residual connections and multi-head self-attention layers in the module, sparsely process the attention scores and generate discrete attention weights, and combine them with the time series features to generate a time series representation vector, use the graph neural network module to extract the topological features of the flow network, combine the dynamic adjacency matrix with the edge weights and node attribute information to construct a graph convolution operator, introduce jump connections and gating mechanisms to adjust the feature transfer strength, generate a spatial representation vector, input the time series and spatial representation vectors into the contrast learning network, construct a multi-view contrast loss function, use the momentum encoder to update the negative sample queue, and output the fusion state representation; Unit 2 is used to build a hierarchical collaborative deep reinforcement learning architecture. It builds an improved encoder-decoder structure in the policy generation network, implants causal convolution modules and multi-scale feature pyramids, combines multi-scale features with fusion state representations into the decoder, embeds adaptive sampling layers and policy heads, generates policy vectors based on timing rules and device features, implants trust region constraint modules in the pre-set multi-agent execution network, builds hierarchical operation spaces and dual valuation networks, performs hierarchical joint training, transfers policy knowledge in combination with online distillation mechanisms, stores experience samples in hierarchical experience pools, trains recurrent neural network prediction models and dynamically adjusts exploration strategies, and determines real-time states. The third unit is used to input the real-time state and the strategy vector into the trained deep reinforcement learning architecture, generate scheduling instructions, build a Bayesian neural network ensemble model to model the logistics state, combine Monte Carlo random sampling to determine the uncertainty of the logistics state, combine the output of the recurrent neural network prediction model to perform hierarchical model predictive control, adjust the step size and constraint boundaries according to the material flow law and equipment cycle characteristics, solve the optimization problem and output scheduling instructions, build a recursive Bayesian estimator to calculate the confidence of the scheduling instruction, if the confidence is lower than a preset confidence threshold, start re-optimization, store the new samples in the hierarchical experience pool and dynamically adjust the sampling probability, continuously optimize the network parameters, and obtain an optimized scheduling plan.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images