Semiconductor manufacturing equipment grouping working method and system based on deep reinforcement learning

Abstract

The present invention provides a semiconductor manufacturing equipment grouping working method and system based on deep reinforcement learning, which relates to the field of semiconductor manufacturing technology, including collecting real-time operation data of equipment, extracting timing statistical features and coupling features, generating equipment dynamic performance scores, and constructing performance feature vectors. The initial grouping scheme is generated using deep reinforcement learning and particle swarm optimization algorithms, and optimized using dual Q networks and temporal difference learning algorithms. Finally, a group collaborative evaluation network based on a graph structure is constructed, and local combination features and global topological features are extracted through a multi-layer graph convolutional network. The grouping scheme is iteratively optimized using a double-layer evaluation criterion until a preset convergence threshold is met, and the final grouping scheme is output. The present invention can effectively improve the efficiency of equipment grouping, optimize resource allocation, and improve overall production efficiency.

Description

Technical Field

[0001] The present invention relates to semiconductor manufacturing technology, and in particular to a semiconductor manufacturing equipment grouping working method and system based on deep reinforcement learning. Background Art

[0002] Semiconductor manufacturing processes are constantly evolving, and multiple manufacturing equipment of different models and performances are often running simultaneously on the production line. In order to improve the overall efficiency of the production line, these devices need to be reasonably grouped to achieve collaborative operation. Existing equipment grouping mainly relies on the experience of process engineers. Although expert knowledge can be used, it is difficult to adapt to the dynamic changes in equipment status, and the optimality of the grouping scheme cannot be guaranteed.

[0003] Another common method is to group equipment by pre-set rules, such as simple classification according to equipment model, processing capacity and other parameters. This method does not take into account the actual operating status of the equipment and is difficult to cope with complex and changing production environments. There are also traditional machine learning methods that use clustering algorithms to group equipment. Although data-driven ideas are introduced, the practicality of the grouping results is poor because the timing characteristics of equipment performance and the collaborative relationship between equipment are not considered.

[0004] Existing technologies lack the ability to perceive and evaluate the dynamic operating status of devices in real time, making it difficult to accurately characterize device performance characteristics; they do not consider the timing dependencies during device state migration, making it impossible to optimize long-term decisions; and they ignore the collaborative optimization issues of performance matching within the device group and resource balancing between groups, making it difficult to achieve overall improvement in group performance. Therefore, it is urgent to propose a device grouping method that can adaptively perceive the dynamic performance of devices, optimize long-term decision benefits, and achieve group collaboration. Summary of the invention

[0005] The embodiments of the present invention provide a semiconductor manufacturing equipment group working method and system based on deep reinforcement learning, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] A semiconductor manufacturing equipment grouping working method based on deep reinforcement learning is provided, comprising:

[0008] Collect real-time operation data of multiple semiconductor manufacturing equipment in a continuous processing cycle, perform segmented sampling on the real-time operation data through a sliding window mechanism, extract the time series statistical characteristics of parameters and the coupling characteristics between parameters in each sliding window, input the time series statistical characteristics and coupling characteristics into a pre-trained recurrent neural network, generate a time series representation of equipment performance, build a hierarchical evaluation system based on the experience of process experts, generate a dynamic performance score for semiconductor manufacturing equipment, and build a performance feature vector with adaptive weights based on the dynamic performance score;

[0009] Input the performance feature vector into a deep reinforcement learning environment, construct a Markov decision process model that considers device state migration, map the device state space and the group action space into a particle swarm optimization problem, encode the grouping scheme into a particle position vector, and obtain an initial grouping scheme through iterative optimization search; construct a dual Q network based on the initial grouping scheme, the dual Q network includes an online evaluation network and a target value network, store historical decision sequences through an experience replay buffer pool, construct a temporal difference learning algorithm based on the historical decision sequences, calculate multidimensional reward values, update the parameters of the online evaluation network according to the multidimensional reward values, and optimize the initial grouping scheme to obtain a candidate grouping scheme;

[0010] A group collaborative evaluation network based on a graph structure is constructed, the equipment group relationship in the candidate grouping scheme is constructed as a dynamic weighted graph, the dynamic weighted graph is input into a multi-layer graph convolutional network, the local combination features within the equipment group and the global topological features between the equipment groups are extracted, and a double-layer evaluation criterion is constructed. When the evaluation result of the double-layer evaluation criterion does not reach a preset threshold, a new equipment allocation scheme is generated according to the local combination features and the global topological features, the execution result of the equipment allocation scheme is input into a deep reinforcement learning environment as a new state matrix for updating, and the equipment reallocation process is repeated until the double-layer evaluation criterion meets the preset convergence threshold, and the final equipment grouping scheme is output.

[0011] In an optional embodiment,

[0012] The timing statistical features and coupling features are input into a pre-trained recurrent neural network to generate a timing representation of equipment performance, and a hierarchical evaluation system is constructed in combination with process expert experience to generate a dynamic performance score for semiconductor manufacturing equipment. The performance feature vector with adaptive weights is constructed based on the dynamic performance score, including:

[0013] Acquire the timing statistical features and coupling features of semiconductor manufacturing equipment, calculate the multi-order statistical moments of the characteristic parameters in each sliding window, and concatenate the multi-order statistical moments with the timing statistical features and coupling features to construct an enhanced feature sequence;

[0014] Constructing a bidirectional recurrent neural network structure based on a capsule network, wherein the capsule network adaptively captures the hierarchical relationship of the feature sequence through a dynamic routing algorithm, the bidirectional recurrent neural network includes an attention-enhanced long short-term memory unit, the attention-enhanced long short-term memory unit introduces an adaptive weight matrix to dynamically adjust the contribution of the input feature, the enhanced feature sequence is input into the capsule network to extract hierarchical features, and the hierarchical features are initially represented in time series through the bidirectional recurrent neural network;

[0015] Designing a multi-head self-attention mechanism to perform feature fusion on the initial time series representation, wherein the multi-head self-attention mechanism includes multiple parallel attention subspaces, each attention subspace independently learns feature association patterns under different perspectives, and the output of each attention subspace is transformed through nonlinear transformation to obtain a fused time series representation;

[0016] Based on the fused timing representation and combined with the experience of process experts, a hierarchical evaluation system is constructed, wherein the hierarchical evaluation system includes evaluation indicators at four levels: equipment health, processing accuracy, production efficiency, and quality consistency. The key evaluation indicators at each level are screened by a recursive feature elimination algorithm, and the key evaluation indicators are combined with the fused timing representation to construct a multidimensional evaluation space;

[0017] Adopting an adaptive weight optimization strategy, a dual loss function including task relevance and weight sensitivity is constructed. Combining the fusion timing representation, a dynamic learning rate adjustment mechanism is designed. The weight distribution of evaluation indicators at each level is iteratively updated through an alternating optimization algorithm to obtain a dynamic performance score.

[0018] The dynamic performance score, the time difference feature of the dynamic performance score, the multi-scale spectrum feature based on wavelet transform and the fused time series representation are combined to construct a performance feature vector with adaptive weights, and the performance feature vector is used to characterize the static level, dynamic change characteristics and potential evolution trend of equipment performance.

[0019] In an optional embodiment,

[0020] Adopting the adaptive weight optimization strategy, the calculation formula of the dual loss function including task relevance and weight sensitivity is as follows:

[0021] ;

[0022] Among them, L(w) represents the dual loss function, n represents the total number of samples, and y i represents the true score of the i-th sample, w represents the weight vector of the evaluation index, T represents the transposition, x i represents the input feature vector of the i-th sample, λ1 represents the hyperparameter of the weight sensitivity loss, and w T x iIndicates the performance score of the device. Indicates the rating w T x i The gradient of the weight vector w of the evaluation index, ||·||1 represents the L1 norm of the gradient, which is used to measure the gradient sensitivity, λ2 represents the hyperparameter of weight regularization, α represents the regularization term balance factor, ||w||1 represents the L1 norm of the weight vector of the evaluation index, The L2 norm of the weight vector representing the evaluation index.

[0023] In an optional embodiment,

[0024] A Markov decision process model considering device state migration is constructed, the device state space and group action space are mapped into a particle swarm optimization problem, the grouping scheme is encoded into a particle position vector, and the initial grouping scheme is obtained through iterative optimization search, including:

[0025] Constructing a device state space and a group action space, wherein the device state space is composed of a performance feature vector, and the group action space contains a plurality of device grouping schemes, each of which defines a group allocation relationship of devices;

[0026] A state transition probability matrix is ​​constructed based on historical performance data, element values ​​of the state transition probability matrix are calculated by conditional probability estimation, and a Bayesian smoothing parameter is introduced to correct the state transition probability matrix to obtain a probability distribution of device state migration;

[0027] Mapping the performance feature vector in the device state space and the device grouping scheme in the grouping action space into a particle swarm optimization problem, encoding the device grouping scheme into a particle position vector representing the device group number, and encoding the change of the device group number into a velocity vector;

[0028] Constructing a multi-objective fitness function based on the device state transition probability distribution, the multi-objective fitness function includes an inter-group performance balance evaluation item, a resource utilization evaluation item, and an intra-group state similarity evaluation item, assigning an adaptive weight coefficient to the multi-objective fitness function, and calculating an evaluation score of the device grouping scheme;

[0029] Iteratively optimize the particle position vector according to the evaluation score, and use the dynamic inertia weight to update the speed vector, calculate the position update amount based on the individual historical optimal solution of the particle position vector and the current optimal solution of the group, and constrain the updated particle position vector to ensure that the limit on the number of devices in the group is met;

[0030] When the continuous iterative improvement of the evaluation score is less than a preset convergence threshold, the current optimal particle position vector is decoded as an initial grouping scheme.

[0031] In an optional embodiment,

[0032] Based on the initial grouping scheme, a dual Q network is constructed, the dual Q network includes an online evaluation network and a target value network, a historical decision sequence is stored through an experience replay buffer pool, a temporal difference learning algorithm is constructed based on the historical decision sequence, a multidimensional reward value is calculated, and the parameters of the online evaluation network are updated according to the multidimensional reward value. The initial grouping scheme is optimized to obtain a candidate grouping scheme, including:

[0033] The network parameters of the online evaluation network are set as initial parameters, and the initial parameters are partially copied to the target value network according to a preset parameter inheritance ratio to form target network parameters;

[0034] Construct an experience replay buffer pool for storing historical decision sequences, wherein each decision sample in the historical decision sequence includes a current state matrix, an execution action, a multidimensional reward value, and a next state matrix, wherein the multidimensional reward value includes intra-group performance balance, resource utilization efficiency, and state migration stability;

[0035] Calculating the time difference error of each decision sample in the historical decision sequence, determining the sampling priority of each decision sample based on the time difference error, and selecting a training sample from the experience replay buffer pool according to the sampling priority;

[0036] Input the training sample into the online evaluation network to obtain a first set of optimal actions, calculate the first Q value corresponding to the first set of optimal actions, input the training sample into the target value network to obtain a second set of optimal actions, calculate the second Q value corresponding to the second set of optimal actions, set an attenuation coefficient for the multi-dimensional reward value in the training sample, combine the attenuation coefficient with the first Q value and the second Q value to obtain a target Q value, calculate the deviation value between the target Q value and the current Q value output by the online evaluation network, and use the deviation value as the optimization target of the online evaluation network;

[0037] Based on the optimization target, network parameters of the online evaluation network are updated by back propagation to obtain updated parameters, and the updated parameters are copied to the target value network according to the preset parameter inheritance ratio;

[0038] Input the current state matrix into the updated online evaluation network, obtain the grouping adjustment scheme with the highest evaluation value, execute the grouping adjustment scheme to adjust the current grouping scheme, and calculate the adjusted state matrix and performance evaluation value;

[0039] The adjusted state matrix, the executed grouping adjustment plan, the performance evaluation value and the next state matrix form a new decision sample and are stored in the experience replay buffer pool. The iteration is repeated until the preset number of iterations is met, and the current grouping plan is determined as the optimized grouping plan.

[0040] In an optional embodiment,

[0041] Constructing a group collaborative evaluation network based on a graph structure, constructing the equipment group relationship in the candidate grouping scheme into a dynamic weighted graph, inputting the dynamic weighted graph into a multi-layer graph convolutional network, extracting local combination features within the equipment group and global topological features between equipment groups, and constructing a double-layer evaluation criterion. When the evaluation result of the double-layer evaluation criterion does not reach a preset threshold, generating a new equipment allocation scheme according to the local combination features and the global topological features includes:

[0042] The device group relationship in the candidate grouping scheme is constructed as a dynamic weighted graph model, wherein the dynamic weighted graph model includes a node set and an edge weight set, each node in the node set corresponds to a device, and the node includes a performance feature vector composed of a computing power index, a storage capacity index, and a communication bandwidth index;

[0043] Calculate the cosine similarity value based on the performance feature vectors of any two nodes in the node set and use it as the corresponding edge weight value;

[0044] The dynamic weighted graph is input into a multi-layer graph convolutional network, which includes a first graph convolutional layer and a second graph convolutional layer. For each node in the dynamic weighted graph model, the performance feature vectors of each node in the neighborhood node set of the current node and the corresponding edge weight values ​​are weighted and converged in the first graph convolutional layer, and the local combined features of the current node are obtained by processing with a first feature transformation matrix and a first activation function;

[0045] For each node in the dynamic weighted graph model, the local combined features of each node in the extended neighborhood node set of the current node and the corresponding edge weight value are weighted and converged in the second graph convolution layer, and the global topological features of the current node are obtained by processing the second feature transformation matrix and the second activation function;

[0046] Calculating the ratio of the minimum value to the maximum value of the local combination characteristics of the nodes in the device group to obtain the performance matching degree within the device group, and constructing an evaluation criterion for the performance matching degree within the group according to the performance matching degree within the device group;

[0047] Calculate the ratio of the standard deviation of the sum of the global topological characteristics of the nodes in all device groups to the mean to obtain the resource balance degree between the device groups, and construct an evaluation criterion for the resource balance degree between the groups according to the resource balance degree between the device groups;

[0048] A double-layer evaluation criterion is obtained by weighted combination of the intra-group performance matching evaluation criterion and the inter-group resource balance evaluation criterion, and a group collaboration evaluation score is calculated based on the double-layer evaluation criterion;

[0049] When the group collaborative evaluation score is less than a preset threshold, nodes with mismatched performance are determined based on local combination features, and a device group with complementary resources is determined based on global topological features, and a device reallocation plan is generated. The candidate grouping plan is adjusted according to the device reallocation plan to obtain a new device allocation plan.

[0050] In an optional embodiment,

[0051] The calculation formula for calculating the group collaborative evaluation score based on the two-layer evaluation criteria is as follows:

[0052] ;

[0053] Among them, G represents the group collaborative evaluation score, μ represents the importance weight of the performance matching within the group, M represents the total number of equipment groups, represents the local performance feature vector of the jth node in the kth device group, represents the smoothing factor, β represents the weight of resource balance between groups, A represents the total number of nodes in all device groups, and a k Indicates the number of nodes in the kth device group, represents the global feature of the jth node in the kth device group, and ||·|| represents the norm of the calculation vector.

[0054] According to a second aspect of the embodiments of the present invention,

[0055] A semiconductor manufacturing equipment grouping work system based on deep reinforcement learning is provided, comprising:

[0056] The first unit is used to collect real-time operation data of multiple semiconductor manufacturing equipment in a continuous processing cycle, perform segmented sampling on the real-time operation data through a sliding window mechanism, extract the time series statistical characteristics of parameters and the coupling characteristics between parameters in each sliding window, input the time series statistical characteristics and coupling characteristics into a pre-trained recurrent neural network, generate a time series representation of equipment performance, build a hierarchical evaluation system based on the experience of process experts, generate a dynamic performance score of the semiconductor manufacturing equipment, and build a performance feature vector with adaptive weights based on the dynamic performance score;

[0057] The second unit is used to input the performance feature vector into a deep reinforcement learning environment, construct a Markov decision process model that considers device state migration, map the device state space and the group action space into a particle swarm optimization problem, encode the grouping scheme into a particle position vector, and obtain an initial grouping scheme through iterative optimization search; construct a dual Q network based on the initial grouping scheme, the dual Q network includes an online evaluation network and a target value network, store historical decision sequences through an experience replay buffer pool, construct a temporal difference learning algorithm based on the historical decision sequences, calculate multidimensional reward values, update the parameters of the online evaluation network according to the multidimensional reward values, and optimize the initial grouping scheme to obtain a candidate grouping scheme;

[0058] The third unit is used to construct a group collaborative evaluation network based on a graph structure, construct the device group relationship in the candidate grouping scheme as a dynamic weighted graph, input the dynamic weighted graph into a multi-layer graph convolutional network, extract local combination features within the device group and global topological features between device groups, and construct a two-layer evaluation criterion. When the evaluation result of the two-layer evaluation criterion does not reach a preset threshold, a new device allocation scheme is generated according to the local combination features and the global topological features, and the execution result of the device allocation scheme is input into the deep reinforcement learning environment as a new state matrix for updating, and the device reallocation process is repeated until the two-layer evaluation criterion meets the preset convergence threshold, and the final device grouping scheme is output.

[0059] According to a third aspect of the embodiments of the present invention,

[0060] An electronic device is provided, comprising:

[0061] processor;

[0062] a memory for storing processor-executable instructions;

[0063] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0064] A fourth aspect of the embodiments of the present invention is:

[0065] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0066] In this embodiment, BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of a process flow of a semiconductor manufacturing equipment grouping working method based on deep reinforcement learning according to an embodiment of the present invention;

[0068] Figure 2This is a schematic diagram of the structure of a semiconductor manufacturing equipment grouping working system based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0071] Figure 1 FIG. 1 is a flow chart of a semiconductor manufacturing equipment grouping working method based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0072] S101. Collect real-time operation data of multiple semiconductor manufacturing equipment in a continuous processing cycle, perform segmented sampling on the real-time operation data through a sliding window mechanism, extract the time series statistical characteristics of parameters and the coupling characteristics between parameters in each sliding window, input the time series statistical characteristics and coupling characteristics into a pre-trained recurrent neural network, generate a time series representation of equipment performance, build a hierarchical evaluation system based on the experience of process experts, generate a dynamic performance score of the semiconductor manufacturing equipment, and build a performance feature vector with adaptive weights based on the dynamic performance score;

[0073] S102. Input the performance feature vector into a deep reinforcement learning environment, construct a Markov decision process model that considers device state migration, map the device state space and the group action space into a particle swarm optimization problem, encode the grouping scheme into a particle position vector, and obtain an initial grouping scheme through iterative optimization search; construct a dual Q network based on the initial grouping scheme, the dual Q network includes an online evaluation network and a target value network, store historical decision sequences through an experience replay buffer pool, construct a temporal difference learning algorithm based on the historical decision sequences, calculate multidimensional reward values, update the parameters of the online evaluation network according to the multidimensional reward values, and optimize the initial grouping scheme to obtain a candidate grouping scheme;

[0074] S103. Construct a group collaborative evaluation network based on a graph structure, construct the device group relationship in the candidate grouping scheme into a dynamic weighted graph, input the dynamic weighted graph into a multi-layer graph convolutional network, extract the local combination features within the device group and the global topological features between the device groups, and construct a two-layer evaluation criterion. When the evaluation result of the two-layer evaluation criterion does not reach a preset threshold, generate a new device allocation scheme according to the local combination features and the global topological features, input the execution result of the device allocation scheme as a new state matrix into the deep reinforcement learning environment for updating, and repeat the device reallocation process until the two-layer evaluation criterion meets the preset convergence threshold, and output the final device grouping scheme.

[0075] Among them, real-time operation data refers to various types of data generated in real time by multiple semiconductor manufacturing equipment during their continuous processing cycle, including temperature, pressure, speed, power consumption, equipment failure information, etc.

[0076] In an optional embodiment,

[0077] The timing statistical features and coupling features are input into a pre-trained recurrent neural network to generate a timing representation of equipment performance, and a hierarchical evaluation system is constructed in combination with process expert experience to generate a dynamic performance score for semiconductor manufacturing equipment. The performance feature vector with adaptive weights is constructed based on the dynamic performance score, including:

[0078] Acquire the timing statistical features and coupling features of semiconductor manufacturing equipment, calculate the multi-order statistical moments of the characteristic parameters in each sliding window, and concatenate the multi-order statistical moments with the timing statistical features and coupling features to construct an enhanced feature sequence;

[0079] Constructing a bidirectional recurrent neural network structure based on a capsule network, wherein the capsule network adaptively captures the hierarchical relationship of the feature sequence through a dynamic routing algorithm, the bidirectional recurrent neural network includes an attention-enhanced long short-term memory unit, the attention-enhanced long short-term memory unit introduces an adaptive weight matrix to dynamically adjust the contribution of the input feature, the enhanced feature sequence is input into the capsule network to extract hierarchical features, and the hierarchical features are initially represented in time series through the bidirectional recurrent neural network;

[0080] Designing a multi-head self-attention mechanism to perform feature fusion on the initial time series representation, wherein the multi-head self-attention mechanism includes multiple parallel attention subspaces, each attention subspace independently learns feature association patterns under different perspectives, and the output of each attention subspace is transformed through nonlinear transformation to obtain a fused time series representation;

[0081] Based on the fused timing representation and combined with the experience of process experts, a hierarchical evaluation system is constructed, wherein the hierarchical evaluation system includes evaluation indicators at four levels: equipment health, processing accuracy, production efficiency, and quality consistency. The key evaluation indicators at each level are screened by a recursive feature elimination algorithm, and the key evaluation indicators are combined with the fused timing representation to construct a multidimensional evaluation space;

[0082] Adopting an adaptive weight optimization strategy, a dual loss function including task relevance and weight sensitivity is constructed. Combining the fusion timing representation, a dynamic learning rate adjustment mechanism is designed. The weight distribution of evaluation indicators at each level is iteratively updated through an alternating optimization algorithm to obtain a dynamic performance score.

[0083] The dynamic performance score, the time difference feature of the dynamic performance score, the multi-scale spectrum feature based on wavelet transform and the fused time series representation are combined to construct a performance feature vector with adaptive weights, and the performance feature vector is used to characterize the static level, dynamic change characteristics and potential evolution trend of equipment performance.

[0084] Exemplarily, firstly, the timing statistical features and coupling features of the semiconductor manufacturing equipment are obtained. For example, the timing statistical features may include the data of the temperature, pressure, current, voltage, etc. of the equipment that change with time, and the coupling features may include the correlation, mutual information, etc. between different sensor data. Assume that the temperature, pressure and current data of the equipment running for 1000 time steps, as well as the correlation coefficient of temperature and pressure, are collected. In each sliding window of length 10, the multi-order statistical moments of each characteristic parameter, such as the mean, standard deviation, skewness and kurtosis, are calculated. The calculated statistical moments of each order are spliced ​​with the original timing statistical features and coupling features to construct an enhanced feature sequence. For example, in a sliding window, the mean of the temperature is 25°C, the standard deviation is 2°C, the skewness is 0.5, and the kurtosis is 3. These statistical moments are spliced ​​with the original temperature, pressure, current data and the correlation coefficient of temperature and pressure in the window to form the enhanced feature vector corresponding to the window. The enhanced feature vectors of all sliding windows are arranged in chronological order to form an enhanced feature sequence.

[0085] Construct a bidirectional recurrent neural network structure based on capsule network. The capsule network adaptively captures the hierarchical relationship of feature sequences through a dynamic routing algorithm. The dynamic routing algorithm iteratively adjusts the connection weights between capsules so that the output of low-level capsules can better match the expectations of high-level capsules. The bidirectional recurrent neural network contains attention-enhanced long short-term memory units. The attention-enhanced long short-term memory unit introduces an adaptive weight matrix to dynamically adjust the contribution of input features. The adaptive weight matrix dynamically adjusts the weights according to the importance of the input features, so that the model can pay more attention to important features. The enhanced feature sequence is input into the capsule network to extract hierarchical features, and then the hierarchical features are input into the bidirectional recurrent neural network to obtain the initial time series representation. For example, the enhanced feature sequence is input into a two-layer capsule network, the first layer of capsules captures local features, and the second layer of capsules captures global features. Then the output of the capsule network is input into a bidirectional LSTM network with an attention mechanism to obtain the initial time series representation of device performance.

[0086] A multi-head self-attention mechanism is designed to perform feature fusion on the initial time series representation. The multi-head self-attention mechanism contains multiple parallel attention subspaces, and each attention subspace independently learns the feature association pattern under different perspectives. For example, 8 attention heads are set, and each attention head learns the correlation between features at different time steps. The output of each attention subspace is transformed through a nonlinear transformation, such as a fully connected layer, to obtain a fused time series representation. The fused time series representation combines the feature association patterns learned from different attention subspaces, and more comprehensively expresses the temporal variation law of device performance.

[0087] Based on the fusion timing representation and combined with the experience of process experts, a hierarchical evaluation system is constructed. The hierarchical evaluation system includes evaluation indicators at four levels: equipment health, processing accuracy, production efficiency, and quality consistency. For example, equipment health can be measured by indicators such as temperature stability and vibration amplitude; processing accuracy can be measured by indicators such as dimensional error and surface roughness; production efficiency can be measured by indicators such as output per unit time and yield rate; quality consistency can be measured by indicators such as the volatility of product performance. The key evaluation indicators of each level are screened by the recursive feature elimination algorithm. For example, the recursive feature elimination algorithm determines that temperature stability and vibration amplitude are the key evaluation indicators of equipment health. The key evaluation indicators are combined with the fusion timing representation to construct a multidimensional evaluation space.

[0088] Adopting an adaptive weight optimization strategy, a dual loss function including task relevance and weight sensitivity is constructed. The loss function can be defined as the weighted sum of the mean square error between the predicted value and the true value of the evaluation indicators at each level, where the weight is determined by the task relevance and weight sensitivity. A dynamic learning rate adjustment mechanism is designed in combination with the fused temporal representation. For example, the learning rate is dynamically adjusted according to the changes in the fused temporal representation, reducing the learning rate when the performance changes dramatically and increasing the learning rate when the performance changes slowly. The weight distribution of the evaluation indicators at each level is iteratively updated through the alternating optimization algorithm to obtain a dynamic performance score.

[0089] The dynamic performance score, the time difference feature of the dynamic performance score, the multi-scale spectrum feature based on wavelet transform, and the fusion time series representation are combined to construct the performance feature vector with adaptive weights. The time difference feature reflects the changing trend of the dynamic performance score, and the multi-scale spectrum feature of the wavelet transform reflects the fluctuation of the dynamic performance score on different time scales. The performance feature vector is used to characterize the static level, dynamic change characteristics, and potential evolution trend of the equipment performance.

[0090] In an optional embodiment,

[0091] Adopting the adaptive weight optimization strategy, the calculation formula of the dual loss function including task relevance and weight sensitivity is as follows:

[0092] ;

[0093] Among them, L(w) represents the dual loss function, n represents the total number of samples, and y i represents the true score of the i-th sample, w represents the weight vector of the evaluation index, T represents the transposition, x i represents the input feature vector of the i-th sample, λ1 represents the hyperparameter of the weight sensitivity loss, and w T x i Indicates the performance score of the device. Indicates the rating w T x i The gradient of the weight vector w of the evaluation index, ||·||1 represents the L1 norm of the gradient, which is used to measure the gradient sensitivity, λ2 represents the hyperparameter of weight regularization, α represents the regularization term balance factor, ||w||1 represents the L1 norm of the weight vector of the evaluation index, The L2 norm of the weight vector representing the evaluation index.

[0094] In this embodiment, the equipment's time series statistical characteristics, coupling characteristics, hierarchical relationships, and multi-scale characteristics are comprehensively considered, which can more comprehensively capture the dynamic changes in equipment performance, thereby improving the accuracy and comprehensiveness of performance evaluation. By constructing a performance feature vector with adaptive weights, the static level, dynamic change characteristics, and potential evolution trends of equipment performance can be characterized, thereby improving the reliability and advanceness of performance prediction. According to the changes in the dynamic performance score, a more reasonable maintenance plan can be formulated to avoid excessive or insufficient maintenance; according to the performance prediction results, potential equipment failures can be warned in advance, and maintenance measures can be taken in time to avoid production interruptions.

[0095] In an optional embodiment,

[0096] A Markov decision process model considering device state migration is constructed, the device state space and group action space are mapped into a particle swarm optimization problem, the grouping scheme is encoded into a particle position vector, and the initial grouping scheme is obtained through iterative optimization search, including:

[0097] Constructing a device state space and a group action space, wherein the device state space is composed of a performance feature vector, and the group action space contains a plurality of device grouping schemes, each of which defines a group allocation relationship of devices;

[0098] A state transition probability matrix is ​​constructed based on historical performance data, element values ​​of the state transition probability matrix are calculated by conditional probability estimation, and a Bayesian smoothing parameter is introduced to correct the state transition probability matrix to obtain a probability distribution of device state migration;

[0099] Mapping the performance feature vector in the device state space and the device grouping scheme in the grouping action space into a particle swarm optimization problem, encoding the device grouping scheme into a particle position vector representing the device group number, and encoding the change of the device group number into a velocity vector;

[0100] Constructing a multi-objective fitness function based on the device state transition probability distribution, the multi-objective fitness function includes an inter-group performance balance evaluation item, a resource utilization evaluation item, and an intra-group state similarity evaluation item, assigning an adaptive weight coefficient to the multi-objective fitness function, and calculating an evaluation score of the device grouping scheme;

[0101] Iteratively optimize the particle position vector according to the evaluation score, and use the dynamic inertia weight to update the speed vector, calculate the position update amount based on the individual historical optimal solution of the particle position vector and the current optimal solution of the group, and constrain the updated particle position vector to ensure that the limit on the number of devices in the group is met;

[0102] When the continuous iterative improvement of the evaluation score is less than a preset convergence threshold, the current optimal particle position vector is decoded as an initial grouping scheme.

[0103] Exemplarily, first, historical device performance data, such as CPU utilization, memory occupancy, network throughput, etc., are collected to form a performance feature vector for each device. These vectors together constitute the device state space. At the same time, multiple possible device grouping schemes are predefined, and each scheme specifies the group to which each device belongs. These grouping schemes constitute a grouping action space. For example, if there are 10 devices, different schemes for dividing the devices into 2 groups, 3 groups, 4 groups, etc. can be predefined, and each scheme has clear provisions for which specific device belongs to which group. Suppose a two-group grouping scheme is: devices 1 to 5 are in the first group, and devices 6 to 10 are in the second group.

[0104] Next, the device state transition probability matrix is ​​calculated based on the collected historical performance data. Specifically, the probability of the device transferring to the next state in the current state is counted. For example, if the current CPU utilization of device A is 20%, then what is the probability that its CPU utilization will become 30%, 40%, or other values ​​in the next time period. In order to avoid the probability of some state transitions being zero due to data sparsity, the Bayesian smoothing parameter is introduced to correct the probability matrix to ensure that each state transition has a non-zero probability value. For example, if a state transition has never appeared in historical data, it can be assigned a small probability value based on the Bayesian smoothing parameter. Finally, the probability distribution of device state transition is obtained, which describes the law of device state evolution over time.

[0105] Then, the device state space and group action space are mapped to a particle swarm optimization problem. Each device grouping scheme is encoded as a position vector of a particle, and each element in the vector represents the group number to which a device belongs. For example, the particle position vectors of the above two grouping schemes can be expressed as [1, 1, 1, 1, 1, 2, 2, 2, 2, 2]. The change in the device group number is encoded as a velocity vector.

[0106] In order to evaluate the advantages and disadvantages of different grouping schemes, a multi-objective fitness function is constructed. The function contains three evaluation items: inter-group performance balance, resource utilization, and intra-group state similarity. Inter-group performance balance refers to the size of the performance difference between different groups. The smaller the difference, the higher the balance. Resource utilization refers to the resource usage of each group. The higher the utilization, the better. Intra-group state similarity refers to the similarity of the performance characteristics of devices in the same group. The higher the similarity, the better. Assign adaptive weight coefficients to these three evaluation items, such as dynamically adjusting the weights according to the current system load, and then calculate the evaluation score of each device grouping scheme. For example, if the current system load is high, the weight of resource utilization can be increased. Assuming that the inter-group performance balance score of a grouping scheme is 0.8, the resource utilization score is 0.9, and the intra-group state similarity score is 0.7, and the weights of the three evaluation items are 0.3, 0.5, and 0.2 respectively, then the final score of the scheme is 0.3*0.8+0.5*0.9+0.2*0.7=0.83.

[0107] Next, the particle swarm optimization algorithm is used to iteratively search for the optimal grouping scheme. The position and velocity of the particle are updated according to the evaluation score of each particle. The position update is based on the historical optimal solution of the individual particle and the current optimal solution of the group, and a dynamic inertia weight is used to balance the global search and local search capabilities. The updated particle position vector is constrained to ensure that the number of devices in each group does not exceed the preset limit. For example, if each group is required to accommodate a maximum of 6 devices, the position vector needs to be adjusted to ensure that the limit is met.

[0108] The iteration process continues until the continuous iterative improvement of the evaluation score is less than the preset convergence threshold. At this point, the current optimal particle position vector is decoded into an initial grouping scheme. For example, if the optimal position vector is [1, 1, 2, 2, 1, 2, 1, 2, 1, 2], it means that devices 1, 2, 5, 7, 9 belong to the first group, and devices 3, 4, 6, 8, 10 belong to the second group.

[0109] In this embodiment, by grouping devices with similar status into the same group, resources within the group can be used more effectively to avoid resource waste. By optimizing the grouping scheme, the performance differences between different groups can be made smaller, avoiding the situation where some groups are overloaded while others are underloaded. Devices with similar status within a group are more conducive to stable operation within the group, reducing the risk of system failure caused by large differences in device status.

[0110] In an optional embodiment,

[0111] Based on the initial grouping scheme, a dual Q network is constructed, the dual Q network includes an online evaluation network and a target value network, a historical decision sequence is stored through an experience replay buffer pool, a temporal difference learning algorithm is constructed based on the historical decision sequence, a multidimensional reward value is calculated, and the parameters of the online evaluation network are updated according to the multidimensional reward value. The initial grouping scheme is optimized to obtain a candidate grouping scheme, including:

[0112] The network parameters of the online evaluation network are set as initial parameters, and the initial parameters are partially copied to the target value network according to a preset parameter inheritance ratio to form target network parameters;

[0113] Construct an experience replay buffer pool for storing historical decision sequences, wherein each decision sample in the historical decision sequence includes a current state matrix, an execution action, a multidimensional reward value, and a next state matrix, wherein the multidimensional reward value includes intra-group performance balance, resource utilization efficiency, and state migration stability;

[0114] Calculating the time difference error of each decision sample in the historical decision sequence, determining the sampling priority of each decision sample based on the time difference error, and selecting a training sample from the experience replay buffer pool according to the sampling priority;

[0115] Input the training sample into the online evaluation network to obtain a first set of optimal actions, calculate the first Q value corresponding to the first set of optimal actions, input the training sample into the target value network to obtain a second set of optimal actions, calculate the second Q value corresponding to the second set of optimal actions, set an attenuation coefficient for the multi-dimensional reward value in the training sample, combine the attenuation coefficient with the first Q value and the second Q value to obtain a target Q value, calculate the deviation value between the target Q value and the current Q value output by the online evaluation network, and use the deviation value as the optimization target of the online evaluation network;

[0116] Based on the optimization target, network parameters of the online evaluation network are updated by back propagation to obtain updated parameters, and the updated parameters are copied to the target value network according to the preset parameter inheritance ratio;

[0117] Input the current state matrix into the updated online evaluation network, obtain the grouping adjustment scheme with the highest evaluation value, execute the grouping adjustment scheme to adjust the current grouping scheme, and calculate the adjusted state matrix and performance evaluation value;

[0118] The adjusted state matrix, the executed grouping adjustment plan, the performance evaluation value and the next state matrix form a new decision sample and are stored in the experience replay buffer pool. The iteration is repeated until the preset number of iterations is met, and the current grouping plan is determined as the optimized grouping plan.

[0119] Exemplarily, first, the dual Q network is initialized. The network parameters of the online evaluation network are randomly initialized, and then according to a preset parameter inheritance ratio (e.g., 50%), part of the initial parameters are copied to the target value network to form the initial parameters of the target network. The two networks have the same structure but different parameters.

[0120] Next, build an experience replay cache pool. The cache pool is used to store historical decision sequences. Each decision sample contains four elements: the current state matrix, the executed action, the multidimensional reward value, and the next state matrix. For example, assuming that there are five members who need to be grouped, the current state matrix can be represented by a 5x5 matrix, in which each element represents the strength of the relationship between members. The action performed can be to move a member from one group to another, represented by a vector. The multidimensional reward value includes three indicators: performance balance within the group, resource utilization efficiency, and state migration stability. It can be represented by a three-dimensional vector, such as [0.8, 0.9, 0.7], which represent the scores of the three indicators respectively. The next state matrix is ​​the member relationship matrix after the action is executed.

[0121] Then, the temporal difference error is calculated and priority sampling is performed. The historical decision sequence in the experience replay buffer pool is traversed to calculate the temporal difference error of each decision sample. The temporal difference error represents the difference between the evaluation value of the current strategy and the target value. The calculation method is: use the target value network to calculate the optimal action of the next state and its corresponding Q value, and combine the multi-dimensional reward value of the current state to calculate a target Q value. Then, calculate the Q value of the current state by the online evaluation network, and define the difference between the target Q value and the current Q value as the temporal difference error. According to the size of the temporal difference error, the sampling priority of each decision sample is determined. The larger the error, the higher the priority. According to the sampling priority, a batch of training samples are selected from the experience replay buffer pool. For example, samples with larger temporal difference errors are preferentially selected for training.

[0122] Next, update the parameters of the online evaluation network. Input the training samples into the online evaluation network to obtain a set of optimal actions and their corresponding Q values. At the same time, input the training samples into the target value network to obtain another set of optimal actions and their corresponding Q values. Set an attenuation coefficient for the multidimensional reward values ​​in the training samples, such as 0.95. Combine the attenuation coefficient, the Q value corresponding to the first set of optimal actions, and the Q value corresponding to the second set of optimal actions to obtain the target Q value. The target Q value can be calculated by multiplying the attenuation coefficient by the Q value corresponding to the second set of optimal actions, and then adding the multidimensional reward value of the current state. Calculate the deviation between the target Q value and the current Q value output by the online evaluation network, and use the deviation value as the optimization target of the online evaluation network. Update the parameters of the online evaluation network according to the optimization target through the back propagation algorithm.

[0123] Update the parameters of the target value network. According to the preset parameter inheritance ratio, copy the updated parameters of the online evaluation network to the target value network. For example, copy 50% of the updated parameters to the target value network so that the parameters of the target value network gradually approach the parameters of the online evaluation network.

[0124] Execute the grouping adjustment plan and update the cache pool. Input the current state matrix into the updated online evaluation network to obtain the grouping adjustment plan with the highest evaluation value. Execute the plan, adjust the current grouping plan, and calculate the adjusted state matrix and new multi-dimensional reward value. The adjusted state matrix, the executed grouping adjustment plan, the new multi-dimensional reward value, and the next state matrix form a new decision sample and store it in the experience replay cache pool.

[0125] The above steps are repeated until a preset number of iterations (eg, 1000) is met. Finally, the current grouping scheme is determined as the optimized grouping scheme.

[0126] In this embodiment, through the learning mechanism of multi-dimensional reward value and dual Q network, the initial grouping scheme can be effectively optimized, and the performance balance within the group, resource utilization efficiency and state migration stability can be improved. The experience replay mechanism and temporal difference learning algorithm enable the method to adapt to the dynamically changing environment, continuously adjust the grouping scheme according to the new state information and feedback, and maintain the effectiveness of the grouping scheme. The structure of the dual Q network can effectively reduce the over-estimation problem and prevent the algorithm from falling into the local optimal solution, thereby finding a better grouping scheme.

[0127] In an optional embodiment,

[0128] Constructing a group collaborative evaluation network based on a graph structure, constructing the equipment group relationship in the candidate grouping scheme into a dynamic weighted graph, inputting the dynamic weighted graph into a multi-layer graph convolutional network, extracting local combination features within the equipment group and global topological features between equipment groups, and constructing a double-layer evaluation criterion. When the evaluation result of the double-layer evaluation criterion does not reach a preset threshold, generating a new equipment allocation scheme according to the local combination features and the global topological features includes:

[0129] The device group relationship in the candidate grouping scheme is constructed as a dynamic weighted graph model, wherein the dynamic weighted graph model includes a node set and an edge weight set, each node in the node set corresponds to a device, and the node includes a performance feature vector composed of a computing power index, a storage capacity index, and a communication bandwidth index;

[0130] Calculate the cosine similarity value based on the performance feature vectors of any two nodes in the node set and use it as the corresponding edge weight value;

[0131] The dynamic weighted graph is input into a multi-layer graph convolutional network, which includes a first graph convolutional layer and a second graph convolutional layer. For each node in the dynamic weighted graph model, the performance feature vectors of each node in the neighborhood node set of the current node and the corresponding edge weight values ​​are weighted and converged in the first graph convolutional layer, and the local combined features of the current node are obtained by processing with a first feature transformation matrix and a first activation function;

[0132] For each node in the dynamic weighted graph model, the local combined features of each node in the extended neighborhood node set of the current node and the corresponding edge weight value are weighted and converged in the second graph convolution layer, and the global topological features of the current node are obtained by processing the second feature transformation matrix and the second activation function;

[0133] Calculating the ratio of the minimum value to the maximum value of the local combination characteristics of the nodes in the device group to obtain the performance matching degree within the device group, and constructing an evaluation criterion for the performance matching degree within the group according to the performance matching degree within the device group;

[0134] Calculate the ratio of the standard deviation of the sum of the global topological characteristics of the nodes in all device groups to the mean to obtain the resource balance degree between the device groups, and construct an evaluation criterion for the resource balance degree between the groups according to the resource balance degree between the device groups;

[0135] A double-layer evaluation criterion is obtained by weighted combination of the intra-group performance matching evaluation criterion and the inter-group resource balance evaluation criterion, and a group collaboration evaluation score is calculated based on the double-layer evaluation criterion;

[0136] When the group collaborative evaluation score is less than a preset threshold, nodes with mismatched performance are determined based on local combination features, and a device group with complementary resources is determined based on global topological features, and a device reallocation plan is generated. The candidate grouping plan is adjusted according to the device reallocation plan to obtain a new device allocation plan.

[0137] Exemplarily, first, prepare a candidate grouping scheme. For example, a candidate scheme divides 10 devices into 3 groups: Group A (devices 1, 2, 3), Group B (devices 4, 5, 6), and Group C (devices 7, 8, 9, 10). Each device has three performance indicators: computing power, storage capacity, and communication bandwidth. For example, the performance indicators of device 1 are (computing power: 10, storage capacity: 50, communication bandwidth: 100).

[0138] Next, a dynamic weighted graph model is constructed. Each device is taken as a node, and the node contains the performance feature vector of the device. For example, the performance feature vector of device 1 is (10, 50, 100). Then, the cosine similarity of the performance feature vectors between any two nodes is calculated, and this value is used as the weight of the edge between the two nodes. For example, the performance feature vectors of device 1 and device 2 are (10, 50, 100) and (12, 40, 90) respectively, and the calculated cosine similarity is 0.98, then the weight of the edge connecting device 1 and device 2 is 0.98.

[0139] Then, the constructed dynamic weighted graph is input into a two-layer graph convolutional network. In the first graph convolutional layer, for each node, the performance feature vector of its neighboring nodes is multiplied by the weight of the corresponding edge and then summed, and then processed through a feature transformation matrix and activation function to obtain the local combination feature of the node. For example, the neighboring nodes of device 1 are device 2 and device 3, then the performance feature vectors of device 2 and device 3 are multiplied by the weight of the edge connecting device 1 and then summed, and then processed through feature transformation and activation function to obtain the local combination feature of device 1.

[0140] In the second graph convolution layer, for each node, the local combination features of its extended neighbor nodes (neighbors of neighbors) are multiplied by the weight of the corresponding edge and then summed, and then processed through a feature transformation matrix and activation function to obtain the global topological features of the node. For example, the extended neighbor nodes of device 1 include device 2, device 3 and their respective neighbor nodes. The local combination features of these nodes are multiplied by the weight of the edge connecting device 1 and then summed, and then processed through feature transformation and activation function to obtain the global topological features of device 1.

[0141] Next, a two-layer evaluation criterion is constructed. First, the intra-group performance matching is calculated. For each device group, the ratio of the minimum to the maximum value of the local combination characteristics of all nodes in the group is calculated. For example, the local combination characteristics of devices 1, 2, and 3 in group A are (0.8, 0.9, and 0.7), respectively, and the performance matching degree of group A is 0.7 / 0.9=0.78. Then the inter-group resource balance is calculated. The ratio of the standard deviation to the mean of the sum of the global topological characteristics of the nodes in all device groups is calculated. For example, the sum of the global topological characteristics of groups A, B, and C is 10, 12, and 11, respectively, then the standard deviation is 0.82, the mean is 11, and the inter-group resource balance is 0.82 / 11=0.07. Finally, the intra-group performance matching and the inter-group resource balance are weightedly combined to obtain the group collaborative evaluation score. For example, assuming the weights are 0.6 and 0.4, respectively, the group collaborative evaluation score is 0.6*0.78+0.4*0.07=0.496.

[0142] If the group collaborative evaluation score is less than the preset threshold (for example, 0.8), a new device allocation plan is generated based on the local combination features and global topological features. For example, if it is found that the local combination feature of device 3 in group A is the smallest and the performance is not matched, while the sum of the global topological features of group C is the largest and the resources are more abundant, then device 3 can be moved from group A to group C. Repeat the above steps until the group collaborative evaluation score reaches the preset threshold.

[0143] In an optional embodiment,

[0144] The calculation formula for calculating the group collaborative evaluation score based on the two-layer evaluation criteria is as follows:

[0145] ;

[0146] Among them, G represents the group collaborative evaluation score, μ represents the importance weight of the performance matching within the group, M represents the total number of equipment groups, represents the local performance feature vector of the jth node in the kth device group, represents the smoothing factor, β represents the weight of resource balance between groups, A represents the total number of nodes in all device groups, and a k Indicates the number of nodes in the kth device group, represents the global feature of the jth node in the kth device group, and ||·|| represents the norm of the calculation vector.

[0147] In this embodiment, by considering the performance characteristics of the equipment and the relationship between groups, the equipment can be more reasonably allocated to different groups to avoid resource waste and performance bottlenecks. By optimizing the equipment grouping, the efficiency of intra-group collaboration and inter-group coordination can be improved, thereby improving the overall work efficiency. The equipment grouping scheme can be automatically evaluated and adjusted to reduce manual intervention and improve the efficiency of group optimization. The two aspects of intra-group performance matching and inter-group resource balance are comprehensively considered, which can more comprehensively reflect the collaborative work level of the equipment group. The weight parameters can be adjusted according to the specific application scenario to meet different evaluation needs.

[0148] The present invention provides another optional embodiment, including:

[0149] Based on the modeling idea of ​​Markov decision process, we must first clarify the construction method of state space. The state space contains two core dimensions: production demand state and equipment state. The production demand state is mainly reflected in the target output data of different product types, including the production target index value of each product, and the processing information of the work-in-progress lot, such as the process formula, priority, delivery date and other characteristics of the lot. The equipment state dimension includes attribute information such as equipment availability, processing capacity, maintenance status, etc.

[0150] The design of the action space focuses on the adjustment decision of the equipment grouping scheme. The grouping scheme can be understood as a configuration scheme that divides the equipment into different functional groups, each of which is responsible for the processing tasks of a specific type of product. In order to transform the grouping problem into an optimization problem, a particle swarm algorithm is used to solve it, in which the position vector encoding of the particle represents the specific grouping scheme, and each dimension of the position vector corresponds to the group a device belongs to.

[0151] The design of the reward function focuses on the achievement of production goals. The main consideration is the completion rate of the production goal, that is, the ratio of actual output to target output; at the same time, the equipment utilization rate is considered, which is reflected in the proportion of the actual processing time of the equipment to the total available time. These two indicators are used to comprehensively evaluate the advantages and disadvantages of the grouping scheme in a weighted manner.

[0152] In the implementation process, the state transition probability is first estimated based on historical data, including the product demand change pattern and the equipment state change law. Then the particle swarm algorithm is used for iterative optimization, and the speed and position of the particles are updated each iteration until a better grouping solution is found.

[0153] Take the grouping of photolithography machines as an example: Assume that there are ten photolithography machines, and three types of products need to be processed. By analyzing historical production data, it is known that the current production target of product one is 100 pieces per day, product two is 80 pieces per day, and product three is 60 pieces per day. The equipment status shows that eight devices are in normal operation and two are in maintenance. The equipment grouping scheme is encoded as a ten-dimensional vector, and the vector elements take values ​​from one to three, representing the product group to which the equipment belongs. Through iterative optimization, it is possible to eventually get a grouping scheme that assigns four devices to product one, three to product two, and three to product three.

[0154] In this embodiment, through the precise modeling of state space and the application of particle swarm optimization algorithm, the intelligent equipment grouping decision is realized, and the grouping plan can be adjusted in time according to the changes in product demand and equipment status; the execution efficiency of the production plan is improved, and the capacity utilization rate is significantly improved; the dependence on manual decision-making is reduced, and the standardization and automation of grouping decisions are realized; it has strong adaptability and can quickly respond to dynamic changes in the production environment; by comprehensively considering multiple production goals, the balance and practicality of the grouping plan are guaranteed.

[0155] Figure 2 Schematic diagram of the structure of a semiconductor manufacturing equipment grouping working system based on deep reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0156] The first unit is used to collect real-time operation data of multiple semiconductor manufacturing equipment in a continuous processing cycle, perform segmented sampling on the real-time operation data through a sliding window mechanism, extract the time series statistical characteristics of parameters and the coupling characteristics between parameters in each sliding window, input the time series statistical characteristics and coupling characteristics into a pre-trained recurrent neural network, generate a time series representation of equipment performance, build a hierarchical evaluation system based on the experience of process experts, generate a dynamic performance score of the semiconductor manufacturing equipment, and build a performance feature vector with adaptive weights based on the dynamic performance score;

[0157] The second unit is used to input the performance feature vector into a deep reinforcement learning environment, construct a Markov decision process model that considers device state migration, map the device state space and the group action space into a particle swarm optimization problem, encode the grouping scheme into a particle position vector, and obtain an initial grouping scheme through iterative optimization search; construct a dual Q network based on the initial grouping scheme, the dual Q network includes an online evaluation network and a target value network, store historical decision sequences through an experience replay buffer pool, construct a temporal difference learning algorithm based on the historical decision sequences, calculate multidimensional reward values, update the parameters of the online evaluation network according to the multidimensional reward values, and optimize the initial grouping scheme to obtain a candidate grouping scheme;

[0158] The third unit is used to construct a group collaborative evaluation network based on a graph structure, construct the device group relationship in the candidate grouping scheme as a dynamic weighted graph, input the dynamic weighted graph into a multi-layer graph convolutional network, extract local combination features within the device group and global topological features between device groups, and construct a two-layer evaluation criterion. When the evaluation result of the two-layer evaluation criterion does not reach a preset threshold, a new device allocation scheme is generated according to the local combination features and the global topological features, and the execution result of the device allocation scheme is input into the deep reinforcement learning environment as a new state matrix for updating, and the device reallocation process is repeated until the two-layer evaluation criterion meets the preset convergence threshold, and the final device grouping scheme is output.

[0159] According to a third aspect of the embodiments of the present invention,

[0160] An electronic device is provided, comprising:

[0161] processor;

[0162] a memory for storing processor-executable instructions;

[0163] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0164] A fourth aspect of the embodiments of the present invention is:

[0165] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0166] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semiconductor manufacturing equipment grouping working method based on deep reinforcement learning, characterized in that: include: Collect real-time operation data of multiple semiconductor manufacturing equipment in a continuous processing cycle, perform segmented sampling on the real-time operation data through a sliding window mechanism, extract the time series statistical characteristics of parameters and the coupling characteristics between parameters in each sliding window, input the time series statistical characteristics and coupling characteristics into a pre-trained recurrent neural network, generate a time series representation of equipment performance, build a hierarchical evaluation system based on the experience of process experts, generate a dynamic performance score for semiconductor manufacturing equipment, and build a performance feature vector with adaptive weights based on the dynamic performance score; Input the performance feature vector into a deep reinforcement learning environment, construct a Markov decision process model considering device state migration, map the device state space and group action space into a particle swarm optimization problem, encode the grouping scheme into a particle position vector, and obtain an initial grouping scheme through iterative optimization search; Based on the initial grouping scheme, a dual Q network is constructed, the dual Q network includes an online evaluation network and a target value network, a historical decision sequence is stored through an experience replay buffer pool, a temporal difference learning algorithm is constructed based on the historical decision sequence, a multidimensional reward value is calculated, parameters of the online evaluation network are updated according to the multidimensional reward value, and the initial grouping scheme is optimized to obtain a candidate grouping scheme; A group collaborative evaluation network based on a graph structure is constructed, the equipment group relationship in the candidate grouping scheme is constructed as a dynamic weighted graph, the dynamic weighted graph is input into a multi-layer graph convolutional network, the local combination features within the equipment group and the global topological features between the equipment groups are extracted, and a double-layer evaluation criterion is constructed. When the evaluation result of the double-layer evaluation criterion does not reach a preset threshold, a new equipment allocation scheme is generated according to the local combination features and the global topological features, the execution result of the equipment allocation scheme is input into a deep reinforcement learning environment as a new state matrix for updating, and the equipment reallocation process is repeated until the double-layer evaluation criterion meets the preset convergence threshold, and the final equipment grouping scheme is output.

2. The method according to claim 1, characterized in that The timing statistical features and coupling features are input into a pre-trained recurrent neural network to generate a timing representation of equipment performance, and a hierarchical evaluation system is constructed in combination with process expert experience to generate a dynamic performance score for semiconductor manufacturing equipment. The performance feature vector with adaptive weights is constructed based on the dynamic performance score, including: Acquire the timing statistical features and coupling features of semiconductor manufacturing equipment, calculate the multi-order statistical moments of the characteristic parameters in each sliding window, and concatenate the multi-order statistical moments with the timing statistical features and coupling features to construct an enhanced feature sequence; Constructing a bidirectional recurrent neural network structure based on a capsule network, wherein the capsule network adaptively captures the hierarchical relationship of the feature sequence through a dynamic routing algorithm, the bidirectional recurrent neural network includes an attention-enhanced long short-term memory unit, the attention-enhanced long short-term memory unit introduces an adaptive weight matrix to dynamically adjust the contribution of the input feature, the enhanced feature sequence is input into the capsule network to extract hierarchical features, and the hierarchical features are initially represented in time series through the bidirectional recurrent neural network; Designing a multi-head self-attention mechanism to perform feature fusion on the initial time series representation, wherein the multi-head self-attention mechanism includes multiple parallel attention subspaces, each attention subspace independently learns feature association patterns under different perspectives, and the output of each attention subspace is transformed through nonlinear transformation to obtain a fused time series representation; Based on the fused timing representation and combined with the experience of process experts, a hierarchical evaluation system is constructed, wherein the hierarchical evaluation system includes evaluation indicators at four levels: equipment health, processing accuracy, production efficiency, and quality consistency. The key evaluation indicators at each level are screened by a recursive feature elimination algorithm, and the key evaluation indicators are combined with the fused timing representation to construct a multidimensional evaluation space; Adopting an adaptive weight optimization strategy, a dual loss function including task relevance and weight sensitivity is constructed. Combining the fusion timing representation, a dynamic learning rate adjustment mechanism is designed. The weight distribution of evaluation indicators at each level is iteratively updated through an alternating optimization algorithm to obtain a dynamic performance score. The dynamic performance score, the time difference feature of the dynamic performance score, the multi-scale spectrum feature based on wavelet transform and the fused time series representation are combined to construct a performance feature vector with adaptive weights, and the performance feature vector is used to characterize the static level, dynamic change characteristics and potential evolution trend of equipment performance.

3. The method according to claim 2, characterized in that Adopting the adaptive weight optimization strategy, the calculation formula of the dual loss function including task relevance and weight sensitivity is as follows: ； Among them, L(w) represents the dual loss function, n represents the total number of samples, and y i represents the true score of the i-th sample, w represents the weight vector of the evaluation index, T represents the transposition, x i represents the input feature vector of the i-th sample, λ1 represents the hyperparameter of the weight sensitivity loss, and w T x i Indicates the performance score of the device. Indicates the rating w T x i The gradient of the weight vector w of the evaluation index, ||·||1 represents the L1 norm of the gradient, which is used to measure the gradient sensitivity, λ2 represents the hyperparameter of weight regularization, α represents the regularization term balance factor, ||w||1 represents the L1 norm of the weight vector of the evaluation index, The L2 norm of the weight vector representing the evaluation index.

4. The method according to claim 1, characterized in that: A Markov decision process model considering device state migration is constructed, the device state space and group action space are mapped into a particle swarm optimization problem, the grouping scheme is encoded into a particle position vector, and the initial grouping scheme is obtained through iterative optimization search, including: Constructing a device state space and a group action space, wherein the device state space is composed of a performance feature vector, and the group action space contains a plurality of device grouping schemes, each of which defines a group allocation relationship of devices; A state transition probability matrix is ​​constructed based on historical performance data, element values ​​of the state transition probability matrix are calculated by conditional probability estimation, and a Bayesian smoothing parameter is introduced to correct the state transition probability matrix to obtain a probability distribution of device state migration; Mapping the performance feature vector in the device state space and the device grouping scheme in the grouping action space into a particle swarm optimization problem, encoding the device grouping scheme into a particle position vector representing the device group number, and encoding the change of the device group number into a velocity vector; Constructing a multi-objective fitness function based on the device state transition probability distribution, the multi-objective fitness function includes an inter-group performance balance evaluation item, a resource utilization evaluation item, and an intra-group state similarity evaluation item, assigning an adaptive weight coefficient to the multi-objective fitness function, and calculating an evaluation score of the device grouping scheme; Iteratively optimize the particle position vector according to the evaluation score, and use the dynamic inertia weight to update the speed vector, calculate the position update amount based on the individual historical optimal solution of the particle position vector and the current optimal solution of the group, and constrain the updated particle position vector to ensure that the limit on the number of devices in the group is met; When the continuous iterative improvement of the evaluation score is less than a preset convergence threshold, the current optimal particle position vector is decoded as an initial grouping scheme.

5. The method according to claim 1, characterized in that Based on the initial grouping scheme, a dual Q network is constructed, the dual Q network includes an online evaluation network and a target value network, a historical decision sequence is stored through an experience replay buffer pool, a temporal difference learning algorithm is constructed based on the historical decision sequence, a multidimensional reward value is calculated, and the parameters of the online evaluation network are updated according to the multidimensional reward value. The initial grouping scheme is optimized to obtain a candidate grouping scheme, including: The network parameters of the online evaluation network are set as initial parameters, and the initial parameters are partially copied to the target value network according to a preset parameter inheritance ratio to form target network parameters; Construct an experience replay buffer pool for storing historical decision sequences, wherein each decision sample in the historical decision sequence includes a current state matrix, an execution action, a multidimensional reward value, and a next state matrix, wherein the multidimensional reward value includes intra-group performance balance, resource utilization efficiency, and state migration stability; Calculating the time difference error of each decision sample in the historical decision sequence, determining the sampling priority of each decision sample based on the time difference error, and selecting a training sample from the experience replay buffer pool according to the sampling priority; Input the training sample into the online evaluation network to obtain a first set of optimal actions, calculate the first Q value corresponding to the first set of optimal actions, input the training sample into the target value network to obtain a second set of optimal actions, calculate the second Q value corresponding to the second set of optimal actions, set an attenuation coefficient for the multi-dimensional reward value in the training sample, combine the attenuation coefficient with the first Q value and the second Q value to obtain a target Q value, calculate the deviation value between the target Q value and the current Q value output by the online evaluation network, and use the deviation value as the optimization target of the online evaluation network; Based on the optimization target, network parameters of the online evaluation network are updated by back propagation to obtain updated parameters, and the updated parameters are copied to the target value network according to the preset parameter inheritance ratio; Input the current state matrix into the updated online evaluation network, obtain the grouping adjustment scheme with the highest evaluation value, execute the grouping adjustment scheme to adjust the current grouping scheme, and calculate the adjusted state matrix and performance evaluation value; The adjusted state matrix, the executed grouping adjustment plan, the performance evaluation value and the next state matrix form a new decision sample and are stored in the experience replay buffer pool. The iteration is repeated until the preset number of iterations is met, and the current grouping plan is determined as the optimized grouping plan.

6. The method according to claim 1, characterized in that Constructing a group collaborative evaluation network based on a graph structure, constructing the equipment group relationship in the candidate grouping scheme into a dynamic weighted graph, inputting the dynamic weighted graph into a multi-layer graph convolutional network, extracting local combination features within the equipment group and global topological features between equipment groups, and constructing a double-layer evaluation criterion. When the evaluation result of the double-layer evaluation criterion does not reach a preset threshold, generating a new equipment allocation scheme according to the local combination features and the global topological features includes: The device group relationship in the candidate grouping scheme is constructed as a dynamic weighted graph model, wherein the dynamic weighted graph model includes a node set and an edge weight set, each node in the node set corresponds to a device, and the node includes a performance feature vector composed of a computing power index, a storage capacity index, and a communication bandwidth index; Calculate the cosine similarity value based on the performance feature vectors of any two nodes in the node set and use it as the corresponding edge weight value; The dynamic weighted graph is input into a multi-layer graph convolutional network, which includes a first graph convolutional layer and a second graph convolutional layer. For each node in the dynamic weighted graph model, the performance feature vectors of each node in the neighborhood node set of the current node and the corresponding edge weight values ​​are weighted and converged in the first graph convolutional layer, and the local combined features of the current node are obtained by processing with a first feature transformation matrix and a first activation function; For each node in the dynamic weighted graph model, the local combined features of each node in the extended neighborhood node set of the current node and the corresponding edge weight value are weighted and converged in the second graph convolution layer, and the global topological features of the current node are obtained by processing the second feature transformation matrix and the second activation function; Calculating the ratio of the minimum value to the maximum value of the local combination characteristics of the nodes in the device group to obtain the performance matching degree within the device group, and constructing an evaluation criterion for the performance matching degree within the group according to the performance matching degree within the device group; Calculate the ratio of the standard deviation of the sum of the global topological characteristics of the nodes in all device groups to the mean to obtain the resource balance degree between the device groups, and construct an evaluation criterion for the resource balance degree between the groups according to the resource balance degree between the device groups; A double-layer evaluation criterion is obtained by weighted combination of the intra-group performance matching evaluation criterion and the inter-group resource balance evaluation criterion, and a group collaboration evaluation score is calculated based on the double-layer evaluation criterion; When the group collaborative evaluation score is less than a preset threshold, nodes with mismatched performance are determined based on local combination features, and a device group with complementary resources is determined based on global topological features, and a device reallocation plan is generated. The candidate grouping plan is adjusted according to the device reallocation plan to obtain a new device allocation plan.

7. The method according to claim 6, characterized in that The calculation formula for calculating the group collaborative evaluation score based on the two-layer evaluation criteria is as follows: ； Among them, G represents the group collaborative evaluation score, μ represents the importance weight of the performance matching within the group, M represents the total number of equipment groups, represents the local performance feature vector of the jth node in the kth device group, represents the smoothing factor, β represents the weight of resource balance between groups, A represents the total number of nodes in all device groups, and a k Indicates the number of nodes in the kth device group, represents the global feature of the jth node in the kth device group, and ||·|| represents the norm of the calculation vector.

8. A semiconductor manufacturing equipment grouping work system based on deep reinforcement learning, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect real-time operation data of multiple semiconductor manufacturing equipment in a continuous processing cycle, perform segmented sampling on the real-time operation data through a sliding window mechanism, extract the time series statistical characteristics of parameters and the coupling characteristics between parameters in each sliding window, input the time series statistical characteristics and coupling characteristics into a pre-trained recurrent neural network, generate a time series representation of equipment performance, build a hierarchical evaluation system based on the experience of process experts, generate a dynamic performance score of the semiconductor manufacturing equipment, and build a performance feature vector with adaptive weights based on the dynamic performance score; The second unit is used to input the performance feature vector into a deep reinforcement learning environment, construct a Markov decision process model considering device state migration, map the device state space and the group action space into a particle swarm optimization problem, encode the grouping scheme into a particle position vector, and obtain an initial grouping scheme through iterative optimization search; Based on the initial grouping scheme, a dual Q network is constructed, the dual Q network includes an online evaluation network and a target value network, a historical decision sequence is stored through an experience replay buffer pool, a temporal difference learning algorithm is constructed based on the historical decision sequence, a multidimensional reward value is calculated, parameters of the online evaluation network are updated according to the multidimensional reward value, and the initial grouping scheme is optimized to obtain a candidate grouping scheme; The third unit is used to construct a group collaborative evaluation network based on a graph structure, construct the device group relationship in the candidate grouping scheme as a dynamic weighted graph, input the dynamic weighted graph into a multi-layer graph convolutional network, extract local combination features within the device group and global topological features between device groups, and construct a two-layer evaluation criterion. When the evaluation result of the two-layer evaluation criterion does not reach a preset threshold, a new device allocation scheme is generated according to the local combination features and the global topological features, and the execution result of the device allocation scheme is input into the deep reinforcement learning environment as a new state matrix for updating, and the device reallocation process is repeated until the two-layer evaluation criterion meets the preset convergence threshold, and the final device grouping scheme is output.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images