An intelligent scheduling optimization system for equipment upgrade in unmanned workshops based on reinforcement learning

Abstract

The present invention relates to the field of intelligent manufacturing, and discloses an intelligent scheduling optimization system for equipment upgrade in an unmanned workshop based on reinforcement learning, comprising: a feature collection module: collecting workshop equipment data, and extracting corresponding data features to obtain production feature data; a space optimization module: performing clustering operations on the production feature data to obtain feature clustering labels, and constructing an optimized multidimensional state space through the feature clustering labels; a strategy optimization module: learning the optimal equipment scheduling action strategies under different states in the multidimensional state space through a neural network model with an optimized structure, and obtaining a series of equipment scheduling strategy sets for different states; a scheduling acquisition module: constructing an improved deep network model, inputting the equipment scheduling strategy set into the improved deep network model, and obtaining the optimal scheduling mode under different states. Compared with the simple rule allocation in the traditional method, the present invention can better optimize the equipment scheduling of unmanned workshops.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing, and in particular to an intelligent scheduling optimization system for equipment upgrade in an unmanned workshop based on reinforcement learning. Background Art

[0002] With the development of intelligent manufacturing, unmanned workshops, as an important manifestation of intelligent manufacturing, have higher and higher requirements for intelligent equipment scheduling. Traditional scheduling methods are difficult to cope with complex and changing production environments and diversified product demands. It is urgent to introduce advanced intelligent scheduling optimization technologies. In modern manufacturing, product types are increasingly diversified and production processes are also different, which makes the equipment scheduling problem of unmanned workshops more complicated.

[0003] At present, in unmanned workshops with many product types and large differences in production processes, when obtaining the state space, the traditional scheduling optimization method will have a problem because the product itself includes a large number of variables of different dimensions, such as product size, weight, precision requirements, etc., which are all variables of different dimensions. These parameter values ​​are different for each product, which greatly expands the state space dimension. The traditional scheduling optimization method usually adopts a rule-based method. For example, for a specific type of product, it is preferentially assigned to a specific device for processing, or tasks are assigned to relatively idle devices according to the load of the equipment. However, this method will be too complex in the state space. As the dimension of the state space increases, the calculation speed gradually slows down. Therefore, an intelligent scheduling optimization system for equipment upgrade in unmanned workshops based on reinforcement learning is proposed. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcoming in the prior art that the calculation speed gradually slows down with the increase of the state space dimension, and to propose an intelligent scheduling optimization system for unmanned workshop equipment upgrade based on reinforcement learning.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] An intelligent scheduling optimization system for equipment upgrade in unmanned workshops based on reinforcement learning, comprising:

[0007] Feature collection module: collects workshop equipment data, extracts corresponding data features, and obtains production feature data;

[0008] Spatial optimization module: clusters production feature data to obtain feature clustering labels, and constructs an optimized multidimensional state space through feature clustering labels;

[0009] Strategy Optimization Module: This module uses a neural network model with an optimized structure to learn the optimal equipment scheduling action strategy under different states in a multi-dimensional state space, and obtains a series of equipment scheduling strategy sets for different states.

[0010] Scheduling acquisition module: Building an improved depth Network model, input device scheduling strategy set to improve depth In the network model, the optimal scheduling mode under different states is obtained.

[0011] The production feature data is obtained from sensors in the workshop and the production management database to construct a feature vector of the product instance, which is expressed as: .

[0012] The steps of obtaining feature clustering labels are:

[0013] Set a neighborhood radius and minimum number of neighbors ;

[0014] Based on production characteristic data , through one of the points , traverse except All production feature data outside, calculate the Euclidean distance with all other points, and obtain a Euclidean distance set ;

[0015] in, Indicates the production characteristic data data points;

[0016] Traverse all Euclidean distance sets, when Any point in the When , this point is represented by Points in the neighborhood;

[0017] Traversing production feature data , determine the number of points in the neighborhood of each data point ,when Greater than the minimum number of neighbors When , mark this point as the core point. Less than the minimum number of neighbors And the neighborhood radius of any point If it is within, mark this point as a boundary point, and if it does not meet all the conditions, mark it as a noise point;

[0018] From an unvisited core point Start, find the point All points in the neighborhood of , add these points to a cluster, and for each point in each newly added cluster, when it is greater than the minimum number of neighbor points When the number of neighbor points is greater than the minimum The points in the neighborhood radius of the point are added to the current cluster until there are no new points to add, and finally multiple non-noise clusters consisting of core points and boundary points are obtained. and pure noise points;

[0019] Compute non-noise clusters The characteristic mean of all points in is calculated by the formula: ,in, is a non-noise cluster The number of all points in is a non-noise cluster The point in , takes the feature mean as the initial clustering center of this cluster;

[0020] By producing characteristic data The distance between each point and each initial cluster center is calculated, and the data points are assigned to the nearest non-noise cluster to obtain a series of new non-noise clusters. The feature means of these new non-noise clusters are obtained, and the new feature means are used as new cluster centers. The production feature data is traversed again. The distance between each point in the cluster and each new cluster center is calculated, and the data point is assigned to the nearest non-noise cluster until the cluster center no longer changes. Finally, each data point is assigned to a specific cluster and an integer cluster label is assigned to each data point, with a label value ranging from 0 to , indicating the cluster category to which the data point belongs, where is the final number of clusters, and the feature clustering label is obtained.

[0021] The steps of constructing an optimized multidimensional state space are:

[0022] The feature cluster labels are organized into a label vector sequence, expressed as: L = [ l 1 ， l 2 ... l n ] T ,in Indicates The cluster labels of the data points are constructed based on this label vector sequence. , expressed as: ,in, Indicates Does the data point belong to cluster, if the The data point belongs to clusters, then ,otherwise ;

[0023] Through the label matrix Get the initial state of each data point and preset a state space , set the number of times each state in the state space transfers to another state to , traverse the label vector sequence , calculate the number of state transitions: ,in, Indicates status arrive The number of times, for each time step , if the current data point The status is , and the next data point The status is ,So The value of is increased by 1;

[0024] The state transition probability is calculated based on the number of state transitions, expressed as: , a state transfer matrix is ​​constructed based on the state transfer profile, and the state space is represented by the state transfer matrix, that is, the multi-dimensional state space is optimized.

[0025] The steps for constructing the neural network model of the optimized structure are:

[0026] Obtaining a multi-dimensional state feature vector from the optimized multi-dimensional state space;

[0027] Use a convolutional neural network and replace the convolutional layer in the convolutional neural network with a depth-separable convolutional layer. For the input multi-dimensional feature vector, it first passes through a depth-convolutional layer. The convolution kernel size of the depth-convolutional layer is ,in, It is the spatial dimension of the convolution kernel. Each convolution kernel operates on only one input feature channel to extract the local features of that channel.

[0028] Next, the outputs of the depthwise convolution are combined using a point-wise convolution layer with a kernel size of , where The number of output channels of the depthwise convolution, is the desired number of output channels;

[0029] After the depthwise separable convolutional layer, a fully connected layer is added, and then the neural network model is trained through a loss function. ,in, is a hyperparameter used to control the error threshold. is the output value of the real value, is the output value of the model value;

[0030] Through a The optimizer optimizes the model, takes the multi-dimensional feature vector as input, and adds the corresponding equipment scheduling action and reward value consisting of training data to the neural network for training. In each training batch, a batch of data is randomly selected, the gradient of the loss function is calculated, and the loss function is used The parameters of the new neural network of the optimizer are adjusted. After multiple trainings, the neural network model learns the value estimation of actions under different states. For a given state, the action with the highest value estimation is selected as the optimal equipment scheduling action strategy, and finally a neural network model with an optimized structure is obtained.

[0031] The multi-dimensional state feature vector is composed of the in-degree, out-degree and stay probability of each state.

[0032] The improvement depth The steps to build the network model are:

[0033] depth The network model includes convolutional layer, fully connected layer, and output layer;

[0034] Set the input state to ,in, is the feature dimension, and a convolutional layer is used to extract the spatial features of the state. The formula is expressed as: ,in, and is the index variable used for convolution operation, m represents the convolution kernel movement step index in the horizontal direction, Represents the convolution kernel moving step index in the vertical direction, Indicated in States in a layer The output, For the The convolution kernel weights of the layer, For the The bias term of the layer, then flatten the output of the convolutional layer and input it into the fully connected layer. The calculation formula of the fully connected layer is: ,in, is the activation function, is the weight matrix, is the bias term, Indicates The output of the layer is the input of the fully connected layer. After the linear transformation of the weight matrix and the processing of the activation function, the output of the current layer is obtained. After being processed by multiple fully connected layers, it finally reaches the output layer. The output layer outputs the final predicted value of each action. .

[0035] The steps of obtaining the optimal scheduling mode under different states are:

[0036] Input data from the equipment scheduling strategy set to the improvement depth In each training batch, a batch of data is randomly selected, and the data in the device scheduling strategy set is divided into a training set, a validation set, and a test set. The optimization algorithm updates the network parameters;

[0037] Store the experience data generated by training and set a priority. The priority calculation formula is: ,in, is the discount factor, For the current reward, ;

[0038] is the maximum predicted value in the next state, is the predicted price of the model;

[0039] A probability distribution is constructed based on the priority of the experience data. The higher the priority of the experience data, the greater the probability of being selected, based on the formula: ,in, To control the priority parameter, It is The probability of an experience being selected, For the The priority of the selected experience. It is the index variable after traversing all the data, based on Perform random sampling of experience values. For each experience data , where Current status, is the action taken, is the reward value, is the next state, and the loss function is calculated: ,in, is the real value, that is, the value obtained by simulating or emulating the equipment scheduling system in a specific state. Take action The true value of Represents the number of empirical data samples used to calculate the loss function. After training, for a given state , input to improve depth In the network model, the model outputs the value estimate of each action and selects the action with the highest value estimate as the optimal device scheduling action strategy.

[0040] The present invention has the following beneficial effects:

[0041] 1. In the present invention, the multi-dimensional variables of products in the traditional method greatly expand the dimension of the state space, while the present invention effectively reduces the dimension of the state space by classifying production feature data with similar characteristics into one category. At the same time, the constructed optimized multi-dimensional state space can better reflect the inherent change law and trend of the data by calculating the number and probability of state transitions. Compared with the static state division in the traditional rule-based method, this dynamic state space can better adapt to the changes in production conditions in unmanned workshops and improve the flexibility and adaptability of scheduling.

[0042] 2. In the present invention, the traditional method adopts a rule-based scheduling strategy, which often relies on manual experience and is difficult to adapt to the complex and changeable production environment. The present invention can automatically learn the optimal equipment scheduling action strategy under different states by using a neural network model with optimized structure and an improved deep network model. Through a large amount of training data, the model can learn the best scheduling plan under different product types and production processes. The neural network model can estimate the value of actions under different states and select the action with the highest value estimate as the optimal equipment scheduling action strategy. This method can comprehensively consider multiple factors, such as production efficiency, equipment utilization, etc., to make more informed decisions. The improved deep Q network can better extract the spatial characteristics of the state, and combine the fully connected layer and the activation function to enhance the nonlinear expression ability. In terms of strategy extraction, the highest value estimated action is selected as the optimal scheduling strategy, and it is adjusted through multi-indicator evaluation to more accurately capture the key information in the equipment scheduling problem and improve the accuracy of value estimation. Compared with the simple rule allocation in the traditional method, it can better optimize the equipment scheduling of unmanned workshops and improve the overall production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a system block diagram of an unmanned workshop equipment upgrade intelligent scheduling optimization system based on reinforcement learning proposed by the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Embodiment 1, as Figure 1 As shown, the present invention proposes an unmanned workshop equipment upgrade intelligent scheduling optimization system based on reinforcement learning, comprising:

[0046] Feature collection module: collects workshop equipment data, extracts corresponding data features, and obtains production feature data;

[0047] Spatial optimization module: clusters production feature data to obtain feature clustering labels, and constructs an optimized multidimensional state space through feature clustering labels;

[0048] Strategy Optimization Module: This module uses a neural network model with an optimized structure to learn the optimal equipment scheduling action strategy under different states in a multi-dimensional state space, and obtains a series of equipment scheduling strategy sets for different states.

[0049] Scheduling acquisition module: Building an improved depth Network model, input device scheduling strategy set to improve depth In the network model, the optimal scheduling mode under different states is obtained.

[0050] The steps to obtain production feature data are:

[0051] Obtain data from sensors in the workshop and the production management database to construct the feature vector of the product instance, which is expressed as: ;

[0052] Furthermore, the product feature vector includes length, width, high-dimensional data, precision data, and features related to the production process.

[0053] The steps to obtain feature cluster labels are:

[0054] Set a neighborhood radius and minimum number of neighbors ;

[0055] Based on production characteristic data , through one of the points , traverse except All production feature data outside, calculate the Euclidean distance with all other points, and obtain a Euclidean distance set ;

[0056] in, Indicates the production characteristic data data points;

[0057] Traverse all Euclidean distance sets, when Any point in the When , this point is represented by Points in the neighborhood;

[0058] Traversing production feature data , determine the number of points in the neighborhood of each data point ,when Greater than the minimum number of neighbors When , mark this point as the core point. Less than the minimum number of neighbors And the neighborhood radius of any point If it is within, mark this point as a boundary point, and if it does not meet all the conditions, mark it as a noise point;

[0059] From an unvisited core point Start, find the point All points in the neighborhood of , add these points to a cluster, and for each point in each newly added cluster, when it is greater than the minimum number of neighbor points When the number of neighbor points is greater than the minimum The points in the neighborhood radius of the point are added to the current cluster until there are no new points to add, and finally multiple non-noise clusters consisting of core points and boundary points are obtained. and pure noise points;

[0060] Compute non-noise clusters The characteristic mean of all points in is calculated by the formula: ,in, is a non-noise cluster The number of all points in is a non-noise cluster The point in , takes the feature mean as the initial clustering center of this cluster;

[0061] Calculate production characteristic data The distance between each point and each initial cluster center is calculated, and the data points are assigned to the nearest non-noise cluster to obtain a series of new non-noise clusters. The feature means of these new non-noise clusters are obtained, and the new feature means are used as new cluster centers. The production feature data is traversed again. The distance between each point in the cluster and each new cluster center is calculated, and the data point is assigned to the nearest non-noise cluster until the cluster center no longer changes. Finally, each data point is assigned to a specific cluster and an integer cluster label is assigned to each data point, with a label value ranging from 0 to , indicating the cluster category to which the data point belongs, where is the final number of clusters, and the feature clustering label is obtained.

[0062] The steps to construct an optimized multidimensional state space are:

[0063] The feature cluster labels are organized into a label vector sequence, expressed as: L = [ l 1 ， l 2 ... l n ] T ,in Indicates The cluster labels of the data points are constructed based on this label vector sequence. , expressed as: ,in, Indicates Does the data point belong to cluster, if the The data point belongs to clusters, then ,otherwise ;

[0064] Through the label matrix Get the initial state of each data point and preset a state space , the state space is defined as the set of all possible states, here refers to the cluster category to which the data point belongs. Therefore, the state space is all possible combinations of cluster categories, and the data points belonging to the same cluster are regarded as states with similar characteristics;

[0065] Set the number of times each state in the state space transitions to another state to , traverse the label vector sequence , calculate the number of state transitions: ,in, Indicates status arrive The number of times, for each time step , if the current data point The status is , and the next data point The status is ,So The value of is increased by 1;

[0066] The state transition probability is calculated based on the number of state transitions, expressed as: , a state transfer matrix is ​​constructed based on the state transfer profile, and the state space is represented by the state transfer matrix, that is, the multi-dimensional state space is optimized.

[0067] Furthermore, each data point is assigned to a cluster, and the cluster identifier is the state or , if the data point Belongs to cluster , and in the next time step , data points Belongs to cluster , then it is said that a change from state To status Compared with static clustering labels or simple state divisions, a single transfer can better reflect the inherent change laws and trends of the data, thereby achieving the optimization of the multi-dimensional state space.

[0068] The steps for constructing a neural network model with optimized structure are:

[0069] Obtaining a multi-dimensional state feature vector from the optimized multi-dimensional state space;

[0070] Use a convolutional neural network and replace the convolutional layer in the convolutional neural network with a depth-separable convolutional layer. For the input multi-dimensional feature vector, it first passes through a depth-convolutional layer. The convolution kernel size of the depth-convolutional layer is ,in, It is the spatial dimension of the convolution kernel. Each convolution kernel operates on only one input feature channel to extract the local features of that channel.

[0071] Next, the outputs of the depthwise convolution are combined using a point-wise convolution layer with a kernel size of , where The number of output channels of the depthwise convolution, is the desired number of output channels. The point-by-point convolution layer can change the number of channels without changing the spatial dimension to achieve feature combination and transformation;

[0072] After the depthwise separable convolutional layer, a fully connected layer is added. The fully connected layer can be used to map the features extracted by the depthwise separable convolutional layer to the prediction space of the final scheduling strategy.

[0073] Then the neural network model is trained through a loss function. ,in, is a hyperparameter used to control the error threshold. is the output value of the real value, is the output value of the model value;

[0074] Furthermore, Indicates the real value. In the scenario of equipment scheduling, it can be understood as the quantitative value of the real effect of performing an action under a specific state, such as the numerical value corresponding to indicators such as real production efficiency and equipment utilization rate. Represents the value of the model prediction, that is, the neural network's prediction of the effect of performing an action in a specific state;

[0075] Through a The optimizer optimizes the model, takes the multi-dimensional feature vector as input, and adds the corresponding equipment scheduling action and reward value consisting of training data to the neural network for training. In each training batch, a batch of data is randomly selected, the gradient of the loss function is calculated, and the loss function is used The parameters of the new neural network of the optimizer are adjusted. After multiple trainings, the neural network model learns the value estimation of actions under different states. For a given state, the action with the highest value estimation is selected as the optimal equipment scheduling action strategy, and finally a neural network model with an optimized structure is obtained.

[0076] Furthermore, the equipment scheduling action can be obtained by simulating or emulating the equipment scheduling system. Different scheduling actions can be tried and their effects can be observed. The reward value can be obtained by assigning a weight to each target and then weighted summing the reward values ​​of each target to obtain the total reward value.

[0077] The multidimensional state feature vector consists of the in-degree of each state (the total number of times other states transfer to this state), the out-degree (the total number of times this state transfers to other states), the state's residence probability (i.e. the probability that the state remains unchanged), and the state's label.

[0078] Improved Depth The steps to build the network model are:

[0079] depth The network model includes convolutional layer, fully connected layer, and output layer;

[0080] In the improved deep network model, the convolutional layer extracts the spatial features of the input state, the fully connected layer integrates and nonlinearly transforms these features, and the output layer outputs action value estimates to make optimal scheduling decisions;

[0081] Set the input state to ,in, is the feature dimension, and a convolutional layer is used to extract the spatial features of the state. The formula is expressed as: ,in, and is the index variable used for convolution operation, where m represents the convolution kernel movement step index in the horizontal direction. Represents the convolution kernel moving step index in the vertical direction, Indicated in States in a layer The output, For the The convolution kernel weights of each layer, For the The bias term of the layer, and then the output of the convolutional layer is flattened and input into the fully connected layer;

[0082] Specifically, Indicated in Layer, relative to the current state Offset in horizontal direction , vertical direction The value of the element at the offset;

[0083] The calculation formula of the fully connected layer is: ,in, is the activation function, is the weight matrix, is the bias term, Indicates The output of the layer is the input of the fully connected layer. After the linear transformation of the weight matrix and the processing of the activation function, the output of the current layer is obtained. After being processed by multiple fully connected layers, it finally reaches the output layer. The output layer outputs the final predicted value of each action. .

[0084] Furthermore, the input state is a multidimensional vector, which contains various feature information describing the current state in the equipment scheduling problem. The role of the convolution layer is to extract the spatial features of the input state, and to capture the local patterns and spatial relationships in the input data by performing local weighted summation operations on the convolution kernel and the input state. In the calculation formula of the convolution layer, different convolution kernel weights determine the degree of attention to different positions and features. The bias term of the convolution layer provides an adjustable offset for the output of each position, and the flattened vector is used as the input of the fully connected layer. In the calculation formula of the fully connected layer, the weight matrix linearly transforms the input vector and maps it to a new feature space. The activation function introduces nonlinearity, enabling the model to learn more complex feature representations. The bias term of the fully connected layer again provides an adjustable offset for the output to better estimate the value of each action. Finally, after being processed by multiple fully connected layers, it finally reaches the output layer. The output layer outputs the value estimate of each action. This value estimate is the model's prediction of the expected return of taking different actions under a given input state. By comparing the value estimates of different actions, the model can select the action with the highest value estimate as the optimal equipment scheduling action strategy.

[0085] The steps to obtain the optimal scheduling mode under different states are:

[0086] Input data from the equipment scheduling strategy set to the improvement depth In each training batch, a batch of data is randomly selected, and the data in the device scheduling strategy set is divided into a training set, a validation set, and a test set. The optimization algorithm updates the network parameters;

[0087] Store the experience data generated by training and set a priority. The priority calculation formula is: ,in, is the discount factor, For the current reward, is the maximum predicted value in the next state, is the predictive value of the model;

[0088] Furthermore, the larger the priority result is, the more important the experience data is for the learning of the model, so the priority result can be used as a measure of priority;

[0089] A probability distribution is constructed based on the priority of the experience data. The higher the priority of the experience data, the greater the probability of being selected, based on the formula: ,in, To control the priority parameter, It is The probability of an experience being selected, For the The priority of the selected experience. It is the index variable after traversing all the data, based on Perform random sampling of experience values. For each experience data , where Current status, is the action taken, is the reward value, is the next state, and the loss function is calculated: ,in, is the real value, that is, the value obtained by simulating or emulating the equipment scheduling system in a specific state. Take action The true value of Represents the number of empirical data samples used to calculate the loss function;

[0090] After training, for a given state , the state Input into the model, the model outputs the value estimate of each action in a series of states, and selects the action with the highest value estimate as the optimal equipment scheduling action strategy.

[0091] Furthermore, by taking the given state Input into a trained deep network model, the model will output value estimates for each possible action. These value estimates are the model's prediction of the expected returns of taking different actions under a given input state. The value estimates of different actions output by the model are compared, and the action with the highest value estimate is selected as the optimal device scheduling action strategy.

[0092] In this embodiment, production feature data is obtained from workshop sensors and a production management database, and clustering operations are performed on the production feature data to obtain feature clustering labels and construct an optimized multidimensional state space. This method can classify data points with similar characteristics into one category, which is helpful for discovering potential patterns and structures in the data. Compared with a single feature representation, the clustered state space can better reflect the actual situation in the production process. Then, the convolutional layer in the convolutional neural network is replaced with a deep separable convolutional layer, which improves the computational efficiency and feature extraction capability of the model. The deep convolutional layer is used to extract local features for each input feature channel, and the point-by-point convolutional layer is used to combine and transform features, which can better adapt to the processing requirements of the multidimensional state feature vector. The model is trained through the loss function and the optimizer, so that the neural network can learn the value of actions in different states. Value estimation: For a given state, the action with the highest value estimation is selected as the optimal scheduling strategy. At the same time, the convolution layer captures the local patterns and spatial relationships in the input data through local weighted sum operations, and can extract the spatial features of the input state. Different convolution kernel weights and bias items can flexibly adjust the degree of attention to different positions and features, providing more valuable input information for the subsequent fully connected layer. The fully connected layer combined with the activation function can learn more complex feature representations, and map the spatial features extracted by the convolution layer to the new feature space. The adjustment of the weight matrix and bias items enables the model to perform adaptive feature transformation according to different input states, improving the processing capabilities of different scheduling situations. Finally, the output layer outputs the value estimation of each action, enabling the model to select the action with the highest value estimation as the optimal device scheduling action strategy.

[0093] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent scheduling optimization system for equipment upgrade in unmanned workshops based on reinforcement learning, characterized in that: include: Feature collection module: collects workshop equipment data, extracts corresponding data features, and obtains production feature data; Spatial optimization module: clusters production feature data to obtain feature clustering labels, and constructs an optimized multidimensional state space through feature clustering labels; The feature cluster labels are generated by The distance between each point and each initial cluster center is calculated, and the data points are assigned to the nearest non-noise cluster to obtain a series of new non-noise clusters. The feature means of these new non-noise clusters are obtained, and the new feature means are used as new cluster centers. The production feature data is traversed again. The distance between each point in the cluster and each new cluster center is calculated, and the data point is assigned to the nearest non-noise cluster until the cluster center no longer changes. Finally, each data point is assigned to a specific cluster and an integer cluster label is assigned to each data point, with a label value ranging from 0 to , indicating the cluster category to which the data point belongs, where is the final number of clusters, and the feature clustering label is obtained; Strategy Optimization Module: This module uses a neural network model with an optimized structure to learn the optimal equipment scheduling action strategy under different states in a multi-dimensional state space, and obtains a series of equipment scheduling strategy sets for different states. The neural network model of the optimized structure is implemented by a The optimizer optimizes the model, takes the multi-dimensional feature vector as input, and adds the corresponding equipment scheduling action and reward value consisting of training data to the neural network for training. In each training batch, a batch of data is randomly selected, the gradient of the loss function is calculated, and the loss function is used The parameters of the new neural network of the optimizer are: after multiple trainings, the neural network model learns the value estimation of actions under different states, and for a given state, selects the action with the highest value estimation as the optimal device scheduling action strategy, and finally obtains a neural network model with an optimized structure; Scheduling acquisition module: Building an improved depth Network model, input device scheduling strategy set to improve depth In the network model, the optimal scheduling mode under different states is obtained; The optimal scheduling model inputs the data in the equipment scheduling strategy set into the improved depth In each training batch, a batch of data is randomly selected, and the data in the device scheduling strategy set is divided into a training set, a validation set, and a test set. The optimization algorithm updates the network parameters.

2. The intelligent scheduling optimization system for equipment upgrade in unmanned workshops based on reinforcement learning according to claim 1 is characterized in that: The production feature data is obtained from sensors in the workshop and the production management database to construct a feature vector of the product instance. The product feature vector includes length, width, and height dimensional data, precision data, and features related to the production process, which are expressed as: .

3. According to claim 2, an unmanned workshop equipment upgrade intelligent scheduling optimization system based on reinforcement learning is characterized in that: The steps of obtaining feature clustering labels are: Set a neighborhood radius and minimum number of neighbors ; Based on production characteristic data , through one of the points , traverse except All production feature data outside, calculate the Euclidean distance with all other points, and obtain a Euclidean distance set ; in, Indicates the production characteristic data data points; Traverse all Euclidean distance sets, when Any point in the When , this point is represented by Points in the neighborhood; Traversing production feature data , determine the number of points in the neighborhood of each data point ,when Greater than the minimum number of neighbors When , mark this point as the core point. Less than the minimum number of neighbors And the neighborhood radius of any point If it is within, mark this point as a boundary point, and if it does not meet all the conditions, mark it as a noise point; From an unvisited core point Start, find the point All points in the neighborhood of , add these points to a cluster, and for each point in each newly added cluster, when it is greater than the minimum number of neighbor points When the number of neighbor points is greater than the minimum The points in the neighborhood radius of the point are added to the current cluster until there are no new points to add, and finally multiple non-noise clusters consisting of core points and boundary points are obtained. and pure noise points; Compute non-noise clusters The characteristic mean of all points in is calculated by the formula: ,in, is a non-noise cluster The number of all points in is a non-noise cluster The point in , takes the feature mean as the initial clustering center of this cluster; Based on production characteristic data Get feature cluster labels with the initial cluster centers.

4. According to claim 1, the intelligent scheduling optimization system for equipment upgrade in unmanned workshop based on reinforcement learning is characterized in that: The steps of constructing an optimized multidimensional state space are: The feature cluster labels are organized into a label vector sequence, expressed as: ,in Indicates The cluster labels of the data points are constructed based on this label vector sequence. , expressed as: ,in, Indicates Does the data point belong to cluster, if the The data point belongs to clusters, then ,otherwise ; Through the label matrix Get the initial state of each data point and preset a state space , set the number of times each state in the state space transfers to another state to , traverse the label vector sequence , calculate the number of state transitions: ,in, Indicates status arrive The number of times, for each time step , if the current data point The status is , and the next data point The status is ,So The value of is increased by 1; The state transition probability is calculated based on the number of state transitions, expressed as: , a state transfer matrix is ​​constructed based on the state transfer profile, and the state space is represented by the state transfer matrix, that is, the multi-dimensional state space is optimized.

5. According to claim 1, the intelligent scheduling optimization system for equipment upgrade in unmanned workshop based on reinforcement learning is characterized in that: The steps for constructing the neural network model of the optimized structure are: Obtaining a multi-dimensional state feature vector from the optimized multi-dimensional state space; Use a convolutional neural network and replace the convolutional layer in the convolutional neural network with a depth-separable convolutional layer. For the input multi-dimensional feature vector, it first passes through a depth-convolutional layer. The convolution kernel size of the depth-convolutional layer is ,in, It is the spatial dimension of the convolution kernel. Each convolution kernel operates on only one input feature channel to extract the local features of that channel. Next, the outputs of the depthwise convolution are combined using a point-wise convolution layer with a kernel size of , where The number of output channels of the depthwise convolution, is the desired number of output channels; After the depthwise separable convolutional layer, a fully connected layer is added, and then the neural network model is trained through a loss function. ,in, is a hyperparameter used to control the error threshold. is the output value of the real value, is the output value of the model value; Through a The optimizer performs model optimization and finally obtains a neural network model with an optimized structure.

6. According to claim 5, the intelligent scheduling optimization system for equipment upgrade in unmanned workshop based on reinforcement learning is characterized in that: The multi-dimensional state feature vector is composed of the in-degree, out-degree and stay probability of each state.

7. According to claim 1, the intelligent scheduling optimization system for equipment upgrade in unmanned workshop based on reinforcement learning is characterized in that: The improvement depth The steps to build the network model are: depth The network model includes convolutional layer, fully connected layer, and output layer; Set the input state to ,in, is the feature dimension, and a convolutional layer is used to extract the spatial features of the state. The formula is expressed as: ,in, Indicated in States in a layer The output, For the The convolution kernel weights of the layer, For the The bias term of the layer, then flatten the output of the convolutional layer and input it into the fully connected layer. The calculation formula of the fully connected layer is: ,in, is the activation function, is the weight matrix, is the bias term, Indicates The output of the layer is the input of the fully connected layer. After the linear transformation of the weight matrix and the processing of the activation function, the output of the current layer is obtained. After being processed by multiple fully connected layers, it finally reaches the output layer. The output layer outputs the final predicted value of each action. .

8. According to claim 1, the intelligent scheduling optimization system for equipment upgrade in unmanned workshop based on reinforcement learning is characterized in that: The steps of obtaining the optimal scheduling mode under different states are: Input data from the equipment scheduling strategy set to the improvement depth Training in the network; Store the experience data generated by training and set a priority. The priority calculation formula is: ,in, is the discount factor, For the current reward, is the maximum predicted value in the next state, is the predictive value of the model; A probability distribution is constructed based on the priority of the experience data. The higher the priority of the experience data, the greater the probability of being selected, based on the formula: ,in, To control the priority parameter, It is The probability of an experience being selected, For the The priority of the selected experience. It is the index variable after traversing all the data, based on Perform random sampling of experience input, for each experience data , where Current status, is the action taken, is the reward value, is the next state, and calculate the loss function: ,in, is the real value, that is, the value obtained by simulating or emulating the equipment scheduling system in a specific state. Take action The true value, after training, for a given state , input to improve depth In the network model, the model outputs the value estimate of each action and selects the action with the highest value estimate as the optimal device scheduling action strategy.

Images