Manufacturing data intelligent analysis method and system based on deep reinforcement learning

Abstract

This application relates to the field of data processing technology and discloses a method and system for intelligent analysis of manufacturing data based on deep reinforcement learning. The method includes: performing time-series processing on manufacturing equipment status, order characteristics, inventory levels, and quality indicator data to obtain a four-dimensional production data matrix; performing strategy learning through the LSTM-Actor-Critic algorithm to obtain a manufacturing decision strategy network; classifying and processing by production cycle to obtain a hierarchical data set; constructing an intelligent experience replay buffer; collaboratively optimizing order scheduling, inventory replenishment, and equipment task allocation decisions to obtain intelligent analysis results of manufacturing data. This application addresses the technical problems of existing manufacturing data analysis methods, which lack adaptive learning capabilities and are unable to handle multi-domain collaborative decision optimization.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for intelligent analysis of manufacturing data based on deep reinforcement learning. Background Art

[0002] Existing manufacturing data analysis methods primarily rely on traditional heuristic algorithms and static optimization models to process production data, such as particle swarm optimization, genetic algorithms, and integer programming models. These methods analyze data such as equipment status, order information, and inventory levels using pre-set rules and constraints, and make corresponding production decisions based on historical experience. Traditional methods perform well in relatively stable manufacturing environments, optimizing production efficiency and resource allocation to a certain extent, providing fundamental data analysis support for manufacturing companies' operational management.

[0003] However, existing technologies have significant shortcomings when dealing with dynamic and complex manufacturing environments. First, traditional heuristic algorithms lack adaptive learning capabilities and cannot automatically adjust decision-making strategies based on real-time changes in the production environment. When market demand fluctuates or equipment status changes, algorithm parameters and optimization objectives need to be manually redesigned. Secondly, static optimization models have difficulty handling complex scenarios of multi-objective conflicts and real-time decision-making in the manufacturing industry. In particular, when multiple decision domains such as order scheduling, inventory management, and equipment allocation require collaborative optimization, traditional methods often lead to local optimal rather than global optimal solutions. In addition, existing technologies do not adequately mine the temporal characteristics and periodic patterns of manufacturing data, and cannot effectively use historical production experience to guide current decision-making.

[0004] Due to the limitations of existing technologies, the manufacturing industry urgently needs a data analysis method capable of intelligent learning and adaptive optimization. Specifically, it needs to address the issues of effective time-series processing and feature extraction of multimodal manufacturing data, the construction of a deep learning network architecture capable of processing both production time-series features and business-related features, the implementation of hierarchical data management and intelligent experience playback based on the characteristics of the manufacturing production cycle, and the realization of collaborative optimization and global consistency constraints across multiple decision domains. Summary of the Invention

[0005] This application provides a method and system for intelligent analysis of manufacturing data based on deep reinforcement learning, which is used to solve the technical problems that existing manufacturing data analysis methods lack adaptive learning capabilities and cannot handle multi-domain collaborative decision-making optimization.

[0006] In the first aspect, the present application provides a method for intelligent analysis of manufacturing data based on deep reinforcement learning, and the method for intelligent analysis of manufacturing data based on deep reinforcement learning includes: performing time series processing on equipment status data, order feature data, inventory level data and quality index data in the manufacturing production environment to obtain a four-dimensional production data matrix; based on the four-dimensional production data matrix, performing strategy learning processing on production time series features and order-inventory association features through the LSTM-Actor-Critic deep reinforcement learning algorithm to obtain a manufacturing decision strategy network; classifying the four-dimensional production data matrix according to the preparation stage, production execution stage, quality inspection stage and delivery stage to obtain a production stage hierarchical data set; based on the production stage hierarchical data set, prioritizing sampling and dynamically updating historical decision experience data to obtain a production cycle-aware intelligent experience replay buffer; based on the manufacturing decision strategy network and the intelligent experience replay buffer, collaboratively optimizing order scheduling decisions, inventory replenishment decisions and equipment task allocation decisions to obtain intelligent analysis results of manufacturing data.

[0007] In a second aspect, the present application provides a manufacturing data intelligent analysis system based on deep reinforcement learning, the manufacturing data intelligent analysis system based on deep reinforcement learning includes:

[0008] A processing module is used to perform time-series processing on equipment status data, order feature data, inventory level data, and quality indicator data in a manufacturing production environment to obtain a four-dimensional production data matrix;

[0009] A learning module is used to perform strategy learning processing on production time series features and order-inventory association features using an LSTM-Actor-Critic deep reinforcement learning algorithm based on the four-dimensional production data matrix to obtain a manufacturing decision strategy network;

[0010] a classification module for classifying the four-dimensional production data matrix into production cycle categories according to the preparation stage, production execution stage, quality inspection stage, and delivery stage, thereby obtaining a hierarchical data set for the production stages;

[0011] An update module is used to perform priority sampling and dynamic update processing on historical decision experience data based on the production stage layered data set to obtain an intelligent experience playback buffer zone that is aware of the production cycle;

[0012] The optimization module is used to collaboratively optimize order scheduling decisions, inventory replenishment decisions, and equipment task allocation decisions based on the manufacturing decision strategy network and the intelligent experience replay buffer, so as to obtain intelligent analysis results of manufacturing data.

[0013] In a third aspect, a manufacturing data intelligent analysis device based on deep reinforcement learning is provided, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the manufacturing data intelligent analysis device based on deep reinforcement learning executes the above-mentioned manufacturing data intelligent analysis method based on deep reinforcement learning.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computer, enables the computer to execute the above-mentioned method for intelligent analysis of manufacturing data based on deep reinforcement learning.

[0015] In the technical solution provided by the present application, a four-dimensional production data matrix is ​​obtained by performing time-series processing on the equipment status data, order feature data, inventory level data and quality index data in the manufacturing production environment, thereby solving the problem that traditional methods cannot effectively integrate multimodal manufacturing data, and enabling heterogeneous data sources to be collaboratively analyzed under a unified time-series framework. The LSTM-Actor-Critic deep reinforcement learning algorithm overcomes the deficiency of the existing technology in the lack of adaptive learning ability in the strategic learning processing of production time-series features and order-inventory association features, enabling the manufacturing decision-making policy network to automatically adjust the optimization strategy according to the dynamic changes of the production environment, avoiding the limitation of the traditional heuristic algorithm that requires manual parameter adjustment. The four-dimensional production data matrix is ​​classified and processed according to the production cycle of the preparation stage, production execution stage, quality inspection stage and delivery stage to obtain a hierarchical data set of the production stage, which effectively solves the problem of insufficient mining of the time-series features and periodic patterns of manufacturing data in the existing technology, and enables historical production experience to be structured and stored and accurately called according to the characteristics of the production stage. The production cycle-aware intelligent experience replay buffer significantly improves the defect of traditional methods in effectively utilizing historical experience through priority sampling and dynamic update processing, ensuring that the deep reinforcement learning model always focuses on the historical decision-making experience most relevant to the current production situation, thereby improving learning efficiency and decision-making quality.

[0016] The time series modeling capability of the LSTM network enables the system to accurately identify the cyclical patterns of equipment operation and the dynamic relationship between orders and inventories. The policy learning mechanism of the Actor-Critic algorithm ensures that the decision network can generate optimal manufacturing decision actions under complex constraints. The introduction of a multi-agent collaborative mechanism enables different decision domains such as order scheduling, inventory management, and equipment allocation to achieve global optimization while maintaining their respective specializations. Compared with traditional particle swarm optimization and integer programming methods, the deep reinforcement learning algorithm of this application has the ability to continuously learn and self-optimize. It can quickly adapt to new production models when the manufacturing environment changes, without the need to redesign algorithm parameters or constraints, thereby providing manufacturing companies with a truly intelligent data analysis solution, effectively improving production efficiency, reducing operating costs, and improving resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 This is a schematic diagram of an embodiment of a method for intelligent analysis of manufacturing data based on deep reinforcement learning in an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of an embodiment of a manufacturing data intelligent analysis system based on deep reinforcement learning in an embodiment of the present application;

[0020] Figure 3 This is a schematic block diagram of the structure of a manufacturing data intelligent analysis device based on deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The embodiments of the present application provide a method and system for intelligent analysis of manufacturing data based on deep reinforcement learning. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices.

[0022] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiments of the present application, an embodiment of the manufacturing data intelligent analysis method based on deep reinforcement learning includes:

[0023] Step S101: Time-series processing is performed on equipment status data, order feature data, inventory level data, and quality index data in a manufacturing production environment to obtain a four-dimensional production data matrix;

[0024] Step S102: Based on the four-dimensional production data matrix, the LSTM-Actor-Critic deep reinforcement learning algorithm is used to perform policy learning processing on the production time series features and the order-inventory association features to obtain a manufacturing decision strategy network;

[0025] Step S103: Classify the four-dimensional production data matrix into production cycles according to the preparation stage, production execution stage, quality inspection stage, and delivery stage to obtain a production stage hierarchical data set;

[0026] Step S104: Prioritize sampling and dynamically update historical decision experience data based on the production stage hierarchical data set to obtain an intelligent experience playback buffer zone that is aware of the production cycle;

[0027] Step S105: Based on the manufacturing decision strategy network and the intelligent experience playback buffer, collaborative optimization processing is performed on order scheduling decisions, inventory replenishment decisions, and equipment task allocation decisions to obtain intelligent analysis results of manufacturing data.

[0028] It is understandable that the execution subject of this application can be a manufacturing data intelligent analysis system based on deep reinforcement learning, or a terminal or server, which is not limited here. The embodiment of this application is described by taking the server as the execution subject as an example.

[0029] Specifically, four types of manufacturing data were time-series processed using a sliding window technique to form a four-dimensional production data matrix. The sliding window technique segments continuous time series data into fixed-length windows, with each window containing data samples from a specific time period. Equipment status data, including parameters such as equipment operating status, temperature, pressure, and speed, is segmented using a sliding window to form an equipment status time-series submatrix. Each time segment records the changes in the equipment's operating parameters during that time period. Discrete order types in the order feature data were converted to continuous values ​​using one-hot encoding, and process parameters were standardized to form a sequence of order feature vectors. Inventory level data was divided into time windows based on the length of the production cycle, recording changes in raw material, semi-finished product, and finished product inventory within each cycle to form an inventory level time-series matrix. Quality indicator data, including metrics such as pass rate, defect rate, and rework rate, was normalized using Z-scores to eliminate dimensional differences between different metrics, resulting in a standardized quality indicator vector. These four data components were combined using a dimension concatenation operation to form a unified four-dimensional production data matrix, with each row representing a time step and each column representing a different type of production feature.

[0030] An LSTM-Actor-Critic deep reinforcement learning network architecture is constructed based on a four-dimensional production data matrix. The LSTM long short-term memory network features a gating mechanism that effectively handles long-term dependencies in time series data. The first-layer LSTM network specifically processes the equipment state time series submatrix. Through the coordinated action of forget gates, input gates, and output gates, it extracts cyclical patterns and abnormal change trends in equipment operation, and outputs a device state feature vector. The order feature vector sequence and the inventory level time series matrix are first fused through element-wise multiplication and concatenation to form an order-inventory joint feature matrix, which reflects the dynamic balance between order demand and inventory supply. The second-layer LSTM network processes the joint feature matrix, learns the impact of order changes on inventory levels, and generates a business-layer time series feature vector. The actor network receives the equipment state feature vector and the business-layer time series feature vector as input. Using a fully connected layer and a softmax activation function, it calculates the probability distribution of various manufacturing decision actions, including discrete action choices such as production batch adjustment, equipment maintenance scheduling, and inventory replenishment. The Critic network adopts a hybrid architecture. The convolutional layer processes the spatial distribution characteristics of the equipment status, and the fully connected layer processes the temporal characteristics. The final output is the state value function to evaluate the quality of the current production status.

[0031] The four-dimensional production data matrix is ​​divided into four stages according to the standard manufacturing production process. The time boundaries of the preparation stage are determined by extracting the order receipt timestamp and the production start timestamp. This stage includes activities such as order analysis, material preparation, and equipment preheating. The production execution stage determines the time range by identifying start and stop signals in the equipment status data. A start signal is characterized by a sudden increase in equipment power and a speed reaching a preset value, while a stop signal is characterized by a gradual reduction in equipment speed to a shutdown state. The quality inspection stage is divided according to the inspection start and completion timestamps in the quality indicator data. The inspection start time corresponds to the moment the product is off the production line, and the completion time corresponds to the moment the quality report is generated. The delivery stage is determined by outbound shipment records and order completion markers in the inventory level data. Outbound shipment records indicate a decrease in finished product inventory, while order completion markers indicate customer confirmation of delivery. Based on these time boundary markers, the four-dimensional production data matrix is ​​sliced ​​into stages. Each slice contains the corresponding stage's equipment status, order information, inventory changes, and quality data, forming a hierarchical dataset for the production stages.

[0032] A production cycle-aware intelligent experience replay buffer is constructed, optimizing experience storage and sampling strategies based on the characteristics of manufacturing production. The hierarchical experience storage structure allocates storage capacity based on the actual time proportions of the four production stages: the preparation stage typically accounts for 15%, the production execution stage for 60%, the quality inspection stage for 15%, and the delivery stage for 10%. Historical decision experience data includes information such as state transition sequences, executed actions, and rewards earned. The temporal difference error (TDE) is calculated as the difference between the current state value and the next state value plus the discounted reward. A larger TDE indicates a more valuable experience for policy improvement. Experience data priority is determined based on the absolute value of the TDE, with experiences with higher TDE receiving a higher sampling probability. Current production cycle information is matched with historical cycle patterns through correlation analysis. Experience samples corresponding to historical patterns with high correlation receive a higher sampling frequency. The sampling frequency adjustment coefficient is dynamically calculated based on the TDE match. A higher TDE match increases the coefficient, prioritizing historical experience from similar scenarios.

[0033] Collaborative optimization is performed based on a manufacturing decision policy network and an intelligent experience replay buffer. The probabilistic output of manufacturing decision actions is decomposed into three decision domains: the order scheduling domain focuses on order prioritization and production planning; the inventory replenishment domain focuses on raw material procurement timing and quantity decisions; and the equipment task allocation domain focuses on equipment load balancing and maintenance scheduling. The policy networks for these three decision domains simultaneously update their parameters through a parallel training mechanism. Each domain samples relevant experience from the intelligent experience replay buffer for gradient descent optimization. Multi-agent collaborative decision-making ensures decision consistency through information sharing and constraint coordination. The order priority allocation sequence determines the production sequence, the inventory replenishment quantity vector determines material requirements, and the equipment task scheduling matrix determines resource allocation. Global consistency constraint verification checks for logical conflicts between the three decision outcomes, such as the compatibility of order scheduling with equipment availability and the synchronization of inventory replenishment time with production demand. Conflicts are resolved through iterative optimization and constraint satisfaction algorithms, generating collaborative optimization solutions. The feasibility and expected performance of these solutions are then evaluated based on the current production status.

[0034] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0035] The device status data is segmented into time series through a sliding window to obtain the device status time series submatrix;

[0036] The discrete order types and process parameters in the order feature data are continuously encoded to obtain an order feature vector sequence;

[0037] The inventory level data is divided into time windows according to the length of the production cycle to obtain the inventory level time series matrix;

[0038] Based on the statistical distribution characteristics of the quality indicator data, standardization and normalization processing are performed to obtain a quality indicator standardized vector;

[0039] The equipment status time series sub-matrix, order feature vector sequence, inventory level time series matrix and quality indicator standardized vector are dimensionally spliced ​​to obtain a four-dimensional production data matrix.

[0040] Specifically, the sliding window's time series segmentation of device status data involves cutting continuously collected device data into fixed time segments. The sliding window technique divides the continuous data stream on the time axis into multiple overlapping time segments. Device status data includes real-time monitoring parameters such as machine tool speed, cutting force, tool wear, and processing temperature. Each sensor collects data at a fixed frequency to form a time series. The sliding window segments the original time series according to a preset window length and step size. The window length determines the time range contained in each data segment, and the step size determines the degree of overlap between adjacent windows. After segmentation, a device status time series submatrix is ​​formed. The rows of the matrix represent time steps, and the columns represent different device parameters. Each submatrix records the complete change process of the device's operating status within a specific time period.

[0041] The discrete order types and process parameters in the order feature data are continuously encoded using a combination of one-hot encoding and numerical mapping. Discrete order types include standard orders, urgent orders, and custom orders. One-hot encoding converts each type into a binary vector, with standard orders encoded as [1,0,0], urgent orders as [0,1,0], and custom orders as [0,0,1]. Process parameters include discrete values ​​such as machining accuracy level, surface roughness requirements, and material hardness level. A mapping table is established to convert discrete levels into continuous values, with machining accuracy levels from rough machining to fine machining mapped to values ​​from 1 to 10, respectively. The continuous encoding process converts all discrete features into numerical vectors, which are arranged in chronological order to form a sequence of order feature vectors. Each vector in the sequence represents the order status and process requirements at a specific moment.

[0042] Inventory level data is processed into time windows based on the length of the production cycle, with the time window boundaries determined according to the manufacturing company's standard production rhythm. The length of the production cycle is determined by the time span between order placement and product delivery, and different product types have different production cycles. Time window partitioning segments inventory data into segments that are integer multiples of the production cycle. Each time window contains inventory change information such as the amount of raw materials entering the warehouse, the amount of semi-finished products flowing through the warehouse, and the amount of finished products leaving the warehouse. The inventory level time series matrix records the inventory change trajectory of each material type within each time window. The matrix rows represent the time window sequence number, and the columns represent the inventory levels of different material types. Through window partitioning, continuous inventory flow data is converted into a structured time series matrix.

[0043] Standardization and normalization of the statistical distribution characteristics of quality indicator data eliminates differences in dimensions and numerical ranges between different indicators. Quality indicator data includes multiple metrics such as product qualification rate, defect density, number of reworks, and customer satisfaction, each with a different numerical range and statistical distribution characteristics. Standardization calculates the mean and standard deviation of each indicator, subtracting the mean from the original value and dividing it by the standard deviation to obtain the standardized value. Normalization maps the standardized value to the range of 0 to 1, using a linear transformation to map the minimum value to 0 and the maximum value to 1. After standardization, the standardized vectors of quality indicators have the same numerical range and distribution characteristics, facilitating unified processing by subsequent deep reinforcement learning algorithms.

[0044] The dimensional splicing process combines the four data components into a unified four-dimensional production data matrix, and the splicing operation is performed according to the principle of time alignment. The equipment status time series submatrix, order feature vector sequence, inventory level time series matrix, and quality indicator normalized vector correspond to different dimensions of information in the manufacturing process. The dimensional splicing operation horizontally connects data with the same timestamp. The splicing process first synchronizes the four data components to ensure that each time step contains corresponding records of the four types of data. Then, the four vectors are connected into a single long vector in a predetermined order. Each row of the four-dimensional production data matrix represents the state of the manufacturing system at a specific moment, including equipment operating parameters, order demand information, inventory supply status, and quality control indicators. Each column represents the change trajectory of a specific feature in the time series.

[0045] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0046] The equipment status time series submatrix in the four-dimensional production data matrix is ​​input into the first-layer LSTM network to extract the equipment operation periodic pattern and obtain the equipment status feature vector;

[0047] Perform feature fusion processing on the order feature vector sequence and inventory level time series matrix in the four-dimensional production data matrix to obtain the order-inventory joint feature matrix;

[0048] The order-inventory joint feature matrix is ​​input into the second-layer LSTM network for dynamic association relationship learning to obtain the business layer time series feature vector;

[0049] Based on the device state feature vector and the business layer time series feature vector, an Actor network is constructed to calculate and process the strategy probability distribution, and the manufacturing decision action probability output is obtained;

[0050] A critic network is constructed based on the four-dimensional production data matrix to perform state value function evaluation and obtain the state value evaluation result, which is combined with the manufacturing decision action probability output to form a manufacturing decision strategy network.

[0051] Specifically, the first-layer LSTM network receives the device state time-series submatrix as input and extracts periodic patterns in device operation. The LSTM network has three gating mechanisms: a forget gate, an input gate, and an output gate. The forget gate determines which information is discarded from the cell state. It uses a sigmoid activation function to linearly transform the previous hidden state and the current input, then outputs a value between 0 and 1. Values ​​closer to 0 indicate that the information is forgotten, while values ​​closer to 1 indicate that the information is retained. The input gate determines which new information is stored in the cell state. It consists of two parts: a sigmoid layer that determines which values ​​are updated and a tanh layer that creates a candidate value vector. The output gate controls which parts of the cell state are output. The hidden state output is obtained through a combination of sigmoid and tanh functions. The device state time-series submatrix contains time-series information such as changes in machine speed, cutting temperature fluctuations, and vibration amplitude. The LSTM network uses gating mechanisms to identify periodic patterns in device operation, such as the start-stabilization-stop cycle during machining, while also detecting abnormal fluctuations and trend changes. The output device state feature vector contains abstract feature information such as device health, operating efficiency, and fault warnings. The feature fusion process of the order feature vector sequence and the inventory level time series matrix achieves data association through element-by-element multiplication and concatenation operations. The order feature vector sequence records order demand information at each moment, including features such as order quantity, urgency, and product specifications. The inventory level time series matrix records the inventory changes of raw materials, semi-finished products, and finished products. Feature fusion first aligns the two data series in time to ensure that data at the same moment is fused. Then, element-by-element multiplication is performed to capture the correlation between order demand and inventory supply. The multiplication result reflects the degree of supply and demand matching. When order demand is high and inventory is low, the multiplication value is small, indicating a prominent supply-demand imbalance. The concatenation operation horizontally connects the order and inventory features to form a higher-dimensional joint feature vector. The concatenated order-inventory joint feature matrix contains information on both the demand and supply sides. Each row represents the supply and demand status at a specific moment, and each column represents the temporal trend of a specific feature.

[0052] The second-layer LSTM network processes the order-inventory joint feature matrix to learn dynamic associations. This network specifically learns how order changes affect inventory levels and how inventory status constrains order execution. The LSTM network captures the causal chain between order placement, production scheduling, inventory consumption, and replenishment decisions through time series modeling. The network's hidden state records historical supply and demand interaction patterns. Current input is combined with historical state to generate predictions about future supply and demand relationships. Dynamic association learning encompasses complex relationships such as the impact of order batch size on inventory turnover, the impact of inventory shortages on order delays, and the impact of urgent orders on inventory allocation. The LSTM network uses a backpropagation algorithm to adjust weight parameters and gradually learn these dynamic association patterns. The output business-level time series feature vector contains abstract business-level features such as supply-demand balance, inventory risk level, and order fulfillment capability. These features reflect the operational status and trends of the manufacturing system at the business execution level.

[0053] The actor network constructs a policy probability distribution calculation module based on the device state feature vector and the business-layer time series feature vector. This actor network utilizes a fully connected neural network architecture. The input layer receives the concatenation of the two feature vectors, the hidden layer extracts the complex relationships between the features through nonlinear transformations, and the output layer generates an action probability distribution using a softmax activation function. The device state feature vector provides device-level constraint information, such as equipment availability, maintenance requirements, and capacity constraints. The business-layer time series feature vector provides business-level requirement information, such as order priority, inventory pressure, and delivery requirements. The policy probability distribution calculation comprehensively considers both device constraints and business requirements, outputting the selection probabilities for various manufacturing decision actions, including discrete action options such as batch size adjustment, equipment maintenance scheduling, and order schedule changes. The manufacturing decision action probability output is in the form of a probability vector, with a length equal to the action space dimension. Each element represents the probability of executing the corresponding action, with higher probability values ​​indicating a more optimal action under the current conditions. The critic network constructs a state-value function evaluation module based on a four-dimensional production data matrix. The critic network utilizes a hybrid architecture of convolutional neural networks and fully connected networks. The convolutional layer processes the spatial distribution characteristics of device status data, extracting local feature patterns by sliding the convolution kernel across the data matrix. The pooling layer performs dimensionality reduction on the convolutional features to reduce the number of parameters. The fully connected layer processes temporal and business features, learning the complex mapping relationships between features through multiple layers of nonlinear transformations. The state value function assesses the expected long-term benefits of the current production status, with evaluation indicators including multi-dimensional performance such as production efficiency, resource utilization, and customer satisfaction. The manufacturing decision-making strategy network combines the actor network and the critic network through a parameter sharing mechanism. The two networks share some underlying feature extraction layers, reducing model parameter redundancy while maintaining their respective functional independence. The actor network is responsible for action selection, and the critic network is responsible for state evaluation. The two work together to form a policy learning framework.

[0054] In a specific embodiment, the process of executing the step of constructing a critic network based on the four-dimensional production data matrix to perform state value function evaluation processing may specifically include the following steps:

[0055] The equipment status time sequence submatrix in the four-dimensional production data matrix is ​​reconstructed into a two-dimensional equipment status matrix and processed by convolutional neural network to obtain the equipment spatial feature map;

[0056] Perform dimensionality reduction processing on the device spatial feature map using the pooling layer to obtain the device state space feature vector;

[0057] The business layer time series feature vector is input into the first fully connected layer for nonlinear transformation to obtain the intermediate feature representation vector;

[0058] Perform feature concatenation based on the device state space feature vector and the intermediate feature representation vector to obtain a comprehensive state feature vector;

[0059] The state comprehensive feature vector is input into the second fully connected layer for state value regression calculation processing to obtain the state value evaluation result, which is combined with the manufacturing decision action probability output through a parameter sharing mechanism to form a manufacturing decision strategy network.

[0060] Specifically, reconstructing the device state time series submatrix into a two-dimensional device state matrix is ​​the process of converting three-dimensional time series data into two-dimensional spatial data. The reconstruction operation encodes the time dimension information into spatial locations. The original device state time series submatrix contains three dimensions: time step, device number, and parameter type. The reconstruction process combines the time step and device number as row indices, and the parameter type as column indices, forming a two-dimensional matrix structure. Convolutional neural networks process the two-dimensional device state matrix by sliding a convolution kernel across the matrix to extract features. The convolution kernel is a small weight matrix that is summed by element-wise multiplication with a local region of the input matrix to obtain eigenvalues. The convolution operation can detect spatial correlation patterns between device parameters, such as the collaborative working status of adjacent devices and the propagation path of parameter anomalies. The device spatial feature map is the output of the convolution operation. Each feature map corresponds to the response of a convolution kernel and contains local feature information of the device state. Multiple feature maps are combined to describe the operating mode and anomaly distribution of a group of devices.

[0061] The pooling layer performs dimensionality reduction processing by sampling the device spatial feature map to reduce the data dimension. The pooling operation defines a fixed-size window on the feature map, and the values ​​within the window are aggregated through maximum pooling or average pooling. Maximum pooling selects the maximum value within the window as the output, retaining the most significant feature response, while average pooling calculates the mean of the values ​​within the window, preserving the overall feature trend. The pooling layer dimensionality reduction processing converts the high-resolution feature map into a compact low-resolution representation while maintaining important spatial feature information. The device state space feature vector is a one-dimensional vector output by the pooling layer. Each element of the vector represents the feature intensity of the device group at a specific spatial location. The vector dimension is much smaller than the number of pixels in the original feature map, but it contains key information about the device's operating status.

[0062] The business layer's time series feature vector is input into the first fully connected layer for nonlinear transformation. A fully connected layer is a fundamental component of a neural network, where each neuron is connected to all neurons in the previous layer. The nonlinear transformation is performed by matrix multiplication of the weight matrix with the input vector. After adding a bias vector, the output is processed through an activation function. Activation functions introduce nonlinearity. Common activation functions include ReLU, Sigmoid, and Tanh. ReLU sets negative values ​​to zero and retains positive values. Sigmoid maps values ​​to the range 0 to 1. Tanh maps values ​​to the range -1 to -1. The business layer's time series feature vector contains abstract business-level features, such as order demand trends, inventory change patterns, and production plan adjustments. The nonlinear transformation of the first fully connected layer maps these business features into a new feature space. The intermediate feature representation vector contains high-level abstractions and combination patterns of these business features.

[0063] The feature concatenation process horizontally connects the device state space feature vector and the intermediate feature representation vector to form a higher-dimensional vector. The concatenation operation simply arranges the elements of the two vectors in sequence into a single long vector. The device state space feature vector reflects the physical state of the manufacturing system, while the intermediate feature representation vector reflects the business state of the manufacturing system. The combined state feature vector contains both physical constraints and business requirement information. The first half of the state comprehensive feature vector corresponds to the device state characteristics, while the second half corresponds to the business state characteristics. The vector dimension is equal to the sum of the two input vector dimensions. The concatenation process establishes an information bridge between the device state and the business state, enabling subsequent value assessments to comprehensively consider both physical feasibility and business rationality.

[0064] The second fully connected layer performs state value regression on the comprehensive state feature vector. Regression is a machine learning task that predicts continuous values. Unlike classification tasks, the output of regression tasks is real-valued rather than discrete categories. State value regression maps the high-dimensional feature vector into a single value score, which represents the expected long-term return of the current manufacturing state. The second fully connected layer typically has only one output neuron, and the activation function uses a linear function or a tanh function to ensure that the output value remains within a reasonable range. The state value assessment result quantifies the quality of the current production state. Positive values ​​indicate that the state is conducive to achieving long-term goals, while negative values ​​indicate potential risks or efficiency issues.

[0065] A parameter sharing mechanism combines the manufacturing decision action probability output with the state value assessment results to form a manufacturing decision policy network. Parameter sharing refers to the use of the same weight parameters by multiple network modules for calculations. The actor network and the critic network share the underlying feature extraction layer, including the parameters of the convolutional layer, pooling layer, and some fully connected layers. The two networks use the same weight matrix and bias vector during the feature extraction stage. Parameter sharing reduces the total number of model parameters, mitigates the risk of overfitting, and ensures that the actor network and the critic network have a consistent understanding of state characteristics. The manufacturing decision policy network coordinates action selection and value assessment through parameter sharing. The actor network generates action probability distributions to guide decision execution, while the critic network evaluates state values ​​to guide policy improvement.

[0066] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0067] The time period is extracted based on the order receiving time and production start time in the order feature vector sequence to obtain the time boundary identifier of the preparation stage;

[0068] Based on the equipment start signal and stop signal in the equipment status timing sub-matrix, the production time period is identified and processed to obtain the time boundary identification of the production execution stage;

[0069] Perform time interval processing on the inspection start timestamp and inspection completion timestamp in the quality indicator standardized vector to obtain the time boundary identifier of the quality inspection stage;

[0070] The time range of the outbound record time and the order completion time in the inventory level time series matrix is ​​determined to obtain the time boundary identifier of the delivery stage;

[0071] According to the time boundary identifiers of the preparation phase, production execution phase, quality inspection phase and delivery phase, the four-dimensional production data matrix is ​​sliced ​​to obtain a hierarchical data set of production phases.

[0072] Specifically, the time segment extraction process in the order feature vector sequence determines the time boundaries of the preparation phase by calculating the difference between timestamps. The order receipt time refers to the time when the order management system records the customer request, and the production start time refers to the time when the production planning system issues the production instruction. The time segment extraction process performs a difference calculation on the two timestamps to calculate the time interval between order receipt and production start. This interval includes preparation activities such as order review, material preparation, process planning, and equipment preheating. The preparation phase time boundary identifier is represented by a tuple of start time and end time. The start time corresponds to the order receipt timestamp, and the end time corresponds to the production start timestamp. The time boundary identifier records the duration and time range of the preparation phase for each order. Different order types and complexities have different preparation times, with standard orders having shorter preparation times and custom orders having longer preparation times. The production time segment identification process in the equipment state time series submatrix uses a signal detection algorithm to identify the nodes at which the equipment's operating state changes. The equipment start signal is represented by a sudden increase in equipment power from standby to operating state, while the equipment stop signal is represented by a gradual decrease in equipment power from operating state to standby state. The signal detection algorithm identifies state transition moments by setting power thresholds and rate-of-change thresholds. A start signal is determined when the device power exceeds the start threshold and the rate of change is greater than the set value. A stop signal is determined when the device power falls below the stop threshold and the rate of change is less than the negative set value. The production execution phase time boundary identifies the continuous operating period from the start to the stop of the device, encompassing a combination of multiple start-stop cycles. Each production batch corresponds to a complete start-stop cycle. The time boundary of the production execution phase is determined by the first start signal and the last stop signal. Short downtimes in between do not affect the overall phase division.

[0073] The time interval division process in the standardized vector of quality indicators is divided into intervals according to the time nodes of the quality inspection process. The inspection start timestamp corresponds to the moment when the product comes off the production line and enters the quality inspection station, and the inspection completion timestamp corresponds to the moment when the quality inspection report is generated and uploaded to the system. The time interval division process uses the inspection start and completion times as interval boundaries to calculate the quality inspection duration of each product. Different product types and inspection items correspond to different inspection durations. The appearance inspection time of simple products is shorter, and the performance test time of complex products is longer. The time boundary identifier of the quality inspection stage includes three elements: the inspection start time, the inspection completion time, and the inspection duration. The time boundary identifier is associated with the product batch number and records the quality inspection time window of each batch of products. The time division of the inspection stage directly affects the learning effect of the subsequent deep reinforcement learning model on the quality control strategy.

[0074] The time range determination process in the inventory level timing matrix determines the time boundary of the delivery phase through correlation analysis between outbound shipment records and order status. The outbound shipment record time refers to the timestamp when the finished product is shipped from the warehouse, and the order completion time refers to the timestamp when the customer confirms receipt and the system marks the order completion. The time range determination process uses the outbound shipment record time as the start time of the delivery phase and the order completion time as the end time of the delivery phase. The time interval between the two includes delivery activities such as logistics transportation, customer acceptance, and signature confirmation. The delivery phase time boundary identifies the time range from product shipment to order closure. This time range is affected by factors such as transportation distance, logistics method, and customer acceptance process. Different customers and product types have different delivery times. The delivery time for local customers is shorter, and the delivery time for remote customers is longer.

[0075] The stage data slicing process performs a time dimension segmentation operation on the four-dimensional production data matrix according to four time boundary markers. Data slicing refers to dividing a continuous data stream into discrete data segments according to time boundaries. The slicing process first sorts the four time boundary markers to ensure the correct time sequence of the preparation stage, production execution stage, quality inspection stage, and delivery stage. Then, the time dimension of the four-dimensional production data matrix is ​​indexed and segmented according to the time boundaries. The production stage hierarchical dataset contains four sub-datasets, each of which corresponds to a data slice of a production stage. The data within the slice retains the original multi-dimensional information such as equipment status, order characteristics, inventory levels, quality indicators, etc. The data between slices is temporally continuous but logically independent. The construction of the hierarchical dataset enables the deep reinforcement learning model to learn specialized decision-making strategies for different production stages.

[0076] Taking the AGV robot scheduling system in a smart manufacturing workshop as an example, the order feature vector sequence records the time when the AGV transport task is received and the time when it starts executing. When the workshop receives a material handling order, the system records the order receipt timestamp. When the AGV scheduling algorithm calculates the path plan and issues the execution instruction, it records the production start timestamp. The difference between these two timestamps constitutes the time boundary of the AGV task preparation phase. The device state time series submatrix contains parameters such as the AGV's battery voltage, motor speed, and position coordinates. When the AGV detaches from the charging station and begins moving, the motor speed suddenly increases from zero, triggering a start signal. When the AGV completes the transport task and returns to the standby position, the motor speed drops to zero, triggering a stop signal. The time period between the start and stop signals constitutes the time boundary of the production execution phase. The standardized quality indicator vector records the AGV's transport accuracy and timeliness indicators. When the AGV arrives at the target location and begins cargo handover, the system records the inspection start timestamp. When the cargo handover is completed and the transport quality is confirmed to be qualified, the system records the inspection completion timestamp. These two timestamps constitute the time boundary of the quality inspection phase. The inventory level time series matrix records the changes in the warehouse's goods entering and leaving the warehouse. When the AGV removes the goods from the source location, the system records the outbound timestamp. When the AGV delivers the goods to the target location and completes the task, the system records the order completion timestamp. These two timestamps constitute the time boundary of the delivery phase. Based on the time boundary identifiers of the four phases, the AGV scheduling system divides the production data matrix into path planning data for the preparation phase, operation monitoring data for the execution phase, accuracy assessment data for the inspection phase, and task confirmation data for the delivery phase. This hierarchical dataset enables the deep reinforcement learning algorithm to learn the optimal scheduling strategy for the AGV at different stages: path optimization strategy for the preparation phase, operation control strategy for the execution phase, quality assurance strategy for the inspection phase, and task coordination strategy for the delivery phase.

[0077] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0078] The storage capacity of the hierarchical data set of the production stage is allocated according to the time ratio of the preparation stage, production execution stage, quality inspection stage and delivery stage to obtain a hierarchical experience storage structure;

[0079] Based on the state transition sequence and reward signal in the historical decision experience data, the time difference error calculation is performed to obtain the experience data priority weight;

[0080] According to the priority weight of the experience data, the probability distribution of the experience samples in the hierarchical experience storage structure is redistributed to obtain the priority sampling probability distribution;

[0081] Match and analyze the current production beat information with the historical production cycle pattern to obtain the sampling frequency adjustment coefficient;

[0082] Based on the priority sampling probability distribution and the sampling frequency adjustment coefficient, batch selection and buffer update processing are performed on the experience samples to obtain an intelligent experience replay buffer that is aware of the production cycle.

[0083] Specifically, storage capacity allocation for hierarchical datasets in the production phase involves partitioning memory space based on the actual production time ratio in the manufacturing industry. Storage capacity allocation involves allocating limited buffer memory to different data categories according to predetermined ratios. The preparation phase, which encompasses activities such as order analysis, material preparation, and equipment preheating, accounts for a relatively small portion of the entire production cycle and is typically allocated 15% of the total storage capacity. The production execution phase, encompassing the actual manufacturing process, is the most critical link in the production cycle and is allocated 60% of the total storage capacity. The quality inspection phase, encompassing activities such as product testing, data recording, and report generation, is allocated 15% of the total storage capacity. The delivery phase, encompassing activities such as packaging, shipping, and transportation, is allocated 10% of the total storage capacity. The hierarchical experience storage structure utilizes a linked list or dynamic array data structure. Each phase corresponds to a separate storage area, which stores historical decision experience data for that phase in chronological order. Storage capacity allocation ensures that experience data from each phase is distributed appropriately according to its actual importance.

[0084] The temporal difference error (TDE) calculation process is based on the value function update mechanism used in reinforcement learning. TDE refers to the difference between the predicted value of the current state and the observed value. Historical decision experience data consists of a four-tuple sequence consisting of state, action, reward, and next state. The state transition sequence records the change from the current state to the next state, and the reward signal reflects the immediate benefit obtained after performing a specific action. TDE is calculated as the difference between the estimated value of the current state and the target value. The target value is equal to the immediate reward plus the discounted value of the next state. The discount factor controls the weighting of future rewards. Experience data priority weights are determined based on the absolute value of the TDE. Larger errors indicate a more valuable experience for policy improvement and are associated with larger priority weights. Experience with smaller errors contributes less to policy improvement and is associated with smaller priority weights. Priority weights are calculated using a power function of the absolute value of the error, with the power exponent controlling the discrimination of the priorities.

[0085] Probability distribution redistribution converts the priority weights of experience data into sampling probabilities. Probability distribution redistribution involves recalculating the probability of each experience sample being selected based on the priority weights. Experience samples in the hierarchical experience storage structure are sorted by priority weight, with samples with higher weights placed first and samples with lower weights placed last. Probability distribution redistribution converts priority weights into probability values ​​through normalization. The sampling probability of each sample is equal to its priority weight divided by the sum of all sample weights. Priority sampling probability distribution ensures that important experiences are sampled more frequently while ensuring that low-priority experiences still have a chance of being selected, thus avoiding completely ignoring certain experience samples. Probability distribution is implemented using a roulette wheel selection algorithm. Samples with higher probabilities occupy larger sectors of the wheel and have a higher probability of being selected during random selection.

[0086] The matching analysis between production cycle information and historical production cycle patterns uses correlation calculations to identify similar production scenarios. Production cycle information includes real-time indicators such as current equipment operating frequency, order processing speed, and material flow rate. Historical production cycle patterns are typical cycle characteristics extracted from past production data, encompassing production patterns for different product types, seasons, and order volumes. The matching analysis calculates the similarity between the current production cycle and each historical pattern using metrics such as Euclidean distance or cosine similarity. Smaller distances or higher similarities indicate a better match. The sampling frequency adjustment coefficient is determined based on the degree of match. Empirical samples corresponding to historical patterns with high matching scores receive a higher sampling frequency, while those corresponding to historical patterns with low matching scores receive a lower sampling frequency. The adjustment coefficient is calculated using linear interpolation or an exponential function to ensure a reasonable mapping between sampling frequency and matching scores.

[0087] Batch selection and buffer update processing perform experience sample selection and storage management based on the priority sampling probability distribution and the sampling frequency adjustment coefficient. Batch selection refers to selecting a certain number of samples from the experience storage structure to form a training batch. The selection process first performs random sampling based on the priority sampling probability distribution. The sampling result is then corrected by applying the sampling frequency adjustment coefficient, giving priority to experience samples that have a high degree of match with the current production rhythm. The buffer update process includes the addition of new experience and the elimination of old experience. When the buffer capacity reaches the upper limit, the experience samples with the lowest priority and the worst match with the current production rhythm are eliminated. The production cycle-aware intelligent experience replay buffer ensures that experience replay always focuses on the historical experience most relevant to the current production situation by dynamically adjusting the storage and sampling strategies, while maintaining the diversity of experience to avoid overfitting.

[0088] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0089] The manufacturing decision action probability output in the manufacturing decision strategy network is probabilistically decomposed into three decision domains: order scheduling, inventory replenishment, and equipment task allocation, to obtain the domain-specific decision probability distribution.

[0090] Based on the experience samples in the intelligent experience replay buffer, the policy networks of the three decision domains are trained and updated in parallel to obtain a set of domain-specific policy parameters.

[0091] Based on the domain-specific decision probability distribution and domain-specific strategy parameter set, multi-agent collaborative decision-making calculations are performed to obtain the order priority allocation sequence, inventory replenishment quantity vector, and equipment task scheduling matrix.

[0092] The order priority allocation sequence, inventory replenishment quantity vector and equipment task scheduling matrix are subjected to global consistency constraint verification to obtain a collaborative optimization decision-making solution.

[0093] Based on the collaborative optimization decision-making scheme, the current production status and expected performance indicators are comprehensively evaluated and calculated to obtain the results of intelligent analysis of manufacturing data.

[0094] Specifically, the probability decomposition process of the manufacturing decision action probability output splits the unified probability vector into three independent decision domain probability distributions through dimensional segmentation. Probability decomposition refers to the division of a high-dimensional probability vector into multiple low-dimensional sub-vectors according to predetermined rules. The probability vector output by the manufacturing decision strategy network contains the probability values ​​of all decision actions and is grouped and segmented according to the action type. The order scheduling domain includes the probabilities of actions such as order priority adjustment, production sequence change, and delivery date rescheduling. The inventory replenishment domain includes the probabilities of actions such as raw material procurement, semi-finished product replenishment, and safety stock adjustment. The equipment task allocation domain includes the probabilities of actions such as equipment load distribution, maintenance plan adjustment, and capacity allocation optimization. The domain-specific decision probability distribution uses normalization to ensure that the sum of the probabilities within each decision domain is 1, maintain the probability independence between each decision domain, and maintain the information integrity of the original probability vector. The probability decomposition process solves the problem that a single decision model cannot effectively handle multi-domain decision conflicts. Each decision domain focuses on a specific manufacturing link to avoid confusion in decision logic.

[0095] Parallel training and update processing optimizes the policy network parameters for each of the three decision domains based on experience samples stored in the intelligent experience replay buffer. Parallel training involves performing gradient descent updates on multiple networks simultaneously without blocking each other. Experience samples in the intelligent experience replay buffer are labeled and categorized by decision domain type. Order scheduling experience includes data related to order status changes and scheduling decisions, inventory replenishment experience includes data related to inventory level changes and replenishment decisions, and equipment task allocation experience includes data related to equipment status changes and task allocation decisions. The three policy networks utilize the same LSTM-Actor-Critic architecture but with different parameter weights. Each network specifically processes experience samples from its corresponding decision domain, and network parameters are independently updated using the backpropagation algorithm. The domain-specific policy parameter set contains the weight matrix, bias vector, and activation function parameters for each decision domain network. The parameter update frequency is dynamically adjusted based on the number and importance of experience samples from each domain, with networks for decision domains with more experience being updated more frequently and networks for decision domains with less experience being updated less frequently.

[0096] Multi-agent collaborative decision-making and computation combines domain-specific decision probability distributions with domain-specific policy parameter sets to generate specific decision outputs. Multi-agent collaboration refers to the coordination of decision outcomes among multiple independent decision-making agents through information exchange. The order scheduling agent calculates order processing priorities based on the probability distribution of the order scheduling domain and the corresponding policy parameters, generating an order priority assignment sequence. Each element in the sequence represents the numerical value of the processing priority of a specific order, with higher values ​​indicating higher priority. The inventory replenishment agent calculates the replenishment requirements for various materials based on the probability distribution and policy parameters of the inventory replenishment domain, generating an inventory replenishment quantity vector. Each element in the vector represents the replenishment quantity for a specific material, which is related to the current inventory level and expected demand. The equipment task allocation agent calculates the matching relationship between equipment and tasks based on the probability distribution and policy parameters of the equipment task allocation domain, generating an equipment task scheduling matrix. The rows of the matrix represent the equipment number, the columns represent the task number, and the matrix elements represent the probability of the equipment executing the task or the execution time.

[0097] Global consistency constraint verification checks for logical and resource constraint conflicts between the three decision outputs. Global consistency constraints refer to unified constraints that must be satisfied by multiple decision outcomes. The consistency constraint between the order priority allocation sequence and the inventory replenishment quantity vector checks whether high-priority orders have sufficient material support. A constraint conflict is triggered when a key material required for a high-priority order is out of stock and the replenishment quantity is insufficient. The consistency constraint between the inventory replenishment quantity vector and the equipment task scheduling matrix checks whether the replenishment plan matches the equipment's available time. When the replenishment time conflicts with equipment maintenance, the replenishment plan or equipment maintenance schedule needs to be adjusted. The consistency constraint between the equipment task scheduling matrix and the order priority allocation sequence checks whether the equipment's capacity meets the order's delivery date requirements. When the delivery date requirements of high-priority orders exceed the equipment's capacity limits, the order priority or equipment task allocation needs to be readjusted. The collaborative optimization decision solution resolves constraint conflicts through an iterative optimization algorithm, employing the Lagrange multiplier method or genetic algorithm for multi-objective optimization.

[0098] The comprehensive evaluation calculation process quantitatively evaluates the current state and expected performance of the manufacturing system based on the collaborative optimization decision-making solution. Comprehensive evaluation refers to the numerical evaluation of the feasibility and superiority of the decision-making solution from multiple dimensions. The current production status evaluation includes an analysis of the current status of key performance indicators such as equipment utilization, order completion rate, inventory turnover rate, and quality pass rate, and judges the quality of the current status by comparing with historical benchmark data. The expected performance indicator evaluation predicts the changing trends of various performance indicators after the implementation of the collaborative optimization decision-making solution, including expected benefits such as the extent of cost reduction, efficiency improvement, and customer satisfaction improvement. The results of the intelligent analysis of manufacturing data are in the form of a weighted comprehensive score. The scores of multiple evaluation dimensions are weighted and summed according to the importance weights to generate a numerical result reflecting the comprehensive value of the decision-making solution.

[0099] The above describes the manufacturing data intelligent analysis method based on deep reinforcement learning in the embodiment of the present application. The following describes the manufacturing data intelligent analysis system based on deep reinforcement learning in the embodiment of the present application. Figure 2 In the embodiments of the present application, an embodiment of the manufacturing data intelligent analysis system based on deep reinforcement learning includes:

[0100] A processing module is used to perform time-series processing on equipment status data, order feature data, inventory level data, and quality indicator data in a manufacturing production environment to obtain a four-dimensional production data matrix;

[0101] A learning module is used to perform strategy learning processing on production time series features and order-inventory association features using an LSTM-Actor-Critic deep reinforcement learning algorithm based on the four-dimensional production data matrix to obtain a manufacturing decision strategy network;

[0102] a classification module for classifying the four-dimensional production data matrix into production cycle categories according to the preparation stage, production execution stage, quality inspection stage, and delivery stage, thereby obtaining a hierarchical data set for the production stages;

[0103] An update module is used to perform priority sampling and dynamic update processing on historical decision experience data based on the production stage layered data set to obtain an intelligent experience playback buffer zone that is aware of the production cycle;

[0104] The optimization module is used to collaboratively optimize order scheduling decisions, inventory replenishment decisions, and equipment task allocation decisions based on the manufacturing decision strategy network and the intelligent experience replay buffer, so as to obtain intelligent analysis results of manufacturing data.

[0105] above Figure 2 The manufacturing data intelligent analysis system based on deep reinforcement learning in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The manufacturing data intelligent analysis device based on deep reinforcement learning in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0106] Reference Figure 3 In an embodiment of the present invention, a manufacturing data intelligent analysis device based on deep reinforcement learning is also provided. The manufacturing data intelligent analysis device based on deep reinforcement learning can be a server, and its internal structure can be as follows: Figure 3 As shown. The manufacturing data intelligent analysis device based on deep reinforcement learning includes a processor, memory, display screen, input device, network interface and database connected through a system bus. Among them, the computer-designed processor is used to provide computing and control capabilities. The memory of the manufacturing data intelligent analysis device based on deep reinforcement learning includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the manufacturing data intelligent analysis device based on deep reinforcement learning is used to store the corresponding data in this embodiment. The network interface of the manufacturing data intelligent analysis device based on deep reinforcement learning is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0107] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the deep reinforcement learning-based manufacturing data intelligent analysis device to which the solution of the present invention is applied.

[0108] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the steps of the method for intelligent analysis of manufacturing data based on deep reinforcement learning.

[0109] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0110] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a manufacturing data intelligent analysis device based on deep reinforcement learning (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0111] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A manufacturing data intelligent analysis method based on deep reinforcement learning, characterized by: The method comprises: The equipment status data, order feature data, inventory level data and quality indicator data in the manufacturing production environment are processed in a time series manner to obtain a four-dimensional production data matrix; Based on the four-dimensional production data matrix, the LSTM-Actor-Critic deep reinforcement learning algorithm is used to perform policy learning processing on the production time series features and the order-inventory association features to obtain a manufacturing decision strategy network; Classify the four-dimensional production data matrix into production cycle categories according to the preparation stage, production execution stage, quality inspection stage, and delivery stage to obtain a production stage hierarchical data set; According to the production stage hierarchical data set, the historical decision experience data is sampled and dynamically updated with priority to obtain a production cycle-aware intelligent experience replay buffer, including: allocating storage capacity of the production stage hierarchical data set according to the time ratio of the preparation stage, the production execution stage, the quality inspection stage and the delivery stage to obtain a hierarchical experience storage structure; performing temporal difference error calculation based on the state transition sequence and the reward signal in the historical decision experience data to obtain the experience data priority weight; performing probability distribution redistribution processing on the experience samples in the hierarchical experience storage structure according to the experience data priority weight to obtain a priority sampling probability distribution; matching and analyzing the current production beat information with the historical production cycle pattern to obtain a sampling frequency adjustment coefficient; performing batch selection and buffer update processing on the experience samples based on the priority sampling probability distribution and the sampling frequency adjustment coefficient to obtain a production cycle-aware intelligent experience replay buffer; Based on the manufacturing decision strategy network and the intelligent experience replay buffer, order scheduling decisions, inventory replenishment decisions and equipment task allocation decisions are collaboratively optimized to obtain intelligent analysis results of manufacturing data.

2. The manufacturing data intelligent analysis method based on deep reinforcement learning according to claim 1 is characterized in that: The equipment status data, order feature data, inventory level data and quality index data in the manufacturing production environment are processed in a time series manner to obtain a four-dimensional production data matrix, including: The device status data is segmented into time series through a sliding window to obtain the device status time series submatrix; The discrete order types and process parameters in the order feature data are continuously encoded to obtain an order feature vector sequence; The inventory level data is divided into time windows according to the length of the production cycle to obtain the inventory level time series matrix; Based on the statistical distribution characteristics of the quality indicator data, standardization and normalization processing are performed to obtain a quality indicator standardized vector; The equipment status time series sub-matrix, order feature vector sequence, inventory level time series matrix and quality indicator normalized vector are dimensionally spliced ​​to obtain a four-dimensional production data matrix.

3. The manufacturing data intelligent analysis method based on deep reinforcement learning according to claim 1 is characterized in that: According to the four-dimensional production data matrix, the production time series features and order-inventory association features are processed by the LSTM-Actor-Critic deep reinforcement learning algorithm to obtain a manufacturing decision strategy network, including: Inputting the equipment state time series submatrix in the four-dimensional production data matrix into the first layer LSTM network to extract the equipment operation periodic pattern and obtain the equipment state feature vector; Performing feature fusion processing on the order feature vector sequence and the inventory level time series matrix in the four-dimensional production data matrix to obtain an order-inventory joint feature matrix; Inputting the order-inventory joint feature matrix into the second-layer LSTM network for dynamic association relationship learning processing to obtain a business layer time series feature vector; Based on the device state feature vector and the business layer time series feature vector, an Actor network is constructed to perform strategy probability distribution calculation and processing to obtain a manufacturing decision action probability output; A critic network is constructed based on the four-dimensional production data matrix to perform state value function evaluation processing, obtain a state value evaluation result, and combine it with the manufacturing decision action probability output to form a manufacturing decision strategy network.

4. The manufacturing data intelligent analysis method based on deep reinforcement learning according to claim 3 is characterized in that: The constructing of a Critic network based on the four-dimensional production data matrix to perform state value function evaluation processing, obtaining a state value evaluation result, and combining it with the manufacturing decision action probability output to form a manufacturing decision strategy network, including: Reconstructing the equipment state time sequence submatrix in the four-dimensional production data matrix into a two-dimensional equipment state matrix and performing convolutional neural network processing to obtain an equipment spatial feature map; Performing a pooling layer dimensionality reduction process on the device space feature map to obtain a device state space feature vector; Inputting the service layer time series feature vector into the first fully connected layer for nonlinear transformation processing to obtain an intermediate feature representation vector; Perform feature splicing processing based on the device state space feature vector and the intermediate feature representation vector to obtain a state comprehensive feature vector; The state comprehensive feature vector is input into the second fully connected layer for state value regression calculation processing to obtain a state value evaluation result, and is combined with the manufacturing decision action probability output through a parameter sharing mechanism to form a manufacturing decision strategy network.

5. The manufacturing data intelligent analysis method based on deep reinforcement learning according to claim 1 is characterized in that: The four-dimensional production data matrix is ​​classified into production cycle according to the preparation stage, production execution stage, quality inspection stage and delivery stage to obtain a production stage hierarchical data set, including: The time period is extracted based on the order receiving time and production start time in the order feature vector sequence to obtain the time boundary identifier of the preparation stage; Based on the equipment start signal and stop signal in the equipment status timing sub-matrix, the production time period is identified and processed to obtain the time boundary identification of the production execution stage; Perform time interval processing on the inspection start timestamp and inspection completion timestamp in the quality indicator standardized vector to obtain the time boundary identifier of the quality inspection stage; The time range of the outbound record time and the order completion time in the inventory level time series matrix is ​​determined to obtain the time boundary identifier of the delivery stage; The four-dimensional production data matrix is ​​subjected to stage data slicing processing according to the preparation stage time boundary identifier, the production execution stage time boundary identifier, the quality inspection stage time boundary identifier and the delivery stage time boundary identifier to obtain a production stage hierarchical data set.

6. The manufacturing data intelligent analysis method based on deep reinforcement learning according to claim 1 is characterized in that: Based on the manufacturing decision strategy network and the intelligent experience playback buffer, the order scheduling decision, inventory replenishment decision and equipment task allocation decision are collaboratively optimized to obtain the intelligent analysis results of the manufacturing data, including: Decomposing the manufacturing decision action probability output in the manufacturing decision strategy network into three decision domains: order scheduling, inventory replenishment, and equipment task allocation, to obtain domain-specific decision probability distributions; Performing parallel training and updating processing on the policy networks of the three decision domains based on the experience samples in the intelligent experience replay buffer to obtain a set of domain-specific policy parameters; Perform multi-agent collaborative decision-making calculations based on the domain-specific decision probability distribution and domain-specific strategy parameter set to obtain an order priority allocation sequence, an inventory replenishment quantity vector, and an equipment task scheduling matrix; Performing global consistency constraint verification on the order priority allocation sequence, inventory replenishment quantity vector, and equipment task scheduling matrix to obtain a collaborative optimization decision-making solution; Based on the collaborative optimization decision-making scheme, the current production status and expected performance indicators are comprehensively evaluated and calculated to obtain intelligent analysis results of manufacturing data.

7. A manufacturing data intelligent analysis system based on deep reinforcement learning, characterized by: Used to implement the manufacturing data intelligent analysis method based on deep reinforcement learning according to any one of claims 1 to 6, the manufacturing data intelligent analysis system based on deep reinforcement learning includes: A processing module is used to perform time-series processing on equipment status data, order feature data, inventory level data, and quality indicator data in a manufacturing production environment to obtain a four-dimensional production data matrix; A learning module is used to perform strategy learning processing on production time series features and order-inventory association features using an LSTM-Actor-Critic deep reinforcement learning algorithm based on the four-dimensional production data matrix to obtain a manufacturing decision strategy network; a classification module for classifying the four-dimensional production data matrix into production cycle categories according to the preparation stage, production execution stage, quality inspection stage, and delivery stage, thereby obtaining a hierarchical data set for the production stages; An update module is used to perform priority sampling and dynamic update processing on the historical decision experience data according to the production stage hierarchical data set to obtain a production cycle-aware intelligent experience replay buffer, including: allocating storage capacity of the production stage hierarchical data set according to the time ratio of the preparation stage, production execution stage, quality inspection stage and delivery stage to obtain a hierarchical experience storage structure; performing time series difference error calculation processing based on the state transition sequence and reward signal in the historical decision experience data to obtain the experience data priority weight; performing probability distribution redistribution processing on the experience samples in the hierarchical experience storage structure according to the experience data priority weight to obtain a priority sampling probability distribution; matching and analyzing the current production beat information with the historical production cycle pattern to obtain a sampling frequency adjustment coefficient; performing batch selection and buffer update processing on the experience samples based on the priority sampling probability distribution and the sampling frequency adjustment coefficient to obtain a production cycle-aware intelligent experience replay buffer; The optimization module is used to collaboratively optimize order scheduling decisions, inventory replenishment decisions, and equipment task allocation decisions based on the manufacturing decision strategy network and the intelligent experience replay buffer, so as to obtain intelligent analysis results of manufacturing data.

8. A manufacturing data intelligent analysis device based on deep reinforcement learning, characterized in that: It includes a memory and a processor, the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, it implements the manufacturing data intelligent analysis method based on deep reinforcement learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the processor executes the manufacturing data intelligent analysis method based on deep reinforcement learning as described in any one of claims 1 to 6.

Images