Manufacturing industry ERP dynamic resource optimization configuration system based on reinforcement learning

Abstract

The invention discloses a manufacturing industry ERP dynamic resource optimization configuration system based on reinforcement learning, and relates to the technical field of manufacturing industry ERP resource configuration, and the system comprises an ERP state data perception module which is in butt joint with a manufacturing industry ERP system, collects multi-source heterogeneous data in real time, fuses the multi-source heterogeneous data in real time by using a Kalman filtering technology, detects an abnormal value, and sends the abnormal value to the ERP state data perception module; an improved Robust Scaling method is adopted for standardization processing, and initial data of a unified structure are output and transmitted to the feature extraction and integration processing representation module; multi-source heterogeneous data are collected in real time through the ERP state data sensing module, fusion is carried out through the Kalman filtering technology, dynamic changes in the production environment can be captured in time, strategies can be continuously adjusted according to the real-time dynamic change data through the reinforcement learning method, and the real-time dynamic change data can be obtained. And outputting an optimal resource allocation action adapting to a dynamic environment, so as to greatly improve the adaptability of the system to a complex production environment.

Description

Technical Field

[0001] The present invention relates to the technical field of manufacturing ERP resource configuration, and in particular to a manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning. Background Art

[0002] Enterprise Resource Planning (ERP) systems in the manufacturing industry are crucial for rationally allocating resources, improving production efficiency, and reducing costs. Traditional ERP systems have long relied on linear programming (LP) or genetic algorithms (GA) for resource allocation.

[0003] For example, the traditional linear programming (LP) method is based on static data settings with fixed order demand and constant equipment efficiency. It solves the optimal resource allocation plan by establishing a linear objective function and linear constraints. However, this method can only handle problems with linear relationships. For complex nonlinear production processes, dynamically changing order demand, inventory levels, and other situations in the manufacturing industry, it is difficult to quickly adjust the resource allocation plan and its adaptability is poor. The genetic algorithm (GA) simulates the biological evolution process and continuously iterates through operations such as selection, crossover, and mutation to find the optimal solution. However, the GA algorithm converges slowly. When it needs to simultaneously optimize conflicting objectives such as inventory cost, delivery on-time rate, and equipment utilization, it is easy to fall into a local optimal solution. In addition, when dealing with large-scale resource allocation problems, the computational complexity is high and the operation time is long.

[0004] The existing technology has the following shortcomings: with the development of the manufacturing industry, the production and manufacturing environment has become more and more complex, order demand has shown dynamic changes, and inventory levels also need to be adjusted in real time. At the same time, the discrete variables and continuous variables displayed in the diversity of manufacturing equipment status, as well as the different skill levels and sudden variables of enterprise skilled personnel have also increased the difficulty of resource allocation. When resources are allocated irrationally, it will not only affect resource utilization efficiency and reduce production capacity, but also increase inventory, manpower and equipment costs, and may lead to delayed delivery of orders and low order fulfillment rates. It cannot better adapt to the complex production environment of the manufacturing industry. Therefore, the existing technology has been difficult to meet the needs of enterprises for optimizing the allocation of efficient and flexible resources.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] The purpose of the present invention is to provide a manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning. By adopting reinforcement learning instead of traditional LP or GA methods, the present invention can continuously adjust the strategy according to the real-time dynamic changing data in the production environment, and output the optimal resource allocation action that adapts to the dynamic environment, so as to solve the problems in the above-mentioned background technology.

[0007] To achieve the above objectives, the present invention provides the following technical solutions: a manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning, comprising an ERP status data perception module: connecting to the manufacturing ERP system, collecting multi-source heterogeneous data in real time, using Kalman filtering technology to fuse the multi-source heterogeneous data in real time, detecting outliers, and using an improved Robust Scaling method for standardization processing, outputting initial data with a unified structure, and transmitting it to a feature extraction, integration, processing and representation module;

[0008] Feature extraction, integration, processing and representation module: Receives initial data and performs cleaning, filtering, and feature extraction. It then integrates and processes heterogeneous data using a multi-head attention mechanism to generate a state vector that can be processed by the reinforcement learning decision module.

[0009] Reinforcement learning decision module: Receives the state vector and inputs the state space, defines the action space, establishes a hybrid action space hierarchical reinforcement learning model, designs a multi-objective reward function, outputs the optimal resource allocation action of the upper and lower layer strategies, and transmits the optimal resource allocation action to the digital twin simulation verification module;

[0010] Digital twin simulation verification module: The built-in discrete event simulation (DES) engine combines the physical Adams dynamics model to build a digital twin model corresponding to the actual resource configuration. It simulates and verifies the optimal resource allocation action in a virtual environment and outputs simulation results, including feasibility scores and benefit prediction indicators.

[0011] Resource allocation execution module: This module executes the optimal resource allocation actions verified by the digital twin simulation verification module in the actual manufacturing production environment, and sets a fault-tolerant mechanism to trigger the reinforcement learning decision module to roll back the policy.

[0012] Reward evaluation and feedback learning module: After the resource allocation execution module executes the optimal resource allocation action, it monitors multiple indicators in the production process in real time, calculates the reward value of each indicator based on the multi-objective reward function pre-set in the reinforcement learning decision module, adopts the learning mechanism of priority experience replay PER and proximal policy optimization PPO, and feeds back the reward value and new state information to the reinforcement learning decision module for further learning and strategy update.

[0013] Optionally, the steps for acquiring the multi-source heterogeneous data are as follows:

[0014] Deploy high-precision sensors on manufacturing equipment and transmit sensor data in real time via industrial Ethernet to obtain equipment status information, marked as Esi;

[0015] Use the API interface to connect to the manufacturing ERP system to obtain basic employee information, marked as Psi. At the same time, extract inventory data and order requirements, marked as Ivd and Orm respectively;

[0016] In each production link of the manufacturing ERP system, obtain the production progress information of product manufacturing and mark it as PPI;

[0017] Multi-source heterogeneous data includes equipment status information, personnel information, inventory data, order requirements and production progress information. The expression of multi-source heterogeneous data is MHD = {Esi, Psi, Ivd, Orm, Ppi…, where MHD represents multi-source heterogeneous data.

[0018] Optionally, the steps of processing multi-source heterogeneous data using the Kalman filter technology are as follows:

[0019] A unified state space model is established for multi-source heterogeneous data MHD to generate state vectors and observation vectors that can be processed by Kalman filtering technology. The expression of the state vector is: Where, It is represented as the state vector of multi-source heterogeneous data MHD at time t, It represents the state parameter of the i-th multi-source heterogeneous data MHD, n represents the dimension of the state parameter of the multi-source heterogeneous data MHD, and T represents the transpose;

[0020] The expression of the observation vector is and Where, It is represented as the observation value of the i-th MHD data source at time t, It is represented as the observation matrix of the i-th MHD data source at time t, is the observation noise of the i-th MHD data source at time t, N is the total number of MHD data sources, and i=1,2,…,N, It is represented as the observation noise covariance matrix of the i-th MHD data source;

[0021] The Kalman filter technology is used to detect outliers, including the prediction stage, residual calculation and outlier judgment. The calculation formula of the prediction stage is: And P t =F t ·P t-1 ·F t T +Q t , where It is expressed as the predicted value of the MHD state of multi-source heterogeneous data at time t based on time t-1, It is expressed as the predicted value of the MHD state of multi-source heterogeneous data at time t-1, Ft It is represented as the state transition matrix of multi-source heterogeneous data MHD at time t, P t It is expressed as the prediction state error covariance matrix of multi-source heterogeneous data MHD at time t, P t-1 It is expressed as the prediction state error covariance matrix of multi-source heterogeneous data MHD at time t-1, Q t It is represented as the process noise covariance matrix of multi-source heterogeneous data MHD transmission at time t;

[0022] The formula for residual calculation is and Where, Expressed as the residual of the i-th MHD data source at time t, It is represented as the residual covariance matrix of the i-th MHD data source at time t;

[0023] The calculation formula for outlier judgment is: and Where, Expressed as the Mahalanobis distance of the residual, is the threshold of the chi-square distribution, m is the residual dimension, α is the significance level, and α = 0.95;

[0024] The detected multi-source heterogeneous data are fused in real time with information matrix and updated in status to obtain fused data. The expression of real-time fusion of information matrix is: and Where, Expressed as the inverse matrix of the updated forecast state error covariance matrix, It is expressed as the weighted fusion of multi-source observation information, K t Expressed as the Kalman gain matrix;

[0025] The expression for state update is Where, It is expressed as the state update value obtained by fusing multi-source observation data and correcting the prediction value to obtain the optimal state estimate. The expression of the fused data is x MHD '.

[0026] Optionally, the standardization processing steps of the improved robust scaling method are as follows:

[0027] According to the fusion data x MHD', use the improved Robust Scaling method to output the initial data of unified structure, and reduce the influence of data outliers through dynamic weight adjustment. The expression of the improved Robust Scaling method is and Where x MHD " indicates that the improved Robust Scaling method standardizes the fused data x MHD 'The initial data, Represented as the sample value of the j-th multi-source heterogeneous data MHD, μ ω Represented as fused data x MHD 'The weighted mean, σ ω Represented as fused data x MHD ' is the standard deviation of the weighted mean, ε is a small constant to prevent division by zero, and ε = e -8 , j, J represent the jth sample data and the total number of samples respectively, ω j It is expressed as the weight of the j-th sample, β is expressed as the weight decay coefficient, and ecp is expressed as the exponential decay function. It is expressed as the median of the sample values ​​of the j-th multi-source heterogeneous data MHD.

[0028] Feature extraction, integration and processing representation module: Receives initial data and performs cleaning, screening and feature extraction, combines the integration and processing of heterogeneous data with the multi-head attention mechanism, and generates a state vector that can be processed by the reinforcement learning decision module.

[0029] Optionally, the feature extraction steps are as follows:

[0030] For the initial data x MHD "Cleaning, screening and feature extraction are performed by using the improved Hampel identifier to remove outliers, using dynamic time warping DTW to fill missing values, and MHD " Extract the static features of personnel information Psi and inventory data Ivd, then the expression of static features is X static ={Psi”,Ivd”}, and the dimension of the static feature is X static ∈R dc , where X static Expressed as starting from the initial data x MHD ”, Psi” represents the feature data extracted from personnel information Psi after being processed by the unified structure of ERP status data perception module, Psi” represents the feature data extracted from inventory data Ivd after being processed by the unified structure of ERP status data perception module, R dc Represented as static feature X static Dimensions;

[0031] Use the long short-term memory autoencoder LSTM-AE to process the time series data of the device status information Esi, calibrated as Esi t , and Esi t ∈R d , for time series data Esi t Perform dynamic feature extraction and output time series features. The encoder in the LSTM-AE network performs dimensionality reduction processing;

[0032] The decoder in the LSTM-AE network performs reconstruction processing to reconstruct the temporal characteristics of the output device state information Esi. The expression of the temporal characteristics of the device state information Esi is X seq ={Esi”}, and X seq ∈R T×ds , where X seq Represented as the time series data Esi of LSTM-AE processing device status information Esi t The output time series characteristics, Esi" represents the time series data Esi of the device status information Esi t After dynamic extraction of LSTM-AE, R T×ds Represented as time series feature X seq Dimensions;

[0033] Use graph convolutional network GCN to build a process route relationship model for order demand Orm and production progress information Ppi to extract graph structure features and output graph features X graph , and X graph ∈R M×dg , where X graph It is represented as the graph features extracted by GCN for order demand Orm and production progress information Ppi, R N×dg Represented as graph feature X graph Dimensions;

[0034] From the initial data x MHD The extracted heterogeneous features are grouped and input into the multi-head attention mechanism for feature normalization and integration.

[0035] Optionally, the integration processing steps of the multi-head attention mechanism are as follows:

[0036] From the initial data x MHD The extracted static feature X static , time series feature X seq and graph feature X graph Perform alignment and projection, where the calculation formula for alignment and projection is: Where, Represented as time series feature X seqAlignment, Represented as graph feature X graph Alignment, MaxPool is expressed as a time pooling dimensionality reduction function, Expressed as node-level attention aggregation calculation, Represented as static feature X static Extension, R ds+dg Represented as a replication dimension;

[0037] Multi-head attention is calculated by splicing heterogeneous features to generate an input splicing feature vector, where the expression of the input splicing feature vector is And R d =R ds+dg+dc , where X concat Represented as the concatenated feature vector, R d Represented as a unified dimension;

[0038] According to the input concatenated feature vector, layer normalization is used to perform feature normalization processing, and the state vector that conforms to the dimension of the reinforcement learning state space is output. The state vector expression after feature normalization is and Where, LN(X concat ) represents the output value of the layer normalization, γ represents the learnable scaling parameter, Represented as the concatenated feature vector X concat The mean of Represented as the concatenated feature vector X concat The standard deviation of , δ represents the learnable translation parameter, s(X) t It is represented as the state vector after feature normalization, MultiHead(X concat ) is represented as the concatenated feature vector X concat The multi-head output splicing value, is represented as the dimension of the reinforcement learning state space, and d RL =256.

[0039] Optionally, the steps for establishing the hybrid action space hierarchical reinforcement learning model are as follows:

[0040] Design a hierarchical architecture for a hybrid action space hierarchical reinforcement learning model, including upper and lower policies. The upper and lower policies are refined to define their time granularity, action space, and state inputs. The upper policy has hourly time granularity, the action space is discrete, and the state input is global information.

[0041] The time granularity of the lower-level strategy is minute-level update, the action space is continuous control, and the state input is local features;

[0042] Define the joint action space of the upper-layer strategy and the lower-layer strategy, and use a dual-branch network architecture to establish the upper / lower-layer strategy network. The upper-layer strategy network uses the PPO algorithm to output the calculation formula for the probability distribution of discrete decision actions: Where, π upper It is represented as the upper policy network using the PPO algorithm to output the probability distribution of discrete decision actions, and Softmax is represented as the probability distribution function that maps the output to discrete actions. upper Represented as the weight matrix of the upper policy network, Represented as a state encoder with 3 layers of fully connected network MLP to extract global information, a upper Expressed as a discrete decision action, s upper It represents the global information of the upper state input, and T represents the transpose;

[0043] The lower layer strategy network uses the DDPG algorithm to output the calculation formula of the probability distribution of continuous control actions: Where, π lower It is represented as the lower layer policy network using the PPO algorithm to output the probability distribution of continuous control actions, tanh is represented as the activation function, ω lower Represented as the weight matrix of the lower policy network, Represented as a state encoder with 3 layers of fully connected network MLP to extract global information, a lower Expressed as a continuous control action, s lower Represents the local features of the lower-level state input;

[0044] The fully connected network MLP is used to evaluate the value of the state-action pair of the lower-level policy network. The expression of the MLP evaluation value is: Where Q φ It represents the value of the state-action pair evaluated by the lower policy network, and MLP represents the Q-value estimate of the MLP output. Represented as a vector concatenation operation, it is used to concatenate the local features of the lower-level state input with the continuous control action and then input them into the MLP;

[0045] Design a multi-objective reward function for inventory cost, on-time delivery rate, and equipment utilization rate, and use a hierarchical weighting method to balance conflicting objectives;

[0046] The steps for outputting the optimal resource allocation actions of the upper and lower layer strategies are as follows:

[0047] Combined with the upper-layer strategy network, the PPO algorithm is used to output the probability distribution of discrete decision actions. The optimal resource allocation action selection calculation formula of the upper-layer strategy is a upper =Gumbel-Softamax(π upper (a upper |s upper)), where Gumbel-Softmax is expressed as the back-propagation activation function of the probability distribution of discrete variables;

[0048] According to the lower-layer strategy network, the DDPG algorithm is used to output the probability distribution of continuous control actions, and exploration noise and action clipping are injected to obtain the optimal resource allocation action of the lower-layer strategy. The calculation formula for selecting the optimal resource allocation action of the layer strategy is a lower =π lower (a lower |s lower )+N(0,σ t 2 ), and a lower '=clip(a lower ,0,1), where N(0,σ t 2 ) is represented by Gaussian noise with a mean of 0 and a variance of σ t 2 , a lower ' represents the optimal resource allocation action of the final lower-level strategy, and a lower '∈[0,1], clip represents the limit of continuous control action a lower Cutting process;

[0049] The optimal resource allocation action output by the upper-level strategy is passed to the lower-level strategy and used as an additional state input of the lower-level strategy, so that the discrete decision-making action of the upper-level decision and the continuous control action of the lower-level decision are synergistically associated.

[0050] Optionally, the reward values ​​of the multiple indicators are calculated as follows:

[0051] Multiple indicators include inventory cost, delivery on-time rate, and equipment utilization rate. The formula for calculating inventory cost is R ic =-κ·m ic , where R ic is expressed as inventory cost, κ is the penalty coefficient for controlling inventory overstock, m ic Expressed as the total inventory cost;

[0052] The formula for calculating on-time delivery rate is: Where R odr It is expressed as the on-time delivery rate, no' is the total number of orders, and exp is the exponential function with the natural constant e as the base. It represents the actual completion time of the no. order. It represents the agreed delivery time of the no. order, t τ It is expressed as a time tolerance parameter that controls the rate at which the reward decays;

[0053] The calculation formula for equipment utilization is: And when η k >η min When ψ(1;0)=1, when η k ≤η min When ψ(1;0)=0, where R eu It is represented by the equipment utilization rate, K is the total number of equipment, η k is the utilization rate of the kth device, ψ(1;0) is the indicator function, η min It is expressed as the minimum utilization threshold of the device;

[0054] The expression of the multi-objective reward function is R total =ω ic ·R ic +ω odr ·R odr +ω eu ·R eu , and ω ic +ω odr +ω eu =1, where R total Expressed as the output value of the multi-objective reward function of inventory cost, delivery on-time rate, and equipment utilization rate, ω ic 、ω odr 、ω eu Respectively expressed as the corresponding inventory cost R ic , delivery on-time rate R odr , equipment utilization rate R eu The weight coefficient of .

[0055] A computer device comprises: a memory and a processor; the memory stores a computer program, and the processor implements the steps of the above-mentioned manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning when executing the computer program.

[0056] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning.

[0057] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0058] The present invention uses the ERP status data perception module to collect multi-source heterogeneous data in real time and uses Kalman filtering technology to fuse them. This allows the system to promptly capture dynamic changes in the production environment. By adopting reinforcement learning methods, the system can continuously adjust strategies based on real-time dynamic change data and output optimal resource allocation actions that adapt to the dynamic environment, thereby greatly improving the system's adaptability to complex production environments.

[0059] Establishing a hybrid action space hierarchical reinforcement learning model and setting a multi-objective reward function. By constructing a two-layer policy network, it can handle nonlinear, discrete and continuous mixed variables, further better adapt to complex production environments, and can quickly adjust resource allocation according to real-time status, optimize multi-objective conflicts, improve resource utilization efficiency, and enhance decision reliability and accuracy, meeting the efficient and flexible resource allocation needs of manufacturing enterprises.

[0060] Using the discrete event simulation DES engine, combined with the physical Adams dynamics model, the optimal resource allocation action is simulated and verified in a virtual environment, and the feasibility score and benefit prediction indicators are output. Combined with the fault-tolerant mechanism, the rollback strategy of the reinforcement learning decision module is triggered, which can reduce the losses caused by decision-making errors and optimize resource allocation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction to the drawings required for use in the embodiments will be given below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0062] Figure 1 This is a module block diagram of the manufacturing ERP dynamic resource optimization configuration system of the present invention. DETAILED DESCRIPTION

[0063] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0064] The present invention provides Figure 1 The manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning shown in the figure includes an ERP status data perception module: it connects to the manufacturing ERP system, collects multi-source heterogeneous data in real time, including equipment status, personnel information, inventory data, order requirements and production progress information, uses Kalman filtering technology to fuse the collected multi-source heterogeneous data in real time, detects outliers, and uses an improved robust scaling method to perform standardization processing, outputs initial data with a unified structure, and transmits it to the feature extraction, integration, processing and representation module;

[0065] Specifically, the steps for obtaining multi-source heterogeneous data are as follows:

[0066] Deploy high-precision sensors on manufacturing equipment and transmit sensor data in real time via industrial Ethernet to obtain equipment status information, marked as Esi. The equipment status information Esi includes operating parameters such as equipment temperature, pressure, vibration frequency, and status information such as normal operation, abnormal operation, shutdown, maintenance, and standby.

[0067] The API interface is used to connect to the manufacturing ERP system to obtain basic employee information, marked as Psi. Personnel information Psi includes name, work number, skill level, work attendance, and task assignment status. At the same time, inventory data and order requirements are extracted and marked as Ivd and Orm respectively. Among them, inventory data Ivd includes the inventory quantity, storage location, and in and out records of raw materials, semi-finished products, and finished products. Order requirements Orm include order number, customer information, product specifications, order quantity, and delivery date.

[0068] In each production link of the manufacturing ERP system, the production progress information of the completed quantity, unfinished quantity and process completion time of product manufacturing is obtained through data statistics and calibrated as PPI;

[0069] Multi-source heterogeneous data includes equipment status information, personnel information, inventory data, order requirements and production progress information. The expression of multi-source heterogeneous data is MHD = {Esi, Psi, Ivd, Orm, Ppi}, where MHD represents multi-source heterogeneous data.

[0070] Specifically, the steps of Kalman filtering technology to process multi-source heterogeneous data are as follows:

[0071] A unified state space model is established for multi-source heterogeneous data MHD to generate state vectors and observation vectors that can be processed by Kalman filtering technology. The expression of the state vector is: Where, It is represented as the state vector of multi-source heterogeneous data MHD at time t, It represents the state parameter of the i-th multi-source heterogeneous data MHD, N represents the dimension of the state parameter of the multi-source heterogeneous data MHD, and T represents the transpose;

[0072] The expression of the observation vector is and Where, It is represented as the observation value of the i-th MHD data source at time t, It is represented as the observation matrix of the i-th MHD data source at time t, is the observation noise of the i-th MHD data source at time t, N is the total number of MHD data sources, and i=1,2,…,N, It is represented as the observation noise covariance matrix of the i-th MHD data source;

[0073] The Kalman filter technology is used to detect outliers, including the prediction stage, residual calculation and outlier judgment. The calculation formula of the prediction stage is: And P t =F t ·P t-1 ·F t T +Q t , where It is expressed as the predicted value of the MHD state of multi-source heterogeneous data at time t based on time t-1, It is expressed as the predicted value of the MHD state of multi-source heterogeneous data at time t-1, F t It is represented as the state transition matrix of multi-source heterogeneous data MHD at time t, P t It is expressed as the prediction state error covariance matrix of multi-source heterogeneous data MHD at time t, P t-1 It is expressed as the prediction state error covariance matrix of multi-source heterogeneous data MHD at time t-1, Q t It is represented as the process noise covariance matrix of multi-source heterogeneous data MHD transmission at time t;

[0074] The formula for residual calculation is and Where, Expressed as the residual of the i-th MHD data source at time t, It is represented as the residual covariance matrix of the i-th MHD data source at time t;

[0075] The calculation formula for outlier judgment is: and Where, Expressed as the Mahalanobis distance of the residual, is the threshold of the chi-square distribution, m is the residual dimension, α is the significance level, and α = 0.95;

[0076] The detected multi-source heterogeneous data are fused in real time with information matrix and updated in status to obtain fused data. The expression of real-time fusion of information matrix is: and Where, Expressed as the inverse matrix of the updated forecast state error covariance matrix, It is expressed as the weighted fusion of multi-source observation information, K t Expressed as the Kalman gain matrix;

[0077] The expression for state update is Where, It is expressed as the state update value obtained by fusing multi-source observation data and correcting the prediction value to obtain the optimal state estimate. The expression of the fused data is X MHD '.

[0078] Specifically, the standardization processing steps of the improved Robust Scaling method are as follows:

[0079] According to the fusion data X MHD ', using the improved Robust Scaling method to output the initial data of a unified structure, reducing the influence of data outliers through dynamic weight adjustment to automatically optimize the sensitivity to outliers, which can improve multimodal adaptability. The expression of the improved Robust Scaling method is and Where x MHD " indicates that the improved RobustScaling method standardizes the fused data x MHD 'The initial data, Represented as the sample value of the j-th multi-source heterogeneous data MHD, μ ω Represented as fused data x MHD 'The weighted mean, σ ω Represented as fused data x MHD ' is the standard deviation of the weighted mean, ε is a small constant to prevent division by zero, and ε = e -8 , j, J represent the jth sample data and the total number of samples respectively, ω j It is expressed as the weight of the j-th sample, β is expressed as the weight decay coefficient, and ecp is expressed as the exponential decay function. It is expressed as the median of the sample values ​​of the j-th multi-source heterogeneous data MHD.

[0080] Feature extraction, integration, processing and representation module: Receives initial data and performs cleaning, screening and feature extraction, including using long short-term memory autoencoders (LSTM-AE) to process time series data of equipment status, using graph convolutional networks (GCN) to establish a process route relationship model for order requirements, and combining the use of a multi-head attention mechanism to integrate and process heterogeneous data to generate a state vector that can be processed by the reinforcement learning decision module.

[0081] Specifically, the steps of feature extraction are as follows:

[0082] For the initial data x MHD"Perform cleaning, screening and feature extraction, remove outliers by using the improved Hampel identifier, fill missing values ​​with dynamic time warping DTW, remove noise data during the cleaning and screening process, retain valid features, and extract features from the initial data x MHD " Extract the static features of personnel information Psi and inventory data Ivd, then the expression of static features is X static ={Psi”,Ivd”}, and the dimension of the static feature is X static ∈R dc , where X static Expressed as starting from the initial data x MHD ”, Psi” represents the feature data extracted from personnel information Psi after being processed by the unified structure of ERP status data perception module, Psi” represents the feature data extracted from inventory data Ivd after being processed by the unified structure of ERP status data perception module, R dc Represented as static feature X static Dimensions;

[0083] Use the long short-term memory autoencoder LSTM-AE to process the time series data of the device status information Esi, calibrated as Esi t , and Esi t ∈R d , for time series data Esi t Perform dynamic feature extraction and output time series features. The encoder in the LSTM-AE network performs dimensionality reduction processing, including using LSTM units to reduce the time series data Esi t Through the Gate of Oblivion Input Gate Candidate cell states Cell status update Output Gate and hidden state output Processing, where Esi t Represents the input device status information at time t, Represents the output value of the forget gate, σ represents the Sigmoid activation function, Represented as the hidden state at time t-1, ω f Represented as the weight matrix of the forget gate, b f Represented as the bias term of the forget gate, Represented as the output value of the input gate, ω i Represented as the weight matrix of the input gate, b i is represented as the bias term of the input gate, Represents the output value of the candidate cell state, tanh represents the tanh activation function, ω CRepresented as the weight matrix of cell state update, b C Represented as the bias term for cell state update, Represents the output value of the cell state update at time t, ⊙ represents element-by-element multiplication, Represents the output value of the cell state update at time t-1, Represented as the output value of the output gate, ω o Represented as the weight matrix of the output gate, b o is represented as the bias term of the output gate, Represented as the hidden state at time t;

[0084] The decoder in the LSTM-AE network performs reconstruction processing, using reverse LSTM and fully connected layers, by defining the loss function Where, L AE It is represented as loss function, t' is represented as time window, Represented as LSTM-AE pair Esi t The reconstructed output, λ is represented by the L2 regularization coefficient, ω' is represented by the set of all weight parameters of the LSTM-AE network, || || 2 Expressed as L2 norm, reconstructing the temporal characteristics of the output device state information Esi, the expression of the temporal characteristics of the device state information Esi is X seq ={Esi”}, and X seq ∈R T×ds , where X seq Represented as the time series data Esi of LSTM-AE processing device status information Esi t The output time series characteristics, Esi" represents the time series data Esi of the device status information Esi t After dynamic extraction of LSTM-AE, R T×ds Represented as time series feature X seq Dimensions;

[0085] Use graph convolutional network GCN to build a process route relationship model for order demand Orm and production progress information Ppi to extract graph structure features and output graph features X graph , and X graph ∈R M×dg , where X graph It is represented as the graph features extracted by GCN for order demand Orm and production progress information Ppi, R N×dg Represented as graph feature X graph The GCN network constructs the nodes, edges and node features of the graph structure, using the graph convolution formula and Where Ppi (l+1)It is represented as the node feature of the production progress information Ppi of the l+1th layer, l represents the layer, It is represented as the degree matrix of node connection strength, ReLU is represented as the ReLU activation function, Expressed as the adjacency matrix with self-loops added, ω (l) Represented as the weight matrix of the lth layer, I Orm Expressed as the adjacency matrix of order demand Orm, Ppi (l) It is represented as the node feature of the production progress information Ppi of the lth layer;

[0086] From the initial data x MHD The extracted heterogeneous features are grouped and input into the multi-head attention mechanism for feature normalization and integration.

[0087] Specifically, the integration processing steps of the multi-head attention mechanism are as follows:

[0088] From the initial data x MHD The extracted static feature X static , time series feature X seq and graph feature X graph Perform alignment and projection, where the calculation formula for alignment and projection is: Where, Represented as time series feature X seq Alignment, Represented as graph feature X graph Alignment, MaxPool is expressed as a time pooling dimensionality reduction function, Expressed as node-level attention aggregation calculation, Represented as static feature X static Extension, R ds+dg Represented as a replication dimension;

[0089] Multi-head attention is calculated by splicing heterogeneous features to generate an input splicing feature vector, where the expression of the input splicing feature vector is And R d =R ds+dg+dc , where X concat Represented as the concatenated feature vector, R d Represented as a unified dimension, it is used to align and fuse heterogeneous feature data;

[0090] According to the input concatenated feature vector, layer normalization is used to perform feature normalization processing, and the state vector that conforms to the dimension of the reinforcement learning state space is output. The state vector expression after feature normalization is and Where, LN(X concat) represents the output value of the layer normalization, γ represents the learnable scaling parameter, μ Xconcat Represented as the concatenated feature vector X concat The mean of Xconcat Represented as the concatenated feature vector X concat The standard deviation of , δ represents the learnable translation parameter, s(X) t It is represented as the state vector after feature normalization, MultiHead(X concat ) is represented as the concatenated feature vector X concat The multi-head output splicing value, is represented as the dimension of the reinforcement learning state space, and d RL =256.

[0091] Reinforcement Learning Decision Module: Receives the state vector and inputs the state space, defines the action space, establishes a hybrid action space hierarchical reinforcement learning model, designs a multi-objective reward function for inventory cost, on-time delivery rate, and equipment utilization rate, outputs the optimal resource allocation actions for the upper and lower layer strategies, and continuously learns and optimizes strategies to achieve efficient resource allocation. The optimal resource allocation actions are then transmitted to the digital twin simulation verification module.

[0092] Specifically, the steps for establishing the hybrid action space hierarchical reinforcement learning model are as follows:

[0093] Design a hierarchical architecture for a hybrid action space hierarchical reinforcement learning model, including upper-level and lower-level strategies. This model refines the temporal granularity, action space, and state inputs of the upper and lower-level strategies. The upper-level strategy has hourly updates, and the action space consists of discrete decisions related to capacity allocation mode selection, supplier scheduling strategy, order quantity planning schedule, inventory strategy adjustment, and personnel shift management. The state inputs are global information, including total order demand, overall production schedule, total inventory, total number of personnel and shift schedules, and comprehensive equipment status.

[0094] The time granularity of the lower-level policy is minute-level updates. The action space is continuous control of equipment speed adjustment, worker real-time location and skills, current production line output, real-time inventory distribution, and order urgency breakdown. State input is local features, including starting and stopping, switching and adjusting equipment operating modes or operating parameters, scheduling personnel work or dynamically adjusting personnel work intensity, inventory entry and exit operations, and order quantity priority adjustment.

[0095] Define the joint action space of the upper-layer strategy and the lower-layer strategy, and use a dual-branch network architecture to establish the upper / lower-layer strategy network. The upper-layer strategy network uses the PPO algorithm to output the calculation formula for the probability distribution of discrete decision actions: Where, π upperIt is represented as the upper policy network using the PPO algorithm to output the probability distribution of discrete decision actions, and Softmax is represented as the probability distribution function that maps the output to discrete actions. upper Represented as the weight matrix of the upper policy network, Represented as a state encoder with 3 layers of fully connected network MLP to extract global information, a upper It is expressed as discrete decision actions, including capacity allocation mode selection, supplier scheduling strategy, order quantity planning schedule, inventory strategy adjustment and personnel shift management, s upper It represents the global information input to the upper-level state, including the total order demand, overall production progress, total inventory, total number of personnel and shift arrangements, and comprehensive equipment status. T represents transposition;

[0096] The lower layer strategy network uses the DDPG algorithm to output the calculation formula of the probability distribution of continuous control actions: Where, π lower It is represented as the lower layer policy network using the PPO algorithm to output the probability distribution of continuous control actions, tanh is represented as the activation function, ω lower Represented as the weight matrix of the lower policy network, Represented as a state encoder with 3 layers of fully connected network MLP to extract global information, a lower It is expressed as a continuous control action, including starting and stopping, switching and adjusting the working mode or operating parameters of equipment, scheduling personnel work or dynamically adjusting personnel work intensity, inventory entry and exit operations, and order quantity priority adjustment. lower Represents local features as input to the lower-level state, including starting and stopping, switching and adjusting equipment operating modes or operating parameters, scheduling personnel work or dynamically adjusting personnel work intensity, inventory entry and exit operations, and order quantity priority adjustment;

[0097] The fully connected network MLP is used to evaluate the value of the state-action pair of the lower-level policy network. The expression of the MLP evaluation value is: Where Q φ It represents the value of the state-action pair evaluated by the lower policy network, and MLP represents the Q-value estimate of the MLP output. Represented as a vector concatenation operation, it is used to concatenate the local features of the lower-level state input with the continuous control action and then input them into the MLP;

[0098] A multi-objective reward function of inventory cost, delivery on-time rate and equipment utilization is designed, and a hierarchical weighted method is used to balance the conflicting objectives.

[0099] Specifically, the output steps of the optimal resource allocation actions of the upper and lower layer strategies are as follows:

[0100] Combined with the upper-layer strategy network, the PPO algorithm is used to output the probability distribution of discrete decision actions. The optimal resource allocation action selection calculation formula of the upper-layer strategy is a upper =Gumbel-Softmax(π upper (a upper |s upper )), where Gumbel-Softmax is expressed as the back-propagation activation function of the probability distribution of discrete variables;

[0101] According to the lower-layer strategy network, the DDPG algorithm is used to output the probability distribution of continuous control actions, and exploration noise and action clipping are injected to obtain the optimal resource allocation action of the lower-layer strategy. The calculation formula for selecting the optimal resource allocation action of the layer strategy is a lower =π lower (a lower |s lower )+N(0,σ t 2 ), and a lower '=clip(a lower ,0,1), where N(0,σ t 2 ) is represented by Gaussian noise with a mean of 0 and a variance of σ t 2 , a lower ' represents the optimal resource allocation action of the final lower-level strategy, and a lower '∈[0,1], clip represents the limit of continuous control action a lower Cutting process;

[0102] The optimal resource allocation action output by the upper-level strategy is transmitted to the lower-level strategy and used as an additional state input of the lower-level strategy, so that the discrete decision action of the upper-level decision and the continuous control action of the lower-level decision are collaboratively associated. The upper-level decision outputs the action space of discrete decisions based on the input global information state, and the lower-level decision outputs the action space of continuous control based on the input local feature state. The upper-level strategy transmits abstract instructions through decision actions, allowing the lower-level strategy to decompose and execute continuous control actions in time-granular actions. The lower-level strategy feeds back the execution results of the actual continuous control actions to trigger the upper-level strategy to dynamically adjust the discrete action decisions.

[0103] Digital twin simulation verification module: The built-in discrete event simulation DES engine combines the physical Adams dynamics model to build a digital twin model corresponding to the actual resource configuration. It simulates and verifies the optimal resource allocation action in a virtual environment, outputs simulation results, including feasibility scores and benefit prediction indicators, and evaluates and judges the simulation results. If the judgment verification passes, the optimal resource allocation action information is passed to the resource allocation execution module; if the judgment verification fails, the problem of the simulation result is fed back to the reinforcement learning decision module for dynamic adjustment;

[0104] Specifically, the steps for outputting simulation results are as follows:

[0105] Using the discrete event simulation DES engine and Adams dynamic model, combined with the actual resource configuration of the system, input the DES event log and input the Adams dynamic state data;

[0106] According to the event scheduling rules of the DES engine, discrete events of resource allocation are processed in sequence. The Adams dynamic model is used to simulate the equipment status. The order completion time, equipment working mode and operating parameter indicators during the simulation process are recorded and analyzed.

[0107] Evaluate the feasibility of resource allocation actions based on recorded indicator information and calculate benefit prediction indicators such as total cost, total revenue, and production efficiency;

[0108] The assessed feasibility scores and calculated benefit prediction indicators are output as simulation results.

[0109] Resource allocation execution module: This module executes the optimal resource allocation actions verified by the digital twin simulation verification module in the actual manufacturing production environment, and sets a fault-tolerant mechanism to trigger the reinforcement learning decision module to roll back the policy.

[0110] Specifically, the setting logic of the fault tolerance mechanism is as follows:

[0111] Develop a fault-tolerant strategy: Real-time monitoring of deviations after executing optimal resource allocation actions. When anomalies are detected, or when the execution results deviate from expectations, a policy rollback is automatically triggered. Monitoring indicators are defined as inventory costs, on-time delivery rates, and equipment utilization rates. Adaptive thresholds are set for these indicators using a sliding window statistical method. Anomalies are compared and analyzed against the adaptive thresholds of the monitoring indicators to determine whether a rollback is necessary.

[0112] Trigger and execute the fault-tolerance mechanism: Use decision tree rules to trigger rollback decisions. When the adaptive threshold is exceeded, the reinforcement learning decision module is triggered to roll back the policy, restore the action selected by the last feasible resource allocation policy, and output the execution effect.

[0113] Evaluate the effectiveness of the fault-tolerance mechanism and make improvements and optimizations: Evaluate the effectiveness of the fault-tolerance mechanism after triggering a policy rollback, generate evaluation results, and provide feedback to relevant personnel for subsequent analysis and improvement.

[0114] Reward Evaluation and Feedback Learning Module: After the resource allocation execution module executes the optimal resource allocation action, it monitors multiple indicators in the production process in real time, including inventory cost, delivery on-time rate, and equipment utilization rate. According to the multi-objective reward function pre-set in the reinforcement learning decision module, the reward value of each indicator is calculated. The learning mechanism of Prioritized Experience Replay (PER) and Proximal Policy Optimization (PPO) is adopted, and the reward value and new state information are fed back to the reinforcement learning decision module for further learning and strategy update.

[0115] Specifically, the steps for calculating the reward values ​​of multiple indicators are as follows:

[0116] Multiple indicators include inventory cost, delivery on-time rate, and equipment utilization rate. The formula for calculating inventory cost is R ic =-κ·m ic , where R ic is expressed as inventory cost, κ is the penalty coefficient for controlling inventory overstock, m ic Expressed as the total inventory cost;

[0117] The formula for calculating on-time delivery rate is: Where R odr It is expressed as the on-time delivery rate, no' is the total number of orders, and exp is the exponential function with the natural constant e as the base. It represents the actual completion time of the no. order. It represents the agreed delivery time of the no. order, t τ It is expressed as a time tolerance parameter that controls the rate at which the reward decays;

[0118] The calculation formula for equipment utilization is: And when η k >η min When ψ(1;0)=1, when η k ≤η min When ψ(1;0)=0, where R eu It is represented by the equipment utilization rate, K is the total number of equipment, η k is the utilization rate of the kth device, ψ(1;0) is the indicator function, η min It is expressed as the minimum utilization threshold of the device;

[0119] The expression of the multi-objective reward function is R total =ω ic ·R ic+ω odr ·R odr +ω eu ·R eu , and ω ic +ω odr +ω eu =1, where R total Expressed as the output value of the multi-objective reward function of inventory cost, delivery on-time rate, and equipment utilization rate, ω ic 、ω odr 、ω eu Respectively expressed as the corresponding inventory cost R ic , delivery on-time rate R odr , equipment utilization rate R eu The weight coefficient of .

[0120] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0121] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0122] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0123] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0124] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning, characterized by: It includes an ERP status data perception module: it connects to the manufacturing ERP system, collects multi-source heterogeneous data in real time, uses Kalman filtering technology to fuse multi-source heterogeneous data in real time, detects outliers, and uses an improved robust scaling method for standardization. It outputs initial data with a unified structure and transmits it to the feature extraction, integration, processing and representation module; Feature extraction, integration, processing and representation module: Receives initial data and performs cleaning, filtering, and feature extraction. It then integrates and processes heterogeneous data using a multi-head attention mechanism to generate a state vector that can be processed by the reinforcement learning decision module. Reinforcement learning decision module: Receives the state vector and inputs the state space, defines the action space, establishes a hybrid action space hierarchical reinforcement learning model, designs a multi-objective reward function, outputs the optimal resource allocation action of the upper and lower layer strategies, and transmits the optimal resource allocation action to the digital twin simulation verification module; Digital twin simulation verification module: The built-in discrete event simulation (DES) engine combines the physical Adams dynamics model to build a digital twin model corresponding to the actual resource configuration. It simulates and verifies the optimal resource allocation action in a virtual environment and outputs simulation results, including feasibility scores and benefit prediction indicators. Resource allocation execution module: This module executes the optimal resource allocation actions verified by the digital twin simulation verification module in the actual manufacturing production environment, and sets a fault-tolerant mechanism to trigger the reinforcement learning decision module to roll back the policy. Reward evaluation and feedback learning module: After the resource allocation execution module executes the optimal resource allocation action, it monitors multiple indicators in the production process in real time, calculates the reward value of each indicator based on the multi-objective reward function pre-set in the reinforcement learning decision module, adopts the learning mechanism of priority experience replay PER and proximal policy optimization PPO, and feeds back the reward value and new state information to the reinforcement learning decision module for further learning and strategy update.

2. The manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning according to claim 1 is characterized in that: The steps for acquiring multi-source heterogeneous data are as follows: Deploy high-precision sensors on manufacturing equipment and transmit sensor data in real time via industrial Ethernet to obtain equipment status information, marked as Esi; Use the API interface to connect to the manufacturing ERP system to obtain basic employee information, marked as Psi. At the same time, extract inventory data and order requirements, marked as Ivd and Orm respectively; In each production link of the manufacturing ERP system, obtain the production progress information of product manufacturing and mark it as PPI; Multi-source heterogeneous data includes equipment status information, personnel information, inventory data, order requirements and production progress information. The expression of multi-source heterogeneous data is MHD = {Esi, Psi, Ivd, Orm, Ppi…, where MHD represents multi-source heterogeneous data.

3. The manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning according to claim 2 is characterized in that: The steps of the Kalman filter technology for processing multi-source heterogeneous data are as follows: A unified state space model is established for multi-source heterogeneous data MHD to generate state vectors and observation vectors that can be processed by Kalman filtering technology. The expression of the state vector is: Where, Represented as the state vector of multi-source heterogeneous data MHD at time t, It represents the state parameter of the i-th multi-source heterogeneous data MHD, n represents the dimension of the state parameter of the multi-source heterogeneous data MHD, and T represents the transpose; The expression of the observation vector is and It is represented as the observation value of the i-th MHD data source at time t, It is represented as the observation matrix of the i-th MHD data source at time t, is the observation noise of the i-th MHD data source at time t, N is the total number of MHD data sources, and i=1,2,…,N, It is represented as the observation noise covariance matrix of the i-th MHD data source; The Kalman filter technology is used to detect outliers, including the prediction stage, residual calculation and outlier judgment. The calculation formula of the prediction stage is: And P t =F t ·P t-1 ·F t T +Q t , where It is expressed as the predicted value of the MHD state of multi-source heterogeneous data at time t based on time t-1, It is expressed as the predicted value of the MHD state of multi-source heterogeneous data at time t-1, F t It is represented as the state transition matrix of multi-source heterogeneous data MHD at time t, P t It is expressed as the prediction state error covariance matrix of multi-source heterogeneous data MHD at time t, P t-1 It is expressed as the prediction state error covariance matrix of multi-source heterogeneous data MHD at time t-1, Q t It is represented as the process noise covariance matrix of multi-source heterogeneous data MHD transmission at time t; The formula for residual calculation is and Where, Expressed as the residual of the i-th MHD data source at time t, It is represented as the residual covariance matrix of the i-th MHD data source at time t; The calculation formula for outlier judgment is: and Where, Expressed as the Mahalanobis distance of the residual, is the threshold of the chi-square distribution, m is the residual dimension, α is the significance level, and α = 0.95; The detected multi-source heterogeneous data are fused in real time with information matrix and updated in state to obtain fused data. The expression of real-time fusion of information matrix is: and Where, Expressed as the inverse matrix of the updated forecast state error covariance matrix, It is expressed as the weighted fusion of multi-source observation information, K t Expressed as a Kalman gain matrix; The expression for state update is Where, It is expressed as the state update value obtained by fusing multi-source observation data and correcting the prediction value to obtain the optimal state estimate. The expression of the fused data is x MHD '.

4. The manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning according to claim 3 is characterized in that: The standardized processing steps of the improved Robust Scaling method are as follows: According to the fusion data x MHD ', use the improved Robust Scaling method to output the initial data of unified structure, and reduce the influence of data outliers through dynamic weight adjustment. The expression of the improved Robust Scaling method is and Where, X MHD " indicates that the improved Robust Scaling method standardizes the fused data x MHD 'The initial data, Represented as the sample value of the j-th multi-source heterogeneous data MHD, μ ω Represented as fused data x MHD 'weighted mean, σ ω Represented as fused data x MHD ' is the standard deviation of the weighted mean, ε is a small constant to prevent division by zero, and ε = e -8 , j, J represent the jth sample data and the total number of samples respectively, ω j It is expressed as the weight of the j-th sample, β is expressed as the weight decay coefficient, and ecp is expressed as the exponential decay function. It is expressed as the median of the sample values ​​of the j-th multi-source heterogeneous data MHD. Feature extraction, integration and processing representation module: Receives initial data and performs cleaning, screening and feature extraction, combines the integration and processing of heterogeneous data with the multi-head attention mechanism, and generates a state vector that can be processed by the reinforcement learning decision module.

5. The manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning according to claim 4 is characterized in that: The steps of feature extraction are as follows: For the initial data x MHD "Cleaning, screening and feature extraction are performed by using the improved Hampel identifier to remove outliers, using dynamic time warping DTW to fill missing values, and MHD " Extract the static features of personnel information Psi and inventory data Ivd, then the expression of static features is X static ={Psi”,Ivd”}, and the dimension of the static feature is X static ∈R dc , where X static Expressed as starting from the initial data x MHD ”, Psi” represents the feature data extracted from personnel information Psi after being processed by the unified structure of ERP status data perception module, Psi” represents the feature data extracted from inventory data Ivd after being processed by the unified structure of ERP status data perception module, R dc Represented as static feature X static Dimensions; Use the long short-term memory autoencoder LSTM-AE to process the time series data of the device status information Esi, calibrated as Esi t , and Esi t ∈R d , for time series data Esi t Perform dynamic feature extraction and output time series features. The encoder in the LSTM-AE network performs dimensionality reduction processing; The decoder in the LSTM-AE network performs reconstruction processing to reconstruct the temporal characteristics of the output device state information Esi. The expression of the temporal characteristics of the device state information Esi is X seq ={Esi”}, and X seq ∈R T×ds , where X seq Represented as the time series data Esi of LSTM-AE processing device status information Esi t The output time series characteristics, Esi" represents the time series data Esi of the device status information Esi t After dynamic extraction of LSTM-AE, R T×ds Represented as time series feature X seq Dimensions; Use graph convolutional network GCN to build a process route relationship model for order demand Orm and production progress information Ppi to extract graph structure features and output graph features X graph , and X graph ∈R M×dg , where X graph It is represented as the graph features extracted by GCN for order demand Orm and production progress information Ppi, R N×dg Represented as graph feature X graph Dimensions; From the initial data x MHD The extracted heterogeneous features are grouped and input into the multi-head attention mechanism for feature normalization and integration.

6. The manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning according to claim 5 is characterized in that: The integration processing steps of the multi-head attention mechanism are as follows: From the initial data x MHD The extracted static feature X static , time series feature X seq and graph feature X graph Perform alignment and projection, where the calculation formula for alignment and projection is: Where, Represented as time series feature X seq Alignment, Represented as graph feature X graph Alignment, MaxPool is expressed as a time pooling dimensionality reduction function, Expressed as node-level attention aggregation calculation, Represented as static feature X static Extension, R ds+dg Represented as a replication dimension; Multi-head attention is calculated by splicing heterogeneous features to generate an input splicing feature vector, where the expression of the input splicing feature vector is And R d =R ds+dg+dc , where X concat Represented as the concatenated feature vector, R d Represented as a unified dimension; According to the input concatenated feature vector, layer normalization is used to perform feature normalization processing, and the state vector that conforms to the dimension of the reinforcement learning state space is output. The state vector expression after feature normalization is and Where, LN(X concat ) represents the output value of the layer normalization, γ represents the learnable scaling parameter, Represented as the concatenated feature vector X concat The mean of Represented as the concatenated feature vector X concat The standard deviation of , δ represents the learnable translation parameter, s(X) t It is represented as the state vector after feature normalization, MultiHead(X concat ) is represented as the concatenated feature vector X concat The multi-head output splicing value, is represented as the dimension of the reinforcement learning state space, and d RL =256.

7. The manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning according to claim 6 is characterized in that: The steps for establishing the hybrid action space hierarchical reinforcement learning model are as follows: Design a hierarchical architecture for a hybrid action space hierarchical reinforcement learning model, including upper and lower policies. The upper and lower policies are refined to define their time granularity, action space, and state inputs. The upper policy has hourly time granularity, the action space is discrete, and the state input is global information. The time granularity of the lower-level strategy is minute-level update, the action space is continuous control, and the state input is local features; Define the joint action space of the upper-layer strategy and the lower-layer strategy, and use a dual-branch network architecture to establish the upper / lower-layer strategy network. The upper-layer strategy network uses the PPO algorithm to output the calculation formula for the probability distribution of discrete decision actions: Where, π upper It is represented as the upper policy network using the PPO algorithm to output the probability distribution of discrete decision actions, and Softmax is represented as the probability distribution function that maps the output to discrete actions. upper Represented as the weight matrix of the upper policy network, Represented as a state encoder with 3 layers of fully connected network MLP to extract global information, a upper Expressed as a discrete decision action, s upper It represents the global information of the upper state input, and T represents the transpose; The lower policy network uses the DDPG algorithm to output the calculation formula of the probability distribution of continuous control actions: Where, π lower It is represented as the lower layer policy network using the PPO algorithm to output the probability distribution of continuous control actions, tanh is represented as the activation function, ω lower Represented as the weight matrix of the lower policy network, Represented as a state encoder with 3 layers of fully connected network MLP to extract global information, a lower Expressed as a continuous control action, s lower Represents the local features of the lower-level state input; The fully connected network MLP is used to evaluate the value of the state-action pair of the lower-level policy network. The expression of the MLP evaluation value is: Where Q φ It represents the value of the state-action pair evaluated by the lower policy network, and MLP represents the Q-value estimate of the MLP output. Represented as a vector concatenation operation, it is used to concatenate the local features of the lower-level state input with the continuous control action and then input them into the MLP; Design a multi-objective reward function for inventory cost, on-time delivery rate, and equipment utilization rate, and use a hierarchical weighting method to balance conflicting objectives; The steps for outputting the optimal resource allocation actions of the upper and lower layer strategies are as follows: Combined with the upper-layer strategy network, the PPO algorithm is used to output the probability distribution of discrete decision actions. The optimal resource allocation action selection calculation formula of the upper-layer strategy is a upper =Gumbel-Softmax(π upper (a upper |s upper )), where Gumbel-Softmax is expressed as the back-propagation activation function of the probability distribution of discrete variables; According to the lower-layer strategy network, the DDPG algorithm is used to output the probability distribution of continuous control actions, and exploration noise and action clipping are injected to obtain the optimal resource allocation action of the lower-layer strategy. The calculation formula for selecting the optimal resource allocation action of the layer strategy is a lower =π lower (a lower |s lower )+N(0,σ t 2 ), and a lower '=clip(a lower ,0,1), where N(0,σ t 2 ) is represented by Gaussian noise with a mean of 0 and a variance of σ t 2 , a lower ' represents the optimal resource allocation action of the final lower-level strategy, and a lower '∈[0,1], clip represents the limit of continuous control action a lower Cutting process; The optimal resource allocation action output by the upper-level strategy is passed to the lower-level strategy and used as an additional state input of the lower-level strategy, so that the discrete decision-making action of the upper-level decision and the continuous control action of the lower-level decision are synergistically associated.

8. The manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning according to claim 7 is characterized in that: The steps for calculating the reward values ​​of the multiple indicators are as follows: Multiple indicators include inventory cost, delivery on-time rate, and equipment utilization rate. The formula for calculating inventory cost is R ic =-k·m ic , where R ic is expressed as inventory cost, κ is the penalty coefficient for controlling inventory overstock, m ic Expressed as the total inventory cost; The formula for calculating on-time delivery rate is: Where R odr It is expressed as the on-time delivery rate, no' is the total number of orders, and exp is the exponential function with the natural constant e as the base. It represents the actual completion time of the no. order. It represents the agreed delivery time of the no. order, t τ It is expressed as a time tolerance parameter that controls the rate at which the reward decays; The calculation formula for equipment utilization is: And when η k >η min When ψ(1;0)=1, when η k ≤η min When ψ(1;0)=0, where R eu It is represented by the equipment utilization rate, K is the total number of equipment, η k is the utilization rate of the kth device, ψ(1;0) is the indicator function, η min It is expressed as the minimum utilization threshold of the device; The expression of the multi-objective reward function is R total =ω ic ·R ic +ω odr ·R odr +ω eu ·R eu , and ω ic +ω odr +ω eu =1, where R total Expressed as the output value of the multi-objective reward function of inventory cost, delivery on-time rate, and equipment utilization rate, ω ic 、ω odr 、ω eu Respectively expressed as the corresponding inventory cost R ic , delivery on-time rate R odr , equipment utilization rate R eu The weight coefficient of .

9. A computer device comprising: memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, the steps of the manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the manufacturing ERP dynamic resource optimization configuration system based on reinforcement learning according to any one of claims 1 to 8 are implemented.

Images