Workshop scheduling method based on digital twin and knowledge graph reinforcement learning

Through the reinforcement learning method of digital twins and knowledge graphs, the problem of insufficient effectiveness and interpretation of workshop scheduling in complex dynamic environments is solved, efficient and intelligent production management and decision-making transparency are achieved, and the overall performance of the system and user trust are improved. .

CN117669984BActive Publication Date: 2024-11-05JINAN UNIVERSITY +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202311750332.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-18
Publication Date
2024-11-05
Estimated Expiration
2043-12-18

AI Technical Summary

Technical Problem

When faced with complex and dynamic production environments, existing workshop scheduling technologies lack effectiveness, flexibility and interpretation, making them difficult to provide high-quality scheduling decisions, and poor data timeliness, affecting system trust and dependence.

Method used

Using reinforcement learning methods based on digital twins and knowledge graphs, through real-time data synchronization and multi-level workshop status representation, an interpretable reinforcement learning model is built to generate transparent scheduling decisions, and improve system intelligence through knowledge graph updates and multi-objective optimization algorithms. and user trust.

Benefits of technology

It realizes efficient, intelligent and interpretable production management, improves production efficiency, reduces costs, and enhances decision-making transparency, solving the problems of data timeliness and understanding of complex relationships.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117669984B_ABST
    Figure CN117669984B_ABST
Patent Text Reader

Abstract

The present invention proposes a workshop scheduling method based on digital twin and knowledge graph reinforcement learning, including: establishing a data receiving and storage structure to collect multimodal data, generating labels and metadata based on multimodal data, and using the connectivity of the graph model to predict labels and metadata related to existing data; performing data quality assessment and filtering on the generated labels and metadata, wherein the data quality assessment is performed based on the data quality score; designing a multi-level workshop state representation and establishing a workshop digital twin model; constructing a knowledge graph and an interpretable reinforcement learning model; generating decision logic and explaining the decision logic based on the interpretable reinforcement learning model; building an experimental environment based on the decision logic and the workshop digital twin model and evaluating the experimental performance. The present invention comprehensively solves the efficiency, cost and interpretability issues of workshop scheduling.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a workshop scheduling method based on reinforcement learning of digital twins and knowledge graphs. Background Art

[0002] As a core link in the manufacturing industry, workshop scheduling plays a vital role. Its performance directly affects multiple key indicators such as manufacturing cost, delivery time and resource utilization. Traditional workshop scheduling solutions mainly focus on priority-based rule scheduling and mathematical programming algorithms, such as integer programming and genetic algorithms. These methods perform well when dealing with static or low-dynamic production environments, but when faced with highly dynamic and complex production environments, these methods usually show obvious limitations.

[0003] For example, priority rules are often based on human settings, and such rules often fail to provide excellent solutions for unseen scenarios or complex multi-objective optimization problems. Although mathematical programming methods can describe problems more systematically, the solution process is computationally complex, especially when encountering large-scale problems, the solution time becomes unacceptable.

[0004] At the same time, with the continuous development of digital and intelligent technologies, emerging technologies such as digital twins and knowledge graphs have gradually come into people's view. Digital twins can provide a highly accurate digital replica of physical systems, but in the field of workshop scheduling, their main use is still limited to the simulation and optimization of a single process, and has not yet been deeply integrated with global scheduling problems. Knowledge graphs help to represent and process complex system relationships, but they have also not been widely used in scheduling problems.

[0005] In addition, most existing scheduling solutions lack explainability, which makes it difficult for workshop operators and managers to understand the decision-making basis of the scheduling algorithm, which in turn affects their trust and reliance on the system.

[0006] Therefore, the existing workshop scheduling technology lacks effectiveness, flexibility and interpretability when facing complex and dynamic production environments. A new solution is urgently needed to make up for these shortcomings. Summary of the invention

[0007] The purpose of this invention is to design a workshop scheduling method based on digital twin and knowledge graph reinforcement learning. Through real-time data synchronization, it can not only achieve a high degree of consistency between the workshop environment and the scheduling system, but also efficiently manage and optimize complex production processes and resource dependencies. More importantly, the interpretable reinforcement learning algorithm used in this solution can generate transparent and easy-to-understand scheduling decisions, greatly improving the intelligence level and user trust of the entire system.

[0008] In order to achieve the above objectives, the present invention provides 1. a workshop scheduling method based on digital twin and reinforcement learning of knowledge graph, the method comprising:

[0009] S1. Establish a data receiving and storage structure to collect multimodal data, generate tags and metadata based on the multimodal data, and use the connectivity of the graph model to predict tags and metadata related to the existing data;

[0010] Generating tags and metadata according to multimodal data specifically refers to:

[0011] For each collected data point, a pre-trained classifier based on a graph model is used to generate labels, as shown below:

[0012] L = f tag (D,G)

[0013] Among them, L represents the generated label, f tag () represents the label generation function based on the graph model, D represents the original data, and G represents the existing knowledge graph;

[0014] Finally, metadata is generated for each data point;

[0015] S2. Perform data quality assessment and filtering on the generated tags and metadata. The data quality assessment is performed based on the data quality score, which is expressed as follows:

[0016] Q = g quality (D,M,T)

[0017] Among them, Q represents the data quality score, g quality represents the scoring function, M represents the metadata of the data, and T represents the time factor;

[0018] S3. Design a multi-level workshop status representation and establish a workshop digital twin model, including:

[0019] S301, identifying key data sources;

[0020] S302, designing data fusion agents to collect data, wherein each data fusion agent is responsible for collecting a specific type of data;

[0021] S303. Define a multi-level workshop status representation and build a workshop digital twin model, including:

[0022] Generate a basic production data matrix using data collected by the data fusion agent Workflow Information Matrix Environmental parameter matrix ε and personnel dynamic matrix Then the basic production data matrix is ​​processed using time series decomposition Workflow Information Matrix Environmental parameter matrix ε and personnel dynamic matrix The workshop status characteristics are obtained, wherein the time series decomposition is expressed as:

[0023]

[0024] in, represents the processed basic production data matrix, λ represents the smoothing parameter of STL, and using these processed data matrices, a high-dimensional tensor T is used MLWSR Perform multi-level workshop state representation by forming a high-dimensional tensor T MLWSR Thus forming a digital twin model of the workshop;

[0025] S4. Build a knowledge graph and implement dynamic updates, including:

[0026] S401. Use natural language processing tools to analyze documents, identify entities and relationships, manually verify and adjust the output of natural language processing tools, and generate a preliminary knowledge graph prototype, as shown below:

[0027]

[0028] in, Represents the knowledge graph relationship, V represents the set of entities, and E represents the set of relationships;

[0029] S402, constructing a dynamic knowledge graph according to the prototype of the knowledge graph;

[0030] S403, using a reinforcement learning-based algorithm to update the dynamic knowledge graph in real time;

[0031] S5. Build and train an interpretable reinforcement learning model;

[0032] S6. Generate decision logic and explain decision logic based on interpretable reinforcement learning model;

[0033] S7. Build the experimental environment based on the decision logic and the workshop digital twin model and evaluate the experimental performance.

[0034] Furthermore, the data quality indicators for evaluating the data quality score Q include: completeness, consistency, timeliness and accuracy;

[0035] The evaluation data quality indicator scores calculated according to the evaluation data quality indicators include: completeness score, consistency score, timeliness score and accuracy score;

[0036] The data quality score Q is obtained as:

[0037] Q=a*Completeness Score+b*Consistency Score+c*Timeliness Score+d*Accuracy Score

[0038] Right now

[0039] g quality (D,M,T)=b*Completeness Score+b*Consistency Score+c*TimelinessScore+d*Accuracy Score

[0040] Among them, a, b, c, d represent the weights of completeness score, consistency score, timeliness score and accuracy score respectively, and a+b+c+d=1; Completeness Score represents the completeness score; Consistency Score represents the consistency score; Timeliness Score represents the timeliness score; Accuracy Score represents the accuracy score.

[0041] Furthermore, the data receiving and storing structure includes:

[0042] Multimodal sensor module, used to collect data from multimodal sensors and standardize IoT devices;

[0043] Real-time data preprocessing module, used to preprocess data;

[0044] A secure data transmission module, used for encrypting and transmitting data;

[0045] A data receiving and decoding module is used to decode and analyze the received data on the central server;

[0046] Dataset storage module, used to store real-time data and configure data in relational data storage;

[0047] Data access module, used to develop RESTful APIs for external applications and services to access data;

[0048] Data analysis module, used to perform in-depth analysis of data.

[0049] Furthermore, the mathematical model of the data fusion agent is expressed as follows:

[0050] DFA i =φ i (DS i ,τ i )

[0051] Among them, DFAi represents the i-th data fusion agent, φ i Denotes the data collection function, DS i represents the responsible data source, τ i Indicates the data collection interval.

[0052] Furthermore, the dynamic knowledge graph update rule is expressed as follows:

[0053]

[0054] in, represents the knowledge graph at time t+1, RL represents the graph update function based on reinforcement learning, MLWSR t+1 Represents the multi-level workshop status representation at time t+1.

[0055] Furthermore, the step S5 specifically includes:

[0056] S501. Use a deep neural network to learn the state representation and dynamic knowledge graph of the workshop, and use a decision tree or rule engine to extract specific rules or patterns to explain the decision logic of the neural network, wherein the interpretable reinforcement learning model is expressed as follows:

[0057] f(x)=DNN(x)×DT(x)

[0058] Among them, f(x) represents the final decision function, DNN(x) represents the deep neural network part, and DT(x) represents the decision tree part;

[0059] S502, pre-training an interpretable reinforcement learning model using the historical data of the workshop and the data generated by the simulator,

[0060] S503. Fine-tune the model using real-time collected data and feedback, and use a multi-objective optimization algorithm to consider model performance and interpretability, wherein the multi-objective optimization algorithm is expressed as follows:

[0061]

[0062] Among them, μ represents the weight coefficient, and They represent the loss functions of performance and interpretability respectively, and θ represents the model parameters.

[0063] Furthermore, the S6 specifically includes:

[0064] S601. Generate decision logic using an interpretable reinforcement learning model and convert the strategy into language or rules, wherein the expression of the decision logic is as follows:

[0065] IF(condition1∧condition2∧…)THEN action

[0066] Among them, condition i is the decision condition, and action is the corresponding operation;

[0067] S602: Using the SHAP method, an in-depth explanation is given to the generated decision logic, wherein the SHAP value is expressed as follows:

[0068]

[0069] Among them, φ j represents the SHAP value of feature j, f(S) represents the predicted value given the feature set S, N represents the set of all features, and S represents various combinations of workshop status and operations.

[0070] Furthermore, the step S7 specifically includes:

[0071] S701. Use hardware-in-the-loop technology to integrate actual workshop equipment and workshop digital twin models to form a physical-digital simulation environment and deploy decision logic and workshop digital twin models;

[0072] S702. Design a multi-objective optimization function F(x) that takes into account multiple production and performance factors, expressed as follows:

[0073] F(x) = α×efficiency+T×energy efficiency-γ×equipment wear

[0074] Among them, α, β, and γ represent the weight coefficients of each indicator;

[0075] S703. Use deep learning and statistical methods for data analysis;

[0076] S704. Evaluate the success of the experimental results based on the collected data and the defined multi-objective optimization function, and generate a detailed experimental report.

[0077] Furthermore, the deep learning and statistical methods include time series analysis and anomaly detection, wherein the model of the time series analysis is expressed as:

[0078] h t =f W (h t-1 ,x t )

[0079] Among them, h t represents the hidden state at time t, x t represents the input at time t, h t-1 represents the hidden state at time t-1.

[0080] Furthermore, it also includes S8, using Docker containerization technology to deploy models and knowledge graphs, real-time processing and feedback of data streams, and dynamically adjusting decision logic, and using deep learning to generate models to detect abnormal behaviors or decision errors.

[0081] The beneficial technical effects of the present invention are at least as follows:

[0082] (1) The present invention realizes efficient, intelligent and explainable production management through a workshop scheduling method based on digital twin and knowledge graph reinforcement learning. First, the data collection and preprocessing steps provide accurate and real-time basic data for the entire system. These data are further used to build a digital twin model, which accurately reflects the actual operating status of the workshop and provides reliable input for the subsequent reinforcement learning model. Then, the knowledge graph is used to describe and analyze the complex relationships within the workshop, thereby providing more advanced analysis and optimization methods. On this basis, an explainable reinforcement learning model is designed and trained, which can not only generate effective scheduling strategies, but also provide clear explanations for each decision. Finally, experimental verification and performance evaluation are used to confirm the advantages of the scheme in improving production efficiency, reducing costs and enhancing decision transparency. Each step is designed to achieve higher scheduling efficiency and decision transparency, and there is a close logical connection and data flow between them, which together serve the core goal of the entire patent.

[0083] (2) The present invention uses digital twin technology to collect workshop data in real time and construct a dynamic workshop status representation. This not only solves the problem of poor data timeliness in traditional methods, but also provides more accurate state input for reinforcement learning.

[0084] (3) Based on the status of the workshop, a knowledge graph is constructed to explicitly express the dependencies between the processes and equipment in the workshop. This not only helps the reinforcement learning model understand the complex relationships in the workshop, but also provides richer information for subsequent optimization.

[0085] (4) Based on the workshop status, the present invention constructs a knowledge graph to explicitly express the dependencies between various processes and equipment in the workshop. This not only helps the reinforcement learning model understand the complex relationships in the workshop, but also provides richer information for subsequent optimization.

[0086] (5) Through the interactive update of digital twins and knowledge graphs, the simultaneous optimization of workshop status and scheduling strategy is achieved. The digital twin provides accurate status information for graph updates, while the graph improves the digital twin model by mining relationships and providing optimization suggestions. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0088] Figure 1 This is a flow chart of the workshop scheduling method based on reinforcement learning of digital twins and knowledge graphs of the present invention. DETAILED DESCRIPTION

[0089] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0090] In one or more embodiments, Figure 1 As shown, the present invention discloses a workshop scheduling method based on digital twin and knowledge graph reinforcement learning, characterized in that the method includes:

[0091] S1. Establish a data reception and storage structure to collect multimodal data, generate tags and metadata based on the multimodal data, and use the connectivity of the graph model to predict tags and metadata related to the existing data.

[0092] Use machine learning algorithms based on graph models to automatically generate labels and metadata for various types of data during the data collection phase. Use the connectivity of graph models to predict labels and metadata related to existing data, thereby increasing the availability of data.

[0093] Specifically, the multimodal data includes using IoT devices to collect environmental parameters and mechanical data, deploying visual sensors to capture image information of each workstation in the workshop, and using audio sensors to capture workshop environmental sounds.

[0094] The generating of tags and metadata according to the multimodal data specifically refers to:

[0095] For each collected data point, a pre-trained classifier based on a graph model is used to generate labels, as shown below:

[0096] L = f tag (D,G) (1)

[0097] Among them, L represents the generated label, f tag () represents the label generation function based on the graph model, D represents the original data, and G represents the existing knowledge graph;

[0098] Metadata is then generated for each data point.

[0099] The label generation function is explained as follows:

[0100] (1) Graph expansion: First, data points D are added to the graph G. Each data point becomes a new node and is connected to existing nodes in the graph through a similarity measure.

[0101] (2) Node feature setting: For each node in G, there is a set of feature vectors, which are calculated based on the data point D.

[0102] (3) Graph Embedding: Next, G is embedded using a graph neural network. In this way, each node (including newly added and original) will have a low-dimensional embedding vector.

[0103] E=GNN(G)

[0104] Where E is the set of embedding vectors.

[0105] (4) Label prediction: Finally, a classifier (decision tree) is used to predict the label for each new embedding vector.

[0106] L = Classifier(E)

[0107] Combining the above steps, f tag The (D,G) function can be written as:

[0108] L = f tag (D,G)=Classifier(GNN(G)).

[0109] Specifically, the structure of the data receiving and storing structure includes:

[0110] Multimodal sensor module, used for data collection of multimodal sensors, standardizes IoT devices; uses multimodal sensors (temperature, humidity, vibration, image and sound, etc.) for data collection. These sensors should be standardized through industrial IoT standards.

[0111] The real-time data preprocessing module is used to preprocess the data; it caches and performs basic cleaning before sending the data to the central server, such as removing outliers and normalizing the data.

[0112] The secure data transmission module is used to encrypt data transmission; the secure transport layer based on TLS / DTLS is used to encrypt data transmission.

[0113] The data receiving and decoding module is used to decode and analyze the received data on the central server; decode and analyze the received data on the central server.

[0114] The data set storage module is used to store real-time data and store configuration data in relational data; a time series database such as InfluxDB is used to store real-time data, and a relational database such as PostgreSQL is used to store configuration data and metadata.

[0115] Data access module, used to develop RESTful APIs for external applications and services to access data; develop RESTful APIs for external applications and services to access data.

[0116] Data analysis module, used to conduct in-depth data analysis. It uses machine learning and data analysis methods to conduct in-depth data analysis.

[0117] Specifically, the generation of tags and metadata is as follows:

[0118] (1) For each collected data point, a pre-trained classifier based on a graphical model is used to generate labels.

[0119] A data point represents an individual or a group of measured values, such as the temperature value generated by a temperature sensor every minute or each frame of an image generated by a camera.

[0120] The actual form of the label is as follows:

[0121] 1. Identify the device or workstation: that is, which device or workstation in the workshop the data comes from.

[0122] 2. Process category: such as drilling, welding, etc., describing the production step represented by this data point.

[0123] 3. Product information: such as product type, batch number, etc.

[0124] 4. Timestamp: records the specific time when data is collected.

[0125] 5. Status label: indicates the operating status of the workshop or equipment, such as "normal", "standby", "fault", etc.

[0126] (2) Generate metadata for each data point based on information such as data source, type, and timestamp. Metadata source label: Indicates which device or workstation the data point comes from. Data type label: Indicates which type the data point belongs to (e.g., temperature reading, pressure reading, etc.). Timestamp: Records when the data point was generated or collected.

[0127] For example, in actual JSON format, it might look like this: {"source":"Machine_1","type":"Temperature","timestamp":"2023-10-14T12:34:56Z"}

[0128] S2. Perform data quality assessment and filtering on the generated tags and metadata.

[0129] The quality of the data is assessed using a real-time algorithm based on time series analysis, and low-quality data is filtered out. If the data quality score is below a predetermined threshold, the data point will be marked as low quality and filtered out. The collected high-quality data is preprocessed using statistical methods (such as normalization and standardization).

[0130] Convert data into a format acceptable to the knowledge graph, such as RDF triples or JSON-LD.

[0131] Output high-quality, labeled, metadata-rich datasets as the basis for building knowledge graphs; data quality and label reports are used for the metadata layer of knowledge graphs.

[0132] Specifically, the real-time quality assessment algorithm will use a machine learning model to make predictions, and the model will be updated as new data is added. The data quality score used in the steps is:

[0133] Q = g quality (D,M,T) (2)

[0134] Among them, Q represents the data quality score, g quality represents the scoring function, M represents the metadata of the data, and T represents the time factor.

[0135] Specifically, g quality The meaning of the function is explained as follows:

[0136] g quality : This is a function to evaluate data quality. It calculates a quality score based on the input data point D, its corresponding metadata M, and the time factor T.

[0137] D: represents the raw data point, which may be readings from shop floor sensors or other data sources.

[0138] M: represents metadata related to data point D, which may include data source, data type, timestamp, etc.

[0139] T: represents the time factor, which may be the timestamp of the data point or other information related to the time when the data was collected.

[0140] The calculation of data quality score is as follows:

[0141] (1) Define a series of indicators for evaluating data quality, such as:

[0142] Completeness: Check that each data point is complete, i.e. there are no missing important fields.

[0143] Consistency: Checking whether data is consistent with other data or predefined standards.

[0144] Timeliness: Evaluate the freshness of data and whether it is updated in a timely manner.

[0145] Accuracy: Evaluate whether the data is accurate and has no errors.

[0146] (2) Calculate the scores of each indicator:

[0147] For each data point D and its corresponding metadata M, the score of each indicator is calculated as follows:

[0148] Completeness score: If all required fields have values, the score is highest. If some fields are missing, the score is reduced in proportion to the significance of the missing fields.

[0149] Concordance score: When compared with other data or standards, the score is based on the proportion of agreement.

[0150] Timeliness score: calculated based on the difference between the data's timestamp T and the current time. The smaller the difference, the higher the score.

[0151] Accuracy score: can be assessed by comparing with a known standard or historical data. If the difference is less than a certain threshold, a high score is given.

[0152] The data quality score Q is obtained as:

[0153] Q=a*Completeness+b*Consistency Scpre+c*Timeliness Score+d*AccuracyScore

[0154] Right now

[0155] g quality (D,M,T)=a*Completeness Score+b*Consistency Score+c*TimelinessScore+d*Accuracy Score

[0156] Among them, a, b, c, d represent the weights of completeness score, consistency score, timeliness score and accuracy score respectively, and a+b+c+d=1; Completeness Score represents the completeness score; Completeness Score represents the consistency score; Timeliness Score represents the timeliness score; Accuracy Score represents the accuracy score.

[0157] If we assume that the indicators are equally important, the weight of each indicator is 0.25. If some indicators are more important, the weights can be adjusted.

[0158] (3) Application scoring threshold:

[0159] Define an acceptance threshold for the quality score, such as 0.75. Only when the Q value of a data point is higher than this threshold will it be used for subsequent analysis and decision making.

[0160] Complex decision logic relies on high-quality data input. This quality scoring scheme ensures that the data entering the reinforcement learning model and the plant digital twin model is accurate and reliable, thereby improving the performance and credibility of the entire system.

[0161] S3. Design a multi-level workshop status representation and establish a workshop digital twin model, including:

[0162] S301, data integration and MLWSR generation, identification of key data sources, data source identification and integration utilizes a distributed data acquisition system, namely, a data fusion agent (DFA), to obtain information from data of various types and sources.

[0163] Among them, identify key data sources: machine status sensors, ERP systems, environmental sensors, employee trajectory systems, etc.

[0164] S302, designing data fusion agents to collect data, wherein each data fusion agent is responsible for collecting a specific type of data;

[0165] The data fusion agent collects data from these data sources according to predefined rules and frequencies.

[0166] Mathematical model of data fusion agent:

[0167] DFA i =φ i (DS i ,τ i ) (3)

[0168] Among them, DFA i is the i-th data fusion agent, φi is its data collection function, DS i is the data source it is responsible for, τ i is the data collection interval.

[0169] Specifically, the specific types mentioned include:

[0170] Equipment operation data: such as the operating status of each machine, fault records, maintenance logs, energy consumption data, etc.

[0171] Production process data: including production line operation status, production speed, product quality control data, key event records in the production process, etc.

[0172] Environmental data: such as the temperature, humidity, noise level and other environmental parameters of the workshop.

[0173] Material and inventory data: raw material usage, inventory levels, material flow rates, etc.

[0174] Personnel data: information such as employees’ work arrangements, operational efficiency, skill levels, etc.

[0175] Quality control data: product quality inspection results, scrap rate, rework rate and other data.

[0176] Each agent may be responsible for one or several types of data collection and processing. For example, one agent may specialize in processing all data related to equipment operation, while another agent may process data related to the production process. This division of labor makes data collection and processing more efficient and specialized, and also provides the necessary data support for building accurate and comprehensive digital twin models.

[0177] S303. Introduce time series decomposition to obtain more accurate workshop status characteristics and define multi-level workshop status representation, including:

[0178] Generate a basic production data matrix using data collected by the data fusion agent Workflow Information Matrix Environmental parameter matrix ε and personnel dynamic matrix Then the basic production data matrix is ​​processed using time series decomposition Workflow Information Matrix Environmental parameter matrix ε and personnel dynamic matrix The workshop status characteristics are obtained, wherein the time series decomposition is expressed as:

[0179]

[0180] in, represents the processed basic production data matrix, λ represents the smoothing parameter of STL, and using these processed data matrices, a high-dimensional tensor T is used MLWSR Carrying out multi-level workshop status representation and obtaining the multi-level workshop status representation tensor TMLWSR is essentially creating a comprehensive, data-driven workshop digital model. This model can accurately reflect and simulate the operating status and dynamics of the actual workshop by integrating information from multiple dimensions such as key production data, work processes, environmental parameters, and personnel dynamics, thereby realizing the establishment of a digital twin model of the workshop.

[0181] Specifically, tensors are often used to represent multidimensional data arrays. A high-dimensional tensor is a tensor with more than two dimensions. For example, a three-dimensional tensor can be viewed as multiple matrices stacked together, a four-dimensional tensor is multiple three-dimensional tensors stacked together, and so on.

[0182] S4. Build and update the knowledge graph, using natural language processing-based tools to automatically identify key entities and relationships in the production process and reduce manual annotation, including:

[0183] S401. Use natural language processing tools to analyze documents, identify entities and relationships, manually verify and adjust the output of natural language processing tools, and generate a preliminary prototype of the knowledge graph, as shown below:

[0184]

[0185] in, Represents the knowledge graph relationship, V represents the set of entities, and E represents the set of relationships.

[0186] Specifically, natural language processing tools are mainly used to do three things:

[0187] Entity Recognition: Identify key entities from documents, such as equipment names, workstations, parts, etc.

[0188] Relation extraction: Determine the relationships between entities mentioned in a document, such as which parts belong to which machine, which tasks are completed by which workstation, etc.

[0189] Document Summarization: Automatically generate a summary or key points of a document.

[0190] Specific tools include: Google's BERT, OpenAI's GPT series, Stanford NLP library, etc.

[0191] S402, constructing a dynamic knowledge graph according to the prototype of the knowledge graph;

[0192] Specifically, the specific steps to generate a knowledge graph prototype are:

[0193] Step 1: Perform a preliminary evaluation of the output of the NLP tool:

[0194] Accuracy check: Check the entities and relationships automatically identified by the NLP tool to evaluate their accuracy.

[0195] Coverage check: Determines whether the NLP tool has covered all the required entities and relations.

[0196] Step 2: Data Audit:

[0197] Entity Audit: Conduct a detailed audit of each identified entity to see its specific role in the production process.

[0198] Relationship Audit: A more detailed review of the identified relationships, including but not limited to, whether they are all necessary or if any are missing.

[0199] Step 3: Manual correction:

[0200] Entity Correction: If you discover errors or omissions in step 1 or step 2, you can manually add or remove entities.

[0201] Relationship modification: Similarly, make necessary additions or deletions to the relationships.

[0202] Step 4: Evaluate again:

[0203] After all manual corrections were completed, accuracy and coverage were evaluated again.

[0204] It may be necessary to go back to step 1 and perform multiple iterations until satisfactory accuracy and coverage are achieved.

[0205] Step 5: Generate a knowledge graph prototype:

[0206] Use the corrected and verified entity and relationship data to generate a preliminary knowledge graph prototype.

[0207] The generated knowledge graph is tested to confirm its validity and accuracy.

[0208] S403. Use a reinforcement learning-based algorithm to update the knowledge graph in real time. Use a reinforcement learning algorithm to dynamically update the knowledge graph so that the graph can adapt to changes in the workshop environment.

[0209] First, a dynamic knowledge graph is constructed based on the multi-level shop floor state representation (MLWSR) generated in the previous steps and the preliminary knowledge graph prototype.

[0210] The dynamic knowledge graph construction process is as follows:

[0211] A. Constructing a multi-level workshop state representation (MLWSR):

[0212] First, list in detail all the factors and conditions in the workshop that may affect production. This includes equipment status (whether it is running, maintenance status, etc.), workload (current task volume, estimated completion time, etc.) and personnel allocation (who works in which position, etc.).

[0213] Through data collection and analysis, these factors are quantified into a vector or matrix form, namely MLWSR.

[0214] B. Build a preliminary knowledge graph prototype:

[0215] Next, model the various entities in the workshop (such as machines, personnel, products, etc.) and the relationships between them:

[0216] Entity definition: Make it clear which entities the nodes in the graph represent.

[0217] Relationship definition: Clarify what types of relationships the edges between nodes represent.

[0218] C. Merge Information:

[0219] After having MLWSR and preliminary knowledge graph, we need to integrate these two pieces of information into a unified framework:

[0220] Weight assignment: Assign weights to nodes and edges in the knowledge graph based on the data in MLWSR.

[0221] Attribute addition: Add the state information in MLWSR as the attribute of the node or edge to the knowledge graph.

[0222] D. Dynamic Update

[0223] At this point, we have a preliminary but well-structured dynamic knowledge graph.

[0224] This dynamic knowledge graph is updated in real time using a reinforcement learning-based algorithm. Whenever a new data point or event enters the system, the knowledge graph is updated accordingly.

[0225] Dynamic knowledge graph update rules:

[0226]

[0227] in, is the knowledge graph at time t+1, RL is the graph update function based on reinforcement learning, MLWSR t+1 It is a multi-level workshop status representation at time t+1.

[0228] This ensures that the knowledge graph is always up-to-date and accurate to support subsequent decision logic generation and interpretation.

[0229] The details of the above specific use of reinforcement learning are as follows:

[0230] 1. State definition: Define each node and edge as a state in the reinforcement learning model.

[0231] 2. Action definition: Define actions that can be applied to individual nodes and edges, such as adding, deleting, or updating entities / relationships.

[0232] 3. Reward function: Design a reward function to quantify the contribution of each action to the accuracy or effectiveness of the knowledge graph.

[0233] 4. Policy Iteration: Use policy iteration algorithms (such as Q-Learning or Deep Q-Network) to optimize reinforcement learning models.

[0234] 5. Real-time updates:

[0235] 5.1. When new data points or events enter the system, use the current reinforcement learning model to select the optimal action.

[0236] 5.2. Apply the selected actions to update the knowledge graph.

[0237] 5.3. Observe the rewards and update the reinforcement learning model based on the rewards.

[0238] S5. Build and train an interpretable reinforcement learning model, specifically by combining deep neural networks and decision trees to build a model that has both strong generalization capabilities and interpretability.

[0239] S501. Use a deep neural network to learn the state representation and dynamic knowledge graph of the workshop, and use a decision tree or rule engine to extract specific rules or patterns to explain the decision logic of the neural network, wherein the interpretable reinforcement learning model is expressed as follows:

[0240] f(x)=DNN(x)×DT(x)

[0241] Among them, f(x) represents the final decision function, DNN(x) represents the deep neural network part, and DT(x) represents the decision tree part;

[0242] S502, pre-training an interpretable reinforcement learning model using the historical data of the workshop and the data generated by the simulator,

[0243] S503. Fine-tune the model using real-time collected data and feedback, and use a multi-objective optimization algorithm to consider model performance and interpretability, wherein the multi-objective optimization algorithm is expressed as follows:

[0244]

[0245] Among them, μ represents the weight coefficient, and They represent the loss functions of performance and interpretability respectively, and θ represents the model parameters.

[0246] The construction steps are as follows:

[0247] A deep neural network (DNN) is used to learn the state representation of the workshop (obtained from the first part) and the dynamic knowledge graph (obtained from the second part).

[0248] Decision trees or rule engines are used to extract specific rules or patterns, which will be used to explain the decision logic of the neural network. The interpretable reinforcement learning model is as follows:

[0249] f(x)=DNN(x)×DT(x) (7)

[0250] Among them, f(x) is the final decision function, DNN(x) is the deep neural network part, and DT(x) is the decision tree part.

[0251] Among them, a multi-objective optimization algorithm is used to optimize performance and interpretability simultaneously during the training process. The steps of training and optimization are as follows:

[0252] A. Pre-train the interpretable reinforcement learning model using historical workshop data and data generated by the simulator.

[0253] B. Online training phase: Use real-time collected data and feedback to fine-tune the model. At the same time, use a multi-objective optimization algorithm to consider both model performance and interpretability. The specific steps are as follows:

[0254] The interpretable reinforcement learning model is pre-trained using historical workshop data and data generated by the simulator.

[0255] Online training phase: Use real-time collected data and feedback to fine-tune the model. At the same time, a multi-objective optimization algorithm is used to consider both model performance and interpretability. The multi-objective optimization function is:

[0256]

[0257] Among them, α is a weight coefficient, and are the loss functions for performance and interpretability respectively, and θ is the model parameter.

[0258] In this way, not only is the performance of the model optimized, but its decision-making process is also ensured to be explainable, thereby better serving the subsequent decision logic generation and interpretation stages.

[0259] S6. Generate decision logic and explain decision logic innovation based on interpretable reinforcement learning model. The strategy obtained through reinforcement learning is further converted into advanced, human-understandable decision rules, so that operators can understand and implement them more easily. The specific steps are as follows:

[0260] S601. Generate a strategy using an interpretable reinforcement learning model (derived from the third part). The strategy defines how to make the best decision given the workshop status and knowledge graph. Use decision trees, logical rules or natural language generation (NLG) methods to convert these strategies into human-understandable language or rules. The resulting decision logic function is:

[0261] IF(condition1∧condition2∧…)THEN action(9)

[0262] Among them, condition i is the decision condition and action is the corresponding operation.

[0263] Specifically, the above decision logic function uses a method similar to LIME (Local Interpretable Model Sensitivity) or SHAP (SHapley Additive exPlanations) to deeply explain the generated decision logic. The specific steps are as follows:

[0264] For each decision rule or operation generated, use the above method to explain its influencing factors, including which state variables or elements of the knowledge graph play a dominant role.

[0265] S602: Generate an explanation report, including an evaluation of the effectiveness and accuracy of each decision logic, and how to implement these logics in actual operations, wherein the SHAP value shown is represented as follows:

[0266]

[0267] Among them, φ jis the SHAP value of feature j, f(S) is the predicted value for a given feature set S, and N is the set of all features; S (feature set) specifically represents the various possible combinations of workshop status and operations. A specific S may include the status of a specific machine, the current task type, and the staffing of a specific workstation. By analyzing the impact on the model output (such as task scheduling decisions) with and without specific features (such as the status of a certain machine), the SHAP value determines the contribution of each factor (feature) to the final decision.

[0268] Through these two main steps, the generated decision logic can not only optimize the workshop operation but also be explainable and verifiable, thus better serving the subsequent experimental verification and performance evaluation stages.

[0269] Explanations): LIME works by approximating the model locally in the input space to understand how the model behaves on a single prediction. This typically involves generating a similar but simplified model (such as a linear model) that approximates the behavior of the target model in that region well.

[0270] SHAP (Shapley Additive Explanations): SHAP assigns a "contribution" to each feature based on the Shapley value in cooperative game theory. Specifically, it evaluates the average contribution of a feature in model prediction, taking into account all possible feature subsets.

[0271] S7. Build an experimental environment and evaluate the experimental performance based on the decision logic and the workshop digital twin model, using a physical-digital dual simulation environment to obtain a more accurate performance evaluation. The specific steps are as follows:

[0272] S701. Use hardware-in-the-loop technology to integrate actual workshop equipment and workshop digital twin models to form a physical-digital simulation environment and deploy decision logic and workshop digital twin models;

[0273] S702. In this environment, the decision logic and workshop digital twin model derived from the first four steps are deployed, and multi-objective optimization is used to consider multiple evaluation indicators such as production efficiency, energy efficiency, and equipment wear. The multi-objective optimization function is expressed as follows:

[0274] F(x) = α × efficiency + β × energy efficiency - γ × equipment wear (11)

[0275] Among them, α, β, and γ represent the weight coefficients of each indicator;

[0276] S703. Use deep learning and statistical methods to perform data analysis.

[0277] Use deep learning and statistical methods to perform complex data analysis, including time series analysis and anomaly detection, including:

[0278] During the experimental run, various data are collected, including production volume, energy consumption, equipment status, etc.

[0279] Use deep learning models for time series analysis to identify potential production bottlenecks or equipment failures, where the time series model is as follows:

[0280] h t =f W (h t-1 ,x t ) (12)

[0281] Among them, h t is the hidden state at time t, x t is the input at time t.

[0282] Specifically, the experimental design is as follows:

[0283] Design multiple experimental scenarios, including normal working conditions, abnormal situations (such as equipment failure, energy price fluctuations, etc.).

[0284] Specifically, production bottlenecks: If the model identifies a significant drop in production or an abnormal increase in energy consumption at a certain point in time or time period, this may be a sign of a production bottleneck. At this time, you can further check the status of other equipment or processes at this time to confirm and solve it.

[0285] Equipment failure: If the model predicts that the status data (such as temperature, vibration, noise, etc.) of a certain equipment is suddenly abnormal, this may be a precursor to an impending failure. Further maintenance or inspection can be scheduled to avoid potential downtime.

[0286] S704. Evaluate the success of the experimental results based on the collected data and the defined multi-objective optimization function, and generate a detailed experimental report.

[0287] Finally, the results are evaluated and reported, and the success of the experimental results is evaluated based on the collected data and the defined multi-objective optimization function. A detailed experimental report is generated, including a comprehensive evaluation of the effectiveness, accuracy and feasibility of the method.

[0288] Through the above steps, the solution aims to provide a comprehensive and detailed approach for validating and evaluating the various components and strategies developed in the previous phases.

[0289] S8. Use Docker containerization technology to deploy models and knowledge graphs, process and feedback data streams in real time, dynamically adjust decision logic, and use deep learning to generate models to detect abnormal behavior or decision errors. Use Docker containerization technology to achieve fast and scalable deployment of models and knowledge graphs. The specific steps are as follows:

[0290] (1) Use Docker to package the knowledge graph and decision logic model into a container.

[0291] (2) Use Kubernetes for container orchestration to ensure high availability and load balancing.

[0292] (3) Real-time data flow and feedback loop: Implement real-time data flow processing and feedback, and dynamically adjust decision logic, including:

[0293] a. Use Apache Kafka or a similar message queue system to establish real-time data streaming.

[0294] b. Implement a feedback loop to automatically adjust decision logic through real-time data.

[0295] Specifically, model monitoring and maintenance uses deep learning to generate models to detect possible abnormal behaviors or decision errors. The specific steps are as follows:

[0296] (1) Run model validity checks regularly.

[0297] (2) Collect real data of daily normal operation and compare it with actual behavior to detect anomalies, where the anomaly detection model is expressed as follows:

[0298]

[0299] Among them, L(x) is the loss function, x i and The data points are the actual and normal operation data points respectively. Through the above steps, the scheme aims to achieve efficient deployment, real-time monitoring and continuous optimization of decision logic and workshop digital twin model. The present invention can not only ensure the real-time performance of the system, but also make it well maintainable and explainable.

[0300] In summary, the present invention realizes efficient, intelligent and explainable production management through a workshop scheduling scheme that comprehensively applies digital twin technology, knowledge graph and explainable reinforcement learning. First, the data collection and preprocessing steps provide accurate and real-time basic data for the entire system. These data are further used to construct a digital twin model of the workshop, which accurately reflects the actual operating status of the workshop and provides reliable input for the subsequent reinforcement learning model. Then, the knowledge graph is used to describe and analyze the complex relationships within the workshop, thereby providing more advanced analysis and optimization means. On this basis, an explainable reinforcement learning model is designed and trained, which can not only generate effective scheduling strategies, but also provide clear explanations for each decision. Finally, experimental verification and performance evaluation are used to confirm the advantages of the scheme in improving production efficiency, reducing costs and enhancing decision-making transparency. Each step is designed to achieve higher scheduling efficiency and decision-making transparency, and there is a close logical connection and data flow between them, which together serve the core goal of the entire patent.

[0301] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0302] Although embodiments of the present invention have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0303] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0304] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0305] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0306] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A workshop scheduling method based on digital twin and knowledge graph reinforcement learning, characterized in that: The method comprises: S1. Establish a data receiving and storage structure to collect multimodal data, generate tags and metadata based on the multimodal data, and use the connectivity of the graph model to predict tags and metadata related to the existing data; Generating tags and metadata according to multimodal data specifically refers to: For each collected data point, a pre-trained classifier based on a graph model is used to generate labels, as shown below: L=f tag (D,G) Among them, L represents the generated label, f tag () represents the label generation function based on the graph model, D represents the original data, and G represents the existing knowledge graph; Finally, metadata is generated for each data point; S2. Perform data quality assessment and filtering on the generated tags and metadata. The data quality assessment is performed based on the data quality score, which is expressed as follows: Q=g quality (D,M,T) Among them, Q represents the data quality score, g quality represents the scoring function, M represents the metadata of the data, and T represents the time factor; S3. Design a multi-level workshop status representation and establish a workshop digital twin model, including: S301, identifying key data sources; S302, designing data fusion agents to collect data, wherein each data fusion agent is responsible for collecting a specific type of data; S303. Define a multi-level workshop status representation and build a workshop digital twin model, including: Generate a basic production data matrix using data collected by the data fusion agent Workflow Information Matrix Environmental parameter matrix ε and personnel dynamic matrix Then the basic production data matrix is ​​processed using time series decomposition Workflow Information Matrix Environmental parameter matrix ε and personnel dynamic matrix The workshop status characteristics are obtained, wherein the time series decomposition is expressed as: in, represents the processed basic production data matrix, λ represents the smoothing parameter of STL, and using these processed data matrices, a high-dimensional tensor T is used MLWSR Perform multi-level workshop state representation by forming a high-dimensional tensor T MLWSR Thus forming a digital twin model of the workshop; S4. Build a knowledge graph and implement dynamic updates, including: S401. Use natural language processing tools to analyze documents, identify entities and relationships, manually verify and adjust the output of natural language processing tools, and generate a preliminary knowledge graph prototype, as shown below: in, Represents the knowledge graph relationship, V represents the set of entities, and E represents the set of relationships; S402, constructing a dynamic knowledge graph according to the prototype of the knowledge graph; S403, using a reinforcement learning-based algorithm to update the dynamic knowledge graph in real time; S5. Build and train an interpretable reinforcement learning model; S6. Generate decision logic and explain decision logic based on the interpretable reinforcement learning model, including: S601. Generate decision logic using an interpretable reinforcement learning model, and convert the decision logic into language or rules, wherein the expression of the decision logic is as follows: IF(condition1∧condition2∧…∧condition i )THEN action Among them, condition i is the decision condition, and action is the corresponding operation; S602: Using the SHAP method, an in-depth explanation is given to the generated decision logic, wherein the SHAP value is expressed as follows: Among them, φ j represents the SHAP value of feature j, f(S) represents the predicted value of a given feature set S, N represents the set of all features, and S represents various combinations of shop floor states and operations; S7. Build the experimental environment and evaluate the experimental performance based on the decision logic and the workshop digital twin model, including: S701. Use hardware-in-the-loop technology to integrate actual workshop equipment and workshop digital twin models to form a physical-digital simulation environment and deploy decision logic and workshop digital twin models; S702. Design a multi-objective optimization function F(x) that takes into account multiple production and performance factors, expressed as follows: F(x) = α×efficiency+β×energy efficiency-γ×equipment wear Among them, α, β, and γ represent the weight coefficients of each indicator; S703. Use deep learning and statistical methods for data analysis; S704. Evaluate the success of the experimental results based on the collected data and the defined multi-objective optimization function, and generate a detailed experimental report.

2. The workshop scheduling method based on digital twin and knowledge graph reinforcement learning according to claim 1 is characterized in that: The data quality indicators for evaluating the data quality score Q include: completeness, consistency, timeliness and accuracy; The evaluation data quality indicator scores calculated according to the evaluation data quality indicators include: completeness score, consistency score, timeliness score and accuracy score; The data quality score Q is obtained as: Q=a*Completeness Score+b*Consistency Score+c*Timeliness Score+d*AccuracyScore Right now g quality (D,M,T)=a*Completeness Score+b*Consistency Score+c*Timeliness Score+d*Accuracy Score Among them, a, b, c, d represent the weights of completeness score, consistency score, timeliness score and accuracy score respectively, and a+b+c+d=1; Completeness Score represents the completeness score; Consistency Score represents the consistency score; Timeliness Score represents the timeliness score; Accuracy Score represents the accuracy score.

3. The workshop scheduling method based on digital twin and knowledge graph reinforcement learning according to claim 1 is characterized in that: The data receiving and storing structure comprises: Multimodal sensor module, used to collect data from multimodal sensors and standardize the data collected by IoT devices; Real-time data preprocessing module, used to preprocess data; A secure data transmission module, used for encrypting and transmitting data; A data receiving and decoding module is used to decode and analyze the received data on the central server; A dataset storage module for storing real-time data and configuring data with a relational data repository; Data access module, used to develop RESTful APIs for external applications and services to access data; Data analysis module, used to perform in-depth analysis of data.

4. The workshop scheduling method based on digital twin and knowledge graph reinforcement learning according to claim 1 is characterized in that: The mathematical model of the data fusion agent is expressed as follows: DFA i =φ i (DS i ,t i ) Among them, DFA i represents the i-th data fusion agent, φ i Denotes the data collection function, DS i represents the responsible data source, τ i Indicates the data collection interval.

5. The workshop scheduling method based on digital twin and knowledge graph reinforcement learning according to claim 1 is characterized in that: The dynamic knowledge graph update rule is expressed as follows: in, represents the knowledge graph at time t+1, RL represents the graph update function based on reinforcement learning, MLWSR t+1 Represents the multi-level workshop status representation at time t+1.

6. The workshop scheduling method based on digital twin and knowledge graph reinforcement learning according to claim 5 is characterized in that: The step S5 specifically includes: S501. Use a deep neural network to learn the state representation and dynamic knowledge graph of the workshop, and use a decision tree or rule engine to extract specific rules or patterns to explain the decision logic of the neural network, wherein the interpretable reinforcement learning model is expressed as follows: f(x)=DNN(x)×DT(x) Among them, f(x) represents the final decision function, DNN(x) represents the deep neural network part, and DT(x) represents the decision tree part; S502, pre-training an interpretable reinforcement learning model using the historical data of the workshop and the data generated by the simulator, S503. Fine-tune the model using real-time collected data and feedback, and use a multi-objective optimization algorithm to consider model performance and interpretability, wherein the multi-objective optimization algorithm is expressed as follows: Among them, μ represents the weight coefficient, and They represent the loss functions of performance and interpretability respectively, and θ represents the model parameters.

7. The workshop scheduling method based on digital twin and knowledge graph reinforcement learning according to claim 1 is characterized in that: The deep learning and statistical methods include time series analysis and anomaly detection, wherein the model of the time series analysis is represented as: h t =f W (h t-1 ,x t ) Among them, h t represents the hidden state at time t, x t represents the input at time t, h t-1 represents the hidden state at time t-1.

8. The workshop scheduling method based on digital twin and knowledge graph reinforcement learning according to claim 1 is characterized in that: It also includes S8, using Docker containerization technology to deploy models and knowledge graphs, real-time processing and feedback of data streams, and dynamically adjusting decision logic, and using deep learning to generate models to detect abnormal behavior or decision errors.

Citation Information

Cited By

  • A shared bottleneck reentrant scheduling reinforcement learning training method based on digital twin operation feedback

    CN123014066A