Intelligent factory decision support system and method thereof

Through the integration of data acquisition, analysis and optimization modules of the intelligent factory decision support system, combined with layered reinforcement learning and multi-agent collaborative algorithm, the problems of data fusion and knowledge precipitation in smart factories are solved, and multi-objective optimization in complex scenarios is achieved and real-time scheduling improves the system's self-perception and decision-making efficiency.

CN118982437APending Publication Date: 2024-11-19GUANGDONG JINGGONG INTELLIGENT SYST CO LTD
View PDF 0 Cites 13 Cited by

Patent Information

Application Number
CN202411043857.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Smart factories have many challenges in data fusion, knowledge precipitation and optimization decision-making, including data silos, knowledge dispersion, and decision-making difficulties in complex scenarios. Existing reinforcement learning algorithms are difficult to deal with multi-device collaboration and multi-constraint optimization.

Method used

An intelligent factory decision support system is proposed, which adopts the integration of data acquisition, data analysis, optimization decision module and knowledge base module. Through layered reinforcement learning-driven multi-agent collaborative optimization algorithm, combined with knowledge guidance strategies, multi-objective optimization and Real-time scheduling.

Benefits of technology

It realizes the full process closed loop of data-driven optimization decision-making, knowledge empowerment and human-computer interaction, improves the system's self-perception, self-learning and self-optimization capabilities, simplifies the state action space, improves generalization performance and adaptability, and enhances the The timeliness and robustness of decision-making.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982437A_ABST
    Figure CN118982437A_ABST
Patent Text Reader

Abstract

The invention discloses an intelligent factory decision support system. The system comprises a data acquisition module, a data analysis module, an optimization decision module, a man-machine interaction module and a knowledge base module, the data acquisition module is used for acquiring data of production equipment, materials, environment and the like and uploading the data to the database; the data analysis module is used for analyzing and mining the collected data and extracting features; the optimization decision module calls a multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning, solves a multi-objective optimization model and outputs a decision scheme; the man-machine interaction module presents a decision result through a visual interface for a user to adjust; and the knowledge base module stores domain knowledge and provides support for optimization decision making. The optimization decision module realizes global resource configuration through reinforcement learning, the man-machine interaction module supports expert knowledge injection and decision closed loop, and the knowledge management module realizes knowledge accumulation and optimization through ontology reasoning and parallel collaboration among multiple modules, so that the limitation of a single technology is avoided, and the gain effect of each module is amplified.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of decision-making systems, and more specifically, to an intelligent factory decision support system and method thereof. Background Art

[0002] Intelligent manufacturing is the core of the new round of industrial revolution. The key is to make full use of emerging technologies such as industrial big data and artificial intelligence, deeply explore the value of data, and realize intelligent optimization and decision-making of the entire manufacturing process. At present, the development of information infrastructure such as industrial Internet, 5G, and edge computing has provided a foundation for the real-time collection, transmission, and processing of massive industrial data. The progress of artificial intelligence technologies such as machine learning and knowledge graphs has made it possible to quickly extract features from data, build models, and optimize decisions.

[0003] However, the current construction of smart factories still faces many technical challenges:

[0004] (1) Lack of effective integration and utilization mechanism for industrial big data. Industrial data are of various types (structured, semi-structured, unstructured), data sources are scattered (equipment, MES, ERP, CRM, etc.), and there is a lack of unified semantic representation and association mechanism between different systems. The data island problem is serious, making it difficult to form a global data view, and the value of data has not been fully released.

[0005] (2) Insufficient accumulation of industrial knowledge, making it difficult to reuse optimization decision-making experience. Industrial knowledge such as mechanism models, operating procedures, and expert experience are scattered in documents, systems, and human brains. They lack formal representation methods and are difficult to accumulate into computable knowledge models. As a result, optimization decisions are highly dependent on manpower, have long cycles, and are difficult to cope with rapidly changing market demands.

[0006] (3) The complexity of industrial scenarios leads to a combinatorial explosion in optimization decisions. The interweaving of uncertain factors such as equipment failure, production capacity fluctuations, material shortages, and personnel mobility makes factory operations in a dynamic and complex environment. Traditional optimization methods such as heuristic rules and mathematical programming are prone to dimensional disasters and lack generalization capabilities in large-scale scheduling scenarios across workshops and factories.

[0007] Especially in the key field of industrial scheduling optimization, the existing technology has obvious shortcomings. Traditional industrial scheduling mainly adopts mathematical programming, meta-heuristic search and other methods, such as mixed integer programming, genetic algorithm, ant colony algorithm, etc. Most of these methods are based on offline modeling, which makes it difficult to handle dynamic emergencies; the optimization goal is single, and it is difficult to take into account multi-dimensional constraints such as equipment, process, logistics, and energy; the search space explodes exponentially with the scale of the problem, and it is difficult to meet the timeliness requirements of real-time scheduling. In recent years, deep reinforcement learning has provided a new idea for solving large-scale sequential decision-making problems. Compared with supervised learning, reinforcement learning does not require a large amount of labeled data, and learns from experience through autonomous exploration of the agent. However, most of the existing reinforcement learning algorithms are designed for single agents, making it difficult to model multi-device collaboration; strategy learning adopts an end-to-end approach, which makes it difficult to use existing scheduling knowledge; facing the high-dimensional state-action space of industrial scenarios, the training time is long and the sample efficiency is low.

[0008] Therefore, there is an urgent need for an intelligent decision-making system that integrates industrial big data, artificial intelligence, and expert knowledge to build a full-process solution from data collection, feature extraction, knowledge modeling to optimized decision-making. In particular, it is necessary to innovate algorithm frameworks and mechanisms for the core optimization problem of industrial scheduling, and achieve large-scale collaborative scheduling optimization under the premise of meeting multiple constraints and strong real-time. Summary of the invention

[0009] The present invention is proposed in this context, aiming to address the limitations of the existing technology and provide a smart factory decision support system and method, which, through the innovative integration of data aggregation, knowledge empowerment, hierarchical optimization, and multi-agent collaborative mechanism, provides a possibility for solving the above-mentioned technical problems and accelerating the development of smart manufacturing.

[0010] The present invention provides a smart factory decision support system, which includes: a data acquisition module, a data analysis module, an optimization decision module, a human-computer interaction module and a knowledge base module; the data acquisition module is used to collect data such as production equipment, materials, and environment and upload them to a database; the data analysis module is used to analyze and mine the collected data and extract features; the optimization decision module calls a multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning to solve a multi-objective optimization model and output a decision plan; the human-computer interaction module presents the decision results through a visual interface for users to adjust; the knowledge base module stores domain knowledge and provides support for optimization decisions.

[0011] Specifically, the data acquisition module includes: a sensor submodule, which is used to collect equipment status data through vibration sensors, temperature sensors, current sensors, etc.; an RFID submodule, which is used to collect material data through the RFID submodule; and a data transmission submodule, which is used to transmit the collected data to the database server via industrial Ethernet.

[0012] Specifically, the data analysis module includes: a data cleaning unit, which is used to perform pre-processing such as denoising and normalization on the data; a feature extraction unit, which is used to extract statistical features and frequency domain features of the data; and a machine learning modeling unit, which is used to build equipment health assessment and quality prediction models.

[0013] Specifically, the human-computer interaction module includes: a visualization unit, which uses a web framework and a chart library to develop an interactive dashboard; an interaction design unit, which provides functions such as dragging and drilling; and a permission management unit, which implements functional permission control based on RBAC.

[0014] Specifically, the knowledge base module includes: an ontology construction unit, which defines domain knowledge using ontology language; a semantic storage unit, which stores knowledge through a graph database; and a knowledge update unit, which updates the optimization results into the database as new knowledge.

[0015] Specifically, the optimization decision module is based on a multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning, which includes: a hierarchical reinforcement learning framework, defining K levels, each level corresponding to a Markov decision process (S k ,A k ,P k ,R k ), where S k is the state space, A k is the action space, P k is the state transition probability, R k is the reward function;

[0016] Use HDQN algorithm to learn the optimal strategy for each layer The Q function is where θ k is the Q network parameter, θ′ k The target network parameter γ∈[0,1] is the discount factor;

[0017] Multi-agent collaborative optimization, define each layer N k agents, and the observation of agent i is Action The local strategy is

[0018] AC-MARL algorithm is used to train multi-agent collaborative strategies, including Actor networks and Critic Network in is the joint observation of all agents, For joint action, is the Actor network parameter of agent i, is the Critic network parameter, the objective function is the reward function of agent i:

[0019] Knowledge-guided strategy search, define the knowledge-guided function f(s, a; Φ), ​​where Φ is the guided knowledge queried from the knowledge base. Embed f into the reinforcement learning objective to obtain the modified objective function is the guidance knowledge obtained from the knowledge base at time t.

[0020] Specifically, the detailed execution steps of the hierarchical reinforcement learning driven multi-agent collaborative optimization algorithm are as follows:

[0021] 1. Input the number of layers K and the number of agents in each layer N k , state space S k , action space A k , the reward function R k , discount factor γ, knowledge base Φ;

[0022] 2. For k from 1 to K, execute:

[0023] 2.1) Define and randomly initialize the k-layer HDQN network Q k (s,a;θ k ), target network Q k (s,a;θ′ k ), parameter θ k ,θ′ k ;

[0024] 2.2) Define and randomly initialize the Actor network of each agent i in layer k Critic Network Q i,k (o k ,a k ; φ i,k ), parameter θ i,k ,φ i,k ;

[0025] 2.3) Initialize the experience playback buffer D k , set the batch size B i Learning rate α;

[0026] 2.4) Repeat until the algorithm converges:

[0027] 2.5) Output k-layer trained Actor network That is, the multi-agent collaborative strategy π at level k k ;

[0028] 3. Output the optimal hierarchical strategy π obtained by hierarchical multi-agent reinforcement learning * =(π 1 ,π 2,...,π K ).

[0029] Specifically, in step 2.1), the Q network Q of the k-layer HDQN k (s,a;θ k ) adopts a two-layer neural network structure, including the input layer, the first hidden layer (H1 neurons), the second hidden layer (H2 neurons) and the output layer, where the input is the state s∈S k , the output is all possible actions a∈A k The Q value estimation, the hidden layer activation function is the ReLU function, that is, f(x) = max(0,x), the output layer activation function is the linear function, and the loss function is the mean square error:

[0030] where (s, a, r, s′) is the sum of the sums from the experience playback D k Transfer samples sampled in .

[0031] Specifically, in step 2.2), the Actor network of agent i and Critic Network Q i,k (o k ,a k ; φ i,k ) both use a two-layer neural network structure, and the input of the Actor network is the local observation of the agent. The output is the action distribution parameters, including mean μ and variance σ, action From the Gaussian distribution N(μ,σ 2 ) is sampled: The input of the Critic network is the joint observation o of all agents k and joint action a k , the output is the Q value estimate, that is in A joint strategy for the Actor network of all agents.

[0032] A smart factory decision support method based on the system comprises the following steps:

[0033] 1) Through the data acquisition module, data on production equipment, materials, environment, etc. are collected and uploaded to the database;

[0034] 2) Analyze and mine the collected data and extract features through the data analysis module;

[0035] 3) Through the optimization decision module, the multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning is called to solve the multi-objective optimization model and output a decision plan;

[0036] 4) Presenting the decision results through the human-computer interaction module using a visual interface for user adjustment;

[0037] 5) Storing domain knowledge through the knowledge base module to provide support for optimization decisions;

[0038] 6) The optimization results are fed back to the knowledge base module to update the knowledge.

[0039] The present invention has the following beneficial effects:

[0040] 1. First, the seamless connection of key modules such as data collection, data analysis, optimization decision-making, human-computer interaction, and knowledge management has formed a full-process closed loop of data-driven optimization, knowledge-enabled decision-making, and human-computer interaction verification, which greatly enhances the self-perception, self-learning, self-optimization, and self-evolution capabilities of industrial systems. The data collection module uses heterogeneous sensors to achieve all-round state perception of equipment, processes, materials, and environment. The data analysis module mines data hidden patterns through machine learning algorithms. The optimization decision-making module achieves global resource allocation through reinforcement learning. The human-computer interaction module supports expert knowledge injection and decision-making closed loops. The knowledge management module achieves knowledge accumulation and optimization through ontology reasoning. The parallel collaboration between multiple modules not only avoids the limitations of a single technology, but also amplifies the gain effect of each module.

[0041] Secondly, the present invention creatively proposes a multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning, which cleverly integrates the ideas of divide and conquer, collaboration, migration, and guidance into the reinforcement learning framework. Traditional single-agent reinforcement learning is difficult to handle complex optimization problems such as smart factory scheduling. However, this algorithm divides the original problem into multiple sub-problems through a hierarchical architecture, simplifying the state-action space; through multi-agent parallel exploration and collaborative learning in each layer, it accelerates strategy search; through the nesting of Markov decision processes and strategy migration across layers, it realizes the evolution from local optimization to global optimization; at the same time, the use of knowledge guidance avoids blind exploration and further improves the convergence speed. The superposition and synergy of multiple mechanisms make the generalization performance and adaptability of the hierarchical multi-agent algorithm far surpass traditional optimization algorithms.

[0042] In addition, the present invention uses digital twins as the basis and improves the timeliness and robustness of decision-making through virtual-real interaction. The data acquisition module and the industrial Internet of Things synchronize the factory operation data in real time, and build a digital mirror of the factory in the virtual space. Core modules such as data analysis, simulation optimization, and knowledge management are all carried out on the digital twin, which greatly reduces the decision-making verification cost under complex working conditions. The human-computer interaction module supports visual monitoring and interactive control of the virtual factory, and managers can correct optimization deviations in time. At the same time, the optimization decision interface is encapsulated into a Web service, which can be flexibly distributed to the workshop terminal through new information infrastructure such as 5G and edge computing to realize the real-time execution of scheduling instructions. The integration of virtual and real improves the efficiency of "strategizing", allowing the decision-making system to cope with the complexity and variability of industrial scenarios at a lower cost, faster speed, and higher robustness.

[0043] During the implementation of the system, the modules cooperated well, but there were also contradictions and conflicts. There were certain conflicts between the diversity of data sources and the consistency of knowledge representation, and between the real-time nature of online learning and the accuracy of offline training. This system realizes the semantic association between heterogeneous data and domain ontology by building a mapping mechanism of industrial data, information and knowledge; and realizes the dynamic update of training and reasoning through incremental learning of algorithms and parameter service of policy networks. Contradictions are transformed into collaboration, and the system thus has the flexibility and adaptability of data fusion and online learning. In addition, the distributed architecture also provides performance guarantees for the system to cope with massive data and high concurrent requests.

[0044] In summary, the present invention has carried out systematic innovative design in terms of fusion optimization, human-machine hybrid intelligence, virtual-reality interaction, and distributed parallelism, and has constructed an intelligent industrial operating system with synergistic efficiency enhancement and superimposed evolution of multiple technologies. It has provided a theoretical and practical carrier for the new generation of industrial intelligent empowerment applications, and is expected to accelerate the large-scale implementation of emerging technologies such as industrial big data and artificial intelligence in intelligent manufacturing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a schematic diagram of the framework of an intelligent factory decision support system of the present invention. DETAILED DESCRIPTION

[0046] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation, structure, features and effects of a smart factory decision support system proposed according to the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0047] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0048] The specific scheme of a smart factory decision support system provided by the present invention is described in detail below with reference to the accompanying drawings.

[0049] Example 1

[0050] See also Figure 1 The present invention proposes a smart factory decision support system, which includes a data acquisition module 1, a data analysis module 2, an optimization decision module 3, a human-computer interaction module 4 and a knowledge base module 5. Through the seamless integration of key steps such as data acquisition, analysis, optimization, interaction, knowledge accumulation, and the innovative application of a multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning in the optimization decision module 3, a complete set of intelligent decision-making solutions is formed.

[0051] Specifically, the data acquisition module 1 includes a sensor submodule, an RFID submodule and a data transmission submodule. The sensor submodule is arranged on the key equipment and production lines in the production workshop, including PCB356A12 vibration sensor, PT100 temperature sensor, CLSM-100 current sensor, etc., which are used to collect the operating status data of the equipment in real time. The RFID submodule attaches RFID electronic tags to raw materials, semi-finished products and finished products respectively, and collects data through the ImpinjR420 reader / writer to achieve accurate traceability and inventory of materials. The data transmission submodule uploads the collected multi-source heterogeneous data to the OracleExadata database server in real time through the Profinet industrial Ethernet for storage, ensuring high reliability and low latency transmission of data.

[0052] Data analysis module 2 includes three sub-units: data cleaning, feature extraction, and machine learning modeling. The data cleaning unit performs denoising, anomaly detection, and normalization on the collected raw data. Denoising uses median filtering to remove random noise and pulse interference, anomaly detection uses the 3σ criterion to identify and remove outliers, and normalization uses the maximum and minimum method to uniformly map the data to the [0,1] interval. The feature extraction unit uses wavelet transform to perform time-frequency analysis on time series data, extracts statistical features (mean, variance, kurtosis, etc.) and energy distribution features (wavelet entropy, energy spectrum, etc.) of wavelet coefficients, and uses Fourier transform to perform spectrum analysis on frequency domain data to extract features such as characteristic frequency and frequency band energy of the spectrum graph. The machine learning modeling unit uses support vector machine SVM to build an equipment health assessment model and random forest RF to build a product quality prediction model. The model hyperparameters (SVM penalty factor C, kernel function parameter γ, RF decision tree number n_estimators, minimum node sample number min_samples_split, etc.) are optimized through grid search to improve the classification / regression performance of the model.

[0053] The human-computer interaction module 3 includes three sub-units: visualization, interactive design, and permission management. The visualization unit develops a visual analysis interface based on the Vue.js front-end framework and the ECharts chart library, and intuitively presents information such as equipment operation status, production KPIs, and decision results through line charts, bar charts, scatter charts, radar charts, and other charts. The interactive design unit provides data exploration functions such as information clustering, drill-down, linkage, pan and zoom, and uses humanized interactive methods such as dragging, right-clicking, and floating to facilitate users to explore data from different dimensions and granularities, thereby improving the usability of the system. The permission management unit is based on

[0054] The RBAC (Role-Based Access Control) model customizes corresponding functional modules and data permissions for different user roles (equipment operators, workshop supervisors, decision-making administrators, etc.) to ensure system security and data privacy.

[0055] The knowledge base module 4 includes three sub-units: ontology construction, semantic storage and knowledge update. The ontology construction unit uses the ontology modeling language OWL (Web Ontology Language) to define the core concepts in the smart factory field (such as equipment, materials, processes, products, etc.), concept attributes (such as equipment model, power, speed, etc.) and relationships between concepts (such as equipment production products, process use equipment, etc.) to form a domain ontology knowledge base. The semantic storage unit uses the graph database Neo4j to store the factory knowledge graph, organizes knowledge in the form of nodes (representing conceptual entities) and edges (representing semantic associations between entities), and the graph database supports flexible and efficient graph query, reasoning, and mining analysis. The knowledge update unit defines the knowledge update trigger rules (such as new equipment installation, process parameter changes, product model switching, etc.). When the rule conditions are met, the optimization decision results (such as new equipment configuration parameters, process optimization parameters, etc.) are automatically added to the knowledge base to achieve continuous accumulation and optimization of knowledge.

[0056] The optimization decision module 5 is the core of the present invention. It adopts a multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning to achieve global optimal scheduling decisions in a complex and changeable production environment. The algorithm first adopts a hierarchical reinforcement learning framework to divide the complex scheduling optimization problem into K levels, each of which focuses on scheduling subtasks of different granularity. Each level corresponds to a Markov decision process (S k ,A k ,P k ,R k ), where S k A is the state space of the k-layer, describing the environmental status of the subtask (such as the working status of each device / production line, the progress of task completion, etc.); k is the k-layer action space, describing the scheduling actions of subtasks (such as task allocation, equipment start and stop, etc.); P k :S k ×A k →S k is the state transition probability, characterized in state s k Take action a k Then transfer to the next state s' k The probability of k :S k ×A k →R is the reward function, measuring the state s k Take action a kInstant rewards (such as increased production and equipment utilization in each scheduling cycle). In the hierarchical framework, the optimization target of the kth layer is given by the k+1th layer, and the optimization result of the kth layer is used as the constraint condition of the k-1th layer. The multiple layers achieve collaborative optimization through target decomposition and constraint transfer. In each layer, the hierarchical deep reinforcement learning algorithm HDQN (Hierarchical DeepQ-Network) is used to learn the optimal scheduling strategy. HDQN uses a deep neural network to approximate the function Q k (s,a;θ k ), where s is the state, a is the action, θ k is the k-th layer Q network parameter. k (s,a;θ k )'s update target is:

[0057]

[0058] That is, take action a in state s and then follow the expected cumulative return of the optimal strategy. k is the target network parameter, and γ∈[0,1] is the discount factor. HDQN uses a dual Q network and experience replay mechanism to improve training stability, that is, to maintain a behavior Q network Q at each layer k. k (s,a;θ k ) and a target Q network Q k (s,a;θ′ k ), the behavior network is used to generate training data and select actions, the target network is used to calculate the training target value, and the two network parameters θ k and θ′ k , periodic synchronization.

[0059] In addition, HDQN uses an experience replay buffer D k Store historical transfer samples (s, a, r, s′) and extract them from D during training k The random sampling of small batches of data breaks the correlation between data. Considering that there are multiple devices (sensors, robots, AGVs, etc.) that make parallel decisions and influence each other in the smart factory workshop, the present invention introduces a multi-agent reinforcement learning paradigm in each layer of scheduling tasks to optimize the global scheduling performance through collaborative learning among agents. Specifically, the kth layer contains N k agents, each agent i based on its local observation (such as the state of the device, the behavior of the surrounding agents, etc.) and strategy functions Generate Schedule Action

[0060] As described, the agents take joint actions Get instant rewards k and the next state observation The present invention adopts the actorcritic multi-agent reinforcement learning algorithm AC-MARL to train the collaborative strategy of the agents. Specifically: each agent i contains an actor network and a critic network in is the joint observation of all agents, is a joint action, and θ i ,k and k are the actor and critic network parameters of agent i respectively.

[0061] Actor network output based on local observation The action probability distribution π i,k , the critic network estimates the joint observation-action pair (o k ,a k )’s global value function Q i,k The optimization goal of agent i is to maximize the expected cumulative reward in is the local reward. AC-MARL optimizes the actor network θ i,k The generated action distribution is such that it can improve the critic network φ as much as possible i,k The global value of evaluation guides the agents to learn collaborative strategies.

[0062] In addition, the present invention also adopts knowledge-guided strategy search, using prior knowledge such as empirical rules and optimal practices in the knowledge base to guide reinforcement learning to explore better strategies and accelerate algorithm convergence.

[0063] Define the knowledge guidance function f(s,a;Φ), where Φ is the guidance knowledge queried from the knowledge base, embed f into the reinforcement learning objective, and obtain the modified objective function:

[0064]

[0065] Guide knowledge According to the state-action pair of each decision step Real-time generation, common forms include: target actions for specific states (for example, performing maintenance actions when equipment fails), evaluation of states (such as cautious operation in critical load states), preferences for exploration directions (such as trying different process parameters for new equipment), etc. In the AC-MARL training process, the knowledge-guided reward f is superimposed on the environmental reward Correcting the Agent’s Policy Gradient The search direction is kept consistent with the prior knowledge to avoid blind exploration. The knowledge-guided strategy adopted by the present invention helps the reinforcement learning algorithm to quickly find feasible solutions in complex environments and improve the cold start performance of the system.

[0066] In summary, the hierarchical reinforcement learning-driven multi-agent collaborative optimization algorithm of the present invention performs the following steps in each scheduling cycle:

[0067] 1. Input the number of layers K and the number of agents in each layer N k , state space S k , action space A k , the reward function R k , discount factor γ, knowledge base Φ;

[0068] 2. For k from 1 to K, execute: define and randomly initialize the HDQN network Q with k layers k (s,a;θ k ), target network Q k (s,a;θ′ k ), contains L hidden layers, the lth layer has M kl neurons, the hidden layer activation function is ReLU, that is, f(x) = max(0,x), the output layer activation function is a linear function, and the loss function is the mean square error

[0069] Define and randomly initialize a k-layer actor network for each agent i Output action distribution π i,k The mean and variance of , and the critic network Q i,k (o k ,a k ; φ i,k ), a two-layer neural network structure is used, the first layer is a fully connected layer, the second layer is an LSTM layer, and the input is the joint observation o of all agents k and joint action a k , the output is the Q value estimate;

[0070] Initialize the experience replay buffer D k , capacity is C k , the batch size is B k , the learning rate is α k ; o Repeat until the algorithm converges: At t = 0, randomly initialize the initial state of k layers - For t = 0, 1, 2, ..., execute: query the current joint observation-action pair from the knowledge base Φ Guidance knowledge For i=1,2,…,N k , agent i is based on local observation Using the actor network μ i,k Output action distribution π i,k , and from π i,k Medium sampling action All agents perform joint actions The environment enters the next state and continues to execute subsequent steps to update network parameters and optimization strategies. The above describes the innovative optimization decision module of the present invention in the field of intelligent factory production scheduling. Through hierarchical reinforcement learning and multi-agent collaborative algorithm, combined with knowledge-guided technology, it aims to achieve global optimal production scheduling decisions and adapt to complex and changeable industrial production environments.

[0071] In each iteration of the algorithm, the following steps are performed:

[0072] The environment enters the next state Return joint rewards at the same time

[0073] Each agent observes a new state Calculating local rewards Transfer samples Deposit D k ;

[0074] From D k Random sampling B k Samples where j = 1, 2, ..., B k ;

[0075] Query the knowledge base and obtain i ,j k arrive Knowledge guidance information Computational Knowledge Guide Rewards Corrected target Q value:

[0076]

[0077] Update the critic network to minimize the loss:

[0078]

[0079] Update the actor network, based on the deterministic policy gradient theorem calculation:

[0080]

[0081] Update θ using Adam optimizer i,k :

[0082]

[0083] per ton kStep synchronization once the target network parameter θ′ k ←θ k ;

[0084] Output k layers of trained actor network μ i,k , forming a collaborative strategy for each agent in level k

[0085] 3. Output the optimal hierarchical scheduling strategy π for cross-layer collaboration * =(π1,π2,…,π K ).

[0086] These steps accurately describe how to gradually train the optimal production scheduling decision model in a smart factory environment through hierarchical reinforcement learning and multi-agent collaborative optimization algorithm combined with knowledge-guided strategies.

[0087] The present invention also provides a smart factory decision support method based on the above system, the main steps of which are as follows:

[0088] Step 1: Through the data acquisition module 1, use sensors, radio frequency identification and other technologies to collect key data such as equipment operation parameters, material consumption, product quality, etc. in the smart factory workshop in real time, and transmit them to the database through a standardized interface for fusion storage;

[0089] Step 2: Through the data analysis module 2, the collected heterogeneous data is cleaned, features are extracted, and correlation analysis is performed to build data models such as equipment health prediction, product quality analysis, and process parameter optimization to assist in optimizing the entire production process;

[0090] Step 3: By optimizing the decision module 3, the multi-source data and domain knowledge are converted into a Markov decision process, and the hierarchical reinforcement learning-driven multi-agent collaborative optimization algorithm of the present invention is used to solve the optimal scheduling decisions of each level of equipment, production line, and workshop under complex working condition constraints;

[0091] Step 4: Through the human-computer interaction module 4, the optimization results are presented to the management personnel in the form of charts, dashboards, etc. using visual analysis technology. The management personnel evaluate and adjust the optimization plan through interactive operations and generate executable scheduling instructions;

[0092] Step 5: Return the optimized decision results to the knowledge base module 5, and use technologies such as ontology reasoning and graph mining to integrate, evaluate, and update the knowledge, so as to continuously accumulate and improve intelligent decision-making knowledge.

[0093] The intelligent factory decision support system of the present invention seamlessly integrates offline data analysis with online real-time optimization through data-driven, knowledge-guided, and human-machine collaborative methods, significantly improving the decision-making level of intelligent factories under complex working conditions. In particular, for the difficult problem of intelligent scheduling in the industrial field, a multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning is innovatively proposed. The algorithm cleverly combines hierarchical learning architecture, multi-agent collaborative mechanism, and knowledge-guided strategy, simplifies the complexity of the problem by divide-and-conquer, improves the optimization performance by collaboration, and guides the acceleration of strategy search, providing new ideas for real-time optimization decisions of complex systems at the theoretical and application levels. The system adopts modular design and standardized interfaces, has good versatility, scalability, and practicality, can be flexibly adapted to intelligent factories of different scales and industries, and provides strong support for the implementation of intelligent manufacturing in industrial scenarios.

[0094] In terms of hardware resources, the system of the present invention can be deployed on a general x86 server or an industrial edge computing gateway. The sensors mainly use industrial vibration sensors (such as PCB356A12, frequency response 0.5Hz850℃, accuracy ±0.15℃), high-bandwidth current sensors (such as CLSM-100, frequency response DC~150kHz, accuracy better than 0.5%), etc. The database uses a combination of relational database (such as OracleExadata) and time series database (such as InfluxDB), and the knowledge base uses a graph database (such as Neo4j) for storage. Both software and hardware are deployed in a modular and plug-in manner, and can be flexibly configured according to actual application scenarios.

[0095] The smart factory decision support system and its method described above integrate industrial big data, artificial intelligence, and expert knowledge to form a complete set of intelligent optimization solutions for complex systems. The system fully utilizes the value of data and knowledge, enhances the self-perception, self-learning, self-adaptation, and self-decision-making capabilities of the industrial system, and is of great significance for improving the digitalization, networking, and intelligence level of the manufacturing industry and promoting industrial transformation and upgrading.

[0096] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, replacement, and improvement made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A smart factory decision support system, characterized in that: The system includes: a data acquisition module, a data analysis module, an optimization decision module, a human-computer interaction module and a knowledge base module; the data acquisition module is used to collect production equipment, materials, and environmental data and upload them to a database; the data analysis module is used to analyze and mine the collected data and extract features; the optimization decision module calls a multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning to solve a multi-objective optimization model and output a decision plan; the human-computer interaction module presents the decision results through a visual interface for users to adjust; the knowledge base module stores domain knowledge and provides support for optimization decisions.

2. The intelligent factory decision support system according to claim 1, characterized in that: The data acquisition module includes: a sensor submodule, which is used to collect equipment status data through a vibration sensor, a temperature sensor, and a current sensor; an RFID submodule, which is used to collect material data through the RFID submodule; and a data transmission submodule, which is used to transmit the collected data to a database server through industrial Ethernet.

3. The intelligent factory decision support system according to claim 1, characterized in that: The data analysis module includes: a data cleaning unit for performing denoising and normalization preprocessing on the data; a feature extraction unit for extracting statistical features and frequency domain features of the data; and a machine learning modeling unit for building equipment health assessment and quality prediction models.

4. The intelligent factory decision support system according to claim 1, characterized in that: The human-computer interaction module includes: a visualization unit, which uses a web framework and a chart library to develop an interactive dashboard; an interaction design unit, which provides drag and drop and drill functions; and a permission management unit, which implements function permission control based on RBAC.

5. The intelligent factory decision support system according to claim 1, characterized in that: The knowledge base module includes: an ontology construction unit, which defines domain knowledge using ontology language; a semantic storage unit, which stores knowledge through a graph database; and a knowledge update unit, which updates the optimization results into the database as new knowledge.

6. The intelligent factory decision support system according to claim 1, characterized in that: The optimization decision module is based on a multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning. The algorithm includes: a hierarchical reinforcement learning framework, defining K levels, each level corresponds to a Markov decision process (S k ,A k ,P k ,R k ), where S k is the state space, A k is the action space, P k is the state transition probability, R k is the reward function; Use HDQN algorithm to learn the optimal strategy for each layer The Q function is Q k (s, a; θ k )=E[R k +γmax a′ Q k (s′, a′; θ′ k )|S k =s,A k =a] i where θ k is the Q network parameter, θ′ k The target network parameter γ∈[0,1] is the discount factor; Multi-agent collaborative optimization, define each layer N k agents, and the observation of agent i is Action The local strategy is AC-MARL algorithm is used to train multi-agent collaborative strategies, including Actor networks and Critic Network in is the joint observation of all agents, For joint action, is the Actor network parameter of agent i, is the Critic network parameter, the objective function is the reward function of agent i: Knowledge-guided strategy search, define the knowledge-guided function f(s, a; Φ), ​​where Φ is the guided knowledge queried from the knowledge base. Embed f into the reinforcement learning objective to obtain the modified objective function is the guidance knowledge obtained from the knowledge base at time t.

7. The intelligent factory decision support system according to claim 6 is characterized in that: The detailed execution steps of the hierarchical reinforcement learning driven multi-agent collaborative optimization algorithm are as follows:

1. Input the number of layers K and the number of agents in each layer N k , state space S k , action space A k , the reward function R k , discount factor γ, knowledge base Φ; 2. For k from 1 to K, execute: 2.1) Define and randomly initialize the k-layer HDQN network Q k (s, a; θ k ), target network Q k (s, a; θ′ k ), parameter θ k ,θ′ k ; 2.2) Define and randomly initialize the Actor network of each agent i in layer k Critic Network Q i,k (o k , a k ; f i,k ), parameters i,k ,f i,k ; 2.3) Initialize the experience playback buffer D k , set the batch size B i Learning rate α; 2.4) Repeat until the algorithm converges: 2.5) Output k-layer trained Actor network That is, the multi-agent collaborative strategy π at level k k ; 3. Output the optimal hierarchical strategy π obtained by hierarchical multi-agent reinforcement learning * =(π 1 , π 2 ,...,π K ).

8. The intelligent factory decision support system according to claim 7, characterized in that: In step 2.1), the Q network Q of the k-layer HDQN k (s, a; θ k ) adopts a two-layer neural network structure, including the first hidden layer (H1 neurons) of the input layer, the second hidden layer (H2 neurons) and the output layer, where the input is the state s∈S k , and all possible actions a∈A k The Q value estimation, the hidden layer activation function is the ReLU function, that is, f(x) = max(0, x), the output layer activation function is the linear function, and the loss function is the mean square error: where (s, a, r, s′) is the sum of the sums from the experience playback D k Transfer samples sampled in .

9. The intelligent factory decision support system according to claim 7, characterized in that: In step 2.2), the Actor network of agent i and Critic Network Q i,k (o k , a k ; φ i,k ) both use a two-layer neural network structure, and the input of the Actor network is the local observation of the agent. The output is the action distribution parameters, including mean μ and variance σ, action From the Gaussian distribution N(μ,σ 2 ) is sampled: The input of the Critic network is the joint observation o of all agents k and joint action a k , the output is the Q value estimate, that is in A joint strategy for the Actor network of all agents.

10. A smart factory decision support method based on the system according to any one of claims 1 to 9, characterized in that: The following steps are involved: 1) Through the data acquisition module, collect production equipment, materials, and environmental data and upload them to the database; 2) Analyze and mine the collected data and extract features through the data analysis module; 3) Through the optimization decision module, the multi-agent collaborative optimization algorithm driven by hierarchical reinforcement learning is called to solve the multi-objective optimization model and output a decision plan; 4) Presenting the decision results through the human-computer interaction module using a visual interface for user adjustment; 5) Storing domain knowledge through the knowledge base module to provide support for optimization decisions; 6) The optimization results are fed back to the knowledge base module to update the knowledge.

Citation Information

Cited By

  • Environment index monitoring method and system for networked storage cabinet

    CN119245740A

  • A method and system for monitoring environmental indicators of networked storage cabinets

    CN119245740B

  • Furniture plate processing equipment control method and control system

    CN119472513A

  • Intelligent scheduling method for collaborative manufacturing of tunnel boring machines

    CN119671136A

  • Intelligent scheduling method for tunnel boring machine collaborative manufacturing

    CN119671136B