Causal-driven intelligent manufacturing strategy optimization method and system and medium

Abstract

The invention relates to the technical field of intelligent manufacturing and reinforcement learning, and discloses a causal-driven intelligent manufacturing strategy optimization method and system, and a medium. Comprising the following steps: collecting state information of an intelligent agent and a production environment, and constructing an environment data set and a zero action data set; constructing a causal decoupling dynamic model comprising a causal intervention module and an environmental evolution module; mixing the environment data set and the zero action data set, and training a causal decoupling dynamic model by using a first mixed data set; performing intervention simulation on the environment data set by using the trained causal decoupling dynamic model to obtain a model data set; mixing the environment data set and the model data set, and training an unbiased strategy optimizer by using a second mixed data set; and deploying the trained unbiased strategy optimizer into a production system, and driving an intelligent agent to execute production operation according to the state information. Through causal-driven modeling and unbiased strategy optimization, high-precision and high-reliability decision control of the intelligent manufacturing system in a complex dynamic environment is realized.

Description

Technical Field

[0001] This application relates to the fields of intelligent manufacturing and reinforcement learning technologies, and in particular to a causal-driven intelligent manufacturing strategy optimization method, system, and medium. Background Technology

[0002] Reinforcement learning, as a core method for training intelligent agents (such as execution units, control modules, or decision agents in intelligent manufacturing systems), learns optimal control decision strategies by interacting with unknown and dynamic industrial operating environments. It has broad application prospects in simulation training, control algorithm development, and digital twin construction within the field of intelligent manufacturing. Model-Based Reinforcement Learning (MBRL) predicts system state transitions by constructing dynamic environmental models, significantly reducing reliance on real-world interaction data and improving sample efficiency. It is particularly suitable for industrial scenarios where data acquisition costs are high or large-scale acquisition of real-world interaction data is difficult, such as complex production line control, process parameter optimization, and production scheduling decisions.

[0003] While MBRL theoretically possesses the advantage of efficient learning, its practical application in intelligent manufacturing control decision-making tasks still faces two major challenges: insufficient accuracy in production system modeling and severe impact from biased strategy generation data during control strategy learning. Regarding the accuracy and generalization issues of production system modeling, in an intelligent manufacturing environment, the dynamic characteristics of a production system are typically determined by the behavior of equipment, production lines, or manufacturing processes. Traditional MBRL methods often model system state transitions as black-box mappings. Although some studies have employed techniques such as model ensemble, Lipschitz constraints, or distributed matching to improve model robustness, these methods still generally couple the influence of control actions (such as equipment setting parameters, robot motion commands, and process adjustments) with system states (such as workpiece processing status, material flow information, environmental disturbances, and other equipment states). They fail to identify the essential attribute of control actions as an external intervention and do not explicitly distinguish the natural evolution process of the task environment itself, such as natural changes in workpiece states, material characteristic evolution, and external order fluctuations. This modeling approach easily introduces confounding bias in complex industrial scenarios (such as multi-machine collaborative operations, precision machining and assembly, continuous process control, and human-machine collaborative tasks), resulting in poor model generalization ability and amplifying prediction errors over long-term decision-making and planning (compounding error). This severely impacts the performance of intelligent manufacturing systems in terms of control safety, execution accuracy, production efficiency, and quality reliability. The policy-induced bias problem in control policy learning arises because, during the policy learning phase, optimization methods based on value functions (such as Q-functions) (e.g., the Actor-Critic framework) rely on data collected from the interaction between the policy and the environment for policy improvement. However, due to the selectivity of the policy itself (such as the explore-exploit mechanism), the distribution of the collected state-action pairs is not random, exhibiting significant selection bias. This bias further distorts the estimation of value functions (such as the expected cumulative reward in the future, corresponding to goals such as production efficiency, product quality, energy consumption, and safety), causing the policy gradient update direction to deviate from the optimal path, ultimately affecting the optimality, robustness, and economy of the learned control policy.

[0004] Current mainstream improvement methods mostly focus on enhancing single-step prediction accuracy (such as model ensemble) or constraining model complexity (such as Lipschitz regularization), failing to fundamentally decouple the intervention effects of control actions from the natural evolution of the environment, and also unable to effectively correct data bias in the policy evaluation stage. Although techniques such as Optimal Transport and Adversarial Learning have made attempts in distribution matching, their effectiveness is limited in dealing with learning biases with specific causal structures in MBRL. Advanced causal inference methods such as Target Maximum Likelihood Estimation (TMLE) have shown potential in correcting observational data bias, but have not yet been systematically introduced and used to correct biases induced by reinforcement learning policies, especially in control scenarios with high reliability and high precision requirements, such as intelligent manufacturing.

[0005] In summary, existing MBRL methods, when dealing with complex control tasks in intelligent manufacturing, still lack explicit modeling of control action intervention attributes, effective decoupling from natural environmental evolution, and structured correction mechanisms for policy evaluation biases. Therefore, there is an urgent need to propose a new model-based reinforcement learning framework that can fundamentally improve the accuracy and generalization ability of production system behavior modeling, enhance the unbiasedness and reliability of control policy learning, and support intelligent manufacturing decision-making scenarios with high safety, high precision, and high efficiency requirements. Summary of the Invention

[0006] The purpose of this invention is to address at least one deficiency in the existing technology and provide a causal-driven intelligent manufacturing strategy optimization method, system, and medium. This invention achieves high-precision and high-reliability decision control of intelligent manufacturing systems in complex dynamic environments through causal-driven modeling and unbiased strategy optimization.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a causal-driven intelligent manufacturing strategy optimization method, the method comprising the following steps: Collect state information of the intelligent agent and the production environment, and construct an environment dataset and a zero-action dataset without interactive actions based on the state information; A causal decoupled dynamic model is constructed, comprising a causal intervention module and an environmental evolution module. The causal intervention module is used to output intermediate representation variables based on the current state variables and action variables. The environmental evolution module is used to receive the intermediate representation variables output by the causal intervention module and output predictions of the state and immediate reward at the next moment. The environmental dataset and the zero-action dataset are combined to form a first mixed dataset, and the causal decoupling dynamic model is trained using the obtained first mixed dataset to obtain the trained causal decoupling dynamic model. The trained causal decoupling dynamic model is used to simulate interventions in the environmental dataset to generate a model dataset. The environment dataset and the model dataset are mixed to form a second mixed dataset, and the unbiased policy optimizer is trained using the obtained second mixed dataset to obtain the trained unbiased policy optimizer. The trained unbiased policy optimizer is deployed in the control unit of the production system to receive status information from sensors and production equipment in real time, calculate the optimal action command, and drive the agent to perform production operations.

[0008] Furthermore, the environmental dataset includes the feature variables at the current time, the action variables, the feature variables at the next time, the reward value after performing the action at the current time, and the probability density of taking the action according to the policy neural network at the current time, denoted as... ,in, For the current moment, the feature variable in the environmental dataset. This represents the action variables in the environmental dataset. This represents the feature variable in the environmental dataset for the next time step. This indicates that the environmental data focuses on the reward value after performing an action at the current moment. This represents the probability density of actions taken by the policy neural network at the current moment, based on the environmental data. The zero-action dataset includes the feature variables at the current moment, the zero-action variables, the feature variables at the next moment, the reward value after performing the zero-action at the current moment, and the probability density of taking the zero-action at the current moment according to the policy neural network, which have the same variable structure as the environmental dataset. This is denoted as... ,in, For the current moment, in the zero-action dataset, For the zero-action variables in the zero-action dataset, For the next time step in the zero-action dataset, The zero-action data focuses on the reward value after performing a zero-action at the current moment. The zero-action data is the probability density of taking a zero action based on the policy neural network at the current moment.

[0009] Furthermore, the causal intervention module is used to output intermediate representation variables based on the current state variables and action variables, and the specific calculation formula is as follows:

[0010] in, For the current state variable, As an action variable, As an intermediate characterization variable, This is a function used to fit a conditional distribution. ,when Zero action variable hour, , This is the first multilayer perceptron; The environmental evolution module receives the intermediate representation variables output by the causal intervention module and outputs predictions for the next time step state and immediate reward. The specific calculation method is as follows:

[0011] in, For predicting the state at the next moment, For the prediction of instant rewards in the next moment, For the second multilayer perceptron, This is a function used to fit a conditional distribution. .

[0012] Furthermore, in the process of training the causal decoupled dynamic model using the obtained first mixed dataset, the loss function is defined as the expectation of the sum of the state prediction error and the reward prediction error, and the specific calculation method is as follows:

[0013] in, For loss function, This is the first mixed dataset.

[0014] Furthermore, the trained causal decoupling dynamic model is used to simulate interventions on the environmental dataset to generate a model dataset, specifically including: (1). Retrieve a current state variable from the environmental dataset. and the current state variable Input into the policy network, output the action variables for that state. and its probability density ; (2). The input is given to the causal intervention module to obtain intermediate characterization variables. ; (3). The intermediate representation variables The input is given to the environment evolution module to obtain a prediction of the state at the next time step. And prediction of instant rewards ; (4). Combine the data generated in steps (1) to (3) into a single data record. , , , , ); (5) Repeat steps (1) to (3) for all or part of the states in the environmental dataset, and summarize all the generated data records to obtain the model dataset. .

[0015] Furthermore, the unbiased policy optimizer includes a main Q-value neural network, a target Q-value neural network, and a policy neural network. The main Q-value neural network is used to estimate the current action state value, the target Q-value neural network is used to estimate the action state value at the next moment, and the policy neural network is used to output the action variable and the corresponding probability density based on the current state variable, thereby guiding the agent to interact with the environment.

[0016] Furthermore, the main Q-value neural network is used to estimate the current action state value, and the specific calculation method is as follows:

[0017] in, This represents the current action state value. , It is the third multilayer perceptron. , It is the fourth multilayer perceptron; The target Q-value neural network is used to estimate the action state value at the next moment, and the specific calculation formula is as follows:

[0018] in, This is the action state value for the next moment. , It is a sixth-layer perceptron. , It is a seventh-layer perceptron, and , It is the fifth multilayer perceptron.

[0019] Furthermore, during the training of the unbiased policy optimizer using the obtained second mixed dataset, the loss function of the main Q-value neural network is trained. as follows:

[0020] in, For the second mixed dataset, It is a discount factor; The parameters of the target Q-value neural network are obtained from the main Q-value neural network via a soft-modulation method:

[0021] And calculate the loss function based on the updated parameters. ,as follows:

[0022] in, The parameters of the target Q-value neural network are... These are the parameters of the main Q-value neural network; The loss function of the policy neural network as follows: .

[0023] Secondly, the present invention also provides a causal-driven intelligent manufacturing strategy optimization system, the system being based on the method described in the first aspect, and comprising the following modules: The data acquisition module is configured to collect state information of the intelligent agent and the production environment, and to construct an environmental dataset and a zero-action dataset without any interactive actions based on the state information. The building module is configured to build a causal decoupled dynamic model that includes a causal intervention module and an environmental evolution module. The causal intervention module is used to output intermediate representation variables based on the current state variables and action variables. The environmental evolution module is used to receive the intermediate representation variables output by the causal intervention module and output a prediction of the state and immediate reward at the next moment. First training module: configured to mix the environment dataset and the zero-action dataset to form a first mixed dataset, and use the obtained first mixed dataset to train the causal decoupling dynamic model to obtain the trained causal decoupling dynamic model; Processing module: configured to use the trained causal decoupling dynamic model to perform intervention simulation on the environmental dataset and generate a model dataset; The second training module is configured to mix the environment dataset and the model dataset to form a second mixed dataset, and use the obtained second mixed dataset to train an unbiased policy optimizer to obtain a trained unbiased policy optimizer. Deployment module: configured to deploy the trained unbiased policy optimizer to the control unit of the production system, for receiving status information from sensors and production equipment in real time, calculating the optimal action command, and driving the agent to perform production operations.

[0024] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0025] Compared with the prior art, the technical advantages of the present invention are as follows: This invention constructs a causal decoupled dynamic model to explicitly separate the intervention effect of control actions from the natural evolution process of the production environment. This effectively overcomes the confounding bias problem caused by action-state coupling modeling in traditional model-based reinforcement learning, significantly improving the accuracy of predicting the dynamic behavior of production systems and the model's generalization ability. Furthermore, an unbiased policy optimizer based on maximum likelihood estimation of the objective is introduced in the policy optimization stage. The propensity score weighting mechanism corrects the selection bias in the policy generation data, thereby achieving structural correction of the Q-function estimation and ensuring the correctness and stability of the policy gradient update direction. Finally, the control policy trained by the above framework exhibits stronger decision reliability, environmental adaptability, and task execution efficiency in real intelligent manufacturing scenarios, and is particularly suitable for industrial control tasks with stringent safety and precision requirements, such as high-precision assembly, flexible scheduling, and human-machine collaboration. Attached Figure Description

[0026] Figure 1 This is a flowchart of a causal-driven intelligent manufacturing strategy optimization method according to Embodiment 1 of the present invention; Figure 2 This is a block diagram of a causal-driven intelligent manufacturing strategy optimization system according to Embodiment 2 of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] Example 1 Please see Figure 1 A preferred embodiment of the present invention provides a causal-driven intelligent manufacturing strategy optimization method, the method comprising the following steps: S1: Collect the state information of the intelligent agent and the production environment, and construct an environment dataset and a zero-action dataset without interactive actions based on the state information; S2: Construct a causal decoupled dynamic model that includes a causal intervention module and an environmental evolution module. The causal intervention module is used to output intermediate representation variables based on the current state variables and action variables. The environmental evolution module is used to receive the intermediate representation variables output by the causal intervention module and output predictions of the state and immediate reward at the next moment. S3: Mix the environmental dataset and the zero-action dataset to form a first mixed dataset, and use the obtained first mixed dataset to train the causal decoupling dynamic model to obtain the trained causal decoupling dynamic model; S4: Use the trained causal decoupling dynamic model to perform intervention simulation on the environmental dataset to generate a model dataset; S5: Mix the environment dataset and the model dataset to form a second mixed dataset, and use the obtained second mixed dataset to train an unbiased policy optimizer to obtain a trained unbiased policy optimizer. S6: Deploy the trained unbiased policy optimizer into the control unit of the production system to receive status information from sensors and production equipment in real time, calculate the optimal action command, and drive the agent to perform production operations.

[0029] This embodiment constructs a causal decoupled dynamic model, explicitly separating the intervention effect of control actions from the natural evolution process of the production environment. This effectively overcomes the confounding bias problem caused by action-state coupling modeling in traditional model-based reinforcement learning, significantly improving the accuracy of dynamic behavior prediction and model generalization ability of the production system. Furthermore, an unbiased policy optimizer based on maximum likelihood estimation of the objective is introduced in the policy optimization stage. The selection bias in the policy generation data is corrected through a propensity score weighting mechanism, thereby achieving structural correction of the Q-function estimation and ensuring the correctness and stability of the policy gradient update direction. Finally, the control policy trained by the above framework exhibits stronger decision reliability, environmental adaptability, and task execution efficiency in real intelligent manufacturing scenarios, and is particularly suitable for industrial control tasks with stringent safety and precision requirements, such as high-precision assembly, flexible scheduling, and human-machine collaboration.

[0030] In an optional embodiment, in step S1, the intelligent agent can be a robot or a robotic arm, etc. It collects state information of the robot (or robotic arm or other intelligent agent) and the production environment, and constructs an environmental dataset and a zero-action dataset based on the state information. The environmental dataset includes feature variables at the current moment, action variables, feature variables at the next moment, the reward value after performing an action at the current moment, and the probability density of taking an action according to the policy neural network at the current moment, denoted as... ,in, For the current moment, the feature variable in the environmental dataset. This represents the action variables in the environmental dataset. This represents the feature variable in the environmental dataset for the next time step. This indicates that the environmental data focuses on the reward value after performing an action at the current moment. This represents the probability density of actions taken by the policy neural network at the current moment, based on the environmental data. The zero-action dataset includes the feature variables at the current moment, the zero-action variables, the feature variables at the next moment, the reward value after performing the zero-action at the current moment, and the probability density of taking the zero-action at the current moment according to the policy neural network, which have the same variable structure as the environmental dataset. This is denoted as... ,in, For the current moment, in the zero-action dataset, For the zero-action variables in the zero-action dataset, For the next time step in the zero-action dataset, The zero-action data focuses on the reward value after performing a zero-action at the current moment. The zero-action data is the probability density of taking a zero action based on the policy neural network at the current moment.

[0031] In this embodiment, the introduction of a zero-action dataset provides a crucial counterfactual benchmark for training the causal decoupled dynamic model, significantly improving the accuracy and unbiasedness of production system modeling. Specifically, the zero-action dataset records system state transitions, rewards, and action probability densities under zero-action (i.e., no intervention) conditions, contrasting them with the normal environment dataset. This allows the causal intervention module to explicitly distinguish the intervention effect of control actions from the natural evolution of the environment. This contrastive learning effectively eliminates the confounding biases caused by action-state coupling in traditional modeling, avoiding the cumulative amplification of prediction errors in long-term decision-making, thereby enhancing the model's generalization ability in complex industrial scenarios. Simultaneously, the introduction of the zero-action dataset provides a more reliable data foundation for the policy optimization stage, ensuring that the unbiased policy optimizer can perform value function estimation and policy gradient updates based on accurate environmental dynamics, ultimately improving the reliability, safety, and efficiency of intelligent manufacturing system control decisions.

[0032] In an optional embodiment, in step S2, the causal decoupling dynamic model includes a causal intervention module and an environmental evolution module, wherein the causal intervention module is used to output intermediate representation variables based on the current state variables and action variables. The specific calculation formula is as follows:

[0033] in, For the current state variable, As an action variable, As an intermediate characterization variable, This is a function used to fit a conditional distribution. ,when Zero action variable hour, , This is the first multilayer perceptron; The environmental evolution module receives the intermediate representation variables output by the causal intervention module and outputs predictions for the next time step state and immediate reward; the specific calculation method is as follows:

[0034] in, For predicting the state at the next moment, For the prediction of instant rewards in the next moment, For the second multilayer perceptron, This is a function used to fit a conditional distribution.

[0035] In this embodiment, by designing a separate structure for the causal intervention module and the environmental evolution module, decoupling modeling of the intervention effect of control actions and the natural evolution mechanism of the environment is achieved, thereby effectively improving the accuracy and interpretability of dynamic prediction of the production system. Specifically, the causal intervention module explicitly distinguishes the impact of normal actions and zero actions through a conditional judgment mechanism: when the action is zero, the intermediate representation variable remains in its current state, effectively establishing a counterfactual baseline; when the action is non-zero, the causal intervention effect of the action on the state is accurately fitted through a multilayer perceptron. The environmental evolution module focuses on learning the autonomous evolution law of the environment under a given intermediate state. This decoupling structure fundamentally avoids the confounding biases caused by the coupling of actions and the environment in traditional black-box models, significantly reducing the accumulation of errors in long-term predictions, providing a stable and reliable environmental dynamic model foundation for subsequent unbiased strategy optimization, and ultimately improving the decision-making reliability and control accuracy of the intelligent manufacturing system under complex working conditions.

[0036] In an optional embodiment, in step S3, the environmental dataset and the zero-action dataset are mixed to obtain a first mixed dataset, as follows:

[0037]

[0038]

[0039]

[0040]

[0041] The first mixed dataset was finally obtained. Then use the resulting first mixed dataset. The causal decoupling dynamic model is trained to obtain a well-trained causal decoupling dynamic model. During training, the loss function is defined as the expectation of the sum of the state prediction error and the reward prediction error, and the specific calculation method is as follows:

[0042] in, For loss function, This is the first mixed dataset. The training process specifically includes: (1). From the first hybrid dataset Sample a batch of training data (in the middle) , ); (2). Input the causal intervention module to obtain intermediate characterization variables. ; (3). The intermediate representation variables Input the environmental evolution module to obtain the predicted value ( ); (4). Calculate the loss value of the batch of data, and calculate the gradient of the loss function with respect to the parameters of the causal intervention module and the environmental evolution module through the gradient descent algorithm, wherein the parameters can be the weights and biases of the first multilayer perceptron and the second multilayer perceptron, etc. (5) Based on the calculated gradient, backpropagate and update all parameters of the causal decoupled dynamic model to reduce the prediction error. Repeat the above steps until the model converges.

[0043] In this embodiment, a first hybrid dataset is formed by mixing the environmental dataset and the zero-action dataset, and used to train a causal decoupled dynamic model. This allows the model to simultaneously learn the system's dynamic behavior under both normal control actions and zero-action (no intervention) conditions. This hybrid data training method ensures that the causal intervention module can accurately identify the intervention effects of control actions, while the environmental evolution module can focus on the natural evolution process of the environment itself. This effectively decouples the influence of actions and the environment, significantly reducing the confounding bias in traditional modeling. By minimizing the loss function of the prediction error of state and reward, and combining it with gradient descent optimization, the model can accurately predict system state transitions and immediate rewards, providing a reliable environmental model foundation for subsequent unbiased policy optimization, ultimately improving the accuracy and robustness of decision-making in intelligent manufacturing systems.

[0044] In an optional embodiment, step S4 specifically includes: (1). Retrieve a current state variable from the environmental dataset. and the current state variable Input into the policy network, output the action variables for that state. and its probability density ; (2). The input is given to the causal intervention module to obtain intermediate characterization variables. ; (3). The intermediate representation variables The input is given to the environment evolution module to obtain a prediction of the state at the next time step. And prediction of instant rewards ; (4). Combine the data generated in steps (1) to (3) into a single data record. , , , , ); (5) Repeat steps (1) to (3) for all or part of the states in the environmental dataset, and summarize all the generated data records to obtain the model dataset. .

[0045] In an optional embodiment, in step S5, the environment dataset and the model dataset are mixed to obtain a second mixed dataset, as follows:

[0046]

[0047]

[0048]

[0049]

[0050] Finally, the second mixed dataset was obtained. An unbiased policy optimizer is trained using a second mixed dataset to obtain a trained unbiased policy optimizer. The unbiased policy optimizer includes a main Q-value neural network, a target Q-value neural network, and a policy neural network. The main Q-value neural network is used to estimate the current action state value, the target Q-value neural network is used to estimate the action state value at the next time step, and the policy neural network is used to output the action variable and the corresponding probability density based on the current state variable, thereby guiding the agent to interact with the environment.

[0051] Furthermore, the main Q-value neural network is used to estimate the current action state value. The main Q-value neural network includes two multilayer perceptrons: a third and a fourth multilayer perceptron. The third multilayer perceptron is used to generate... Values, the fourth multilayer perceptron is used to generate The value is calculated as follows:

[0052] in, This represents the current action state value. , It is the third multilayer perceptron. , It is the fourth multilayer perceptron; The target Q-value neural network is used to estimate the action state value at the next moment, and the specific calculation formula is as follows:

[0053] in, This is the action state value for the next moment. , It is a sixth-layer perceptron. , It is a seventh-layer perceptron, and , It is the fifth multilayer perceptron.

[0054] Furthermore, during the training of the unbiased policy optimizer using the obtained second mixed dataset, the loss function of the main Q-value neural network is trained. as follows:

[0055] in, For the second mixed dataset, It is a discount factor; The parameters of the target Q-value neural network are obtained from the main Q-value neural network via a soft-modulation method:

[0056] And calculate the loss function based on the updated parameters. ,as follows:

[0057]

[0058]

[0059]

[0060] in, The parameters of the target Q-value neural network are... These are the parameters of the main Q-value neural network; The loss function of the policy neural network as follows: .

[0061] In this embodiment, an unbiased policy optimizer is designed, wherein the main Q-value neural network adopts the target maximum likelihood estimation framework, decomposing the Q-value into basic values. Sum of deviation correction items And weighted by propensity score (i.e., divided by action probability density) This optimizer effectively corrects selection bias in the policy generation data, thus achieving unbiased estimation of the current action state value. The target Q-value neural network synchronizes parameters from the main network through a soft update mechanism, providing a stable future Q-value target, avoiding fluctuations in value function estimation, and enhancing the stability of the learning process. The policy neural network optimizes by maximizing the expected Q-value, ensuring that the policy update direction is towards higher rewards. Overall, this optimizer significantly improves the accuracy and robustness of policy learning through the synergistic effect of bias correction, stable target value, and policy gradient optimization, enabling the agent to make more reliable and efficient control decisions in complex manufacturing environments.

[0062] In an optional embodiment, in step S6, the trained unbiased policy optimizer is deployed to the control unit of the production system. The control unit is an industrial computer or embedded controller (such as a PLC, industrial PC, or edge computing device) with real-time computing capabilities. The communication interface between the control unit and sensors (such as vision sensors, force sensors, encoders) and execution devices (such as industrial robots, servo drives, pneumatic components) in the production environment is configured to ensure real-time acquisition of state information and low-latency issuance of control commands. The control unit continuously receives real-time state information from sensors (such as robot end-effector pose, workpiece position, equipment operating parameters, etc.). The current state is input into the deployed policy neural network, which outputs the optimal action command and its probability density based on the learned policy. The control unit converts the action command into specific equipment control signals (such as joint angle setpoints, speed commands, torque outputs, etc.) and sends them to the execution device via an industrial bus or Ethernet protocol, driving the intelligent agent (such as a robotic arm, mobile robot) to complete production operations (such as precision assembly, material handling, trajectory tracking, etc.). During operation, the system continuously records the state transition data after actual execution and stores it in a local cache for subsequent online model fine-tuning or performance evaluation. To ensure production safety, the control unit integrates an anomaly detection module (such as setting action range limits and state change thresholds). If an abnormal state or action exceeds the limit, a safety protocol is triggered (such as emergency stop, speed reduction, or switching to a preset conservative strategy).

[0063] Example 2 Please see Figure 2 An embodiment of the present invention provides a causal-driven intelligent manufacturing strategy optimization system, comprising the following modules: The data acquisition module is configured to collect state information of the intelligent agent and the production environment, and to construct an environmental dataset and a zero-action dataset without any interactive actions based on the state information. The building module is configured to build a causal decoupled dynamic model that includes a causal intervention module and an environmental evolution module. The causal intervention module is used to output intermediate representation variables based on the current state variables and action variables. The environmental evolution module is used to receive the intermediate representation variables output by the causal intervention module and output a prediction of the state and immediate reward at the next moment. First training module: configured to mix the environment dataset and the zero-action dataset to form a first mixed dataset, and use the obtained first mixed dataset to train the causal decoupling dynamic model to obtain the trained causal decoupling dynamic model; Processing module: configured to use the trained causal decoupling dynamic model to perform intervention simulation on the environmental dataset and generate a model dataset; The second training module is configured to mix the environment dataset and the model dataset to form a second mixed dataset, and use the obtained second mixed dataset to train an unbiased policy optimizer to obtain a trained unbiased policy optimizer. Deployment module: configured to deploy the trained unbiased policy optimizer to the control unit of the production system, for receiving status information from sensors and production equipment in real time, calculating the optimal action command, and driving the agent to perform production operations.

[0064] The system proposed in this embodiment is based on the method proposed in Embodiment 1. Therefore, the options proposed in Embodiment 1 are also applicable to this embodiment. To avoid repetition, they will not be described again here.

[0065] In addition, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in embodiment 1.

[0066] The readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks, optical storage, digital video discs (DVDs), Blu-ray discs (BDs), high-definition universal discs (HVDs), etc.) and semiconductor storage (such as read-only memory (ROMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), non-volatile memory (NAND flash), solid-state drives, etc.).

[0067] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A cause-and-effect driven intelligent manufacturing strategy optimization method, characterized in that, The method comprises the following steps: Collecting state information of the agent and the production environment, and constructing an environment dataset and a zero-action dataset of non-interaction actions based on the state information; A causal decoupling dynamic model comprising a causal intervention module and an environment evolution module is constructed, wherein the causal intervention module is used to output intermediate representation variables according to current state variables and action variables, and the environment evolution module is used to receive the intermediate representation variables output by the causal intervention module and output predictions of the next time state and immediate rewards; The environment dataset and the zero-action dataset are mixed to form a first mixed dataset, and the causal decoupling dynamic model is trained using the obtained first mixed dataset to obtain a trained causal decoupling dynamic model; The trained causal decoupling dynamic model is used to simulate intervention on the environment dataset to generate a model dataset; The environment dataset and the model dataset are mixed to form a second mixed dataset, and the obtained second mixed dataset is used to train an unbiased policy optimizer to obtain a trained unbiased policy optimizer; The trained unbiased policy optimizer is deployed into a control unit of a production system to receive state information from sensors and production equipment in real time and calculate optimal action instructions to drive the agent to perform production operations.

2. The method of claim 1, wherein, The environment data set includes a feature variable at a current moment, an action variable, a feature variable at a next moment, a reward value after performing an action at the current moment, and a probability density of taking an action according to the policy neural network at the current moment, denoted as wherein, is the feature variable at the current moment in the environment data set, represents the action variable in the environment data set, represents the feature variable at the next moment in the environment data set, represents the reward value after performing an action at the current moment in the environment data set, represents the probability density of taking an action according to the policy neural network at the current moment in the environment data set; the zero action data set includes a feature variable at a current moment, a zero action variable, a feature variable at a next moment, a reward value after performing a zero action at the current moment, and a probability density of taking a zero action according to the policy neural network at the current moment, denoted as , wherein, is the feature variable at the current moment in the zero action data set, is the zero action variable in the zero action data set, is the feature variable at the next moment in the zero action data set, is the reward value after performing a zero action at the current moment in the zero action data set, is the probability density of taking a zero action according to the policy neural network at the current moment in the zero action data set.

3. The method of claim 1, wherein, The causal intervention module is used to output intermediate representation variables according to current state variables and action variables, and the specific calculation formula is as follows: wherein, is a current state variable, is an action variable, is an intermediate representation variable, is a function function for fitting a conditional distribution when is a zero action variable , , is a first multi-layer perceptron; The environment evolution module is used to receive the intermediate representation variables output by the causal intervention module and output predictions of the next time state and immediate rewards, and the specific calculation method is as follows: wherein, is a prediction of the state at the next time instant, is a prediction of the immediate reward at the next time instant, is a second multi-layer perceptron, is a function function for fitting the conditional distribution .

4. The method according to any one of claims 1 to 3, characterized in that, In the process of training the causal decoupling dynamic model using the obtained first mixed dataset, the loss function is defined as the expectation of the sum of state prediction errors and reward prediction errors, and the specific calculation method is as follows: wherein, is a loss function, is the first mixed dataset.

5. The method of claim 4, wherein, The trained causal decoupling dynamic model is used to simulate intervention on the environment dataset to generate a model dataset, which specifically comprises: (1). taking out a current state variable from the environment data set and inputting the current state variable to a policy network to output an action variable in the state and its probability density ; (2). The intermediate representation is input to the causal intervention module to obtain the causal intervention variable ;​ (3). inputting the intermediate representation variables to the environment evolution module to obtain a prediction of the next time state and a prediction of the immediate reward ; (4). Combining the data generated in steps (1) to (3) into one data record , , , , ) (5). Repeating steps (1) to (3) for all or part of the states in the environment dataset, and aggregating all the generated data records to obtain a model dataset .

6. The method of claim 1, wherein, The unbiased policy optimizer comprises a main Q-value neural network, a target Q-value neural network and a policy neural network, the main Q-value neural network is used to estimate current action state values, the target Q-value neural network is used to estimate action state values at the next time, and the policy neural network is used to output action variables and corresponding probability densities according to current state variables, thereby guiding the agent to interact with the environment.

7. The method of claim 6, wherein, The main Q-value neural network is used to estimate current action state values, and the specific calculation method is as follows: wherein, is a current action state value, , is a third multi-layer perceptron, , is a fourth multi-layer perceptron; The target Q-value neural network is used to estimate action state values at the next time, and the specific calculation formula is as follows: wherein, is the action state value for the next time instant, , is a sixth multi-layer perceptron, , is a seventh multi-layer perceptron, and , is a fifth multi-layer perceptron.

8. The method of claim 7, wherein, In training the unbiased policy optimizer using the resulting second mixed dataset, a loss function for training the main Q-value neural network As follows: wherein, is a second mixed data set, is a discount factor; The parameters of the target Q-value neural network are obtained from the main Q-value neural network through soft updating: and compute a loss function based on the updated parameters As follows: wherein, are parameters of the target Q-value neural network, are parameters of the main Q-value neural network; loss function of the policy neural network As follows: 。 9. A causal-driven intelligent manufacturing strategy optimization system, based on the method of any one of claims 1-8, characterized in that, The following modules are included: The acquisition module is configured to collect state information of the agent and the production environment, and construct an environment dataset and a zero-action dataset of non-interaction actions based on the state information; The construction module is configured to construct a causal decoupling dynamic model comprising a causal intervention module and an environment evolution module, wherein the causal intervention module is configured to output an intermediate representation variable according to a current state variable and an action variable, and the environment evolution module is configured to receive the intermediate representation variable output by the causal intervention module and output a prediction of a next time state and an immediate reward; The first training module is configured to mix the environment dataset and the zero action dataset to form a first mixed dataset, and train the causal decoupling dynamic model using the obtained first mixed dataset to obtain a trained causal decoupling dynamic model; The processing module is configured to use the trained causal decoupling dynamic model to simulate intervention on the environment dataset to generate a model dataset; The second training module is configured to mix the environment dataset and the model dataset to form a second mixed dataset, and train an unbiased policy optimizer using the obtained second mixed dataset to obtain a trained unbiased policy optimizer; The deployment module is configured to deploy the trained unbiased policy optimizer to a control unit of a production system, to receive state information from sensors and production equipment in real time, and calculate optimal action instructions to drive the agent to perform production operations.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the steps of the method of any one of claims 1 to 8.

Images