Complex heavy equipment process optimization method based on reinforcement learning and digital twinning

Abstract

The complex heavy equipment process optimization method based on reinforcement learning and digital twinning disclosed by the application has the following steps: 1. collecting data information of an actual physical workshop and preprocessing the data information; 2. constructing a digital twinning model; 3. establishing a mathematical model of a physical workshop production line problem; 4. optimizing the mathematical model to obtain a theoretical optimal decision scheme; 5. evaluating the theoretical optimal decision scheme to determine an optimal decision scheme; 6. simulating the production line and synchronously realizing the real-time physical workshop state in the digital twinning model, and completing complex heavy equipment process optimization; and 7. storing data to provide historical data for a reinforcement learning model and a scheduling scheme adopted for a corresponding fault state. The complex heavy equipment digital twinning scheduling method based on reinforcement learning can relieve the excessive dependence of an aluminum extrusion production line on a scheduler, and the scheduler cannot make decisions on production scheduling schemes according to the state of the workshop in real time; the scheduling scheme obtained by the method can better adapt to a complex production process.

Description

Technical Field

[0001] This invention belongs to the field of industrial intelligent scheduling and control technology, and relates to a method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins. Background Technology

[0002] With the rapid rise of my country's manufacturing industry, modern manufacturing is increasingly trending towards intelligent, lean, and personalized customization. Aluminum extrusion production lines, as crucial for producing complex heavy industrial materials, play a significant role in the national economy through their intelligent and information-based development. The core issue in industrial production scheduling is coordinating production batches of raw materials with different processing times and facilitating collaborative processing between production equipment while ensuring production line efficiency. Because the singular nature of traditional industrial manufacturing can no longer meet market demand, aluminum production orders are trending towards small-batch, customized, and multi-batch production. In current aluminum extrusion production lines, scheduling in complex production environments often relies on the subjective experience of schedulers, requiring repeated adjustments and troubleshooting, which consumes considerable time and impacts scheduling efficiency. Therefore, utilizing modern intelligent technologies to enable managers to obtain real-time workshop status and improve industrial production efficiency is of significant practical importance.

[0003] Existing solutions to production scheduling problems mostly utilize production scheduling algorithms, such as heuristic algorithms and swarm intelligence optimization algorithms, which can effectively schedule actual production problems. However, the scheduling rules generated by these methods are relatively complex and tend to overlook the value of experience production records. They lack attention to the historical production and processing scheduling data of the enterprise in the past, and managers cannot promptly discover the current workshop status and make appropriate adjustments, which cannot meet the needs of building current intelligent manufacturing systems.

[0004] Given that existing methods for solving production scheduling problems are complex and tend to overlook the value of experience-based production records, lack attention to historical production and processing scheduling data, and prevent managers from promptly identifying and adjusting the current workshop status, this application proposes a process optimization method for complex heavy equipment based on reinforcement learning and digital twins. Summary of the Invention

[0005] The purpose of this invention is to provide a process optimization method for complex heavy equipment based on reinforcement learning and digital twins, which solves the problems in the prior art where the scheduling of aluminum extrusion production lines fails to make full use of historical production data, resulting in low efficiency of production decision-making and the inability to monitor the workshop production status in real time.

[0006] The technical solution adopted in this invention is a method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins, which is implemented according to the following steps:

[0007] Step 1: Collect data from the actual physical workshop of complex heavy equipment and preprocess it;

[0008] Step 2: Construct a digital twin model of the complex heavy equipment production line workshop based on the preprocessed data.

[0009] Step 3: Determine the description of the physical workshop production line scheduling problem and the constraints during actual processing and production, and establish a mathematical model of the physical workshop production line problem;

[0010] Step 4: Optimize the mathematical model to obtain the theoretically optimal decision scheme;

[0011] Step 5: Evaluate the theoretically optimal decision-making scheme and determine the optimal decision-making scheme;

[0012] Step 6: Input the optimal decision-making scheme into the digital twin model to simulate the production line, and synchronize the real-time physical workshop status in the digital twin model to complete the process optimization of complex heavy equipment.

[0013] Step 7: Store the actual physical workshop implementation status, fault status, brainstorming optimization algorithm parameters, reinforcement learning model and the optimal decision scheme information of the complex heavy equipment, and provide historical data for the reinforcement learning model and scheduling scheme adopted for the corresponding fault status.

[0014] The invention is further characterized in that,

[0015] The data information in step 1 includes personnel information, equipment information, material information, method information, and environmental information. The personnel information mainly includes worker ID, worker name, worker skill type, and worker workload information. The equipment information mainly includes equipment ID, equipment name, equipment type, equipment workload, and equipment wear and tear information. The material information mainly includes material ID, material name, and material type information. The method information mainly includes method ID, method name, method type, and specific method information. The environmental information mainly includes environmental ID, environmental name, environmental type, and specific environmental information.

[0016] The preprocessing step in step 1 involves removing redundant and erroneous information from the data.

[0017] In step 2, the digital twin model is constructed using PlantSimulation in the form of man, machine, material, method, and environment. The digital twin model includes worker twin model, equipment twin model, material twin model, method twin model, and environment twin model.

[0018] Step 3 involves the production line scheduling problem in the physical workshop. Under the condition of satisfying production constraints, the goal is to determine the processing sequence of each batch of workpieces with the minimum production time over time (OPT) as the scheduling objective. The mathematical model for production scheduling is established as follows:

[0019]

[0020] Among them, c i This represents the completion time of the i-th batch of workpieces, i = 1, 2…n, S1, S2…S… n The production scheduling is sorted for all batches of workpieces, and OPT is the processing time required to process all batches of workpieces.

[0021] The constraints in actual processing and production in step 3 are as follows: each batch is processed continuously in the production operation, that is, the workpiece can be sent to the next process in the batch process flow after the current process is completed. The following four constraints exist in the batch production process:

[0022] C(σ y ,j)=B(σ y ,j)+Q(σ y , j); y=1, 2,..., Y; j=1, 2,..., j; (2)

[0023] C(σ y ,j)≤B(σ y , j+1); y=1, 2,..., Y; j=1, 2,..., j; (3)

[0024] C(σ y ,j)-C(σ e ,j)≥Q(σ y , j); y=1, 2,..., Y; j=1, 2,..., j; (4)

[0025] C(σ y ,j)-B(σ y ,j)=Q(σ y , j); y=1, 2,..., Y; j=1, 2,..., j; (5)

[0026] Where, Q(σ) y B(σ) represents the processing time of batch y on machine j; y,j) is the start time of batch y on machine j. Equation (2) means that if there is a processing step in the same batch of workpieces, the completion time of the step is equal to the sum of the start time and the processing time of the step; otherwise, the completion time is equal to the start time. Equation (3) means that the completion time of the previous step and the start time of the next step of the same workpiece are related, that is, the next step can only be processed after the previous step is completed. Equation (4) means that the processing equipment in any processing step can only process one batch of workpieces at a time, that is, any two different batches of workpieces y and e cannot be processed on the same step at the same time. Equation (5) means that the completion time of any step minus its start time cannot be less than its processing time.

[0027] Step 4, the mathematical model optimization process employs both brainstorming and deep reinforcement learning algorithms. The specific optimization process is as follows:

[0028] 4.1 Brainstorming to optimize algorithm settings

[0029] 4.1.1 Encoding / Decoding Settings

[0030] The encoding is performed using integer encoding. The decoding process determines the workpiece processing time based on the encoding result. By reading the encoding result from left to right, the batch number of the workpiece to be processed is obtained. The production scheduling is simulated in the order of the encoding results, and the maximum completion time obtained is the processing time (OPT) required to process all batches of workpieces under this encoding order.

[0031] 4.1.2 Initialization Settings

[0032] Define the parameters of the brainstorming optimization algorithm for solving the production scheduling problem, let the number of iterations be T, the population size be N, and the perturbation probability parameter be (P). c ,P np ,P 1ctr ,P 2ctr N initial individuals representing scheduling problem solutions are randomly generated using integer encoding. The initial individuals x are generated according to the following formula. i,j :

[0033] x i,j =min + (max - min) × rand() (6)

[0034] Where max and min are the upper and lower bounds of the initial population parameters, respectively;

[0035] 4.1.3 Population Evaluation Settings

[0036] Calculate the initial individual x in the initialized population. i,j The fitness value is calculated as the reciprocal of the maximum completion time.

[0037]

[0038] 4.1.4 Population update settings;

[0039] With a preset probability parameter (P) c ,P np ,P 1ctr ,P 2ctr The initial individuals in the population are updated separately. The specific process of population update is to compare the newly generated individuals with the old individuals at the same position and retain the individuals with higher fitness values.

[0040] Among them, P c The probability of choosing a cluster center instead of the actual cluster center; P np The probability of generating a new individual based on whether to choose one class or both classes; P 1ctr ,P 2ctr The probability of generating a new individual is determined by choosing the class center of one class or the class centers of two classes, respectively.

[0041] 4.1.5. Termination condition settings;

[0042] If the initial individual update reaches the maximum number of iterations T, then stop the iteration; otherwise, repeat steps 4.1.2-4.1.4 to reinitialize the population and start a new round of iteration.

[0043] 4.2 Deep reinforcement learning algorithm settings;

[0044] 4.2.1 Constructing a deep reinforcement learning model;

[0045] The state S in the agent's state space consists of an array of fitness values ​​of the population, which is represented in a deep reinforcement learning model as:

[0046] S=(f avg ,f max ,f min ,f MSN ,f SD (8)

[0047] Among them, f avg f is the average fitness value of the population. max ,f min f represents the maximum and minimum fitness values. MSN f is the mean squared error of the fitness values. SD For standard deviation and other related data;

[0048] 4.2.2 Strategy Mechanism Settings;

[0049] Let the action space be the set of actions that the agent can take, and let the probability of the current perturbation (P) be...c ,P np ,P 1ctr ,P 2ctr Adjustments are made, and the current disturbance probability is set to three operations: +0.05, +0, and -0.05. Each time, an action 'a' is selected from the action space using an ε-greedy decreasing strategy, which represents the execution of the current process of the workpiece.

[0050] 4.2.3 Reward Mechanism Setup;

[0051] Let the reward function at time k be:

[0052] r(k)=v(k)-v(k-1) (9)

[0053] Where r(k) is the reward function at time k, v(k) is the value function, and the value function v(k) is the cumulative reward function r(k);

[0054] The immediate reward r is used as the current state s. t Select action a below t Based on the feedback value, the agent continuously optimizes its strategy, selects appropriate actions, and saves the current state s. t The selected action a t The immediate reward r and the next state s t+1 Define the stored quadruple as (s t ,a t ,r t ,s t+1 );

[0055] 4.2.4 Update agent policy settings;

[0056] The agent update strategy adopts an ε-greedy decreasing strategy, which selects the action with the larger reward value based on the quadruple stored in the memory pool with a probability of 1-ε, and selects a random action with a probability of ε to weigh the probability between exploration and exploitation, and sets ε to decrease as the number of iterations increases.

[0057]

[0058] Where ε0 represents the initial ε value, set to 0.9; T cur T represents the current iteration number; DRL Indicates the maximum number of iterations;

[0059] 4.2.5. Termination condition settings;

[0060] The condition for ending agent updates is reaching the maximum iteration value; the current episode reaches the maximum iteration episode T. DRLIf the result is positive, the deep reinforcement learning training is complete, and a theoretically optimal decision scheme is obtained; otherwise, the current state is initialized and training continues.

[0061] The evaluation process of the theoretical optimal solution in step 5 is as follows: compare the state difference between the current physical workshop operating state and the digital twin model operating state. If the state difference is within the preset threshold range, the real-time state of the physical workshop is normal, and the theoretical optimal decision scheme is the optimal decision scheme. If the state difference is not within the preset threshold range, the real-time state of the physical workshop is abnormal. Synchronize the abnormal situation in the digital twin model, obtain and store the physical workshop fault information, and repeat steps 1-4 until the optimal decision scheme is obtained.

[0062] The specific process of synchronizing the real-time physical workshop status in the digital twin model in step 6 is as follows: Based on the real-time status of the physical workshop at the current moment, adjust the digital twin workshop model at the current moment to realize real-time interaction of status data between the physical workshop and the digital twin model.

[0063] The beneficial effects of this invention are:

[0064] (1) Compared with traditional production scheduling decision-making methods, the complex heavy equipment process optimization method based on reinforcement learning and digital twins in this application makes decisions based on real-time information of the production workshop of the heavy equipment physical workshop production line. This makes the method able to cope well with the impact of various disturbances in the aluminum extrusion production line workshop, with high stability and adaptability, and can effectively improve the production efficiency of the production line.

[0065] (2) In this invention, a digital twin model is established for the physical workshop, which enables managers to transcend the constraints of physical space and observe the production status of the production workshop anytime and anywhere. In case of sudden failures, they can also make timely decisions, which is more in line with the actual management needs of the production workshop, thereby maximizing production efficiency and increasing the economic benefits of the factory. Attached Figure Description

[0066] Figure 1 This is a flowchart of the method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins, as described in this invention.

[0067] Figure 2 This is a neural network structure diagram of the complex heavy equipment process optimization method based on reinforcement learning and digital twins of the present invention. Detailed Implementation

[0068] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0069] This invention provides a method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins. Taking an aluminum extrusion production workshop as an example, the process is as follows: Figure 1As shown, it includes the following steps:

[0070] Step 1: Collect workshop information from five aspects—personnel, equipment, materials, methods, and environment—on the aluminum extrusion production line and remove redundant and erroneous information.

[0071] Personnel information mainly includes worker ID, worker name, worker skill type, and worker workload.

[0072] Equipment information mainly includes equipment number, equipment name, equipment type, equipment load level, and equipment wear level;

[0073] Material information mainly includes material number, material name, material type, etc.

[0074] Method information mainly includes method number, method name, method type, and specific method information;

[0075] Environmental information mainly includes environmental number, environmental name, environmental type, and specific environmental information.

[0076] Step 2: Construct a digital twin model of the complex heavy equipment production line workshop;

[0077] The digital twin model is constructed using PlantSimulation based on the physical workshop information of the aluminum extrusion production line workshop in complex heavy equipment, in the manner of man, machine, material, method, and environment. It includes worker twin model, equipment twin model, material twin model, method twin model, and environment twin model.

[0078] Step 3: Determine the description of the aluminum extrusion production line scheduling problem and the constraints during actual processing and production, and establish a mathematical model of the aluminum extrusion production line problem;

[0079] The specific implementation method for step 3 is as follows:

[0080] Aluminum extrusion production lines are crucial for producing high-performance, large-scale industrial materials. The main body of the line consists of three parts: pre-extrusion equipment, the extruder, and post-extrusion equipment. The processing sequence of raw materials into finished products on the aluminum extrusion production line is: aluminum ingot heating → extrusion → quenching → stretching → finished product sawing → length setting → packaging, totaling seven processes. In actual production, different batches of raw materials may have different processing times at the same processing steps. As the number of batches increases, a production method that sequentially arranges batch numbers may lead to excessively long production cycles and delays. Therefore, before undertaking production tasks, the production workshop needs to optimize the scheduling of batches to be processed, aiming to minimize production time by using appropriate decision-making methods to determine production batches, thereby shortening the production cycle and improving production efficiency.

[0081] Based on the production scheduling information obtained from the aluminum extrusion production line workshop, a mathematical model of the problem is established. The aluminum extrusion production line scheduling problem is described as arranging the processing of aluminum materials under limited resources, so that each process of the aluminum material to be processed is assigned to suitable processing equipment and the start time of processing is determined, and finally a production scheduling decision scheme that meets the scheduling optimization objective is obtained.

[0082] The production scheduling problem addressed by this method is to determine the processing sequence of each batch under the condition of satisfying production constraints, with the goal of minimizing the production line's overtime period (OPT). The optimization objective of the production scheduling mathematical model can be established as follows:

[0083]

[0084] Among them, c i This represents the completion time of the i-th batch of aluminum materials, where i = 1, 2...n, S1, S2...S... n The production scheduling is arranged for all batches of aluminum materials, and OPT is the processing time required to process all batches of aluminum materials.

[0085] In the production process, each batch of aluminum is processed continuously, meaning that once the aluminum is completed in the current process, it can be sent to the next process in the same batch flow for further processing. The following four constraints exist in batch production:

[0086] C(σ y ,j)=B(σ y ,j)+Q(σ y , j); y=1, 2,..., Y; j=1, 2,..., j; (2)

[0087] C(σ y ,j)≤B(σ y , j+1); y=1, 2,..., Y; j=1, 2,..., j; (3)

[0088] C(σ y ,j)-C(σ e ,j)≥Q(σ y , j); y=1, 2,..., Y; j=1, 2,..., j; (4)

[0089] C(σ y ,j)-B(σ y ,j)=Q(σ y , j); y=1, 2,..., Y; j=1, 2,..., j; (5)

[0090] Where, Q(σ) y B(σ) represents the processing time of batch y on machine j; y,j) is the start time of batch y on machine j. Equation (2) means that if there is a processing step in the same batch of workpieces, the completion time of the step is equal to the sum of the start time and the processing time of the step; otherwise, the completion time is equal to the start time. Equation (3) means that the completion time of the previous step and the start time of the next step of the same workpiece are related, that is, the next step can only be processed after the previous step is completed. Equation (4) means that the processing equipment in any processing step can only process one batch of workpieces at a time, that is, any two different batches of workpieces y and e cannot be processed on the same step at the same time. Equation (5) means that the completion time of any step minus its start time cannot be less than its processing time.

[0091] Step 4: Optimize the mathematical model using a brainstorming algorithm based on deep reinforcement learning to obtain the theoretically optimal decision scheme;

[0092] The specific implementation method for step 4 is as follows:

[0093] 4.1 Brainstorming to optimize algorithm settings

[0094] 4.1.1 Encoding / Decoding Settings

[0095] Encoding and decoding are the connection between the scheduling scheme and the algorithm. It is the most important step in using the BSO algorithm to solve scheduling problems. Taking into account the characteristics of workshop production scheduling problems, the algorithm adopts integer encoding to prevent the influence of illegal codes.

[0096] For example, if there are 12 batches of workpieces, the coding is as follows, representing the production order in the workshop from left to right. The following coding indicates that workpiece number 5 will be processed first, followed by workpiece number 3, workpiece number 10, and so on, up to workpiece number 8:

[0097] Columns 1 to 12 5 3 10 11 7 12 4 1 2 6 9 8

[0099] The decoding process is to determine the workpiece processing time through the encoding result. Specifically, the encoding result is read from left to right to obtain the batch number of the workpiece to be processed. The production is simulated according to the order of the encoding result. The maximum completion time obtained is the processing time OPT required to process all batches of workpieces under this encoding order.

[0100] 4.1.2 Initialization Settings

[0101] Define the relevant parameters of the brainstorming optimization algorithm for solving the production scheduling problem, such as the number of iterations T, the population size N, and the perturbation probability parameter (P). c ,P np ,P 1ctr ,P 2ctrN initial individuals representing scheduling problem solutions are randomly generated using integer encoding. The initial individuals are generated according to the following formula, where max and min are the upper and lower limits of the initial population parameters, respectively.

[0102] x i,j =min + (max - min) × rand() (6)

[0103] 4.1.3 Population evaluation settings;

[0104] Calculate the initial individual x in the initialized population. i,j The fitness value is calculated by decoding the initial individual and using the reciprocal of the processing time (OPT) required to process all batches of workpieces in the decoded encoding order as the fitness value. The fitness value is calculated using the following formula:

[0105]

[0106] 4.1.4 Population Update Settings

[0107] With a preset probability parameter (P) c ,P np ,P 1ctr ,P 2ctr The initial individuals in the population are updated respectively, where P c The probability of choosing a cluster center instead of the actual cluster center; P np The probability of generating a new individual based on whether to choose one class or both classes; P 1ctr ,P 2ctr The probability of generating a new individual is determined by choosing the class center of one class or the class centers of two classes, respectively.

[0108] There are four ways to update individuals in a population: (a) adding a random perturbation to a random class center, i.e., the best individual in that class, to generate a new individual; (b) randomly selecting an individual in a random class and adding a random perturbation to generate a new individual; (c) randomly merging two class centers and adding a random perturbation to generate a new individual; (d) randomly merging two random individuals from two classes and adding a random perturbation to generate a new individual.

[0109] The formula for generating new individuals is as follows: compare the newly generated individual with the old individual at the same location, and retain the individual with the higher fitness value;

[0110] x s =w1×x s1 +w2×x s2 (11)

[0111] Where, x s1 ,xs2 All are candidate individuals, and w1 and w2 are perturbation factors, where w1 and w2 ∈ (0, 1);

[0112] 4.1.5. Termination condition settings;

[0113] If the initial individual update reaches the maximum number of iterations T, then stop the iteration; otherwise, repeat steps 4.1.2-4.1.4 to reinitialize the population and start a new round of iteration.

[0114] 4.2 Deep reinforcement learning algorithm settings;

[0115] 4.2.1 Constructing a Deep Reinforcement Learning Model

[0116] To optimize the production line scheduling problem using deep reinforcement learning, the production line scheduling problem first needs to be constructed as a deep reinforcement learning model. The state S of the agent's state space consists of an array of fitness values ​​of the population, including the average fitness value, maximum and minimum fitness values, mean squared deviation and standard deviation of the fitness values, and other relevant data. The deep reinforcement learning model is represented as follows:

[0117] S=(f avg ,f max ,f min ,f MSN ,f SD (8)

[0118] Among them, f avg f is the average fitness value of the population. max ,f min f represents the maximum and minimum fitness values. MSN f is the mean squared error of the fitness values. SD For standard deviation and other related data; in order to prevent the experience pool storing historical actions from suffering from the curse of dimensionality due to the complexity of the problem, a convolutional neural network is defined to extract features from the original input data, process the features of the data, and optimize the algorithm performance.

[0119] 4.2.2 Strategy Mechanism Settings;

[0120] A policy is the action an agent takes in response to a given state; it is a mapping from a state to an action. The action space is the set of actions an agent can take, which can be set as the probability of the current perturbation (P). c ,P np ,P 1ctr ,P 2ctr The adjustment of the current disturbance probability is set to three operations: +0.05, +0, and -0.05. Each time, an action 'a' is selected from the action space using an ε-greedy decreasing strategy, which indicates the execution of the current process of the aluminum material.

[0121] 4.2.3 Reward Mechanism Setup;

[0122] Reward is the objective of reinforcement learning. At each time step, the value emitted by the environment to the reinforcement learning system is the reward, which reflects the quality of the agent's action. The agent's task is to maximize the total reward value over a given time period. The immediate reward function r(k) is represented by the improvement in optimal fitness of the current population compared to the previous generation; that is, the difference between the optimal fitness value after the action and the optimal fitness value before the action. The value function v(k) is the cumulative reward function r(k).

[0123] The reward function at time k is expressed as:

[0124] r(k)=v(k)-v(k-1) (9)

[0125] Where r(k) is the reward function at time k, v(k) is the value function, and the value function v(k) is the cumulative reward function r(k);

[0126] With instant rewards r t As the current state s t Select action a below t Based on the feedback value, the agent continuously optimizes its strategy and selects appropriate actions, and saves the current state s. t The selected action a t The instant reward obtained r t and the next state s t+1 Define the stored quadruple as (s t ,a t ,r t ,s t+1 );

[0127] 4.2.4 Update agent policy settings;

[0128] The agent update strategy adopts an ε-greedy decreasing strategy, which selects the action with the larger reward value based on the quadruple stored in the memory pool with a probability of 1-ε, and selects a random action with a probability of ε to weigh the probability between exploration and exploitation, and sets ε to decrease as the number of iterations increases.

[0129]

[0130] Where ε0 represents the initial ε value, set to 0.9; T cur T represents the current iteration number; DRL Indicates the maximum number of iterations;

[0131] 4.2.5. Termination condition settings;

[0132] When the production line reaches the final scheduling completion state, that is, when the agent completes its first phase, the brainstorming optimization algorithm based on deep reinforcement learning is optimized, resulting in a theoretically optimal decision scheme. The optimization operation ends when the maximum iteration value is reached, and the maximum number of iterations T is set. DRL If the current episode reaches the maximum iteration episode T DRL If the current state is not initialized, the deep reinforcement learning training is completed; otherwise, the current state is initialized and training continues.

[0133] Step 5: Evaluate the theoretically optimal decision-making scheme and determine the optimal decision-making scheme;

[0134] Compare the current operating status of the physical workshop of the aluminum extrusion production line with the operating status of the digital twin model. If the difference is within the preset threshold (0-0.5), it indicates that the real-time status of the workshop is normal, and the theoretically optimal decision scheme is the optimal decision scheme. The optimal decision scheme is then used to simulate the aluminum extrusion production line in the digital twin model, and the status of the physical workshop model is synchronously scheduled. If the difference is outside the preset threshold, the real-time status of the workshop is abnormal. The twin workshop model is then synchronously abnormal, the fault information of the physical workshop is obtained, and sent to the database for storage. Steps 1-4 are repeated to reschedule the aluminum profile production line until the optimal decision scheme is obtained.

[0135] Step 6: Input the optimal decision-making scheme into the digital twin model to simulate the production line, and synchronize the real-time physical workshop status in the digital twin model to complete the process optimization of complex heavy equipment.

[0136] After a specific time interval, the state difference between the physical workshop operation status and the digital twin model operation status of the aluminum extrusion production line at the current moment is used to adjust the digital twin workshop model at the current moment based on the real-time operation status of the physical workshop of the aluminum extrusion production line. This enables real-time interaction of state data between the physical workshop and the digital twin model, realizes the digital twin of the complex heavy equipment workshop, and completes the process optimization of complex heavy equipment.

[0137] Step 7: Store the real-time status, fault status, brainstorming optimization algorithm parameters, reinforcement learning model, and activated scheduling scheme of the aluminum profile production line into the database to provide historical data for the reinforcement learning model and scheduling scheme adopted for the corresponding fault status.

[0138] The specific advantages of the method of the present invention are as follows:

[0139] 1) It has strong intelligence and can intelligently select the most suitable scheduling decision scheme according to the current workshop status, avoiding the excessive reliance on scheduling personnel in the traditional way, while making full use of historical production data and improving the efficiency of production decision-making.

[0140] 2) It is highly practical. Dispatchers can remotely monitor the physical production line workshop in real time by observing the situation in the twin production line workshop, and can also handle sudden abnormal situations in a timely manner.

[0141] 3) It has strong universality and can be used not only for the scheduling decision problem of aluminum extrusion production line with complex heavy equipment, but also for the scheduling decision problem of other production lines, and can still achieve relatively ideal results.

[0142] This invention provides a process optimization method for complex heavy equipment based on reinforcement learning and digital twins. It solves the problems in the prior art where the scheduling of aluminum extrusion production lines fails to make full use of historical production data, resulting in low efficiency of production decision-making and the inability to observe the workshop production status in real time. It alleviates the excessive reliance of aluminum extrusion production lines on schedulers, who cannot make decisions on scheduling plans based on the workshop status in real time. The scheduling plan obtained by this method can better adapt to complex production processes.

Claims

1. A method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins, characterized in that, The specific steps are as follows: Step 1: Collect data from the actual physical workshop of complex heavy equipment and preprocess it; Step 2: Construct a digital twin model of the complex heavy equipment production line workshop based on the preprocessed data. Step 3: Determine the description of the physical workshop production line scheduling problem and the constraints during actual processing and production, and establish a mathematical model of the physical workshop production line problem; Step 4: Optimize the mathematical model to obtain the theoretically optimal decision scheme; The mathematical model optimization process employs brainstorming optimization algorithms and deep reinforcement learning algorithms; The optimization process is as follows: 4.1 Brainstorming to optimize algorithm settings 4.1.1 Encoding / Decoding Settings Integer encoding is used for encoding. The decoding process determines the workpiece processing time based on the encoding result. By reading the encoding result from left to right, the batch number of the workpiece to be processed is obtained. Production scheduling is simulated according to the order of the encoding results, and the maximum completion time obtained is the processing time required to process all batches of workpieces under that encoding order. ; 4.1.2 Initialization Settings Define the parameters of the brainstorming optimization algorithm for solving the production scheduling problem, and let the number of iterations be... Population size is The perturbation probability parameter is Randomly generated using integer encoding An initial individual representing a scheduling problem solution is generated according to the following formula. : （6） in, , These are the upper and lower bounds of the initial population parameters, respectively; 4.1.3 Population Evaluation Settings Calculate each initial individual in the initialized population. fitness value , It is expressed as the reciprocal of the maximum completion time, and the specific calculation formula is as follows: （7） in, For fitness value, The processing time required to process all batches of workpieces; 4.1.4 Population update settings; With preset probability parameters The initial individuals in the population are updated separately. The specific process of population update is to compare the newly generated individuals with the old individuals at the same position and retain the individuals with higher fitness values. in, To replace the selection probability of cluster centers; The probability of generating a new individual based on whether to choose one class or both classes; The probability of generating a new individual is determined by choosing the class center of one class or the class centers of two classes, respectively. 4.1.

5. Termination condition settings; If the initial individual update reaches the maximum number of iterations If the condition is met, the iteration stops; otherwise, repeat steps 4.1.2-4.1.4 to reinitialize the population and start a new round of iteration. 4.2 Deep reinforcement learning algorithm settings; 4.2.1 Constructing a deep reinforcement learning model; The state of the agent's state space The deep reinforcement learning model is represented by an array of fitness values ​​of the population: （8） in, The average fitness value of the population. These are the maximum and minimum fitness values. The mean squared deviation of the fitness values. Standard deviation; 4.2.2 Strategy Mechanism Settings; Let the action space be the set of actions that the agent can take, and the probability of the current perturbation. Adjustments are made, setting the current perturbation probability to three values: +0.05, +0, and -0.

05. Each operation... The strategy is to select an action from the action space. This is used to adjust the current perturbation probability; 4.2.3 Reward Mechanism Setup; set up The reward function at time step is: （9） in, for The reward function at time step, Value function, value function For reward function The cumulative total; With instant rewards r t As the current state Select action below Based on the feedback value, the agent continuously optimizes its strategy, selects appropriate actions, and saves the current state. Selected Action Instant rewards received r t and the next state Define the stored quadruple as ,in, Indicates the current state. Indicates the current action. r t Indicates an immediate reward. Indicates the state at the next moment; 4.2.4 Update agent policy settings; Agent update strategy adopts Strategic approach, with 1- The probability is based on selecting the action with the largest reward value according to the four-tuples stored in the memory pool. The probability of choosing a random action is used to balance the probabilities of exploration and exploitation, and is set... An update strategy that increases with the number of iterations; （10） in, Indicates the initial Set the value to 0.9; Indicates the current iteration number; Indicates the maximum number of iterations; 4.2.

5. Termination condition settings; The condition for ending the agent update is reaching the maximum iteration value; the current episode reaches the maximum iteration episode. If the result is positive, the deep reinforcement learning training is complete, and a theoretically optimal decision scheme is obtained; otherwise, the current state is initialized and training continues. Step 5: Evaluate the theoretically optimal decision-making scheme and determine the optimal decision-making scheme; Step 6: Input the optimal decision-making scheme into the digital twin model to simulate the production line, and synchronize the real-time physical workshop status in the digital twin model to complete the process optimization of complex heavy equipment. Step 7: Store the actual physical workshop implementation status, fault status, brainstorming optimization algorithm parameters, reinforcement learning model and the optimal decision scheme information of the complex heavy equipment, and provide historical data for the reinforcement learning model and scheduling scheme adopted for the corresponding fault status.

2. The method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins according to claim 1, characterized in that, The data information mentioned in step 1 includes personnel information, equipment information, material information, method information, and environmental information. The personnel information mainly includes worker ID, worker name, worker skill type, and worker workload information. The equipment information mainly includes equipment ID, equipment name, equipment type, equipment workload, and equipment wear and tear information. The material information mainly includes material ID, material name, and material type information. The method information mainly includes method ID, method name, method type, and specific method information. The environmental information mainly includes environmental ID, environmental name, environmental type, and specific environmental information.

3. The method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins according to claim 1, characterized in that, The preprocessing process described in step 1 is to remove redundant and erroneous information from the data.

4. The method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins according to claim 1, characterized in that, The digital twin model described in step 2 is constructed using PlantSimulation in the manner of man, machine, material, method, and environment; the digital twin model includes worker twin model, equipment twin model, material twin model, method twin model, and environment twin model.

5. The method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins according to claim 1, characterized in that, The production scheduling problem of the physical workshop production line described in step 3 is to determine the production line processing time while meeting production constraints. To determine the processing sequence of each batch of workpieces based on the minimum scheduling objective, the production scheduling mathematical model is established as follows: （1） in, Indicates the first Completion time of a batch of workpieces, , Sort the production schedule for all batches of workpieces. The processing time required to process all batches of workpieces.

6. The method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins according to claim 1, characterized in that, The constraints in the actual processing and production described in step 3 are as follows: each batch in the production operation is continuously processed, that is, the workpiece can be sent to the next process in the batch process flow after the current process is completed. The following four constraints exist in the batch production process: in, For batch In the machine Processing time; For batch In the machine Equation (2) indicates that if there is a processing step in the same batch of workpieces, the completion time of that step is equal to the sum of the start time and the processing time; otherwise, the completion time is equal to the start time. Equation (3) indicates the relationship between the completion time of the previous step and the start time of the next step for the same workpiece, that is, the next step can only be processed after the previous step is completed. Equation (4) indicates that the processing equipment in any processing step can only process one batch of workpieces at a time, that is, any two different batches can be processed at the same time. and The workpieces cannot be processed simultaneously on the same process; Equation (5) means that the completion time of any process minus its start time equals its processing time.

7. The method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins according to claim 1, characterized in that, The evaluation process of the theoretical optimal solution in step 5 is as follows: compare the state difference between the current physical workshop operating state and the digital twin model operating state. If the state difference is within the preset threshold range, the real-time state of the physical workshop is normal, and the theoretical optimal decision scheme is the optimal decision scheme. If the state difference is not within the preset threshold range, the real-time state of the physical workshop is abnormal. Synchronize the abnormal situation in the digital twin model, obtain and store the physical workshop fault information, and repeat steps 1-4 until the optimal decision scheme is obtained.

8. The method for optimizing the process of complex heavy equipment based on reinforcement learning and digital twins according to claim 1, characterized in that, The specific process of synchronizing the real-time physical workshop status in the digital twin model described in step 6 is as follows: Based on the real-time status of the physical workshop at the current moment, adjust the digital twin workshop model at the current moment to realize real-time interaction of status data between the physical workshop and the digital twin model.

Images