Operational optimization system based on digital twin simulation and device operational system comprising the same

Abstract

The present disclosure provides an operation optimization system and a plant operation system. The operation optimization system includes a digital twin simulation module that performs a digital twin simulation based on operation rules, real-time data, and equipment constraint switching conditions and outputs daily production operation information, a reinforcement learning module that selects one of a plurality of operation rules, learns a selection for maximizing final production in a specific time period based on the daily production operation information based on the selected operation rule, and determines an operation plan by using a learned algorithm, and an interface module that transmits information between the digital twin simulation module and the reinforcement learning module.

Description

[0001] This application claims priority to and all benefits arising therefrom of Korean Patent Application No. 10-2025-0050413, filed on April 17, 2025, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] One or more embodiments relate to an operation optimization system based on digital twin simulation and a device operating system including the operation optimization system based on digital twin simulation. Background Technology

[0003] Digital twins are a technology used to reproduce equipment and / or processes as digital images, just as they would in an actual factory. Conditions that might occur in a real factory can be tested in a virtual digital twin factory using a digital twin simulation module, and the condition of equipment in the actual factory can be predicted and diagnosed using data obtained from the simulation.

[0004] Digital twin simulation modules can comprehensively reflect static data such as equipment specifications, operating conditions, drawings, and technical documents in a factory, as well as real-time data collected from sensors and equipment in the actual factory. Therefore, digital twin simulation modules can provide a platform where staff can monitor and collaborate in real time from various locations such as a central control room, maintenance technical room, and the field.

[0005] Operation optimization systems based on digital twin simulation have been studied to maximize equipment operating efficiency in various fields such as the production of display devices and semiconductor devices. Summary of the Invention

[0006] In related technologies, digital twin simulation modules struggle to respond immediately to changing environments (such as equipment maintenance), thus performing only limited functions (such as data analysis and short-term prediction of equipment status). However, these problems are merely examples, and the scope of disclosure is not limited to this.

[0007] One or more embodiments include an operation optimization system and a device operating system including the operation optimization system, the operation optimization system being used to derive an optimal operation plan by training a reinforcement learning module using a digital twin simulation module that takes into account device constraints (preventive tracking (TIP)).

[0008] According to one or more embodiments, an operation optimization system includes: a digital twin simulation module that performs digital twin simulation based on operation rules, real-time data, and equipment constraint switching conditions, and outputs daily production operation information; a reinforcement learning module that selects one of a plurality of operation rules, learns a selection to maximize the final output for a specific period based on the daily production operation information of the digital twin simulation performed based on the selected operation rule, and determines an operation plan by using the learned algorithm; and an interface module that transmits information between the digital twin simulation module and the reinforcement learning module.

[0009] In this embodiment, the digital twin simulation module can assign a level to each of the multiple devices based on selected operating rules, and select the working device based on the level.

[0010] In an embodiment, the digital twin simulation module can select the highest-level device among multiple devices as the working device, and evaluate the device constraint switching conditions of the selected device to perform work using the selected device or select a device of the next lower level as the working device.

[0011] In this embodiment, the reinforcement learning module can select one of multiple operating rules for a round each day, including a specific time period. The digital twin simulation module can output daily production operation information based on the selected operating rule. Furthermore, the reinforcement learning module can repeat the learning process for multiple rounds and determine the combination of operating rules that maximizes the final output as the operation plan.

[0012] In this embodiment, the reinforcement learning module can generate an operation state based on daily production operation information, predict the expected reward obtained by selecting one of multiple operation rules in the operation state, and learn by comparing the expected reward with the daily output of the daily production operation information.

[0013] In one embodiment, the reinforcement learning module can store the experience gained during the learning process in a playback memory and learn by randomly sampling small batches from the playback memory.

[0014] In this embodiment, the reinforcement learning module may include a main network and a target network, and the target network may copy the weights of the main network at a specific time period, maintain the copied weights until the next time period, and output the maximum expected reward.

[0015] In an embodiment, the reinforcement learning module may further include a loss calculator, which calculates a loss function based on the difference between the expected reward and the maximum expected reward, and updates the weights of the main network based on the calculated loss.

[0016] In this embodiment, the main network can estimate the expected reward in the operating state based on the maximum expected reward.

[0017] In an embodiment, the reinforcement learning module can estimate the expected reward of level n based on the nth largest value among the expected rewards of the next operating state, where n is a natural number greater than or equal to 1.

[0018] In an embodiment, when the device constraints of the selected device based on the operation rules cannot be released, the reinforcement learning module can determine the applicable suboptimal operation rules.

[0019] According to one or more embodiments, an equipment operating system includes: a data collection system that collects real-time data from a production line; a data management system that stores the real-time data; and an operation optimization system that performs digital twin simulation based on the real-time data and determines the production line's operation plan based on the results of the digital twin simulation. The operation optimization system includes: a digital twin simulation module that performs digital twin simulation based on operation rules, real-time data, and equipment constraint switching conditions, and outputs daily production operation information; a reinforcement learning module that selects one of multiple operation rules, learns a selection to maximize the final output for a specific period based on the daily production operation information from the digital twin simulation performed based on the selected operation rule, and determines the operation plan using the learned algorithm; and an interface module that transmits information between the data management system, the digital twin simulation module, and the reinforcement learning module.

[0020] In this embodiment, the interface module can send real-time data from the data management system to the digital twin simulation module, send daily production operation information from the digital twin simulation module to the reinforcement learning module, and send the operation rules selected from the reinforcement learning module to the digital twin simulation module.

[0021] In one embodiment, the data collection system may include sensors and an Internet of Things (IoT) network.

[0022] In this embodiment, the data management system can store equipment information, product information, planning information, process information, and operation information.

[0023] In this embodiment, the digital twin simulation module can assign a level to each of the multiple devices based on selected operating rules, and select the working device based on the level.

[0024] In an embodiment, the digital twin simulation module can select the highest-level device among multiple devices as the working device, and evaluate the device constraint switching conditions of the selected device to perform work using the selected device or select a device of the next lower level as the working device.

[0025] In this embodiment, the reinforcement learning module can select one of multiple operating rules for a round each day, including a specific time period. The digital twin simulation module can output daily production operation information based on the selected operating rule. Furthermore, the reinforcement learning module can repeat the learning process for multiple rounds and determine the combination of operating rules that maximizes the final output as the operation plan.

[0026] In this embodiment, the reinforcement learning module can generate an operation state based on daily production operation information, predict the expected reward obtained by selecting one of multiple operation rules in the operation state, and learn by comparing the expected reward with the daily output of the daily production operation information.

[0027] In an embodiment, when the device constraints of the working device based on the operation rules cannot be released, the reinforcement learning module can determine the applicable suboptimal operation rules.

[0028] These general and specific features of the embodiments can be implemented by using a system, method, computer program, or any combination of a system, method, and computer program. Attached Figure Description

[0029] The above and other features of certain disclosed embodiments will become more apparent from the following description taken in conjunction with the accompanying drawings, in which: Figure 1 This is a schematic diagram illustrating a device operating system based on digital twin simulation according to an embodiment; Figure 2 This is a schematic diagram illustrating an operation optimization system according to an embodiment; Figure 3 This is a diagram used to schematically illustrate the reinforcement learning module according to an embodiment; Figure 4 This is a flowchart schematically illustrating a method for deriving an operation plan using an operation optimization system according to an embodiment; Figure 5 This is a flowchart schematically illustrating the execution operations of a digital twin simulation module according to an embodiment; Figure 6 This is a diagram illustrating an example of a device according to an embodiment; Figure 7 This is a flowchart schematically describing the process of deriving operations from an operation plan according to an embodiment; and Figure 8This is a graph showing the learning curve of the reinforcement learning module according to an embodiment. Detailed Implementation

[0030] The invention will now be described more fully below with reference to the accompanying drawings, in which various embodiments are illustrated. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. The same reference numerals throughout denote the same elements.

[0031] The disclosure may include various embodiments and modifications, and specific embodiments thereof are shown in the accompanying drawings and will be described in detail herein. See the following and accompanying drawings. Figure 1 The effects and features disclosed, as well as methods for implementing them, will become apparent from the embodiments described in detail below. However, the disclosure is not limited to the embodiments described below and can be implemented in various forms.

[0032] It will be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or parts, these elements, components, regions, layers, and / or parts should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or part from another. Therefore, without departing from the teaching herein, “first element,” “first component,” “first region,” “first layer,” or “first part” discussed below may be referred to as a second element, second component, second region, second layer, or second part.

[0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, “a,” “the,” and “at least one” do not indicate a limitation on quantity but are intended to include both singular and plural. Thus, a reference to “the” element following a reference to “a” element in the claims includes one element and multiple elements. For example, unless the context clearly indicates otherwise, “a single element” has the same meaning as “at least one element.” “At least one” is not to be construed as limiting “a.” “Or” means “and / or.” As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Furthermore, “at least one of A and B” or “selected from at least one of A and B” means A, B, or A and B. It will also be understood that when the terms “comprising” and / or variations thereof or “including” and / or variations thereof are used in this specification, it indicates the presence of the said feature, region, integral, step, operation, element and / or component, but does not exclude the presence or addition of one or more other features, regions, integrals, steps, operations, elements, components and / or groups thereof.

[0034] It will be understood that when an element is referred to as being "on" another element, the element may be directly on the other element, or there may be an intermediate element between the element and the other element. Conversely, when an element is referred to as being "directly on" another element, there is no intermediate element.

[0035] It will be understood that when a layer, region, or component is referred to as being “connected to” another layer, region, or component, that layer, region, or component may be “directly connected to” the other layer, region, or component, or may be “indirectly connected to” the other layer, region, or component, which has one or more intermediate layers, regions, or components between it and the other layer, region, or component.

[0036] Furthermore, relative terms such as “lower” or “bottom” and “upper” or “top” may be used to describe the relationship between one element and another as shown in the figures. It will be understood that, in addition to the orientation depicted in the figures, relative terms are intended to cover different orientations of the device. For example, if a device in a figure is flipped, an element described as being “below” the other elements will be positioned “above” the other elements. Thus, the term “below” can cover both “below” and “above” orientations depending on the specific positioning in the figure. Similarly, if a device in a figure is flipped, an element described as being “below” or “under” the other elements will be positioned “above” the other elements. Thus, the terms “below” or “under” can cover both above and below orientations.

[0037] Here, when specific embodiments can be implemented differently, a particular process sequence can be performed differently than the described sequence. For example, two consecutively described processes can be performed substantially simultaneously or in the reverse order of the described sequence.

[0038] For ease of description, the dimensions of the elements in the accompanying drawings may be exaggerated. In other words, because the dimensions and shapes of the elements in the accompanying drawings are arbitrarily shown for ease of description, the disclosure is not limited thereto.

[0039] Here, some embodiments can be described as functional blocks and various processing operations. Some or all of these functional blocks can be implemented by any number of hardware and / or software components performing a specific function. Functional blocks can be implemented by one or more processors, or by a circuit configuration for a specific function. Optionally, functional blocks can be implemented using various programming or scripting languages. Optionally, functional blocks can be implemented as algorithms that execute in one or more processors.

[0040] Here, a function performed by one function block can be performed by multiple function blocks, and similarly, a function performed by multiple function blocks can be performed by one function block. Furthermore, the disclosure may employ related techniques for electronic environment setup, signal processing, and / or data processing.

[0041] Here, terms such as “unit” or “module” may refer to a unit used to perform at least one function or operation, which may be implemented by hardware, software or a combination of hardware and software.

[0042] Taking into account the measurements discussed and the errors associated with the measurement of a particular quantity (i.e., the limitations of the measurement system), the terms “approximately” or “roughly” as used herein include stated values ​​and indicate an acceptable range of deviation from a particular value as determined by one of ordinary skill in the art. For example, “approximately” may mean within one or more standard deviations, or within ±30%, ±20%, ±10%, or ±5% of the stated value.

[0043] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will also be understood that terms (such as those defined in a general dictionary) shall be interpreted as having the same meaning as they have in the context of the relevant field and in the context of this disclosure, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0044] Figure 1 This is a schematic diagram illustrating a device operating system based on digital twin simulation according to an embodiment.

[0045] Reference Figure 1 According to the embodiment, the device operating system 1 may include a data collection system 100, a data management system 200, and an operation optimization system 300.

[0046] The data collection system 100 may include sensors 110 and an Internet of Things (IoT) network 120 deployed in the factory and / or production line. The data collection system 100 can obtain information generated from various devices, apparatuses, components, etc. included in the factory, as well as information provided by workers, through the sensors 110 and the IoT network 120.

[0047] Sensor 110 may include various sensors for detecting various information about the plant, such as environmental information, equipment information, and personnel information. In embodiments, for example, sensor 110 may include at least one selected from temperature sensors, humidity sensors, pressure sensors, flow sensors, magnetic sensors, light sensors, sound sensors, image sensors, current sensors, and gas sensors arranged in the plant.

[0048] In an embodiment, sensor 110 may include a positioning sensor for real-time detection of the location information of workers or work-in-process (WIP). Sensor 110 may include photoelectric sensors, ultrasonic sensors, gate sensors, thermal sensors, laser sensors, lidar sensors, WiFi sensors, beacon sensors, ultra-wideband (UWB) sensors, Bluetooth sensors, image analysis sensors, radio frequency identification (RFID) sensors, or cameras.

[0049] The factory's equipment and sensors 110 can be connected to the IoT network 120 via various communication methods such as WiFi, Bluetooth, RFID, and internal network cables to collect and share data in real time. Through the IoT network 120, sensors 110, equipment, or smart devices can interact with each other. In one embodiment, for example, the operation of another device can be automatically controlled based on the status or output of a particular device.

[0050] The data management system 200 can systematically store and manage real-time data DATA1 obtained by the data collection system 100 through sensors 110 and IoT network 120 in the actual factory. Furthermore, the data management system 200 can store and manage technical data (or static data) such as factory equipment specifications, operating conditions, drawings, and technical documents. In an embodiment, for example, the data management system 200 can systematically store and manage real-time data DATA1 and technical data by comprehensively classifying them into equipment information 210, product information 220, planning information 230, process information 240, and operation information 250.

[0051] Equipment information 210 may include the location, installation date, standards, specifications, operating information, maintenance history, or equipment constraint status information of the equipment included in the factory and / or production line. Here, equipment constraints may refer to operational constraints in the production process that restrict the input of WIP to specific equipment. In embodiments, for example, equipment constraints may refer to constraints that can only be released when specific conditions are met due to equipment maintenance, quality issues, etc.

[0052] Product information 220 may include specifications of products manufactured by the factory and / or production line, specifications of intermediate products, design information, production information, or inventory information. Planning information 230 may include production plans, process plans, material management plans, or equipment maintenance plans. Process information 240 may include process parameter data (such as real-time measured temperature, pressure, and flow rate), process progress stages, process control settings, process control algorithms, process history data, or process manuals. Operational information 250 may include the factory's operational plan, operational status and output, operational efficiency analysis and optimization results, equipment constraint switching conditions, or equipment failure prediction data. Here, equipment constraint switching conditions may refer to data and / or logic related to specific conditions used to release equipment constraints.

[0053] In an embodiment, the operation optimization system 300 may include a digital twin simulation module 310, an interface module 320, and a reinforcement learning module 330.

[0054] Equipment information 210, product information 220, planning information 230, process information 240, and operation information 250 from the data management system 200 can be sent to the interface module 320. The interface module 320 can convert this information into standard information DATA2 for configuring the simulation environment through the digital twin simulation module 310, and then send the standard information DATA2 to the digital twin simulation module 310. The standard information DATA2 may include equipment constraint switching conditions and real-time data DATA1 reflecting the actual plant status.

[0055] Interface module 320 can transmit information between digital twin simulation module 310 and reinforcement learning module 330. In an embodiment, for example, interface module 320 can convert an action selected from reinforcement learning module 330 into an operation rule suitable for digital twin simulation module 310, and send the operation rule to digital twin simulation module 310. Here, the operation rule may refer to a standard used to select one of a plurality of devices constituting a production line. In the following, selecting an operation rule via reinforcement learning module 330 may include: converting the action selected from reinforcement learning module 330 into an operation rule suitable for digital twin simulation module 310 or the actual factory via interface module 320, and sending the operation rule to digital twin simulation module 310 or the actual factory.

[0056] In addition, the interface module 320 can convert the daily production operation information output by the digital twin simulation module 310 into information usable by the reinforcement learning module 330, and send the information to the reinforcement learning module 330.

[0057] The digital twin simulation module 310 can generate a digital twin factory by digitally replicating the same environment as the actual factory based on the standard information DATA2 received from the interface module 320. The digital twin factory may include virtual equipment, devices, and components corresponding to those included in the actual factory.

[0058] The digital twin simulation module 310 can perform simulations based on standard information DATA2 and operating rules, and output daily production operation information. The digital twin simulation module 310 can select work equipment based on operating rules, and when the selected equipment is under equipment constraints, it evaluates the equipment constraint switching conditions of the selected equipment to either utilize the selected equipment to perform work or select the next level of equipment as the work equipment.

[0059] Interface module 320 can send daily production operation information to reinforcement learning module 330. Reinforcement learning module 330 can learn choices to maximize the final output for a specific period based on the daily production operation information, and determine the operation plan by using the learned algorithm. A specific period can be defined as an episode, and in an embodiment, an episode can include 10 days.

[0060] The reinforcement learning module 330 can select operating rules daily, and the digital twin simulation module 310 can output new daily production operation information by re-executing the simulation based on the newly synchronized standard information DATA2 and the operating rules. By repeating the round a certain number of times, the reinforcement learning module 330 can learn to derive an optimal operating plan to maximize the final output of a round. Here, the operating plan can be a selection combination of operating rules for a round. In an embodiment, for example, when a round consists of 10 days, the operating plan can be a selection combination of operating rules selected for 10 days. By using the learning algorithm, the reinforcement learning module 330 can determine the optimal (or optimized) operating plan applicable to the actual factory and / or production line.

[0061] In an embodiment, when the device constraints of the device for selecting the optimal operating plan to maximize the final output for a round are not released, the reinforcement learning module 330 can determine and propose an applicable suboptimal operating plan.

[0062] According to an embodiment, since the digital twin simulation module 310 performs simulations based on operating rules and equipment constraint switching conditions, the reinforcement learning module 330, trained through simulation, can derive an optimal operating plan by considering equipment constraints. The operating plan derived by the reinforcement learning module 330 can be sent to the actual factory to be directly reflected in the operation of the actual equipment or used as decision-making information for operations. The equipment operating system 1 is not limited to the operation of a specific process or production line and can be widely used throughout the entire production plant.

[0063] Figure 2 This is a schematic diagram illustrating an operation optimization system according to an embodiment.

[0064] Reference Figure 2 An embodiment of the operation optimization system 300 may include a digital twin simulation module 310, an interface module 320, and a reinforcement learning module 330.

[0065] The digital twin simulation module 310 may include a simulation model 311 and a simulation engine 312. The simulation model 311 may include operation logic, movement control logic, or device constraint switching logic. The operation logic can model the process sequence, working time, resource allocation, etc. of the digital twin factory, and the movement control logic can model the transmission path, movement time, speed, etc. of WIP. When the selected device is in a device constraint state, the device constraint switching logic can evaluate whether the device constraint can be released. If the condition is met, the device constraint of the selected device is released and switched to an available state; otherwise, if the condition is not met, the device constraint state is maintained.

[0066] The simulation engine 312 may include a synchronization unit, an operation plan application unit, a simulation execution unit, or a simulation termination unit. The synchronization unit can synchronize the data of the actual factory with the data of the digital twin factory in real time by reflecting the standard information DATA2 received through the interface module 320 to the digital twin factory. The operation plan application unit can reflect the operation rules selected by the reinforcement learning module 330 to the simulation. The simulation execution unit can execute the simulation model 311 and generate the result value of the simulation model 311, and the simulation termination unit can terminate the execution of the simulation model 311 and output the result value of the simulation model 311.

[0067] Interface module 320 may include a digital twin simulation standard model 321, a first interface application programming interface (API) 322, and a second interface API 323. The digital twin simulation standard model 321 may conform to the ISO23247 international standard, and interface module 320 may support the data management system 200, digital twin simulation module 310, and reinforcement learning module 330 to exchange data with each other in a consistent and efficient manner.

[0068] The digital twin simulation standard model 321 may include a simulation sub-model 321a, an interface sub-model 321b, and a reinforcement learning sub-model 321c. The simulation sub-model 321a may define the information input to the digital twin simulation module 310 and the information output from the digital twin simulation module 310. The information input to the digital twin simulation module 310 may include equipment information 210, product information 220, planning information 230, process information 240, and operation information 250 received from the data management system 200. The information output from the digital twin simulation module 310 may include simulation result values. The simulation result values ​​can be converted into daily production operation information, which includes date, turnaround time (TAT), equipment constraint release count, work-in-process (WIP), WIP count, or daily output.

[0069] To define data exchange between external systems or sub-models, such as the data management system 200, the interface sub-model 321b can perform input / output mapping by assigning input data (ID) and output data (OD). Furthermore, the interface sub-model 321b can define the state switching conditions of the digital twin simulation module 310 and the conditions for interaction between the digital twin simulation module 310 and the data management system 200 or the reinforcement learning module 330.

[0070] The reinforcement learning sub-model 321c may include information input to the reinforcement learning module 330 and information output from the reinforcement learning module 330. The information input to the reinforcement learning module 330 may include state information, reward information, or parameters, and the information output from the reinforcement learning module 330 may include action information. The state information may be configured based on daily production operation information output by the digital twin simulation module 310. The reward information may be the daily output output by the digital twin simulation module 310. The parameters may be values ​​learned or set by the reinforcement learning module 330 and may include weights. The action information may be information about the action of selecting one operation rule from multiple operation rules.

[0071] The first interface API 322 and the second interface API 323 can mediate data exchange between the data management system 200 and the digital twin simulation module 310, and between the digital twin simulation module 310 and the reinforcement learning module 330, based on the digital twin simulation standard model 321. The interface module 320 can ensure compatibility between external systems and sub-modules to allow for smooth data exchange between them. Furthermore, the interface module 320 can independently develop and maintain individual modules, or minimize the impact on the entire system when adding new components.

[0072] The reinforcement learning module 330 may include a reinforcement learning system 331 and an algorithm 332. The reinforcement learning system 331 may include an agent, a neural network, and a replay memory (or replay buffer). The agent can learn based on the algorithm and find an optimal policy. The neural network can receive input from the current state to predict the expected reward (or Q-value) for each action. The replay memory can store the agent's experience gained during the learning process, and the agent can improve the stability of the neural network by learning from mini-batches randomly sampled from the replay memory.

[0073] Algorithm 332 may include an initialization algorithm, an action selection algorithm, an experience storage algorithm, or a Q-value update algorithm. The initialization algorithm initializes the neural network's weights, replay memory, parameters, etc. The action selection algorithm selects the optimal action based on the expected reward. The experience storage algorithm stores the agent's experience gained during the learning process in the replay memory. By using the experience retrieved from the replay memory, the Q-value update algorithm updates the neural network's weights in the direction that minimizes the loss function.

[0074] Figure 3 This is a diagram used to schematically illustrate a reinforcement learning module according to an embodiment.

[0075] Reference Figure 3 An embodiment of the reinforcement learning module 330 may include a playback memory 410, a main network 420, a target network 430, and a loss calculator 440.

[0076] The reinforcement learning module 330 can receive daily production operation information output by the digital twin simulation module 310 from the interface module 320 (see...). Figure 2 Based on daily production operation information, reinforcement learning module 330 can derive operation state s, action a, the immediate reward r obtained when action a is performed in the current state, and the next operation state s' to be reached after performing action a. The operation state s, action a, reward r, and next operation state s' obtained by the agent of reinforcement learning module 330 during the learning process can be stored in playback memory 410.

[0077] The neural network of the reinforcement learning module 330 may include a main network 420 and a target network 430. The main network 420 may receive inputs of operation state s and action a, and output the expected reward Q(s, a; θ) for each action to the loss calculator 440. Here, θ may be the weights of the neural network.

[0078] The target network 430 can be a neural network with the same structure as the main network 420, and can replicate the weights θ of the main network 420 for a specific time period and keep them unchanged until the next time period. The target network 430 can receive the input of the next operating state s' and output the maximum expected reward maxQ(s', a'; θ) to the loss calculator 440. - The maximum expected reward is maxQ(s', a'; θ). - θ is the maximum expected reward of action a' that can be selected in the next operation state s'. Here, θ - These can be the weights of the target network 430. By using the target network 430, the Bellman equation can be applied stably.

[0079] In other words, the reinforcement learning module 330 can predict the expected reward Q(s, a; θ) using Equation 1 below. The reinforcement learning module 330 can learn by comparing the expected reward Q(s, a; θ) with the daily output of the daily production operation information.

[0080] [Equation 1]

[0081] In Equation 1, γ represents a discount factor used to determine the importance of future rewards and satisfies the following inequality: 0 ≤ γ ≤ 1. The function deriving the optimal expected reward Q(s, a; θ) can be computed as a value satisfying the Bellman equation. The Bellman equation provides an equation for updating the state-action value function (Q function).

[0082] The loss calculator 440 can be based on the expected reward Q(s, a; θ) and the maximum expected reward maxQ(s', a'; θ). - The loss function is calculated using the difference between the values ​​of θ and θ, and the weights θ of the main network 420 are updated based on the calculated loss. By using the loss function, the temporal difference (TD) loss (or TD error) can be minimized. The loss function of the loss calculator 440 can be the same as the loss function in Equation 2 below.

[0083] [Equation 2]

[0084] Figure 4 This is a flowchart schematically illustrating an operation plan derivation method using an operation optimization system according to an embodiment.

[0085] Reference Figure 4 An embodiment of the operation plan export method may include a standard information synchronization operation S110, an operation status sending operation S120, a digital twin simulation model execution operation S130, a daily production operation information generation operation S140, a reinforcement learning and action information determination operation S150, and an operation plan export operation S160.

[0086] In standard information synchronization operation S110, the data management system 200 can systematically classify and manage real-time data and technical data collected from the actual factory and / or production line. Interface module 320 can transmit equipment information 210 (see...) Figure 1 Product Information 220 (see product information) Figure 1 ), Plan Information 230 (see) Figure 1 ), Process Information 240 (see) Figure 1 ) and Operation Information 250 (see Figure 1This is converted into standard information DATA2 (see) used for configuring the simulation environment via the digital twin simulation module 310. Figure 1 The system then sends the standard information DATA2 to the digital twin simulation module 310. The digital twin simulation module 310 can reproduce the production line in the digital twin factory based on the standard information DATA2.

[0087] In the operation status transmission operation S120, the digital twin simulation module 310 can configure daily production operation information, including date, TAT, equipment constraint release count, WIP, WIP count, or daily output, based on standard information DATA2, and send the daily production operation information to the reinforcement learning module 330 through the interface module 320. The reinforcement learning module 330 can configure the operation status s based on the daily production operation information and perform learning using a neural network. The reinforcement learning module 330 can determine the learned action 'a' based on the operation status s and send the learned action 'a' to the digital twin simulation module 310 through the interface module 320. The interface module 320 can convert the action 'a' information from the reinforcement learning module 330 into an operation rule suitable for the digital twin simulation module 310 and send the operation rule to the digital twin simulation module 310.

[0088] In the digital twin simulation model execution operation S130, the digital twin simulation module 310 can perform digital twin simulation by applying the operation rules received through the interface module 320. The digital twin simulation module 310 can select the next working device to move the WIP (Work In Progress) to based on the operation rules. In an embodiment, for example, the digital twin simulation module 310 can assign a level to each of multiple devices based on the operation rules and select a working device based on the level. In this operation, when the selected device is in a device constraint state, device constraint switching conditions can be evaluated to either utilize the selected device to perform the work or select a device of the next level as the working device.

[0089] In the daily production operation information generation operation S140, the digital twin simulation module 310 can virtually reproduce the working status of the equipment and export the daily production operation information. In this operation, the daily production operation information may include the daily output, and the interface module 320 can convert the daily output into reward r information for the reinforcement learning module 330 to learn, and send the reward r information to the reinforcement learning module 330.

[0090] In the reinforcement learning and action information determination operation S150, the reinforcement learning module 330 can store operation state s information and action a information in a playback memory by using daily output as a reward r, and update the neural network based on the stored operation state s, action a, and reward r. By using the continuously updated neural network, the reinforcement learning module 330 can return daily action a information based on daily operation state s information and derive a combination of actions a for maximizing the total output for a specific time period. In an embodiment, the specific time period may include 10 days. This process can be configured as a round, and learning can be performed by repeating the round a preset number of times (n times).

[0091] In the operation plan export operation S160, the reinforcement learning module 330, which has completed its learning, can determine the combination of actions to maximize the final output, i.e., the operation plan, and can send the determined operation plan to the actual plant and / or production line for direct application or as reference data for decision-making.

[0092] Figure 5 This is a flowchart schematically illustrating the execution operations of the digital twin simulation module according to an embodiment, and Figure 6 This is a diagram illustrating an example of a device according to an embodiment.

[0093] Reference Figure 5 In the digital twin simulation model execution operation S130, the digital twin simulation module 310 can assign a level to each of the multiple devices according to the operation rules received through the interface module 320, and select the first level (or the highest level) device as the working device based on the level (S131).

[0094] The digital twin simulation module 310 can determine whether the selected device is in a device constraint (i.e., preventative tracking (TIP)) state (S132). When the selected device is not in a device constraint state, the process can be executed by using the selected device as the working device (S139). When the selected device is in a device constraint state, the digital twin simulation module 310 can determine whether the device constraint can be released by considering the device constraint switching conditions (S133). When the device constraint can be released, the device constraint can be released (S134), and the process can be executed by using the selected device as the working device (S139). When the device constraint of the selected device cannot be released, the digital twin simulation module 310 can select the next level of device according to (based on) operating rules (S135).

[0095] The digital twin simulation module 310 can determine whether the selected next-level device is under device constraints (S136). When the selected next-level device is not under device constraints, the process can be executed by using the selected device as the working device (S139). When the selected next-level device is under device constraints, the digital twin simulation module 310 can determine whether the device constraints can be released by considering the device constraint switching conditions (S137). When the device constraints can be released, the device constraints can be released (S138), and the process can be executed by using the selected device as the working device (S139). When the device constraints of the selected device cannot be released, the digital twin simulation module 310 can select the next-level device according to (based on) operating rules (S135).

[0096] Figure 6 This is an illustrative diagram used to illustrate device switching based on operating rules. However, the disclosure is not limited to this. Various modifications can be made to the type, quantity, arrangement, etc., of the equipment and mobile units. (See reference...) Figure 6 A digital twin factory may include a mobile unit 600 and multiple devices obtained by reproducing the equipment of the actual factory in a digital environment. The mobile unit 600 can transport WIP batches from one device to the next.

[0097] In an embodiment, such as Figure 6 As shown, the multiple devices may include a first device 501, a second device 511, a third device 512, a fourth device 513, a fifth device 521, and a sixth device 522. The first device 501 may be a device for performing a first process, and the second device 511, the third device 512, the fourth device 513, the fifth device 521, and the sixth device 522 may be devices for performing a second process performed after the first process. The moving unit 600 may, based on operating rules, transport batches that have passed through the first device 501 to one of the second device 511, the third device 512, the fourth device 513, the fifth device 521, and the sixth device 522.

[0098] Multiple operating rules may include a first operating rule, a second operating rule, a third operating rule, and a fourth operating rule. The first operating rule (i.e., operating rule 1) may assign a higher score to a device due to a decrease in the number of batches in the queue. The second operating rule (i.e., operating rule 2) may assign a score to devices located at the same layer (or level) as the previous process device, and may not assign a score to devices located at different layers (or levels). The third operating rule (i.e., operating rule 3) may assign a higher score to a device due to a decrease in the distance between each device and the previous process device. The fourth operating rule (i.e., operating rule 4) may allocate a score to each device based on the sum of the scores based on the first operating rule, the scores based on the second operating rule, and the scores based on the third operating rule.

[0099] [Table 1]

[0100] In this embodiment, for example, based on operation rule 1, the second device 511 with the highest score can be the highest-level device. Therefore, the digital twin simulation module 310 can select the second device 511 as the working device and determine the device constraint state of the second device 511. When the second device 511 is in a device constraint state, the digital twin simulation module 310 can evaluate the device constraint switching conditions. When the device constraint of the second device 511 can be released, the digital twin simulation module 310 can perform work by using the second device 511, and when the device constraint of the second device 511 cannot be released, the digital twin simulation module 310 can select the fourth device 513, which is the next-level device, as the working device. Based on operation rule 4, the second device 511 can be the highest-level device, and the fifth device 521 can be the next-level device.

[0101] The digital twin simulation module 310 can output daily output based on the operating rules selected by the reinforcement learning module 330 by performing work equipment selection and equipment switching.

[0102] Figure 7 This is a flowchart that schematically describes the process of deriving an operation from an operation plan according to an embodiment.

[0103] Reference Figure 7In this embodiment, the reinforcement learning module 330 can select the optimal operating rule with the maximum output derived using Equation 1 (S161). In this case, the device selected based on the optimal operating rule can be in a device constraint (TIP) state. The reinforcement learning module 330 can determine whether the device constraint of the selected device can be released (S162). When the device constraint of the selected device can be released, the reinforcement learning module 330 can derive an operating plan based on the optimal solution (S167). When the device constraint of the selected device cannot be released, the reinforcement learning module 330 can select an operating rule with the second highest output (S163).

[0104] The reinforcement learning module 330 can determine whether the device constraints of the selected device, based on the operating rule with the second highest output, are releaseable (S164). When the device constraints of the selected device are releaseable, the reinforcement learning module 330 can derive an operating plan based on the optimal solution (S167). When the device constraints of the selected device are not releaseable, the reinforcement learning module 330 can select an operating rule with the third highest output (S165). The reinforcement learning module 330 can determine whether the device constraints of the selected device, based on the operating rule with the third highest output, are releaseable (S166). In other words, the reinforcement learning module 330 can determine the applicable suboptimal operating rule when the device constraints of the selected device, based on a higher-level operating rule with higher output, are not releaseable.

[0105] By using Equation 3 below, the reinforcement learning module 330 can estimate the expected reward Q by using the nth largest value among the k higher expected rewards (Q(s', a'; θ)) of the next operating state s'. k,i (s, a; θ).

[0106] [Equation 3]

[0107] here, Let [i] be the descending sort of the k highest expected rewards Q(s', a'; θ) for the next operating state s', where [i] represents the i-th value in the sorted list. Here, i can be a natural number greater than or equal to 0, and i = n-1.

[0108] For example, the second largest expected reward could be And the third largest expected reward could be Expected value V k (s) can be expressed as equation 4 below.

[0109] [Equation 4]

[0110] In this way, reinforcement learning module 330 can derive and propose the nth level operation rule by using the nth largest expected reward as the expected value instead of choosing the maximum expected reward maxQ(s', a'; θ) as the optimal solution.

[0111] As a comparative example, when the reinforcement learning module only obtains the optimal solution, the applicability and suitability of the operation plan will decrease by proposing an operation plan that does not consider the actual operation of the production line. According to the embodiment, when the selected equipment cannot be operated due to equipment constraints, the reinforcement learning module 330 can improve the applicability and suitability of the operation plan by proposing a suboptimal operation plan.

[0112] Figure 8 This is a graph showing the learning curve of the reinforcement learning module according to an embodiment.

[0113] Figure 8 The reinforcement learning module can be used to perform the drying unit process, and the hyperparameters used in the reinforcement learning module can be the same as those in Table 2 below.

[0114] [Table 2]

[0115] Reference Figure 8 The reinforcement learning module was repeated for approximately 500 rounds, and it can be seen that the final output corresponding to the reward gradually increases based on the progress of the rounds.

[0116] [Table 3]

[0117] Derive the operation plan that yields the highest final output for round 375 out of all rounds. For example, derive the optimal operation plan to apply operation rule 4 on the first day of the simulation, operation rule 1 on days 2 through 5, operation rule 3 on the sixth day, operation rule 2 on the seventh day, and operation rule 3 on days 8 through 10.

[0118] [Table 4]

[0119] Table 4 shows the digital twin simulation results when the optimal operation plan derived from the reinforcement learning module is applied to the drying unit process, and when different operation rules are applied for 10 days without any changes.

[0120] As can be seen, compared to simply applying Operation Rule 4 for 10 days, applying the optimal operation plan derived from the reinforcement learning module increased the number of glass pieces by 1133, resulting in a 4.0% increase in final yield. Furthermore, it can be observed that the equipment constraint release count decreased by approximately 27%, increasing yield and allowing for effective management of equipment constraints.

[0121] The operation optimization system according to the embodiment can be applied not only to a single unit process but also to the entire process of producing the product. Compared to performing optimization by applying the operation optimization system to only one unit process, applying the operation optimization system to the entire process results in a further 1.1% increase in final yield, thus increasing the total final yield by 5.2%.

[0122] As described above, according to the embodiments, an operation optimization system that derives the optimal operation plan by considering equipment constraints (i.e., TIPs) and an equipment operating system including the operation optimization system can be implemented. According to the embodiments, by using the operation optimization system and the equipment operating system including the operation optimization system, equipment operation and decision-making can be supported to improve production line output.

[0123] The invention should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the inventive concept to those skilled in the art.

[0124] Although the invention has been specifically shown and described with reference to embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit or scope of the invention as defined by the claims.

Claims

1. An operation optimization system, comprising: A digital twin simulation module, which performs digital twin simulation based on operation rules, real-time data and equipment constraint switching conditions, and outputs daily production operation information; The reinforcement learning module selects one of a plurality of operating rules, learns the choice to maximize the final output for a specific period based on the daily production operation information of the digital twin simulation based on the selected operating rule, and determines the operation plan by using the learned algorithm. as well as An interface module is provided for transmitting information between the digital twin simulation module and the reinforcement learning module.

2. The operation optimization system according to claim 1, wherein, The digital twin simulation module assigns a level to each of the multiple devices based on the selected operating rules, and selects the working device based on the level.

3. The operation optimization system according to claim 2, wherein, The digital twin simulation module selects the highest-level device among the plurality of devices as the working device, and evaluates the device constraint switching conditions of the selected device to perform work using the selected device or select a device of the next lower level as the working device.

4. The operation optimization system according to claim 1, wherein, The reinforcement learning module selects one of the multiple operation rules for each round, including the specific time period. The digital twin simulation module outputs the daily production operation information based on the selected operation rules, and The reinforcement learning module repeats the learning process for multiple rounds and determines the combination of operation rules that maximizes the final output as the operation plan.

5. The operation optimization system according to claim 4, wherein, The reinforcement learning module generates an operation state based on the daily production operation information, predicts the expected reward obtained by selecting one of the multiple operation rules in the operation state, and learns by comparing the expected reward with the daily output of the daily production operation information.

6. The operation optimization system according to claim 5, wherein, The reinforcement learning module stores the experience gained during the learning process in a replay memory and learns by randomly sampling small batches from the replay memory.

7. The operation optimization system according to claim 5, wherein, The reinforcement learning module includes a main network and a target network, and The target network replicates the weights of the main network during a specific time period, maintains the replicated weights until the next time period, and outputs the maximum expected reward.

8. The operation optimization system according to claim 7, wherein, The reinforcement learning module also includes a loss calculator, which calculates a loss function based on the difference between the expected reward and the maximum expected reward, and updates the weights of the main network based on the calculated loss.

9. The operation optimization system according to claim 7, wherein, The main network estimates the expected reward in the operating state based on the maximum expected reward.

10. The operation optimization system according to claim 7, wherein, The reinforcement learning module estimates the expected reward of level n based on the nth largest value among the expected rewards of the next operating state, where n is a natural number greater than or equal to 1.

11. The operation optimization system according to claim 4, wherein, When the device constraints of the selected device based on the operation rules cannot be released, the reinforcement learning module determines the applicable suboptimal operation rules.

12. A device operating system, comprising: A data collection system that collects real-time data from the production line; A data management system is established to store the real-time data. as well as An operation optimization system performs digital twin simulation based on the real-time data and determines the operation plan for the production line based on the results of the digital twin simulation. The operation optimization system includes: A digital twin simulation module performs digital twin simulation based on operation rules, real-time data, and equipment constraint switching conditions, and outputs daily production operation information; A reinforcement learning module selects one of multiple operating rules, learns from daily production operation information from a digital twin simulation performed based on the selected operating rule to maximize the final output for a specific period, and determines the operating plan using a learned algorithm; and An interface module is provided for transmitting information between the data management system, the digital twin simulation module, and the reinforcement learning module.

13. The device operating system according to claim 12, wherein, The interface module sends the real-time data from the data management system to the digital twin simulation module, sends the daily production operation information from the digital twin simulation module to the reinforcement learning module, and sends the selected operation rules from the reinforcement learning module to the digital twin simulation module.

14. The device operating system according to claim 12, wherein, The data collection system includes sensors and an Internet of Things (IoT) network.

15. The device operating system according to claim 12, wherein, The data management system stores device information, product information, planning information, process information, and operation information.

16. The device operating system according to claim 12, wherein, The digital twin simulation module assigns a level to each of the multiple devices based on the selected operating rules, and selects the working device based on the level.

17. The device operating system according to claim 16, wherein, The digital twin simulation module selects the highest-level device among the plurality of devices as the working device, and evaluates the device constraint switching conditions of the selected device to perform work using the selected device or select a device of the next lower level as the working device.

18. The device operating system according to claim 12, wherein, The reinforcement learning module selects one of the multiple operation rules for each round, including the specific time period. The digital twin simulation module outputs the daily production operation information based on the selected operation rules, and The reinforcement learning module repeats the learning process for multiple rounds and determines the combination of operation rules that maximizes the final output as the operation plan.

19. The device operating system according to claim 18, wherein, The reinforcement learning module generates an operation state based on the daily production operation information, predicts the expected reward obtained by selecting one of the multiple operation rules in the operation state, and learns by comparing the expected reward with the daily output of the daily production operation information.

20. The device operating system according to claim 18, wherein, When the device constraints of the working device based on the operation rules cannot be released, the reinforcement learning module determines the applicable suboptimal operation rules.

Images