Program, control device, and control method

Abstract

To provide a program, a control device, and a control method capable of determining an optimum kneading condition for controlling evaluation values such as a foreign substance amount in a kneaded material and a kneading state of a resin.SOLUTION: Provided is a program to control a kneading machine for producing a kneaded material, which, when executed by a computer, acquires state data including an operation amount of the kneading machine and a foreign substance amount contained in the kneaded material, and which inputs the acquired state data to a first model that is reinforcement-learned based on a reward function defined based on the foreign substance amount and a constraint condition related to the kneading machine, and which determines an operation amount to be applied to the kneading machine on the basis of an output result of the first model to which the acquired state data is input.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a program, a control device, and a control method for controlling a kneader for producing a kneaded product.

Background Art

[0002] Conventionally, in a kneader for producing a kneaded product such as a polymer material, research has been conducted on techniques for improving the quality of the kneaded product. For example, Patent Document 1 discloses a method of using a model in which kneading conditions are learned by reinforcement learning, which is one of machine learning, in order to determine the kneading conditions of a kneader. In this method, the kneading conditions are acquired as state variables, a reward for the determination result of the kneading conditions is calculated based on the acquired state variables, and the kneading conditions are determined from the state variables based on the calculated reward.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When using reinforcement learning technology, it is important to prepare a highly accurate learning model suitable for the manufacturing process of the kneader. Patent Document 1 employs Q-learning as reinforcement learning. However, since Q-learning can only perform discrete control, there may be a problem with the accuracy of the model when applied to the manufacturing process of a kneader that requires continuous control. Also, in reinforcement learning, it is known that the way of giving rewards has a great influence on the accuracy of the model, but no study has been made on the way of giving rewards considering the manufacturing process of the kneader.

[0005] Furthermore, there is a need to reduce the time and effort required to generate the model. Patent Document 1 describes reinforcement learning using data directly obtained from actual equipment. However, when using data obtained from such actual equipment, it is necessary to adjust the operating conditions of the equipment each time in order to obtain various types of data, which makes model preparation time-consuming.

[0006] The object of the present invention is to provide a program, a control device, and a control method that can determine optimal control conditions for controlling evaluation values ​​such as the amount of foreign matter in a kneaded product and the degree of kneading of the resin. [Means for solving the problem]

[0007] (1) A program according to one aspect of the present invention is a program for controlling a kneading machine for manufacturing a compound, which is executed on a computer to acquire state data including the amount of operation of the kneading machine and the amount of foreign matter contained in the compound, inputs the acquired state data to a first model that has been reinforced learning based on a reward function defined based on the amount of foreign matter and constraints on the kneading machine, and determines the amount of operation to be applied to the kneading machine based on the output result of the first model to which the acquired state data has been input.

[0008] (2) In the program described in (1) above, the first model is reinforced using the output data of the second model, which has been trained to output an estimated value of the amount of foreign matter at the next time when the state data at a certain time and the amount to be applied to the kneader at the next time are input.

[0009] (3) In the program described in (2) above, the second model is trained using training data in which the amount of foreign matter at the next time is labeled for combinations of state data at a certain time previously acquired and the manipulated variable at the next time.

[0010] (4) In any of the programs described in (1) to (3) above, the first model is reinforced to output an operator to be applied to the kneader, according to a policy that has been updated so that the reward calculated by the reward function is large when the state data is input.

[0011] (5) In any of the programs described in (1) to (4) above, the reward function is defined such that the reward increases as the amount of foreign matter decreases, and decreases as the amount of foreign matter increases.

[0012] (6) In any of the programs described in (1) to (5) above, the reward function is defined such that the reward decreases when the constraints relating to the kneader are violated, or so on, that the reward decreases as the constraints relating to the kneader are deviated from.

[0013] (7) In any of the programs described in (1) to (6) above, the constraints relating to the kneader are defined based on past experience at the site where the kneader is used.

[0014] (8) In any of the programs described in (1) to (6) above, the constraints relating to the kneader are defined based on the equipment of the kneader or the conditions relating to the production of the kneaded product.

[0015] (9) In any of the programs described in (1) to (8) above, the reward function is defined by a step function, a linear function, or a quadratic function.

[0016] (10) In any of the programs described in (1) to (9) above, the first model is reinforced using a continuous-value control algorithm.

[0017] (11) In any of the programs described in (1) to (10) above, the amount of operation to be applied to the kneader, which has been determined, is displayed on the display unit.

[0018] (12) A control device according to another aspect of the present invention is a control device for controlling a kneading machine for manufacturing a kneaded product, comprising: an acquisition unit that acquires state data including an amount of operation of the kneading machine and an amount of foreign matter contained in the kneaded product; and a determination unit that inputs the acquired state data to a first model that has been reinforced learning based on a reward function defined based on the amount of foreign matter and constraints relating to the kneading machine, and determines an amount of operation to be applied to the kneading machine based on the output result of the first model to which the acquired state data has been input.

[0019] (13) A control method according to one aspect of the present invention is a control method for controlling a kneading machine for manufacturing a kneaded product, comprising: using a computer to acquire state data including the amount of operation of the kneading machine and the amount of foreign matter contained in the kneaded product; inputting the acquired state data to a first model that has been reinforced learning based on a reward function defined based on the amount of foreign matter and constraints relating to the kneading machine; and determining the amount of operation to be applied to the kneading machine based on the output result of the first model to which the acquired state data has been input. [Effects of the Invention]

[0020] According to the program, control device, and control method of the present invention, it is possible to determine optimal control conditions for controlling evaluation values ​​such as the amount of foreign matter in the kneaded material and the degree of kneading of the resin. Furthermore, during the training of the first model, which is a reinforcement learning model, it is possible to advance reinforcement learning at high speed by applying a second model based on supervised learning to the environment. In addition, the first model can perform continuous control by utilizing a continuous value control algorithm. Moreover, in the first model, by setting a reward function for the manipulated variable, it is possible to suppress undesirable operations and to reflect on-site knowledge in the reward function. [Brief explanation of the drawing]

[0021] [Figure 1] It is a schematic diagram showing an example of a processing step in the kneader K according to the embodiment. [Figure 2] It is a diagram showing an example of a device configuration used in the kneading step P3 according to the embodiment. [Figure 3] It is a functional block diagram showing an example of the configuration of the control device 1 according to the embodiment. [Figure 4] It is a flowchart showing an example of the learning process of the environment model M1 according to the embodiment. [Figure 5] It is a diagram for explaining a set of teacher data according to the embodiment. [Figure 6] It is a flowchart showing an example of the learning process of the control model M2 according to the embodiment... [Figure 7] It is a diagram for explaining the flow of reinforcement learning according to the embodiment. [Figure 8A] It is a diagram for explaining the case where the reward function according to the embodiment is defined by a step function. [Figure 8B] It is a diagram for explaining the case where the reward function according to the embodiment is defined by a linear function. [Figure 8C] It is a diagram for explaining the case where the reward function according to the embodiment is defined by a quadratic function. [Figure 9] It is a flowchart showing an example of the operation process using the control model M2 according to the embodiment. [Figure 10] It is a diagram showing an example of the main operation instruction screen PG1 according to the embodiment. [Figure 11] It is a diagram showing an example of the time-series data screen (state prediction screen) PG2 according to the embodiment. [Figure 12] It is a diagram showing an example of the setting screen PG3 according to the embodiment. [Figure 13A] It is a diagram showing the evaluation result of the control model M2 (step function) according to the embodiment. [Figure 13B] It is a diagram showing the evaluation result of the control model M2 (linear function) according to the embodiment. [Figure 13C]This figure shows the evaluation results of the control model M2 (quadratic function) according to the embodiment. [Modes for carrying out the invention]

[0022] Hereinafter, the programs, control devices, and control methods (hereinafter referred to as "programs, etc.") of embodiments of the present invention will be described with reference to the drawings. The programs, etc. of the embodiments are intended to control a kneader that manufactures kneaded products such as polymer materials. The programs, etc. of the embodiments focus on the "amount of foreign matter" contained in the kneaded product, which is an important factor in determining the quality of the kneaded product, and support the determination of optimal control conditions for controlling this amount of foreign matter and evaluation values ​​such as the degree of kneading of the resin. However, the present invention is not limited to the embodiments described below. Furthermore, the components in the embodiments described below include those that can be easily imagined by those skilled in the art, those that are substantially the same, and those within a so-called equivalent range. Moreover, the components disclosed in the embodiments described below can be combined as appropriate.

[0023] [Configuration of mixing machine K] Figure 1 is a schematic diagram showing an example of the processing steps in a kneader K according to an embodiment. The kneader K is, for example, a continuous process type screw kneader. The kneader K is, for example, a melt kneader having one or two screws. The kneader K produces a film-like kneaded material, which is the final product, in rod units (winding) from raw materials such as polymer materials. The processing steps in the kneader K include, for example, a weighing step P1, a blending step P2, a kneading step P3, a heat treatment step P4, a metal detection step P5, a calendering step P6, a cooling step P7, a thickness measurement step P8, a defect detection step P9, a trimmer step P10, and a winding step P11. In Figure 1, the calendering step is shown as an example, but the kneader can also be suitably used in processing steps that include a stretching step instead of a calendering step, or in processing steps such as the production of resin composition pellets.

[0024] For example, in the weighing process P1, raw materials (virgin material) and raw materials (recycled material) are received, and the raw materials are fed into the kneader K by controlling the supply rate per hour. In the blending process P2, raw materials are produced by blending virgin material and recycled material. The weight (proportion) of each raw material, and the respective weights (proportions) of virgin material and recycled material in each raw material are controlled by the control device 1, which will be described later.

[0025] Next, in the kneading process P3, the raw materials are kneaded. Figure 2 shows an example of the equipment configuration used in the kneading process P3 according to this embodiment. In this kneading process P3, equipment comprising a feed section F and a first kneading section N1 and a second kneading section N2 is used. The raw materials blended in the blending process P2 are supplied into the cylinder C by the feed section F, and are crushed into a powder by the rotational movement of the screw S, and then sent from the feed section F to the first kneading section N1 and the second kneading section N2 for kneading. During this time, heat is supplied to the raw materials from the screw S, and further heat supply brings the raw materials into a plasticized state. The raw materials extruded from the second kneading section N2 are sent to the next heat input process P4. The rotational speed and temperature of the screw S and the temperature of the cylinder C are controlled by the control device 1, which will be described later.

[0026] Next, in the heat treatment process P4, the raw materials kneaded in the kneading process P3 are subjected to heat treatment. Then, in the metal detection process P5, metal particles larger than a certain size contained in the kneaded material are detected and removed. Next, in the calendering process P6 and the cooling process P7, film formation treatment is performed on the raw materials. Next, in the thickness measurement process P8, the thickness of the formed film kneaded material is measured, for example, using a beta-ray thickness gauge. Next, in the defect detection process P9, foreign matter contained in the formed film kneaded material is inspected using a camera or the like, and in the trimming process P10, the formed film kneaded material is trimmed. Finally, in the winding process P11, the formed film-like kneaded material is wound up to produce the final product in rod units.

[0027] Various types of foreign matter can be generated in the series of processes described above. Such foreign matter includes, for example, carbides, metallic particles, and fibrous foreign matter. Carbides are formed when the raw materials degrade due to prolonged exposure to heat, causing the resin to burn. For example, in the manufacturing process of kneader K, if polyvinyl chloride (PVC) is included in the raw materials, such carbides are likely to be generated because the thermal decomposition temperature and processing temperature of PVC are close. Metallic particles are, for example, metal parts in kneader K that have been worn away. Fibrous foreign matter consists of debris mixed into the raw materials or during the processes described above. In this embodiment, in particular, optimal control conditions are determined to reduce the amount of foreign matter generated in the kneading process P3.

[0028] [Configuration of Control Device 1] Next, a control device 1 for controlling the operation of the kneader K will be described. Figure 3 is a functional block diagram showing an example of the configuration of the control device 1 according to this embodiment. The control device 1 is connected to the kneader K and the inspection machine IM (foreign object inspection machine) via a communication network NW. The communication network NW includes, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, a wireless base station, etc. The control device 1 may also be connected to other server devices (inspection data recording server, operation data recording server, etc.) via the communication network NW.

[0029] The control device 1 is operated by users such as an administrator who manages the kneader K and inspection machine IM, or on-site workers who operate the kneader K and inspection machine IM on-site. The control device 1 can be implemented using, for example, a personal computer, tablet device, smartphone, or dedicated equipment.

[0030] The control device 1 includes, for example, a control unit 10, a communication interface 20, a display unit 30, an input interface 40, and a storage unit 50. The communication interface 20 is a NIC or the like for performing communication with a kneader K, etc.

[0031] The display unit 30 displays various types of information. For example, the display unit 30 displays a GUI (Graphical User Interface) that accepts various operations from the user. The display unit 30 can be, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or a touch panel. The display unit 30 may be provided separately from the control device 1 and communicate with the control device 1 to display various types of information. Furthermore, if the display unit 30 is implemented as a touch panel, it may also incorporate the functions of the input interface 40.

[0032] The input interface 40 receives various input operations from the user and outputs an electrical signal indicating the content of the received input operation to the control unit 10. The input interface 40 can be implemented, for example, by a keyboard, mouse, touch panel, etc.

[0033] The control unit 10 includes, for example, an acquisition unit 11, a first learning unit 12, a second learning unit 13, a determination unit 14, a display control unit 15, and a kneader control unit 16. Each functional unit of the control unit 10 is realized by a computer such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) executing a program that has been pre-stored in a storage unit 50 (a storage device equipped with a non-transient storage medium). Furthermore, some or all of the functional units of the control unit 10 may be realized by hardware such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), or FPGA (Field-Programmable Gate Array).

[0034] The acquisition unit 11 acquires state data of the kneader K and inspection data of the inspection machine IM via the communication network NW, and stores them in the storage unit 50 as state data SD. The state data SD includes, for example, operation data OD (operational values) for controlling the kneader K, and inspection data IDs indicating the inspection results performed on the kneaded material. The operation data OD includes, for example, the weight of each raw material, the rotation speed of the screw S, the temperature of the screw S, and the temperature of the cylinder C (if there are multiple cylinders, the temperature of each cylinder, such as cylinder 1 (e.g., the cylinder on the inlet side of the kneader K), cylinder 2 (e.g., the cylinder on the outlet side of the kneader K)). The inspection data ID includes, for example, the amount of foreign matter contained in the kneaded material detected in the defect detection process P9. The amount of foreign matter is managed by an index value called foreign matter feature quantity. The foreign matter feature quantity is defined based on a rank assigned according to the size of each foreign matter and a weight for each rank. For example, the foreign matter feature quantity is defined as the sum of the numbers of foreign matter for each rank multiplied by their weights. The acquisition unit 11 is an example of an "acquisition unit". In other words, the acquisition unit 11 acquires state data including the amount of operation of the kneader K and the amount of foreign matter contained in the kneaded product.

[0035] The first learning unit 12 generates an environmental model M1 by performing a learning process using previously acquired state data as training data, and stores it in the storage unit 50. The environmental model M1 is machine-learned using training data in which the amount of foreign matter at the next time is labeled as the correct answer data, based on state data at a certain time (time) and the manipulated quantity at the next time (time) acquired in the past. The environmental model M1 thus generated is trained to output an estimated value of the amount of foreign matter at the next time when state data at a certain time and the manipulated quantity (operation to be performed) to be applied to the kneader at the next time are input.

[0036] Environmental model M1 is a model that algorithmically reproduces the response of a real-world kneader K and reproduces the black-box response using supervised learning (offline simulation environment). Environmental model M1 predicts the state at the next time point, given the state and action at a certain time point, for example, using a gradient boosting decision tree (XGBoost). Environmental model M1 uses only the previous state as an explanatory variable based on a Markov decision process and utilizes a gradient boosting decision tree that enables non-nonlinear prediction. Environmental model M1 is an example of a "second model". Details of the processing of the first learning unit 12 will be described later.

[0037] The second learning unit 13 generates a control model M2 by performing reinforcement learning based on a reward function defined based on the amount of foreign matter and constraints related to the kneader K, and stores it in the memory unit 50. The control model M2 is learned to output an operation to be performed on the kneader K that increases the reward calculated by the reward function when state data for a certain time is input.

[0038] The second learning unit 13 performs reinforcement learning using an algorithm capable of manipulating continuous values. Examples of such algorithms include Actor-Critic algorithms like PPO (Proximal Policy Optimization). PPO is an Actor-Critic model that introduces an Actor who determines an action and a Critic who evaluates the action, and learns the probability distribution of the policy that yields the highest reward using a neural network. PPO stabilizes learning by restricting the ratio of new and old policies within a certain range. Other possible algorithms include A2C / A3C and DDPG. The control model M2 is an example of the "first model." Details of the processing in the second learning unit 13 will be described later.

[0039] The decision unit 14 uses the control model M2 to determine the operation (action) to be performed on the kneader K in the next time step, based on the state data at a given time. The decision unit 14 is an example of a "decision unit". That is, the decision unit 14 inputs the acquired state data into a first model (control model M2) that has been reinforced learning based on a reward function defined based on the amount of foreign matter and constraints on the kneader K, and determines the operation amount to be applied to the kneader K based on the output result of the first model into which the acquired state data has been input. Details of the processing of the decision unit 14 will be described later.

[0040] The display control unit 15 controls the display unit 30 to display an operation instruction screen, etc., which includes information on the amount of operation to be applied to the kneader K determined by the determination unit 14. The display control unit 15 also displays a GUI on the display unit 30 for receiving various inputs and instructions from the user. Details of the processing of the display control unit 15 will be described later.

[0041] The kneader control unit 16 may transmit control signals to the kneader K via the communication network NW to control various operation amounts of the kneader K. Alternatively, the various operation amounts of the kneader K may be set based on instructions entered into the kneader K by a user who has confirmed the operation instruction screen displayed on the display unit 30. Details of the processing of the kneader control unit 16 will be described later.

[0042] The storage unit 50 stores, for example, state data SD, environment model M1, control model M2, etc. The storage unit 50 is a storage device such as an HDD (Hard Disk Drive), RAM (Random Access Memory), or flash memory. The storage unit 50 may also be implemented by other storage devices connected via a communication network NW, such as a NAS (Network Attached Storage) or an external storage server device.

[0043] [Training process for environmental model M1] The learning process of the environment model M1 by the control device 1 will be described below. Figure 4 is a flowchart of an example of the learning process of the environment model M1 according to the embodiment. The process shown in Figure 4 is started, for example, when a user inputs an instruction to execute the learning process of the environment model M1 via the input interface 40. It is assumed that the storage unit 50 stores a set of state data acquired in the past.

[0044] First, the acquisition unit 11 acquires a set of state data SD previously acquired from the storage unit 50 (step S101).

[0045] Next, the first learning unit 12 generates a set of training data using the acquired state data SD (step S103). Figure 5 is a diagram illustrating the set of training data according to the embodiment. The set of training data includes training data TD1, training data TD2, etc. Training data TD1 uses the driving data ((1) to (6)) and inspection data ((7)) at a certain time t-1, and the driving data ((8) to (13)) at the next time t as explanatory variables, with the inspection data ((14)) at the next time t as the dependent variable. Similarly, training data TD2 uses the driving data ((1) to (6)) and inspection data ((7)) at a certain time t, and the driving data ((8) to (13)) at the next time t+1 as explanatory variables, with the inspection data ((14)) at the next time t+1 as the dependent variable. Similarly, training data is generated for combinations of time t+n and time t+n+1. Note that time t is the time required to produce one batch of the mixture, and time t+1 may be the time required to produce the next batch of the mixture.

[0046] Next, the first learning unit 12 performs supervised learning using the set of training data prepared as described above to generate an environmental model M1 (step S105). That is, the first learning unit 12 performs machine learning using training data in which the amount of foreign matter in the next state is labeled to state data at a certain time and state data at the next time that were acquired in the past. The environmental model M1 thus generated is trained to output an estimated value of the amount of foreign matter at the next time when state data at a certain time and the manipulated amount to be applied to the kneader at the next time are input. This completes the processing of this flowchart.

[0047] In the above explanation, we used as an example the case where all items of the driving data ((1) to (6)) and inspection data ((7)) at a certain time, and the driving data ((8) to (13)) at the next time, are used as explanatory variables, but this is not the only example. For example, based on feature engineering or multicollinearity avoidance techniques, one of the combinations with high correlation coefficients from among the explanatory variables (1) to (13) may be excluded and used as training data.

[0048] [Training process for control model M2] The learning process of the control model M2 by the control device 1 will be described below. Figure 6 is a flowchart showing an example of the learning process of the control model M2 according to the embodiment. The process shown in Figure 6 is started, for example, when a user inputs an instruction to execute the learning process of the control model M2 via the input interface 40. It is assumed that the storage unit 50 stores a set of state data acquired in the past and a learned environment model M1.

[0049] First, the acquisition unit 11 acquires state data from the storage unit 50 at a certain time t-1 (initial value) that was acquired in the past (step S201).

[0050] Next, the second learning unit 13 provides the agent (acting entity) with the acquired state data at time t-1 (initial value) to select the operation (action) to be performed on the kneader K at the next time t (step S203). Figure 7 is a diagram illustrating the flow of reinforcement learning according to the embodiment. Initially, after the start of learning, the agent AG selects an action randomly, but as learning progresses, it updates its policy for deciding actions so that it selects the action that maximizes the sum of future rewards.

[0051] Next, the second learning unit 13 inputs the dataset DS1, which includes the acquired state data at time t-1 and the selected action at time t, into the environment model M1 (step S205). As a result, the second learning unit 13 obtains an estimate of the foreign object feature at time t as the output of the environment model M1 (step S207). The second learning unit 13 provides the acquired foreign object feature at time t and the calculated reward rt to the agent AG (step S209).

[0052] Next, the second learning unit 13 determines whether or not the termination conditions are met (step S211). The termination conditions are predetermined by the user or the like. The termination conditions may include the number of repetitions (number of episodes) of the series of processes from steps S204 to S209 above, the learning time, the upper limit of the foreign object features, etc. If the second learning unit 13 determines that the termination conditions are not met (step S211; NO), it acquires the action (time t) selected by the agent AG in step S203 of the previous process as state data for time t (step S213). After that, the second learning unit 13 repeats the processes from steps S203 to S211 again. On the other hand, if the second learning unit 13 determines that the termination conditions are met (step S211; YES), it terminates the learning process.

[0053] In other words, the control model M2 (first model) is reinforced learning using the output data of the second model, which is trained to output an estimated value of the amount of foreign matter at the next time when state data at a certain time and a manipulated variable to be applied to the kneader K at the next time are input. The environment model M1 (second model) is machine learning using training data in which the amount of foreign matter at the next time is labeled for combinations of state data at a certain time and a manipulated variable at the next time that have been acquired in the past.

[0054] [Reward function] The design of the reward function used in the learning process of the control model M2 described above is explained below. The reward function (reward r) in this embodiment is defined as shown in equation (1) below. That is, in addition to the perspective of the amount of foreign matter, the reward r reflects the perspectives of constraints from on-site experience and production / equipment constraints.

[0055] Reward r=r 異物 +r 制約1 +r 制約2 +...Equation (1)

[0056] The "r" in equation (1) above 異物 The section "" shows the reward based on the amount of foreign matter. 異物 The term "for foreign object features" is defined such that, for example, the reward is increased when there are few foreign object features, by adding a value obtained by multiplying the foreign object features by (-1 × coefficient).

[0057] The "r" in equation (1) above 制約1 "r 制約2Each of the following items indicates a reward based on constraints derived from on-site experience or production / equipment constraints. Constraints derived from on-site experience include, for example, if an operation that affects heat generation—cylinder temperature, screw temperature, and screw rotation speed—is performed simultaneously, a negative reward (penalty) is given for the amount increased. Constraints of production / equipment include, if the total amount of recycled material and virgin material exceeds X ± upper and lower limits, a negative reward (penalty) is given for the excess amount, or if an operation item exceeds the upper and lower limit of the production / equipment condition management, a negative reward (penalty) is given based on the set rules.

[0058] In this embodiment, reinforcement learning is performed using multiple patterns of reward functions regarding the magnitude of the reward when the above-mentioned negative reward (penalty) is given. For example, three patterns of reward functions are defined: a step function, a linear function, and a quadratic function. Figures 8A to 8C show the three patterns of reward functions according to the embodiment. Figure 8A is a diagram illustrating the case where the reward function is defined as a step function. In this example, a certain penalty is given when a state variable (for example, an operation variable for a certain operation item) exceeds an upper or lower limit. Figure 8B is a diagram illustrating the case where the reward function is defined as a linear function. In this example, the penalty increases linearly as a state variable moves away from the center of the upper and lower limits. Figure 8C is a diagram illustrating the case where the reward function is defined as a quadratic function. In this example, the penalty increases quadratically as a state variable moves away from the center of the upper and lower limits.

[0059] For example, the second learning unit 13 generates three control models M2 with different definitions of the reward function in the reinforcement learning described above, and stores them in the memory unit 50.

[0060] In other words, the control model M2 (first model) undergoes reinforcement learning to output the manipulated variable to be applied to the mixer K, following an updated policy that increases the reward calculated by the reward function when state data is input. The reward function is defined such that the reward increases as the amount of foreign matter decreases and decreases as the amount of foreign matter increases. The reward function is defined such that the reward decreases when the constraints on the mixer K are violated, or the reward decreases as the constraints on the mixer K are deviated from. The constraints on the mixer K are defined based on past experience in the field where the mixer is used. The constraints on the mixer K are defined based on the equipment of the mixer K or the conditions related to the production of the mixed product. The reward function is defined by either a step function, a linear function, or a quadratic function.

[0061] [Operational processing using control model M2] The following describes the operation process using the control model M2 by the control device 1. Figure 9 is a flowchart showing an example of the operation process using the control model M2 according to this embodiment. The process shown in Figure 9 is started, for example, when a user inputs an execution instruction for the operation process via the input interface 40. It is assumed that the memory unit 50 stores the learned control model M2.

[0062] First, the acquisition unit 11 acquires the current time t status data from the kneader K (step S301). If the current time t status data (inspection data, operation data) is stored in another server device (inspection data recording server, operation data recording server), the acquisition unit 11 may acquire the status data from this other server device.

[0063] Next, the decision unit 14 inputs the acquired time t state data into the control model M2 (provides it to the agent) (step S303). As a result, the decision unit 14 determines the operation (action) to be performed on the kneader K at the next time t+1 based on the output of the control model M2 (step S305).

[0064] Next, the display control unit 15 displays a main operation instruction screen containing information on the operation (action) to be performed on the determined kneader K on the display unit 30 (step S307). Figure 10 is a diagram showing an example of the main operation instruction screen PG1 according to the embodiment. In the main operation instruction screen PG1 shown in Figure 10, a learning model (for example, one of the control models M2 with different definitions of the reward function) can be selected by operating the pull-down menu PD. Depending on the model selection operation, the operation (operation amount) to be performed on the kneader K at the next time t+1 is displayed in area AR1. Also, when the trend display button B1 is pressed, the display control unit 15 displays a time series data screen (state prediction screen) on the display unit 30.

[0065] Figure 11 shows an example of a time-series data screen (state prediction screen) PG2 according to the embodiment. In the time-series data screen (state prediction screen) PG2 shown in Figure 11, the prediction results of foreign object features of multiple learning models are displayed. The user can select the most preferred model on this time-series data screen (state prediction screen) PG2.

[0066] When the user presses the setting button B2 on the main operation instruction screen PG1 in Figure 10, or the setting button B3 on the time-series data screen (state prediction screen) PG2 shown in Figure 11, the screen transitions to the setting screen PG3 shown in Figure 12. The setting screen PG3 has a learning model load / learning model determination button for selecting one of the learning models (control model M2) registered in the memory unit 50, and a learning model reset button for resetting the learning model. When one lot is wound up on the main operation instruction screen PG1 in Figure 10, it receives a signal from the outside and outputs a control signal to the kneader K indicating the amount to be applied to the kneader K at the next time t+1, determined based on the output of the selected learning model (control model M2) (step S309). As a result, the kneader K sets the kneading conditions based on the control signal. Alternatively, the kneader control unit 16 may not output a control signal, only notify the field worker of the determined operating data, and the field worker may set the kneader K. In the time-series data screen (state prediction screen) PG2 in Figure 11, the manipulated amount to be applied to the kneader K and the change in foreign object features from time t+1 onwards are displayed, based on the output of the learning model (control model M2) selected by the time-series data acquisition button. This allows the user to confirm the reduction of foreign object features predicted by the learning model. With this, the processing of this flowchart is completed.

[0067] [Model Evaluation (Simulation Results)] We evaluated control models M2 with a step function, a linear function, and a quadratic function as the reward function. Figure 13A shows the evaluation results of control model M2 (step function) according to the embodiment. Figure 13B shows the evaluation results of control model M2 (linear function) according to the embodiment. Figure 13C shows the evaluation results of control model M2 (quadratic function) according to the embodiment. In Figures 13A to 13C, bar graph BG shows foreign material features, graph L1 shows the amount of virgin material supplied, graph L2 shows the amount of recycled material supplied, graph L3 shows the screw rotation speed, graph L4 shows the screw temperature, graph L5 shows the cylinder 1 (cylinder on the inlet side of kneader K) temperature, and graph L6 shows the cylinder 2 (cylinder on the outlet side of kneader K) temperature.

[0068] In all results from Figures 13A to 13C, simulation evaluations showed that all control models M2 were able to reduce foreign object features in the short term. However, when evaluated up to 10 lots ahead, the foreign object features increased with the control model M2 (step function) shown in Figure 13A. Furthermore, it was confirmed that the control model M2 (quadratic function) shown in Figure 13C was the most stable in reducing foreign object features.

[0069] According to the embodiments described above, it is possible to determine optimal mixing conditions for controlling evaluation values ​​such as the amount of foreign matter in the mixture and the degree of resin mixing. Furthermore, when training the control model M2, which is a reinforcement learning model, it is possible to advance the learning at high speed by applying the environment model M1, which is based on supervised learning, to the environment. In addition, the control model M2 can perform continuous control by using, for example, a continuous value control algorithm. As a continuous value control algorithm, an algorithm using Actor-Critic is particularly preferred. Furthermore, in the control model M2, it is possible to suppress undesirable operations by setting a reward function for the manipulated variable, and it is also possible to reflect on-site knowledge in the reward function.

[0070] Although the present invention has been described above with reference to specific embodiments, each embodiment is presented as an example and does not limit the scope of the present invention. Each embodiment described herein can be modified in various ways within the scope in which the effects of the invention are achieved, and can be combined with features described in other embodiments to the extent that is feasible. [Explanation of symbols]

[0071] 1...Control device, 10...Control unit, 11...Acquisition unit, 12...First learning unit, 13...Second learning unit, 14...Decision unit, 15...Display control unit, 16...Kneader control unit, 20...Communication interface, 30...Display unit, 40...Input interface, 50...Storage unit, K...Kneader, F...Feed unit, S...Screw, C...Cylinder, N1...First kneading unit, N2...Second kneading unit

Claims

1. A program for controlling a kneading machine that manufactures a mixture, which is executed by a computer. The state data including the amount of operation of the kneader and the amount of foreign matter contained in the kneaded product is acquired. The acquired state data is input to a first model that has been reinforced and learned based on a reward function defined based on the amount of foreign matter and the constraints related to the kneader. Based on the output result of the first model into which the acquired state data has been input, the amount to be applied to the kneader is determined. program.

2. The first model is reinforced using the output data of the second model, which has been trained to output an estimated value of the amount of foreign matter at the next time when the state data at a certain time and the manipulated amount to be applied to the kneader at the next time are input. The program according to claim 1.

3. The second model is machine-learned using labeled training data for combinations of state data acquired in the past at a certain time and the manipulated variable at the next time, with respect to the amount of foreign matter at the next time. The program according to claim 2.

4. The first model is reinforced to output the manipulated quantity to be applied to the kneader, according to a policy updated so that the reward calculated by the reward function increases when the state data is input. The program according to any one of claims 1 to 3.

5. The reward function is defined such that the reward increases as the amount of foreign matter decreases, and decreases as the amount of foreign matter increases. The program according to claim 4.

6. The reward function is defined such that the reward decreases as the constraints on the kneader are violated, or so it is defined such that the reward decreases as the constraints on the kneader are deviated from. The program according to claim 4.

7. The constraints relating to the aforementioned kneader are defined based on past experience at the site where the kneader is used for work. The program according to any one of claims 1 to 3.

8. The constraints relating to the kneader are defined based on the equipment of the kneader or the conditions relating to the production of the kneaded product. The program according to any one of claims 1 to 3.

9. The aforementioned reward function is defined by one of the following: a step function, a linear function, or a quadratic function. The program according to any one of claims 1 to 3.

10. The first model is reinforced using a continuous value control algorithm. The program according to any one of claims 1 to 3.

11. Furthermore, the amount of operation to be applied to the kneader, which has been determined, is displayed on the display unit. The program according to any one of claims 1 to 3.

12. A control device for controlling a kneading machine that manufactures a kneaded product, An acquisition unit that acquires state data including the amount of operation of the kneading machine and the amount of foreign matter contained in the kneaded product, A determination unit inputs the acquired state data to a first model that has been reinforced learning based on a reward function defined based on the amount of foreign matter and the constraints related to the kneader, and determines the amount of operation to be applied to the kneader based on the output result of the first model into which the acquired state data has been input. A control device equipped with the following features.

13. A control method for controlling a kneading machine used to manufacture a compounded product, using a computer, The state data including the amount of operation of the kneader and the amount of foreign matter contained in the kneaded product is acquired. The acquired state data is input to a first model that has been reinforced and learned based on a reward function defined based on the amount of foreign matter and the constraints related to the kneader. Based on the output result of the first model into which the acquired state data has been input, the amount to be applied to the kneader is determined. Control method.

Images