Anomaly Determination Device and Anomaly Determination Method

Abstract

Based on the current state of production and logistics, the degree of abnormality in production and logistics that could lead to future problems will be appropriately determined. [Solution] The abnormality determination device (200) comprises a feature acquisition unit (202) that acquires feature quantities output from the intermediate layer of a neural network model that plans production logistics, and an abnormality determination unit (204) that determines the degree of abnormality in the production logistics based on the feature quantities.

Description

Technical Field

[0001] The present disclosure relates to an abnormality degree determination device and an abnormality degree determination method that can be used for determining the abnormality degree in production logistics.

Background Art

[0002] In production logistics management, it is important to judge whether the state of production logistics is normal or abnormal. For example, if the inventory quantity stored in front of production equipment is less than an appropriate level, equipment stoppage due to material shortage will occur, and if it is more than the appropriate level, the storage space capacity will be tight, which may induce stoppage of equipment in the upstream process. In addition to the inventory quantity, various factors such as the manufacturing specifications of intermediate products, equipment processing sequence, placement method in the storage area, and the operating state of conveying equipment and facilities will affect the system, leading to logistics abnormalities that result in logistics troubles such as equipment stoppage. Therefore, in order to prevent logistics troubles in advance, it is critical to detect logistics abnormalities at an early stage in the management of production logistics.

[0003] However, since abnormalities in production logistics occur as a result of the combined influence of various factors, it is difficult to determine a logistics abnormality only by monitoring a specific index (such as the inventory quantity of a certain process). For this reason, logistics abnormalities cannot be automatically detected by a computer or the like, and reliance has had to be placed on the intuition and experience of skilled operators at the production site.

[0004] Patent Document 1 discloses a bottleneck visualization device for visualizing a logistics stagnation state without executing complex simulations for work performed by a large number of various types of equipment.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the bottleneck visualization device described in Patent Document 1 displays information about bottlenecks in production logistics based on calculated simulation results. In other words, bottleneck visualization depends on the simulation (generated work plan). Therefore, it is not possible to appropriately determine the degree of abnormality in production logistics that could lead to future problems based on the current state of production logistics.

[0007] One aspect of this disclosure aims to appropriately determine the degree of abnormality in production logistics that could lead to future problems, based on the current state of production logistics. [Means for solving the problem]

[0008] To solve the above problems, an abnormality determination device according to one aspect of the present disclosure comprises a feature acquisition unit that acquires feature quantities output from an intermediate layer of a neural network model that plans production logistics, and an abnormality determination unit that determines the degree of abnormality in the production logistics based on the feature quantities.

[0009] To solve the above problems, an abnormality determination method according to one aspect of the present disclosure includes a feature acquisition step of acquiring feature quantities output from an intermediate layer of a neural network model that plans production logistics, and an abnormality determination step of determining the degree of abnormality in the production logistics based on the feature quantities. [Effects of the Invention]

[0010] According to one aspect of this disclosure, it is possible to appropriately determine the degree of abnormality in production logistics that could lead to future problems, based on the current state of production logistics. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example of a production logistics system according to Embodiment 1 of this disclosure. [Figure 2]This is a block diagram showing the configuration of a learning system that trains the planning model used by the planning device. [Figure 3] This is a schematic diagram illustrating an example of what a production and logistics system will determine the degree of abnormality of. [Figure 4] This figure shows an example of a neural network model that constitutes a planning model. [Figure 5] This flowchart shows an example of the learning process for a planning model using the planning device disclosed in this disclosure. [Figure 6] This flowchart shows an example of the abnormality determination process performed by the planning device disclosed herein. [Figure 7] This figure shows an example of a table stored in the third memory unit of the learning system. [Figure 8] This figure shows an example of a table stored in the third memory unit of the learning system. [Figure 9] This figure shows an example of a matrix that represents the current state information to be input into the planning model. [Figure 10] This figure shows an example of a graph output by the abnormality score output unit. [Figure 11] This is a block diagram showing an example of a production logistics system according to Embodiment 2 of this disclosure. [Figure 12] This figure shows an example of a table stored in the third memory unit of the learning system. [Figure 13] This figure shows an example of a table stored in the third memory unit of the learning system. [Modes for carrying out the invention]

[0012] [Embodiment 1] One embodiment of this disclosure will be described in detail below.

[0013] [Overview of the production and logistics system] With reference to Figures 1 to 4, an overview of the production logistics system 1 including the abnormality degree determination device 200 will be described. Figure 1 is a block diagram showing an example of the production logistics system 1. The production logistics system 1 is a system that performs production management including the production volume of each product, processing timing, processing sequence, and conveyance of articles including intermediate products from previous processes and materials used for the production thereof, in a facility such as a factory to which a production plan is applied. In the production logistics system 1, production logistics is performed in accordance with a production plan drafted by a drafting model 20 described later.

[0014] As shown in Figure 1, the production logistics system 1 includes a production logistics device 80, a planning device 100, an abnormality degree determination device 200, a learning system 50, and a display unit 60.

[0015] The production logistics device 80 includes, for example, a conveyance device 300 and a manufacturing device 400. The conveyance device 300 is a device that conveys articles. The manufacturing device 400 is a device that performs various processes for producing products. Each device included in the production logistics device 80 is not particularly limited as long as it is a device used for production logistics.

[0016] Figure 3 is a schematic configuration diagram showing an example of a target for which the production logistics system 1 performs abnormality degree determination. The production logistics system 1 is provided, for example, in association with a finishing factory that finishes slabs as semi-finished products in a steel production process. In each of the storage areas P01 to P09 divided in the yard, one or more slabs are stacked and placed, or although not illustrated, the slabs are not placed for rearrangement. The slabs placed in each of the storage areas P01 to P09 are carried into the finishing line by the first crane device 320A. In the following description, the number of slabs that the first crane device can lift at one time is described as one, but the present invention is not limited to this case.

[0017] In the finishing line, the slabs brought in by the first crane device 320A are placed on the first trolley 310A. The first trolley 310A transports the placed slabs. The first processing step is performed on the slabs placed on the first trolley 310A by the first manufacturing device 410A. In the first processing step, processes such as grinding with a grinder or melting with a scarfer are carried out.

[0018] After the first processing step is completed, the slab placed on the first trolley 310A is transferred to the second trolley 310B in a transfer process. The second trolley 310B transports the placed slab. After the transfer process, the slab placed on the second trolley 310B is subjected to a second processing step by the second manufacturing device 410B. In the second processing step, processes such as grinding with a grinder or melting with a scarfer are performed. After the second processing step is completed, the slab placed on the second trolley 310B is placed on one of the storage areas P01 to P09 by the second crane device 320B.

[0019] The trolleys 310A and 310B, and the crane devices 320A and 320B shown in Figure 3 are examples of the transport device 300. The manufacturing devices 410A and 410B are examples of known manufacturing devices 400. Furthermore, the production logistics system 1 may include not only the first and second processing steps, but also a greater number of steps or different types of steps, as targets for abnormality determination for the factory to which the production plan is applied. For example, in addition to the steps described above, a step (hereinafter simply referred to as the relocation step) may be performed by, for example, the first crane device 320A or the second crane device 320B to change the location of the slabs placed in the yard.

[0020] Returning to Figure 1, the planning device 100 is a device that plans production and logistics in a factory to which the production plan is applied, using a neural network model (planning model 20) for planning production and logistics. Furthermore, it has the function of controlling each device provided by the production and logistics system 1. The planning device 100 includes a state acquisition unit 101, an output unit 102, and a first storage unit 120.

[0021] The status acquisition unit 101 is a functional unit that acquires status information, which is information indicating the current state of the factory to which the production plan is applied, that is, information indicating the production status and operation information of the production logistics equipment 80 at the current time. For example, the status acquisition unit 101 acquires location information indicating the position of each transport device 300 and manufacturing equipment 400 from signals obtained from GPS (Global Positioning System), cameras, or position sensors as status information. The status acquisition unit 101 may also acquire information indicating the operation status of the transport device 300 and manufacturing equipment 400 from their respective controllers as status information. The status acquisition unit 101 provides the acquired status information to the planning model 20, which will be described later.

[0022] The output unit 102 is a functional unit that outputs to the transport devices 300 and manufacturing devices 400 of the production logistics system 80 the next action that the factory (its transport devices 300 and manufacturing devices 400) should take, as selected by the planning model 20. More specifically, the output unit 102 outputs to the transport devices 300 and manufacturing devices 400 of the production logistics system 80 an action command corresponding to the action selected by the planning model 20. As a result, the transport devices 300 and manufacturing devices 400 of the production logistics system 80 are controlled according to the action selected by the planning model 20.

[0023] The first memory unit 120 is a memory or other storage medium that stores data and programs used by the planning device 100 in a readable format, and stores a planning model 20, which is a neural network model for planning production logistics in a factory to which the production plan is applied.

[0024] The planning model 20 is a reinforcement learning agent model that has been reinforced to output actions in response to information indicating the production status at the factory to which the production plan is applied. The neural network model that constitutes the planning model 20 is not limited to a reinforcement learning agent model, but may also be a machine learning model that has been trained by machine learning, such as supervised learning. For example, it may be a machine learning model that has learned the relationship between input and output using supervised learning with an input / output dataset that includes the production status as input and the optimal action determined by past actions or optimization methods, etc. In the production logistics system 1, various devices equipped in the production logistics system 1 are controlled according to the production plan formulated by the planning model 20 used by the planning device 100.

[0025] The planning model 20 can be configured to learn the action-value function such that the TD (Temporal Difference) error, which is calculated from the state-action value calculated before the action and the reward of the action, approaches zero. The action-value function is a function that shows the expected value of the cumulative reward when a certain action is taken in a certain initial state under a certain policy.

[0026] Figure 2 is a block diagram showing the configuration of the learning system 50 that learns the planning model 20 used by the planning device 100. The learning system 50 generates the planning model 20 for creating production plans using reinforcement learning. Hereinafter, the factory to which the production plan is applied (i.e., the factory to which the degree of abnormality is determined) will be simply referred to as a factory. In this explanation, the factory will be described as producing products or semi-finished products (hereinafter referred to as products, etc.), but the items produced in the factory are not particularly limited and may be parts or intermediates.

[0027] As shown in Figure 2, the learning system 50 comprises a learning control unit 10 and a third storage unit 30, and is connected to the planning model 20 stored in the first storage unit 120. The learning control unit 10 comprises an action design unit 11, a reward design unit 12, a plan information acquisition unit 13, a plan information setting unit 14, a current state input unit 15, an action calculation unit 16, an action setting unit 17, a state transition unit 18, a reward calculation unit 19, and a planning model update unit 21.

[0028] In reinforcement learning, generally, an agent (corresponding to the planning model 20 in this embodiment) observes the state of the environment (corresponding to the current state of the factory calculated by the production model 34 in this embodiment) and selects a certain action, which causes the environment to change based on that action. As a result of these environmental changes, the agent is given some kind of reward, which helps it learn to select better actions.

[0029] The behavior design unit 11 is a functional unit that acquires behavior design information 31, which is information indicating the selectable actions (go left, go right, place, rearrange, melt, etc.) of the planning model 20, which is an agent, from, for example, the third storage unit 30. The learning control unit 10 can define the actions that the agent learned by the learning control unit 10 can select by acquiring the behavior design information 31 via the behavior design unit 11. The actions that the agent can select may be dynamically changed according to the current state of the factory to which the production plan is applied. In addition, the behavior design information 31 may be input by the operator via an input unit (not shown).

[0030] As described above, the behavioral design information 31 is information that indicates the selectable actions of the planning model 20, which is the agent. The behavioral design information 31 is generated in advance by the user for each transport device 300 and manufacturing device 400 of the factory to which the production plan is applied, and is stored in the third storage unit 30. The selectable actions of the planning model 20 can be determined according to the various tasks performed in the factory to which the production plan is applied. In this embodiment, the agent (planning model 20) will select the following actions. Job type ∋{Loading, Loading, Relocation} Slab IDs: {001, 002, ...} Storage area ∋{01, 02, ...}

[0031] If you select "Export" as the job type, you do not need to select the slab ID or location. If you select "Inbound" as the job type, select the slab to be transported from the slab ID, and you do not need to select the location. If you select "Relocation" as the job type, select the slab to be relocated from the slab ID and select the destination location.

[0032] The reward design unit 12 is a functional unit that defines a reward function for calculating the reward obtained as a result of the planning model 20 outputting an action in a given state. The reward function is a function that outputs a reward when at least one of the action and the transitioned state is input. The reward function may be, for example, a mapping with a rule for one-to-one correspondence, a mathematical formula, or a method that replaces calculation by showing values ​​corresponding to the calculation result of the reward function in a table format.

[0033] The reward design unit 12 obtains reward design information 32 from the third storage unit 30, which is information indicating the arguments and format for defining the reward function, and generates the reward function by determining the arguments and format. The reward design information 32 is generated in advance by the user and stored in the third storage unit 30.

[0034] If the reward function is a function, the arguments to the reward function may include the elapsed time in the current state and the elapsed time when all tasks are completed (makespan). The reward function may calculate a positive reward if the elapsed time is below a standard numerical range, and a negative or zero reward if the elapsed time exceeds the standard numerical range. In addition, the method of giving rewards can be set as a reward obtained for each action (immediate reward), the sum of rewards that will be obtained by the final action (cumulative reward), or a reward obtained at the time of the final action (episodic reward).

[0035] The planning information acquisition unit 13 is a functional unit that acquires planning information 33, which is information necessary when creating a production plan at a factory to which the production plan is applied, from the third storage unit 30 or from outside the production logistics system 1. The planning information 33 is stored in the third storage unit 30 by being generated in advance by the user or acquired from an external source. Preferably, the planning information 33 includes information related to future schedules and (directly or indirectly) time when manufacturing products, and for example, the identifier (ID) of the slab to be processed, the location of each slab, information indicating which layer each slab is stacked on, the delivery date of each slab, and information indicating the time required for the manufacturing process of each slab (for example, the first processing process and the second processing process).

[0036] The planning information acquisition unit 13 only needs to acquire appropriate planning information 33 according to the production plan being formulated, and the content of the planning information 33 is not particularly limited as long as it is information related to the production plan. The planning information acquisition unit 13 may use past performance as planning information 33, may randomly generate planning information using random numbers, or may combine multiple of these. The planning information acquisition unit 13 acquires the state information acquired by the state acquisition unit 101 together with the planning information 33.

[0037] The planning information setting unit 14 is a functional unit that sets (inputs) the acquired or generated planning information 33 and status information into the production model 34.

[0038] The current state input unit 15 is a functional unit that inputs information indicating the current state, that is, the current production status at the factory to which the production plan simulated in the production model 34 is applied, into the planning model 20. In other words, the current state input unit 15 outputs information indicating the current production status at the factory as the state of the environment to the planning model 20, which acts as an agent.

[0039] Figure 7 shows an example of a table stored in the third memory unit 30 of the learning system 50. In the first learning cycle for training the design model 20, for example, the information shown in T1 and T2 in Figure 7, which represent the initial state, is input. The current state input unit 15 converts the current state into a vector, matrix, tensor, etc., so that it can be used as input data for the design model 20, and then inputs it into the design model 20.

[0040] Figure 9 shows an example of a matrix representing the current state information to be input into the design model 20. T4 and T5 in Figure 9 represent information about the slabs placed in each storage area P01 to P09. T4 in Figure 9 shows the time required for the slab manufacturing process (first processing step and second processing step). In T4, the rows represent storage areas P01 to P09, and the columns represent information indicating which layer the slab is stacked on. For example, in T4, the value "6" shown in the first row from the bottom and the first column from the left represents the time required for the manufacturing process (first processing step and second processing step) of the slab placed at the bottom of storage area 01.

[0041] Figure 9 shows T5, which indicates the delivery date of the slabs. In T5, the rows represent storage locations P01 to P09, and the columns represent information indicating the delivery date of the slabs. For example, in T5, the value "3" shown in the first column from the left represents the delivery date of the slab placed in storage location 01.

[0042] The "-1" shown in T4 and T5 indicates that a slab is not present. For example, in T4 and T5, no slab is placed in storage area P02.

[0043] The current status input unit 15 can use information that defines the production status of the factory to which the production plan is applied, such as information indicating which process the production plan has reached (internal clock information in the simulator), or the current status of resources related to the production of products, etc. The status of the resources may include the progress of individual manufacturing processes carried out within the factory, the status of transportation of products or materials, the availability of storage space at the destination, the operating status of factory equipment, or whether or not there are any abnormalities in the factory equipment.

[0044] The action calculation unit 16 uses the planning model 20, which contains information indicating the current state, to calculate the next action that the factory (its transport equipment 300 and manufacturing equipment 400) to which the production plan applies should take, via the planning model 20.

[0045] The actions output by the planning model 20 correspond to one or more processes performed as a series of operations among the multiple processes that constitute the production plan being planned, and correspond to the operation of some device in the factory equipment to which the production plan is applied or to work instructions given to workers. That is, when the planning model 20 receives status information indicating the production status, it calculates information as action information that identifies which device to operate and how, or what work to instruct workers to do, in the next process in the production status, and outputs it to the action calculation unit 16. The size of the action group output by the planning model 20 is not particularly limited, and may be the smallest action to be performed in the production plan, or it may be a group containing multiple such smallest actions. The smallest action may be, for example, the action of selecting a slab to be applied to the manufacturing process (first processing process and second processing process). The action calculation unit 16 may also select an action by selecting one of the pre-set dispatching rules.

[0046] In this embodiment, for example, we assume that the planning model 20 calculates the action of bringing slab ID 003 onto the line from the state shown in Figure 7. In this case, the next action output by the planning model 20 will be {job type, slab ID, location} = {bring in, 003, -}. Note that when the planning model 20 has never been trained (updated), random numbers are (initially) set for the weights and biases of the neural network that constitutes the planning model, so the planning model 20 outputs random actions.

[0047] The action calculation unit 16 may select the next action to take from production rules in order to model the dynamic aspects of the factory. As production rules, for example, the planning model 20 may select an action from parts input rules that determine the timing of inputting materials such as products, or from dispatching rules. The dispatching rule is a rule that determines which processing target to assign when the manufacturing equipment becomes ready for processing, when multiple processing targets are waiting to be processed in a buffer in front of the manufacturing equipment in the factory.

[0048] The action setting unit 17 is a functional unit that sets the action calculated by the action calculation unit 16 into the production model 34. The state transition unit 18 is a functional unit that calculates the state to which the system transitions after an action via the production model 34. Figures 12 and 13 show an example of a table stored in the third storage unit 30 of the learning system 50.

[0049] The action setting unit 17 acquires, for example, T2 in Figure 7, which shows the line status being processed on the current finishing line, and T1 in Figure 7, which shows the slab inventory status, and inputs (sets) the action into the production model 34 to generate the next state of the factory (the new current state). Based on the action {job type, slab ID, location} = {delivery, 003, -} set in the production model 34 by the action setting unit 17, the state transition unit 18 calculates in the production model 34 the transitioned state, which is as shown in Figure 12. Since there are a maximum of two slabs on the finishing line at the same time, when the processing of slab ID 003 is completed, slab ID 002 is placed on trolley No. 2, and trolley No. 1 returns to the line delivery position, slab ID 003 is put into the finishing line. The trolley origin return time indicates the time when the trolley that transported the slab returns to its original position (i.e., the time when the next slab can be transported to the trolley). These times are calculated by the simulator, i.e., the production model 34.

[0050] To elaborate, if the next action selected is to move slab ID 004 from location 01 to location 04, the state after the transition is shown in Figure 13. In this case, since no slab is delivered to the line, T2 in Figure 13, which shows the line state at the destination, does not change from T2 in Figure 12, which shows the line state up to the transition.

[0051] The reward calculation unit 19 is a functional unit that calculates the reward to be received based on the reward function defined in the reward design unit 12. Specifically, the reward calculation unit 19 calculates the reward by inputting at least one of the following into the reward function defined in the reward design unit 12: the planning information 33 and state information set in the production model 34 by the planning information setting unit 14, or the state to which the production model 34 will transition after an action, as calculated by the state transition unit 18. In calculating the reward, it is also possible to calculate the reward obtained for each action (immediate reward), or to calculate future rewards such as the total reward that will be obtained by the final action (cumulative reward), or the reward obtained at the time of the final action (episodic reward). The calculated reward serves as an indicator of whether the action calculated by the production model 34 is appropriate or not. The calculated reward is output to the planning model update unit 21.

[0052] The planning model update unit 21 updates the parameters of the action value function in the planning model 20 based on the rewards obtained from the reward calculation unit 19 and the rewards that can be received in the future. In addition, the parameters of the state value function may be updated separately from or in addition to the action value function. The action value function represents the expected value of the cumulative reward when a certain action is taken in a certain state. The state value function is the expected value of the cumulative reward calculated after considering all possible actions. Parameters refer to weights and biases when the value function is a neural network.

[0053] The planning model update unit 21 performs action calculations, simulation execution, and updates to the planning model 20 (updating the weighting coefficients in the network of the planning model 20) based on the new current state set in the production model 34, until the episode is completed.

[0054] In this embodiment, the period from the start of production until the completion of production of all slabs is treated as one episode, but the episode may be completed based on the elapsed time in the simulation (when it reaches a predetermined time). After one episode is completed, the current state is returned to the initial state, and the same process is repeated. The planning information 33 may be changed for each episode, or it may be generated randomly. When the termination condition is satisfied, the learning of the planning model 20 is completed. The learning termination condition may be determined based on the number of calculations in the iterations, the error in the action value function and the state value function, or the planning model 20 may terminate learning when the total reward obtained in one planning reaches a predetermined value or more.

[0055] In this way, the learning control unit 10 generates (trains) a planning model 20 by performing reinforcement learning to preferentially select and output actions that result in a higher total reward determined by the reward function.

[0056] The third memory unit 30 is a memory or other storage medium that stores data and programs used in the learning system 50 in a readable format. The third memory unit 30 stores behavior design information 31, reward design information 32, and planning information 33 that have been generated in advance by the user or acquired from an external source. The third memory unit 30 also stores the production model 34.

[0057] Production model 34 is a logistics simulator that simulates a factory production line where the planning device 100 formulates a production plan.

[0058] The information modeled as production model 34 may include, for example, information on resources related to the production of products, information on the production methods of products (machinery used, workers, processing time, transport time, good product rate, reprocessing rate, etc.), and information on products (production volume, input time, delivery date, etc.). The resource information may include, for example, information on manufacturing equipment (type of equipment, number of units, capacity, failure rate, etc.), information on manufacturing workers (shifts, capacity, number of people, etc.), and information on the location of products.

[0059] The production model 34 includes constant (fixed) information in the factory, and the planning information 33 may include information about production processes that change with each plan. Figure 8, T3 shows an example of constant (fixed) information in the factory included in the production model 34.

[0060] The abnormality determination device 200 is a device that determines the degree of abnormality in the production logistics of the production logistics system 1. The degree of abnormality is an index for determining whether the current state of production logistics is normal or abnormal, and since it is a numerical representation of the deviation from normal, monitoring the degree of abnormality can prevent future logistics problems. In other words, the abnormality determination device 200 determines whether the state of production logistics in the production logistics device 80 is normal or abnormal. In this embodiment, when simply referred to as the degree of abnormality, it means the degree of abnormality in production logistics. The abnormality determination device 200 comprises a feature acquisition unit 202, an abnormality score calculation unit 203, an abnormality determination unit 204, an abnormality score output unit 205, and a second storage unit 220.

[0061] The feature acquisition unit 202 is a functional unit that acquires feature quantities output from the intermediate layer of the planning model 20, which is a neural network model that plans production and logistics. More specifically, the feature acquisition unit 202 acquires the output values ​​of the activation functions that make up the artificial neurons in the intermediate layer of the planning model 20 as feature quantities. The feature quantities acquired by the feature acquisition unit 202 are feature quantities that indicate the state of production and logistics.

[0062] More specifically, the feature acquisition unit 202 acquires the output value from the activation function of the hidden layer of the planning model 20 when state information is input to the input layer of the planning model 20. The feature acquisition unit 202 can acquire features from any layer, but it is preferable to acquire features from a layer that is closer to the final conclusion, such as the layer immediately preceding the output layer of the planning model 20, as this allows for the acquisition of information that is more closely related to the final conclusion. The feature acquisition unit 202 acquires features when they are output from the hidden layer during the process in which the planning model 20 outputs an action from the output layer based on the state information input to the input layer. The feature acquisition unit 202 provides the acquired features to the anomaly score calculation unit 203.

[0063] Figure 4 shows an example of a neural network model that constitutes the proposed model 20. The neural network model has an input layer into which specific information is input, an output layer into which specific information is output, and one or more intermediate layers located between the input and output layers. Each layer of the neural network model has one or more artificial neurons. Figures 4A and 4B show different neural network models, respectively.

[0064] The neural network model M1, shown as reference numeral 4A in Figure 4, is an example of a neural network model and has a first layer as an input layer, second to n-1 layers as hidden layers, and the nth layer as an output layer. In the neural network model M1, state information acquired by, for example, the state acquisition unit 101 is input to the first layer, and the nth layer outputs, for example, an action corresponding to the input state information. In the neural network model M1, the feature acquisition unit 202 acquires features output from any of the second to n-1 layers. Preferably, the feature acquisition unit 202 acquires features output from the n-1 layer.

[0065] Another example of a neural network model, shown as 4B in Figure 4, is the neural network model M2, which has two output layers that output different types of output information, with the hidden layer branching midway. In the neural network model M2, the first layer is the input layer, and state information acquired by, for example, the state acquisition unit 101 is input. In the neural network model M2, the m-th layer is the output layer of the neural network NN2 that determines the state value, and outputs a state value corresponding to the input state information. The state value is the expected value of the total cumulative reward calculated after considering all possible actions. In the neural network model M2, the n-th layer is the output layer of the neural network NN3 that determines the action, and outputs an action value corresponding to the input state information. For example, the neural network model M2 is a common neural network NN1 from the first to the third layer. For example, the neural network model M2 branches from the third layer into the neural network NN2 and the neural network NN3.

[0066] In the neural network model M2, the feature acquisition unit 202 acquires features output from any of the layers from the 2nd to the (m-1)th layer, or from any of the layers from the 2nd to the (n-1)th layer. Preferably, the feature acquisition unit 202 acquires features output from the intermediate layers of the neural network NN2. More preferably, the feature acquisition unit 202 acquires features output from the (m-1)th layer.

[0067] Furthermore, when the feature acquisition unit 202 says it acquires features from the intermediate layers of the neural network model, it means that it acquires features output from all or some of the artificial neurons included in a specific single or multiple layers within the intermediate layers of the neural network model. For example, when the feature acquisition unit 202 acquires features output from the (n-1)th layer of the neural network model M1, it acquires features output from all the artificial neurons included in the (n-1)th layer.

[0068] The anomaly score calculation unit 203 is a functional unit that performs calculations to transform the features output from the design model 20. The features output from the intermediate layer of the design model 20 are vectors with multiple elements, and it is not practical to monitor each feature as an anomaly score. Therefore, the anomaly score calculation unit 203 performs dimensionality reduction or conversion to scalar values ​​of the vectors. The anomaly score calculation unit 203 performs dimensionality reduction or conversion to scalar values ​​of the vectors by, for example, using principal component analysis. Principal component analysis is a method that captures the linear relationships between features and aggregates information into a small number of variables (principal components). By selecting principal components from the principal components obtained as a result of principal component analysis in order of the size of their corresponding eigenvalues, the dimensionality of the vectors can be reduced, and these can be used as anomaly scores. Alternatively, the T2 statistic, which is the sum of the squared distances from the origin after dividing the obtained principal components by their respective standard deviations, or the Q statistic, which is the squared difference between the data and the actual data after mapping to the principal component axis, can also be used as anomaly scores. Alternatively, the features of the intermediate layer can be used directly as anomaly scores. In this embodiment, the feature vector is converted into a T2 statistic, which is used as the anomaly score.

[0069] The anomaly determination unit 204 is a functional unit that acquires feature quantities output from the intermediate layer of the planning model 20 as an anomaly score and determines the degree of anomaly in production logistics based on the anomaly score. That is, the anomaly determination unit 204 determines whether the state of production logistics is normal or abnormal based on the anomaly score calculated by the anomaly score calculation unit 203. The anomaly determination unit 204 may also determine the degree of anomaly based on an anomaly score obtained by reducing the dimensionality of the feature quantities or by converting them into scalar values. The anomaly score is a score calculated based on feature quantities output from the intermediate layer of the planning model 20 as an index used to determine the degree of anomaly. The anomaly score may be calculated for each production process that is executed. The anomaly determination unit 204 may also determine the degree of anomaly using a predetermined threshold.

[0070] The anomaly detection unit 204 may determine the degree of anomaly using a trained model that takes features output from the intermediate layer of the design model 20 as input and outputs the degree of anomaly. The trained model may be constructed by unsupervised learning that classifies the degree of anomaly as either normal or abnormal based on the features output from the intermediate layer of the design model 20. For example, in addition to determination methods based on statistics such as principal component analysis, distance-based methods such as the k-nearest neighbors method and the LOF (Local Outlier Factor) method, or probability distribution-based methods such as multivariate Gaussian distribution and kernel density estimation may also be used.

[0071] Furthermore, the trained model may be constructed by supervised learning using training data that combines features output from the intermediate layer of the planning model 20 with labels indicating whether the logistics are normal or abnormal. The anomaly determination unit 204 can determine the degree of anomaly more accurately by using the trained model.

[0072] Alternatively, the degree of abnormality in logistics may be determined based on multiple abnormality scores calculated for each of the multiple production processes.

[0073] The abnormality score output unit 205 is a functional unit that outputs information showing the time-series changes in the abnormality score. The abnormality score output unit 205 outputs graph data showing the time-series changes in the abnormality score to the display unit 60. The abnormality score output unit 205 may also output the abnormality determination result to the display unit 60 along with the information showing the time-series changes in the abnormality score. The abnormality score output unit 205 may store the information showing the time-series changes in the abnormality score in the second storage unit 220.

[0074] The display unit 60 is a functional unit connected to the abnormality determination device 200 that displays various information. The display unit 60 may be, for example, a display or the like, and may be provided on the abnormality determination device 200. By displaying information as appropriate, the display unit 60 can communicate to the operator whether or not an abnormality has occurred in production logistics.

[0075] [Learning flow of the planning model] Next, the learning process of the planning model 20 by the planning device 100 according to this embodiment will be explained with reference to Figure 5. Figure 5 shows a flowchart illustrating an example of the learning process of the planning model by the planning device 100 of this disclosure.

[0076] For example, when a factory is selected as the target of production planning and the planning process by the planning device 100 begins, the process proceeds to step S1.

[0077] In step S1, the behavior design unit 11 obtains behavior design information 31 from the third memory unit 30, which shows the selectable actions of the planning model 20, which is the agent. Once the behavior design unit 11 has finished obtaining the behavior design information 31, the process proceeds to step S2.

[0078] In step S2, the reward design unit 12 obtains reward design information 32 from the third memory unit 30, which is information indicating the arguments and format that define the reward function, and generates the reward function. Once the generation of the reward function is complete, the process proceeds to step S3.

[0079] In step S3, the planning information acquisition unit 13 acquires planning information 33, which is necessary for creating production plans in factories, etc., from the third storage unit 30. Once the planning information has been acquired, the process proceeds to step S4.

[0080] In step S4, (since the processes described in steps S5 to S9 have not yet been completed), the current state of the production status at the time of planning in the factory to which the production plan is applied is entered into the planning model 20. Once the current state has been entered into the planning model 20, the process proceeds to step S5.

[0081] In step S5, the planning information setting unit 14 sets the acquired or generated planning information 33 and status information (current status) in the production model 34. Once the information setting is complete, the process proceeds to step S6.

[0082] In step S6, the current state input unit 15 inputs information indicating the current state, i.e., the current production status, into the planning model 20. Then, the action calculation unit 16 uses the planning model 20 to calculate the next action to take based on the information indicating the current state. Once the action calculation is complete, the process proceeds to step S7.

[0083] In step S7, the action setting unit 17 sets the action calculated by the action calculation unit 16 in the production model 34. Then, the state transition unit 18 uses the production model 34 to calculate the (new) state to which the factory subject to the production plan has transitioned as a result of the set action. Once the new state is calculated, the process proceeds to step S8.

[0084] In step S8, the reward calculation unit 19 calculates the reward to be received based on the reward function defined in the reward design unit 12, corresponding to the actions entered into the planning model 20 in step S6. Then, based on these calculation results, the planning model update unit 21 updates the parameters (weighting coefficients) of the action value function in the planning model 20. Once the parameters of the action value function in the planning model 20 have been updated, the process proceeds to step S9.

[0085] In step S9, it is determined whether or not the learning of one episode has been completed. If it is determined that the learning of one episode has been completed, the process proceeds to step S10 (YES in S9). On the other hand, if it is determined that the learning of one episode has not been completed, the process proceeds to step S5 (NO in S9). In this case, if the process returns to step S5, unlike when S5 was passed earlier, the current state has been newly calculated in step S7, so this state is set in the production model 34, and the subsequent processing continues.

[0086] In step S10, it is determined whether the planning model 20 has been able to learn sufficiently (update parameters). If it is determined that sufficient learning has been performed, the learning process of the planning device 100 for the planning model 20 is terminated (YES in step S10). On the other hand, if it is determined that the learning has not reached a sufficient level, the process proceeds to step S4 (NO in S9).

[0087] The planning model 20 is learned through the repetition of these processes (especially steps S4 to S10).

[0088] [Abnormality Determination Flowchart] Next, the flow of the abnormality determination process by the planning device 100 according to this embodiment will be explained with reference to Figure 6. Figure 6 is a flowchart showing an example of the flow of the abnormality determination process by the planning device 100 of this disclosure.

[0089] When the abnormality determination process by the planning device 100 is started, the process proceeds to step S11.

[0090] In step S11, the current state input unit 15 acquires the current state, that is, the current production status at the factory to which the production plan simulated by the production model 34 is applied.

[0091] In step S12, the feature acquisition unit 202 acquires features output from the hidden layers of the planning model 20, which is a neural network model that plans production and logistics. That is, by inputting the current state into the planning model 20, the output values ​​from the activation function of the hidden layers of the planning model 20 are acquired as features. If it is an initial state that has not experienced the processing in step S7, the current state can also be acquired from the state acquisition unit 101. Once the features have been acquired, the process proceeds to step S13.

[0092] In step S13, the anomaly score calculation unit 203 performs calculations to transform the features output from the design model 20. Since the features output from the intermediate layer of the design model 20 are vectors with multiple elements, and it is not practical to monitor each feature as an anomaly score, the anomaly score calculation unit 203 performs dimensionality reduction of the vector or conversion to a scalar value. After the calculation in step S13 is completed, the process proceeds to step S14.

[0093] In step S14, the anomaly determination unit 204 uses a trained model that takes features output from the intermediate layer of the planning model 20 as input and outputs an anomaly score to determine the degree of anomaly. More specifically, the anomaly determination unit 204 determines whether the production and logistics status is normal or abnormal based on the anomaly score calculated by the anomaly score calculation unit 203. Once the determination is complete, the process proceeds to step S15.

[0094] In step S15, the abnormality score output unit 205 outputs information showing the time-series change of the abnormality score, and if necessary, the display unit 60 can display whether or not an abnormality has occurred to the operator or others to prompt them to take action.

[0095] Figure 10 shows an example of a graph output by the abnormality score output unit 205. In the graph shown in Figure 10, the vertical axis represents the abnormality score, and the horizontal axis represents time. The graph shown in Figure 10 shows, for example, the time-series change of the abnormality score when the finishing of slab ID "015" is performed. That is, the abnormality score in the graph shown in Figure 10 shows the abnormality score calculated in each production process shown in the production model 34 during the production of slab ID "015". For example, the graph shown in Figure 10 may also show the threshold that served as the criterion for determining the degree of abnormality by the abnormality determination unit 204. In this way, by displaying the abnormality score on the display unit 60, the user can recognize the current state of production logistics and recognize abnormalities in production logistics.

[0096] Once the processes up to step S15 described above are completed, the abnormality detection process is finished.

[0097] As described above, the abnormality determination device 200 according to this embodiment determines the degree of abnormality based on the feature quantities output from the intermediate layer of the neural network model. In other words, the abnormality determination device 200 can determine the degree of abnormality, which is a concept different from the behavior itself, in a neural network used to infer behavior according to the production situation. This makes it possible to appropriately determine the degree of abnormality that could lead to future problems based on the current state of production and logistics, without predicting the future state of production and logistics.

[0098] Furthermore, by using a neural network model trained through reinforcement learning, the planning model 20 can plan production and logistics with a quality level equal to or higher than that of a skilled operator (user). This improves the accuracy of anomaly detection.

[0099] Furthermore, the feature acquisition unit 202 acquires features output from the intermediate layer of the neural network NN2, which estimates the value of the production and logistics state. This allows for the acquisition of features strongly related to the cumulative reward derived from the current production and logistics situation. This further improves the accuracy of anomaly detection.

[0100] Furthermore, the anomaly score calculation unit 203 converts the vectors representing the features into scalar values, allowing the scalar values ​​to be treated as an anomaly score, which is an index used to determine the degree of anomaly. This makes it easier for users to recognize the degree of anomaly.

[0101] Although this embodiment describes the slab finishing process, the present invention is not limited to this and can be used in any production logistics process having at least one of a transport device 300 and a manufacturing device 400, in which a planning model 20 can be generated and utilized by reinforcement learning or machine learning. The present invention is particularly suitable when the planning model 20 is a neural network.

[0102] [Embodiment 2] Other embodiments of this disclosure will be described below with reference to Figure 11. Figure 11 is a block diagram showing an example of a production logistics system 1A. For the sake of convenience, components having the same function as those described in the above embodiments will be denoted by the same reference numerals, and their descriptions will not be repeated. The production logistics system 1A according to Embodiment 2 differs from the production logistics system 1 in that it does not include a planning device 100.

[0103] As shown in Figure 11, the abnormality determination device 200A of the production logistics system 1A has a configuration that is substantially equivalent to the combination of the planning device 100 and the abnormality determination device 200 in Embodiment 1. The abnormality determination device 200A includes a state acquisition unit 701 and an output unit 702. The state acquisition unit 701 and the output unit 702 have the same configuration as the state acquisition unit 101 and the output unit 102 described in Embodiment 1, respectively, so their description is omitted. The planning model 20 is stored in the second storage unit 220 of the abnormality determination device 200A. With this configuration, the production logistics system 1A does not need to include a planning device 100.

[0104] [Examples of implementation using software] The functions of the learning control unit 10, the planning device 100, and the abnormality determination devices 200, 200A (hereinafter referred to as "devices") can be realized by programs that cause computers to function as devices, and by programs that cause computers to function as each control block of the devices.

[0105] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0106] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the above device. In the latter case, the program may be supplied to the above device via any wired or wireless transmission medium.

[0107] Furthermore, some or all of the functions of each of the above control blocks can also be implemented by logic circuits. For example, an integrated circuit in which logic circuits functioning as each of the above control blocks are formed is also included in the scope of this disclosure. In addition, it is also possible to implement the functions of each of the above control blocks by, for example, a quantum computer.

[0108] 〔summary〕 An abnormality determination device according to Embodiment 1 of this disclosure comprises a feature acquisition unit that acquires feature quantities output from an intermediate layer of a neural network model that plans production logistics, and an abnormality determination unit that determines the degree of abnormality in the production logistics based on the feature quantities.

[0109] According to the above embodiment 1, the degree of anomaly in production logistics is determined based on the features output from the intermediate layer of the neural network model. This makes it possible to appropriately determine the degree of anomaly in production logistics that could lead to future problems, based on the current state of production logistics, without predicting the future state of production logistics.

[0110] In the abnormality determination device according to aspect 2 of this disclosure, the neural network model may be configured to be a reinforcement learning agent model that has been reinforced to output actions in accordance with information indicating the production status, as described in aspect 1 above.

[0111] According to the above embodiment 2, by using a neural network model trained through reinforcement learning, production and logistics planning can be performed with a quality level equivalent to or higher than that of a skilled operator (user). This improves the accuracy of anomaly detection in production and logistics.

[0112] In the third aspect of this disclosure, the abnormality determination device may be configured such that the feature acquisition unit acquires features output from the intermediate layer of a neural network that estimates the value of the production logistics state, as described in the second aspect above.

[0113] According to the above embodiment 3, by obtaining features output from the intermediate layer of a neural network that estimates the value of the state of production and logistics, it is possible to obtain features that are strongly related to the cumulative reward derived from the current state of production and logistics. This makes it possible to further improve the accuracy of determining the degree of anomaly in production and logistics.

[0114] An abnormality determination device according to aspect 4 of the present disclosure may include an abnormality score calculation unit that converts a vector representing the feature quantity into a scalar value in any of aspects 1 to 3 above, and the abnormality determination unit may be configured to determine the degree of abnormality in the production logistics based on the scalar value.

[0115] According to embodiment 4 described above, the scalar value can be treated as an anomaly score, which is an index used to determine the degree of anomaly in production logistics. This makes it easier for users to recognize the degree of anomaly in production logistics.

[0116] The abnormality determination device according to aspect 5 of the present disclosure may be configured such that, in any of aspects 1 to 4 above, the abnormality determination unit uses a trained model that takes the feature quantities as input and outputs the abnormality in the production logistics to determine the abnormality in the production logistics.

[0117] According to embodiment 5 described above, by using a trained model to determine the degree of anomaly in production logistics, the degree of anomaly in production logistics can be determined with greater accuracy.

[0118] An abnormality determination method according to aspect 6 of this disclosure is a method comprising: a feature acquisition step of acquiring feature quantities output from an intermediate layer of a neural network model that plans production logistics; and an abnormality determination step of determining the degree of abnormality in the production logistics based on the feature quantities.

[0119] This disclosure is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of this disclosure. [Explanation of Symbols]

[0120] 1. 1A Production and Logistics System 10 Learning Control Unit 11 Behavior Design Department 12 Compensation Design Department 13. Planning Information Acquisition Department 14. Planning Information Setting Department 15 Current Status Input Section 16 Behavior calculation department 17 Action Setting Section 18 State transition section 19 Remuneration Calculation Department 20 Planning Models 21 Planning Model Update Department 30 Third memory section 50 Learning Systems 60 Display 80 Production and logistics equipment 100 Planning device 101 Status acquisition unit 102 Output section 120 1st memory section 200, 200A abnormality determination device 202 Feature Acquisition Unit 203 Anomaly Score Calculation Unit 204 Abnormality determination section 205 Anomaly Score Output Unit 220 2nd memory section 300 Conveying equipment 400 Manufacturing equipment M1, M2 Neural Network Models

Claims

1. A feature acquisition unit that acquires features output from the intermediate layer of a neural network model used for planning production and logistics, The system includes an abnormality determination unit that determines the degree of abnormality in the production logistics based on the aforementioned feature quantities. Abnormality level determination device.

2. The aforementioned neural network model is a reinforcement learning agent model that has been reinforced to output actions in response to information indicating the production status. An abnormality determination device according to claim 1.

3. The feature acquisition unit acquires features output from the intermediate layer of the neural network that estimates the value of the production and logistics state. The abnormality determination device according to claim 2.

4. The system includes an anomaly score calculation unit that converts a vector representing the aforementioned feature into a scalar value, The abnormality determination unit determines the degree of abnormality in the production logistics based on the scalar value. An abnormality determination device according to claim 1.

5. The anomaly detection unit uses a trained model that takes the feature quantities as input and outputs the degree of anomaly in the production and logistics to determine the degree of anomaly in the production and logistics. An abnormality determination device according to claim 1.

6. A feature acquisition step that obtains features output from the intermediate layer of a neural network model used for planning production and logistics, The process includes an anomaly determination step that determines the degree of abnormality in the production logistics based on the aforementioned feature quantities, Abnormality determination method.

Images