Output method and output system

Abstract

To provide a technique capable of inferring the state of an object with high accuracy and explaining the inference result in a language.SOLUTION: This outputting method is a method for outputting an inference result Dr obtained by inferring a state of a target object and an explanation De related to the inference result Dr, and includes (a) inputting a target dataset of the target object into a machine-learned model M1 and outputting the inference result Dr from the machine-learned model M1, and (b) inputting an intermediate dataset Dm obtained from an intermediate layer of the machine-learned model M1 in step (a) into a large-scale language model 40 that has learned a relationship between the intermediate dataset Dm and the explanation De, and outputting the explanation De from the large-scale language model 40. As a result, it is possible to prevent the inference process from becoming a black box.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an output method and an output system for outputting an inference result obtained by inferring the state of an object and an explanation related to the inference result. [Background technology]

[0002] In recent years, in various industrial fields, machine learning has been used to infer the state of an object based on various data related to the object, and predictive maintenance, anomaly detection, and cause analysis of defects have been carried out based on the inference results. However, when performing such machine learning, there is a problem in that the inference process is unclear because many machine learning models are black boxes. For this reason, there is a growing demand for technology that can explain the results of machine learning in order to understand the inference process.

[0003] For example, Patent Document 1 discloses that a machine learning algorithm is used to predict the response of individual cells of a patient to a specific drug, and that the activation vectors used in the prediction are used in machine learning to explain the results of the prediction. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2023-549614 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the activation vectors in Patent Document 1 are values ​​expressed as distances between the sequencing data samples and the centers of gravity of each of the multiple gene modules, and are not linguistic. Therefore, in order to explain the results of machine learning in language, the workers themselves must verbalize the activation vectors, which is a problem that remains.

[0006] Recently, there has been great progress in technology related to large-scale language models (LLMs). Large-scale language models are language models that have been machine-learned in advance based on huge amounts of text data or text corpora, ranging from hundreds of millions to hundreds of billions of data, and are known to be able to output language in response to input. However, when large-scale language models themselves infer the state of an object, the accuracy of the output tends to be low.

[0007] The present invention has been made in consideration of the above circumstances, and aims to provide a technology that enables the state of an object to be inferred with high accuracy and the inference results to be explained in language. [Means for solving the problem]

[0008] In order to solve the above problem, the first invention of the present application is an output method for outputting an inference result obtained by inferring the state of an object and an explanation related to the inference result, comprising the steps of: (a) inputting object data, which is data on the object, into a trained model created by machine learning, and outputting the inference result from the trained model; and (b) inputting intermediate data obtained from an intermediate layer of the trained model in step (a) into a large-scale language model that has been trained to determine the relationship between the intermediate data and the explanation, and outputting the explanation from the large-scale language model.

[0009] A second invention of the present application is an output method of the first invention, further comprising (c) a learning step of, before step (b), inputting the intermediate data obtained by inputting the target data for training into the trained model into the large-scale language model, thereby training the large-scale language model so that the explanation output from the large-scale language model approaches a known explanation corresponding to the target data.

[0010] A third invention of the present application is an output method according to the first or second invention, in which the trained model is configured by a neural network.

[0011] A fourth invention of the present application is an output method of the second invention, wherein the target data is image data obtained by photographing the target object, and in the learning process, multiple parameters included in the large-scale language model are updated so as to minimize the cross-entropy error between the known explanation corresponding to the image data and the explanation output from the large-scale language model.

[0012] A fifth invention of the present application is an output system that outputs an inference result obtained by inferring the state of an object and an explanation related to the inference result, the output system including an inference device and a large-scale language model. The inference device inputs target data, which is data on the object, into a trained model created by machine learning and outputs the inference result from the trained model. The large-scale language model outputs the explanation by receiving intermediate data obtained from an intermediate layer of the trained model. Furthermore, the large-scale language model has been trained by machine learning so that the explanation it outputs approaches a known explanation corresponding to the target data by receiving the intermediate data obtained from the intermediate layer of the trained model when the target data for training is input to the trained model.

[0013] The sixth invention of the present application is an output system of the fifth invention, further comprising a data acquisition device that inputs the target data to the inference device via a network, and the large-scale language model is connected to the inference device via the network. [Effects of the Invention]

[0014] According to the first to sixth aspects of the present invention, the state of an object can be inferred with high accuracy using a trained model. Furthermore, by inputting intermediate data obtained from the intermediate layer of the trained model into a large-scale language model, an explanation related to the inference result can be output. This makes it possible to explain the inference result in language, preventing the inference process from becoming a black box. [Brief explanation of the drawings]

[0015] [Figure 1]FIG. 1 is a diagram conceptually illustrating a configuration of an output system. [Figure 2] FIG. 2 is a block diagram conceptually illustrating the information processing performed by the output system. [Figure 3] FIG. 1 is a schematic diagram of a neural network that constitutes a trained model. [Figure 4] FIG. 10 is a block diagram conceptually showing how a first learning step is performed. [Figure 5] FIG. 10 is a block diagram conceptually showing how an intermediate data accumulation step is performed. [Figure 6] FIG. 10 is a block diagram conceptually showing how a second learning step is performed. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components described in these embodiments are merely examples and are not intended to limit the scope of the present invention. Furthermore, in the drawings, the dimensions and numbers of each part may be exaggerated or simplified as necessary to facilitate understanding.

[0017] <1. Output system configuration> 1 is a conceptual diagram illustrating the configuration of an output system 1 capable of executing an output method according to one embodiment of the present invention. This output system 1 is a system that infers the state of a certain object and outputs the inference result Dr and an explanation De related to the inference result Dr. Furthermore, the output system 1 of this embodiment performs machine learning to infer whether the tablet t is a good or bad product, for example, using image data 9 obtained by photographing a tablet t as an object.

[0018] However, the object is not limited to the tablet t. The object may be, for example, a substrate such as paper or film, a semiconductor wafer or printed circuit board, or other object. Furthermore, the image data 9 of the tablet t described above is merely an example of the object data related to the object used by the output system 1. The output system 1 may infer the state of the object through machine learning, for example, using the ambient temperature and humidity values ​​of the substrate as the object as object data. In other words, the output system 1 may infer the state of the object through machine learning using object data, which is some data related to the object, and output the inference result Dr and an explanation De related to the inference result Dr.

[0019] 1, the output system 1 includes a computer main body 10, a display unit 20, an input unit 30, and a large-scale language model 40. The computer main body 10 is an information processing device having a processor 11 such as a CPU, a memory 12 such as a RAM, and a storage unit 13 such as a hard disk drive.

[0020] The storage unit 13 stores a computer program 131. The computer program 131 is application software that causes the computer main body 10, the display unit 20, the large-scale language model 40, and the data acquisition device 50 (described later) to execute various processes (described later). The computer program 131 is read from a storage medium M such as a CD or DVD and installed on the computer main body 10. However, the computer program 131 may also be downloaded to the computer main body 10 via a network N such as the Internet. The computer program 131 may also be programmed and created within the computer main body 10. The computer program 131 may also be installed or downloaded to the computer main body 10 and then further modified, updated, or updated.

[0021] 2 is a block diagram conceptually showing information processing executed by the output system 1. As shown in FIG. 2, the computer main body 10 has an inference unit 110 and an intermediate data accumulation unit 120. The inference unit 110 is a function realized by the processor 11 executing a computer program 131. The intermediate data accumulation unit 120 is a function realized by the memory unit 13.

[0022] The inference device 110 has a trained model M1. The trained model M1 is stored in the storage unit 13. The trained model M1 is, for example, an inference program created by a first learning step (described later) using a so-called supervised machine learning algorithm. The trained model M1 of this embodiment is configured by a neural network. The output system 1 uses the trained model M1 to output an inference result Dr that infers the state of the target object.

[0023] Figure 3 is a schematic diagram of a neural network that constitutes the trained model M1. As shown in Figure 3, the neural network has an input layer Li, an intermediate layer Lm, and an output layer Lo. Image data 9, which is the target data, is input to the input layer Li. The output layer Lo outputs an inference result Dr that infers the state of a tablet t, which is the target object. The intermediate layer Lm performs various calculations on the image data 9 input to the input layer Li.

[0024] The display unit 20 is a device that displays various information output from the computer main body 10. For example, a liquid crystal display is used for the display unit 20. The input unit 30 is a device that inputs various information to the computer main body 10. For example, a keyboard or a mouse is used for the input unit 30. The display unit 20 and the input unit 30 are each electrically connected to the computer main body 10. Note that the display unit 20 and the input unit 30 may be realized by a single device such as a touch panel display.

[0025] 1, in the output system 1 of this embodiment, the computer main body 10, the display unit 20, and the input unit 30 are each connected to each other without a network N. However, in the output system 1, the computer main body 10 may be provided on a server or on the cloud, and the display unit 20 and the input unit 30 may each be connected to the computer main body 10 via the network N.

[0026] The computer main body 10 is also connected to large-scale language models 40 via a network N. The large-scale language model 40 is connected to an inference device 110 via the network N. The large-scale language model 40 is a language model that has been machine-learned based on a huge amount of text data or a text corpus, ranging from hundreds of millions to hundreds of billions of data. The large-scale language model 40 is, for example, a deep neural network based on a self-attention mechanism called a Transformer. It is also desirable that the large-scale language model 40 be commercially available and re-trainable. The output system 1 uses the large-scale language model 40 to generate an explanation De related to an inference result Dr that infers the state of an object.

[0027] Furthermore, the computer main body 10 is connected to a data acquisition device 50 via a network N. The data acquisition device 50 of this embodiment is an imaging unit that acquires image data 9 by photographing a tablet t. The data acquisition device 50 outputs the image data 9 obtained by photographing to the inference device 110 of the computer main body 10 via the network N. In other words, the data acquisition device 50 inputs target data to the inference device 110 via the network N.

[0028] <2. Pre-learning flow> Next, we will explain the pre-learning performed in the output system 1. The pre-learning consists of a "first learning process" in which machine learning of the trained model M1 of the inference unit 110 is performed, an "intermediate data accumulation process," and a "second learning process" in which machine learning of the large-scale language model 40 is performed.

[0029] <2-1. First learning process> First, the first learning step will be described. FIG. 4 is a block diagram conceptually showing how the first learning step is performed. As shown in FIG. 4, in the first learning step, a learning model M0 that serves as a prototype of the learned model M1 is prepared. The learning model M0 is stored in the storage unit 13.

[0030] In the first learning step, a plurality of learning image data 9 obtained by photographing tablets t in various states and a result of an operator or the like visually or otherwise determining the quality of the tablets t for each of the plurality of learning image data 9 are prepared. Then, a plurality of sets of image data 9 and labels L1 are input to the learning model M0. For example, the operator or the like operates the input unit 30 and inputs a plurality of sets of image data 9 and labels L1 to the learning model M0 via an external storage device or the like (not shown).

[0031] Then, when multiple sets of image data 9 and labels L1 are input to the learning model M0, the inference device 110 trains the learning model M0 by machine learning so that the quality of the tablet t associated with the image data 9 can be inferred with high accuracy based on each image data 9. Specifically, the inference device 110 trains the learning model M0 by machine learning for calculating with high accuracy the quality of the tablet t associated with the image data 9, using the corresponding label L1 as training data (correct answer data) for each image data 9.

[0032] As described above, when the machine learning is performed, a so-called supervised machine learning algorithm, such as a neural network including deep learning, is used. Then, when the machine learning of the learning model M0 is completed as described above, a trained model M1 is obtained.

[0033] <2-2. Intermediate data accumulation process> Next, the intermediate data accumulation process will be described. The intermediate data accumulation process is performed using the trained model M1 after the first learning process is completed. FIG. 5 is a block diagram conceptually illustrating the manner in which the intermediate data accumulation process is performed. As shown in FIG. 5, in the intermediate data accumulation process, multiple pieces of training image data 9 obtained by photographing tablets t in various states using a data acquisition device 50 are input to the trained model M1. A worker or the like inputs multiple pieces of training image data 9 to the trained model M1, for example, by operating the input unit 30. Note that the image data 9 input to the trained model M1 in the intermediate data accumulation process may be the same as the training image data 9 input to the training model M0 in the first learning process.

[0034] Then, the trained model M1 outputs an inference result Dr of the quality of the tablet t for each of the multiple training image data 9. That is, the inference device 110 inputs the image data 9 as target data into the trained model M1 created by machine learning, and outputs the inference result Dr from the trained model M1. However, the inference result Dr is not used in the intermediate data accumulation process.

[0035] Instead, in the intermediate data accumulation step, data (hereinafter referred to as "intermediate data Dm") obtained from the intermediate layer Lm (see FIG. 3) of the trained model M1 is accumulated in the intermediate data accumulation unit 120. The intermediate data Dm is, for example, a numerical array represented by a multidimensional vector. In addition to the intermediate data Dm, a known explanation (hereinafter referred to as "label L2") corresponding to each image data 9 that is the basis of the intermediate data Dm is also accumulated in the intermediate data accumulation unit 120 together with the intermediate data Dm. A worker or the like inputs the explanation for each piece of intermediate data Dm into the intermediate data accumulation unit 120 by, for example, operating the input unit 30.

[0036] The explanation is a string of language (characters), such as, for example, "There is a black stain on the top surface of the tablet t," "The edge of the tablet t is chipped," or "There is a secant line tilted at 142 degrees in the center of the top surface of the tablet t, but this is not a defect," for the tablet t related to each image data 9. In other words, the explanation is a string of characters that explains why the inference result Dr output from the inference device 110 was reached by inputting each image data 9 that is the basis of the intermediate data Dm into the trained model M1, and reflects the characteristics of each image data 9. In other words, the explanation is a string of characters related to the inference result Dr output from the trained model M1.

[0037] <2-3. About the second learning process> Next, the second learning process will be described. The second learning process corresponds to the "learning process" of the present invention. The second learning process is performed using a plurality of sets of intermediate data Dm and labels L2 stored in the intermediate data storage unit 120 in the intermediate data storage process. Fig. 6 is a block diagram conceptually showing how the second learning process is performed.

[0038] 6, in the second learning step, multiple sets of intermediate data Dm and labels L2 stored in the intermediate data storage unit 120 are input to the large-scale language model 40. A worker or the like operates the input unit 30, for example, to input multiple sets of intermediate data Dm and labels L2 from the intermediate data storage unit 120 to the large-scale language model 40 via the network N.

[0039] Then, when multiple sets of intermediate data Dm and labels L2 are input, the large-scale language model 40 performs machine learning based on each intermediate data Dm so that it can output, with high accuracy, an explanation De corresponding to each image data 9 that is the basis of each intermediate data Dm. Specifically, the large-scale language model 40 performs machine learning for each intermediate data Dm, using the corresponding label L2 as training data (correct answer data) to output, with high accuracy, an explanation De related to the inference result Dr of the tablet t related to each image data 9 that is the basis of each intermediate data Dm. That is, in the second learning step, the intermediate data Dm obtained by inputting image data 9 as target data into the trained model M1 is input into the large-scale language model 40, and the large-scale language model 40 is trained so that the explanation De output from the large-scale language model 40 approaches a known explanation corresponding to the target data.

[0040] More specifically, in the second learning step, for example, the cross-entropy error L is used as a criterion for the error between the explanation De output from the output layer Lo of the large-scale language model 40 and a known explanation that is the correct answer. The cross-entropy error L can be calculated, for example, by the following equation (1):

[0041]

number

[0042] Note that "l(x, y)" in formula (1) is calculated using, for example, the following formula (2). Here, in formulas (1) and (2), "x" indicates an N×C two-dimensional tensor (array) that represents the score of a token (a unit of data used when processing language data) output from the output layer Lo of the large-scale language model 40. Also, "y" indicates a one-dimensional tensor that represents a token in a known explanation that is the correct answer. Also, "C" indicates the number of tokens handled by the large-scale language model 40. Also, "N" indicates the batch size (the size of data used in one learning session). Also, "x n,yn " is "x" whose first dimension is "n" and whose second dimension is "y" n" indicates the elements that fit the criteria. n,c " indicates the element of "x" whose first dimension is "n" and whose second dimension is "c".

[0043]

number

[0044] Then, parameters (weighting coefficients or biases) included in the deep neural network of the large-scale language model 40 are updated so that the cross-entropy error L becomes smaller (approaching zero). Furthermore, by repeating this type of learning, the parameters are optimized. That is, in the second learning step, multiple parameters included in the large-scale language model 40 are updated so as to minimize the cross-entropy error L between the known explanation corresponding to each image data 9 that is the basis of each intermediate data Dm and the explanation De output from the large-scale language model 40.

[0045] As described above, when the machine learning of the large-scale language model 40 is completed, the second learning process ends. That is, when image data 9, which is the target data for learning, is input to the trained model M1, the large-scale language model 40 receives intermediate data Dm obtained from the intermediate layer Lm of the trained model M1 and is input to itself, so that the explanation De it outputs approaches a known explanation corresponding to the target data. Also, as described above, in the second learning process, rather than inputting the image data 9 itself to the large-scale language model 40 for machine learning, the intermediate data Dm obtained from the intermediate layer Lm of the trained model M1 is input to the large-scale language model 40 for machine learning. As a result, the inference result Dr for the tablet t related to the image data 9 can be explained in language with higher accuracy. Then, the output system 1 uses the trained model M1 to output the inference result Dr obtained by inferring the state of the tablet t based on the image data 9 of the tablet t with high accuracy, and can use the large-scale language model 40 to output the explanation De related to the inference result Dr with higher accuracy.

[0046] <3. What to do after completing the pre-learning> Next, the flow of processing in the output system 1 that is executed after the pre-learning is completed will be described.

[0047] Specifically, first, image data 9 is acquired by photographing a tablet t. Then, the acquired image data 9 is input to a trained model M1, and an inference result Dr that infers the state of the tablet t is output from the trained model M1. Furthermore, by inputting intermediate data Dm obtained from the intermediate layer Lm of the trained model M1 when the image data 9 is input to the trained model M1 into a large-scale language model 40, an explanation De for the inference result Dr can be output from the large-scale language model 40 with high accuracy.

[0048] That is, the present invention is an output method for outputting an inference result obtained by inferring the state of an object and an explanation related to the inference result, comprising the steps of: (a) inputting object data, which is data on the object, into a trained model created by machine learning, and outputting the inference result from the trained model; and (b) inputting intermediate data obtained from an intermediate layer of the trained model in step (a) into a large-scale language model that has been trained to determine the relationship between the intermediate data and the explanation, and outputting the explanation from the large-scale language model.

[0049] The training of the large-scale language model may be performed by step (c), which is a training step in which, before step (b), the intermediate data obtained by inputting the target data for training into the trained model is input into the large-scale language model, and the large-scale language model is trained so that the explanation output from the large-scale language model approaches a known explanation corresponding to the target data.

[0050] The present invention also provides an output system that outputs an inference result obtained by inferring a state of an object and an explanation related to the inference result, the output system including an inference device and a large-scale language model. The inference device inputs target data, which is data on the object, into a trained model created by machine learning and outputs the inference result from the trained model. The large-scale language model outputs the explanation by receiving intermediate data obtained from an intermediate layer of the trained model. The large-scale language model has been trained by machine learning so that the explanation it outputs approaches a known explanation corresponding to the target data by receiving the intermediate data obtained from the intermediate layer of the trained model when the target data for training is input to the trained model.

[0051] This allows the state of the object to be inferred with high accuracy using a trained model. Furthermore, by inputting intermediate data obtained from the intermediate layer of the trained model into a large-scale language model, an explanation related to the inference results can be output. This makes it possible to explain the inference results in language with high accuracy, preventing the inference process from becoming a black box.

[0052] The detailed configuration of the output system may differ from that shown in the drawings of the present application. The elements appearing in the above embodiments may be combined as appropriate within the scope of the present invention. [Explanation of symbols]

[0053] 1 Output System 9 Image data 10 Computer main body 20 Display section 30 Input section 40 Large-scale language models 50 Data acquisition device 110 Reasoner 120 Intermediate Data Storage Unit De Description Dm intermediate data Dr inference result L cross entropy error L1 Label L2 Label Li input layer Lm middle layer Lo output layer M0 Model under training M1 pre-trained model N Network t tablets

Claims

1. An output method for outputting an inference result of inferring a state of an object and an explanation related to the inference result, comprising: (a) inputting target data, which is data of the target object, into a trained model created by machine learning and outputting the inference result from the trained model; (b) inputting intermediate data obtained from the intermediate layer of the trained model in step (a) into a large-scale language model that has trained the relationship between the intermediate data and the explanation, and outputting the explanation from the large-scale language model; An output method having:

2. 2. The output method according to claim 1, (c) a learning step of, before the step (b), inputting the intermediate data obtained by inputting the target data for training into the trained model into the large-scale language model, thereby training the large-scale language model so that the explanation output from the large-scale language model approaches a known explanation corresponding to the target data. The output method further comprises:

3. 3. The output method according to claim 1 or 2, An output method in which the trained model is composed of a neural network.

4. 3. The output method according to claim 2, the target data is image data obtained by photographing the target, An output method in which, in the learning process, a plurality of parameters included in the large-scale language model are updated so as to minimize the cross-entropy error between the known explanation corresponding to the image data and the explanation output from the large-scale language model.

5. An output system that outputs an inference result of inferring a state of an object and an explanation related to the inference result, an inference device that inputs target data, which is data of the object, into a trained model created by machine learning and outputs the inference result from the trained model; a large-scale language model that outputs the explanation in response to input of intermediate data obtained from an intermediate layer of the trained model; and An output system in which the large-scale language model has been trained through machine learning so that when the target data for training is input to the trained model, intermediate data obtained from an intermediate layer of the trained model is input to the large-scale language model so that the explanation it outputs approaches a known explanation corresponding to the target data.

6. 6. The output system of claim 5, A data acquisition device that inputs the target data to the inference device via a network and an output system, wherein the large-scale language model is connected to the reasoner via a network;

Images