Information processing device, information processing method, and information processing program

Abstract

To enable a user to easily determine correctness of an estimated result of a model.SOLUTION: An information processing device 10 inputs input data to a learned model learned by learning data, and acquires an output result from the learned model. Subsequently, the information processing device 10 calculates a degree of deviation between a distribution of the input data and a distribution of the learning data. Then, the information processing device 10 displays the acquired output result, and the calculated degree of deviation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Traditionally, machine learning involves learning input-output relationships from training data and making inferences (such as classification, regression, and prediction) for new data. When considering the application of a trained model to a real-world field, the estimation accuracy of the model becomes an important concern in operation. In other words, rather than simply making an estimation, users need to determine the validity of the estimation results (guarantee of the model's operation).

[0003] In addition, when a trained model is applied to a real field, the estimation accuracy of the trained model may be reduced. For example, the estimation accuracy of a trained model may be reduced when the input-output relationship itself of the estimation target changes or when data that is not included in the dataset at the time of training is input to the model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-142654 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology has a problem that a user cannot easily judge the validity of the estimation result of the model. In other words, in the past, a user could not judge the validity of the estimation result of the model by simply applying a trained model to the field, and it was unclear to what extent the user could trust the estimated result.

[0006] The present invention has been made in consideration of the above, and aims to provide an information processing device, an information processing method, and an information processing program that enable a user to easily determine the validity of the estimation results of a model. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the objective, the information processing device of the present invention includes an acquisition unit that inputs input data into a trained model trained with training data and acquires output results from the trained model, a calculation unit that calculates the deviation between the distribution of the input data and the distribution of the training data, and displays the output results acquired by the acquisition unit and the deviation calculated by the calculation unit. Effect of the Invention

[0008] According to the present invention, it is possible for a user to easily determine the validity of the estimation results of a model. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating an example of a configuration of an information processing device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a process of calculating the degree of deviation between the distribution of input data and the distribution of training data while inputting input data to a model. [Diagram 3] FIG. 3 is a diagram for explaining a process for displaying the output result of a model and the distribution deviation degree in a case where the input data and learning data are time-series data of a sensor or the like. [Figure 4] FIG. 4 is a diagram for explaining a process for displaying the output result of a model and the distribution deviation degree in a case where the input data and the learning data are image data. [Diagram 5] FIG. 5 is a diagram for explaining a process for displaying the output result of the model and the distribution deviation degree in a case where the input data and the learning data are image data. [Figure 6]FIG. 6 is a flowchart showing a display process in the information processing device according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing the re-learning process in the information processing device according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating a computer that executes a program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an information processing device, an information processing method, and an information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiments.

[0011] In the following embodiments, the process flow of an information processing device, an information processing method, and an information processing program according to the embodiments will be described in order, and finally, the effects of the embodiments will be described.

[0012] [Embodiment Mode] In the following embodiment, the configuration of an information processing device 10 according to the embodiment and the flow of processing by the information processing device 10 will be described in order, and finally, the effects of the embodiment will be described.

[0013] [Information processing device] First, the configuration of an information processing device 10 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an information processing device according to an embodiment. As shown in Fig. 1, the information processing device 10 includes a communication processing unit 11, a control unit 12, and a storage unit 13.

[0014] The communication processing unit 11 communicates with other devices wirelessly or via wires. The communication processing unit 11 is a communication interface that transmits and receives various information to and from other devices connected via a network or the like. The communication processing unit 11 is realized by a NIC (Network Interface Card) or the like, and communicates between other devices and a control unit 12 (described later) via electric communication lines such as a LAN (Local Area Network) or the Internet.

[0015] The storage unit 13 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. The storage unit 13 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM). The storage unit 13 stores an operating system (OS) and various programs executed by the information processing device 10. Furthermore, the storage unit 13 stores various information used in the execution of the programs. The storage unit 13 has a learning data storage unit 13a and an input data storage unit 13b.

[0016] The learning data storage unit 13a stores learning data used when learning the trained model. For example, the learning data storage unit 13a stores process data acquired from a plurality of sensors as the learning data. The learning data storage unit 13a is not limited to storing process data, and may store images, signals, and the like. The learning data storage unit 13a may store not only raw data but also processed data as the learning data.

[0017] The input data storage unit 13b stores input data, which is data collected by the collection unit 12a described later and is input to the trained model. For example, the input data storage unit 13b stores process data acquired from a plurality of sensors as input data. The process data is, for example, data acquired by various sensors installed in a reactor or various devices in a factory or plant. The process data may also be, for example, data such as various signals for operating the reactor or various devices. The process data is time-series data acquired at a predetermined time interval, for example, every second, in each sensor. The input data storage unit 13b is not limited to storing process data, and may store images, signals, and the like. The input data storage unit 13b may also store processed data as well as raw data as input data.

[0018] The control unit 12 controls the entire information processing device 10. The control unit 12 is, for example, an electronic circuit such as a central processing unit (CPU) or a micro processing unit (MPU), or an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control unit 12 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory. The control unit 12 also functions as various processing units by the operation of various programs. The control unit 12 has a collection unit 12a, an acquisition unit 12b, a calculation unit 12c, a display unit 12d, a determination unit 12e, a re-learning unit 12f, and a deletion unit 12g.

[0019] The collecting unit 12a collects input data. For example, the collecting unit 12a collects process data representing time-series data at a predetermined time interval from a plurality of sensors installed in a monitored facility such as a factory or a plant, and stores the collected data in the input data storage unit 13b. In addition, when a predetermined condition is satisfied, for example, when one hour's worth of process data is stored in the input data storage unit 13b, the collecting unit 12a notifies the acquiring unit 12b of an instruction to input the input data to the trained model. Note that the collecting unit 12a may acquire input data manually input by a user.

[0020] The acquisition unit 12b inputs the input data into a trained model trained with the training data, and acquires an output result from the trained model. Note that the input data and trained model may be of any type and are not limited thereto. For example, as an output result from the trained model, a predicted value of process data may be acquired, an abnormality level of the target equipment may be acquired, or a recognition result of image data may be acquired.

[0021] More specifically, for example, when the input data is process data and the output result is an anomaly degree, the acquiring unit 12b, upon receiving an instruction from the collecting unit 12a, reads the process data from the input data storage unit 13b and cuts out the process data of a predetermined width using a sliding window of a predetermined width. Then, the acquiring unit 12b inputs the cut out process data of the predetermined width into a trained model and acquires an output result from the trained model. Here, the predetermined width is a width of a sliding window set in advance, and a value such as one minute can be used. That is, the calculating unit 15 cuts out the process data of the predetermined width by shifting the sliding window of the predetermined width in the time axis direction by a predetermined shift width. The processing by the acquiring unit 12b may be performed offline or online using streaming data.

[0022] The calculation unit 12c calculates the degree of deviation between the distribution of the input data and the distribution of the learning data. Specifically, the calculation unit 12c reads the learning data from the learning data storage unit 13a and reads the process data from the input data storage unit 13b, and calculates the degree of deviation between the distribution of the input data and the distribution of the learning data. Furthermore, when there are multiple types of input data, the calculation unit 12c may calculate the degree of deviation between the distribution of each input data and the distribution of the same type of learning data. For example, when there is data (input data) from multiple sensors, the calculation unit 12c calculates the degree of deviation between the distribution of the input data and the distribution of the learning data for each sensor.

[0023] Here, the calculation unit 12c may use any method as long as it can calculate the degree of deviation of distribution between data. For example, the density ratio is calculated by using a density ratio estimation method such as uLSIF (Unconstrained Least-Squares Importance Fitting) or KLEP, or Generative Adversarial Networks. In addition, the deviation may be a measure such as Kullback-Leibler divergence or Pearson distance that can be calculated naturally from density ratio estimation. In addition to the density ratio, the deviation may be a Mahalanobis distance that measures the distance between distributions or a Euclidean distance used in the nearest neighbor method. In addition, the density ratio is output as a continuous value, but may be a discrete value (such as large, medium, and small) by setting an appropriate threshold value. In addition, the calculation unit 12c may compress the input data and learning data in advance using a dimension compression method such as PCA, and calculate the density ratio after compression. In addition, the algorithm of the density ratio estimation method may be automatically or manually switched.

[0024] Here, a process of calculating the degree of deviation between the distribution of input data and the distribution of learning data while inputting input data to a model will be described with reference to Fig. 2. Fig. 2 is a diagram for explaining a process of calculating the degree of deviation between the distribution of input data and the distribution of learning data while inputting input data to a model. The example of Fig. 2 illustrates a case where there are 100,000 or more pieces of process data as learning data, and about 300 pieces of process data as input data.

[0025] The acquiring unit 12b shifts a sliding window of a predetermined width in the time axis direction by a predetermined shift width to extract process data of a predetermined width, and inputs the extracted process data to the trained model to obtain an output result of the trained model. The calculating unit 12c calculates the degree of deviation between the distribution of the input data and the distribution of the trained data.

[0026] The display unit 12d displays the output result acquired by the acquisition unit 12b and the deviation calculated by the calculation unit 12c. For example, the display unit 12d displays, by a GUI (Graphical User Interface) application, a graph showing a time series change in the output result of the trained model and a graph showing a time series change in the deviation between the distribution of the input data and the distribution of the trained data, side by side so that the time series are the same.

[0027] Here, a process of displaying the model output result and distribution deviation degree when the input data and learning data are time-series data will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining a process of displaying the model output result and distribution deviation degree when the input data and learning data are time-series data. As illustrated in Fig. 3, the display unit 12d displays a graph showing the time-series change of the output result obtained by inputting new input data to the trained model.

[0028] Furthermore, the display unit 12d displays a graph showing a time series change in the distribution deviation obtained by the distribution deviation estimation process of the new input data and the learning data. The display unit 12d displays these two graphs vertically arranged so that the time series are the same. This allows the user to determine that the output result of the section A, in which the distribution deviation between the input data and the learning data is large, is low in reliability, for example. Furthermore, when there are multiple types of input data, the display unit 12d may display the deviation for each type of input data. For example, when there is data (input data) from multiple sensors, the display unit 12d can display the deviation for each sensor, allowing the user to know which data deviates from the learning data.

[0029] The display method is not limited to this. Also, the input data and the learning data are not limited to time-series data, and may be image data, for example. Figures 4 and 5 are diagrams for explaining the process of displaying the model output result and the distribution deviation when the input data and the learning data are image data.

[0030] 4 and 5 illustrate a case where the trained model is trained using image data of the numbers "3", "5", "0", "7", etc. as training data. In addition, in the examples of FIG. 4 and FIG. 5, the trained model outputs an estimation result of the number contained in the input image data. In such a case, as illustrated in FIG. 4, the information processing device 10 inputs image data containing the number "9" as input data into the trained model, thereby acquiring and displaying the output result "9". In addition, the information processing device 10 also displays the deviation "0.03" between the distribution of the input data and the distribution of the training data. A user who sees this display can determine that the reliability of the output result "9" is high because the deviation value is low.

[0031] On the other hand, as illustrated in FIG. 5, the information processing device 10 inputs image data including "horse" as input data into the trained model, thereby acquiring and displaying the output result "9". The information processing device 10 also displays the degree of discrepancy "0.98" between the distribution of the input data and the distribution of the trained data. A user who sees this display can determine that the reliability of the output result "9" is low because the degree of discrepancy is high. Also, for example, the display unit 12d of the information processing device 10 may sort and display the multiple input image data in order of the degree of discrepancy, so that the input images can be compared and displayed.

[0032] Furthermore, the display unit 12d may display a message to the effect that the trained model is to be retrained when the determination unit 12e, which will be described later, determines that the degree of deviation is equal to or greater than a predetermined threshold. In other words, when the degree of deviation between the distributions of the input data and the trained data is large, data that is not included in the trained data of the trained model is input, so that the accuracy of the output result of the trained model may be low, and the display unit 12d prompts the trained model to be retrained so that it can be adapted to the new input data.

[0033] The determination unit 12e determines whether the deviation calculated by the calculation unit 12c is equal to or greater than a predetermined threshold. For example, when the input data and the learning data are time-series data, the determination unit 12e may determine whether the deviation continues to be equal to or greater than the predetermined threshold for a predetermined period of time (e.g., 5 minutes) or more, or whether the deviation is equal to or greater than the predetermined threshold at a certain rate (e.g., 50%) during the predetermined period of time. In this manner, the determination unit 12e does not determine that the deviation is equal to or greater than the threshold when the deviation only increases momentarily, but determines that the deviation is equal to or greater than the threshold only when the deviation continues for a certain period of time, thereby preventing re-learning, which will be described later, from being performed more frequently than necessary.

[0034] The determination unit 12e may change the predetermined threshold value according to the accuracy of the output result acquired by the acquisition unit 12b. For example, when the accuracy of the output result of the trained model can be acquired, the determination unit 12e changes the threshold value to a higher value when the accuracy is equal to or greater than the predetermined threshold value, since re-learning or the like is not required in a hurry, and changes the threshold value to a lower value when the accuracy is less than the predetermined threshold value, since re-learning or the like is required. Note that, when multiple deviation degrees are calculated by the calculation unit 12c, the determination unit 12e determines whether each deviation degree is equal to or greater than the predetermined threshold value.

[0035] When the determination unit 12e determines that the deviation is equal to or greater than a predetermined threshold, the re-learning unit 12f executes re-learning of the trained model. Any re-learning method may be executed by the re-learning unit 12f. For example, the re-learning unit 12f executes re-learning by using the input data stored in the input data storage unit 13b as training data.

[0036] The deletion unit 12g identifies the type of input data whose deviation is determined by the determination unit 12e to be equal to or greater than a predetermined threshold, and deletes the identified type of input data from the input data to be input to the trained model. For example, when the input data is a plurality of sensor data, the deletion unit 12g identifies the sensor data whose deviation is determined by the determination unit 12e to be equal to or greater than a predetermined threshold, and deletes the identified sensor data from the input data to be input to the trained model. In this way, when only some of the input data have a high deviation, the deletion unit 12g can prevent a decrease in prediction accuracy by deleting only the corresponding data.

[0037] [Processing procedure of information processing device] Next, an example of a processing procedure by the information processing device 10 according to the embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a flowchart showing a display process in the information processing device according to the embodiment. Fig. 7 is a flowchart showing a re-learning process in the information processing device according to the embodiment.

[0038] 6, when the collection unit 12a of the information processing device 10 collects input data (Yes in step S101), the collection unit 12a stores the input data in the input data storage unit 13b (step S102). Then, the acquisition unit 12b inputs the input data to a trained model trained with the training data (step S103) and acquires an output result from the trained model (step S104).

[0039] Then, the calculation unit 12c calculates the degree of deviation between the distribution of the input data and the distribution of the learning data (step S105). Specifically, the calculation unit 12c reads the learning data from the learning data storage unit 13a and reads the process data from the input data storage unit 13b, and calculates the degree of deviation between the distribution of the input data and the distribution of the learning data.

[0040] After that, the display unit 12d displays the output result acquired by the acquisition unit 12b and the deviation calculated by the calculation unit 12c (step S106). For example, the display unit 12d displays a graph showing the time series change of the output result of the trained model and a graph showing the time series change of the deviation between the distribution of the input data and the distribution of the trained data side by side so that the time series are the same.

[0041] 7, when the calculation unit 12c of the information processing device 10 calculates the deviation (Yes at step S201), the determination unit 12e determines whether the calculated deviation is equal to or greater than a predetermined threshold (step S202). For example, the determination unit 12e may determine whether the deviation is equal to or greater than the predetermined threshold continuously for a predetermined period of time (e.g., 5 minutes) or more, or whether the deviation is equal to or greater than the predetermined threshold at a certain rate or more during the predetermined period of time (e.g., 50%).

[0042] As a result, if the determination unit 12e determines that the calculated deviation is equal to or greater than the predetermined threshold (Yes in step S202), the re-learning unit 12f executes re-learning of the trained model (step S203) if the determination unit 12e determines that the deviation is equal to or greater than the predetermined threshold. If the determination unit 12e determines that the calculated deviation is not equal to or greater than the predetermined threshold (No in step S202), the process of this flow ends as it is.

[0043] [Effects of the embodiment] In this manner, the information processing device 10 according to the present embodiment inputs input data to a trained model trained with training data, and acquires an output result from the trained model. Next, the information processing device 10 calculates the degree of deviation between the distribution of the input data and the distribution of the training data. Then, the information processing device 10 displays the acquired output result and the calculated degree of deviation. This enables the information processing device 10 to allow the user to easily determine the validity of the estimation result of the model.

[0044] In other words, the information processing device 10 displays the degree of deviation between the distribution of the training data set and the data newly input at the operation site in accordance with the estimation result of the trained model, thereby making it possible to clearly show the validity of the estimation result of the model to the user. For example, the user can determine that the greater the degree of deviation between the distributions of the training data and the new data, the closer the input data is to unknown data, and therefore the lower the validity of the estimation result of the model.

[0045] [System configuration, etc.] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Furthermore, each processing function performed by each device can be realized in whole or in part by a CPU or GPU and a program analyzed and executed by the CPU or GPU, or can be realized as hardware using wired logic.

[0046] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0047] [program] Also, a program in which the processing executed by the information processing device 10 described in the above embodiment is written in a language executable by a computer can be created. For example, a program in which the processing executed by the information processing device 10 in the embodiment is written in a language executable by a computer can be created. In this case, the same effect as in the above embodiment can be obtained by the computer executing the program. Furthermore, such a program may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read and executed by a computer to realize the same processing as in the above embodiment.

[0048] Fig. 8 is a diagram showing a computer that executes a program. As shown in Fig. 8, a computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070, and these components are connected by a bus 1080.

[0049] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012, as exemplified in Fig. 8. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090, as exemplified in Fig. 8. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0050] 8, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the above programs are stored in, for example, the hard disk drive 1090 as program modules in which instructions to be executed by the computer 1000 are written.

[0051] Moreover, the various data described in the above embodiment are stored as program data, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes various processing procedures.

[0052] Note that the program module 1093 and program data 1094 relating to the program are not limited to being stored in the hard disk drive 1090, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via a disk drive or the like. Alternatively, the program module 1093 and program data 1094 relating to the program may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)) and read by the CPU 1020 via the network interface 1070.

[0053] The above-described embodiments and their modifications are included in the technology disclosed in this application, as well as in the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0054] 10. Information processing device 11 Communication processing unit 12 Control section 12a Collection Section 12b Acquisition part 12c calculation part 12d Display section 12e Judgment section 12f Re-learning section 12g Deleted part 13 Storage section 13a Learning data storage unit 13b Input data storage section

Claims

1. An acquisition unit that inputs input data into a trained model trained with training data and acquires an output result from the trained model; a calculation unit that calculates a degree of deviation between the distribution of the input data and the distribution of the training data; a display unit that displays the output result acquired by the acquisition unit and the deviation degree calculated by the calculation unit; having the calculation unit calculates, when there are multiple types of input data, a degree of deviation between a distribution of each input data and a distribution of training data of the same type; a determination unit that determines whether each deviation calculated by the calculation unit is equal to or greater than a predetermined threshold; and a deletion unit that identifies a type of input data for which the determination unit has determined that the deviation is equal to or greater than a predetermined threshold, and deletes the identified type of input data from the input data to be input to the trained model.

2. The method further includes a determination unit that determines whether the deviation calculated by the calculation unit is equal to or greater than a predetermined threshold value, The information processing device according to claim 1 , wherein the display unit displays a message indicating that the trained model will be retrained when the determination unit determines that the degree of deviation is equal to or greater than the predetermined threshold.

3. a determination unit that determines whether the deviation calculated by the calculation unit is equal to or greater than a predetermined threshold; The information processing device according to claim 1 , further comprising a re-learning unit that executes re-learning of the trained model when the determination unit determines that the deviation is equal to or greater than the predetermined threshold.

4. The information processing device according to any one of claims 1 to 3, characterized in that the judgment unit judges whether the deviation degree is equal to or greater than the predetermined threshold value continuously for a predetermined period of time or more, or whether the deviation degree is equal to or greater than the predetermined threshold value at a certain rate or more during the predetermined period of time.

5. 4. The information processing apparatus according to claim 1, wherein the determination unit changes the predetermined threshold value depending on the accuracy of the output result acquired by the acquisition unit.

6. An information processing method executed by an information processing device, An acquisition step of inputting input data into a trained model trained with training data and acquiring an output result from the trained model; a calculation step of calculating a deviation between the distribution of the input data and the distribution of the training data; a display step of displaying the output result acquired in the acquisition step and the deviation degree calculated in the calculation step; Including, In the calculation step, when there are a plurality of types of input data, a degree of deviation between a distribution of each input data and a distribution of training data of the same type is calculated, a determination step of determining whether each deviation calculated by the calculation step is equal to or greater than a predetermined threshold value; an information processing method further comprising: identifying a type of input data for which the deviation is determined to be equal to or greater than a predetermined threshold by the determination step; and deleting the identified type of input data from the input data to be input to the trained model.

7. An acquisition step of inputting input data into a trained model trained with training data and acquiring an output result from the trained model; a calculation step of calculating a degree of deviation between the distribution of the input data and the distribution of the training data; a display step of displaying the output result acquired in the acquisition step and the deviation degree calculated in the calculation step; on the computer, In the calculation step, when there are a plurality of types of input data, a degree of deviation between a distribution of each input data and a distribution of training data of the same type is calculated, a determination step of determining whether each deviation calculated by the calculation step is equal to or greater than a predetermined threshold; an information processing program further comprising: identifying a type of input data for which the deviation is determined to be equal to or greater than a predetermined threshold by the determination step; and deleting the identified type of input data from the input data to be input to the trained model.

Images