Information processing system, method or program

Abstract

To calculate a contribution degree to a model output drift of each of dimensions belonging to input data to a model as one purpose of the present disclosure.SOLUTION: A model output drift detection part 102 determines whether an increase in a model output drift showing the uncertainty of an output of a model 101 occurs. A baseline data setting part 103 selects baseline data 117 from an input or a feature amount 112 to the model at time before time corresponding to input data 111 or the feature amount to the model being an object in which the increase in the model output drift is detected. A contribution degree calculation part 104 calculates a contribution degree 118 to the model output drift of each of a plurality of dimensions about the dimensions belonging to the input data 111 or the feature amount 112 to the model on the basis of the baseline data 117.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technique for calculating the contribution of each dimension of an input to a model to the uncertainty of the output from the model. [Background technology]

[0002] A classification model is known that outputs information about classes (classifications) (e.g., information about the probability of belonging to each class) according to input data or features obtained by a conversion process on the input data (hereinafter, these may be collectively referred to as "input data, etc." or "input, etc."). In addition, not only in classification models but also in models constructed by machine learning, the uncertainty of the output results from the model may increase (hereinafter, such uncertainty may be referred to as "model output drift"). In order to deal with the uncertainty of the output results from the model, it is sometimes necessary to identify the cause of the increase in uncertainty. Patent Document 1 discloses a prior art that defines the degree of change in time series data input to a model as a change score for each piece of time series data. Non-Patent Document 1 discloses a prior art that uses baseline data and current input data as inputs to a model to calculate the contribution of each dimension of the input to a classification model to the uncertainty of the output result from the classification model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-144493 A [Non-patent literature]

[0004] [Non-Patent Document 1] Iker Perez, et al., "Attribution of Predictive Uncertainties in Classification Models", Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence (UAI 2022) ,arXiv,Submitted on 19 Jul 2021, last revised 8 Jun 2022,pp.1582-1591 Summary of the Invention [Problem to be solved by the invention]

[0005] This is a system configuration in which time-series data having multiple dimensions corresponding to a single time obtained from a certain system is input to a classification model, and information regarding the class (classification) is output from the model. When it is assumed that one of the causes of model output drift is due to changes in the state of the input data, etc. to the model during operation (hereinafter, changes in the state of the input data, etc. may be referred to as "model input drift"), it may be desirable to identify which dimension of model input drift contributed to the model output drift. The prior art disclosed in Patent Document 1 defines the degree of change of time series data for each time series data input to the model as a change score. However, a change in the state of the input data to the model does not necessarily lead to a deterioration in the accuracy of the model output. For example, if the input data to the model has multiple dimensions corresponding to one time, the model input drift for a dimension that has a low contribution to the model output is unlikely to cause model output drift (deterioration of the model accuracy). Therefore, the prior art disclosed in Patent Document 1 may not be able to correctly evaluate the influence of the model input drift of each dimension on the model output drift when the input data to the model has multiple dimensions. In other words, the prior art disclosed in Patent Document 1 may not be able to identify which dimension the model input drift contributed to the model output drift. The prior art disclosed in Non-Patent Document 1 uses baseline data and current input data as input data to the model in order to calculate the contribution of each dimension of the input data to the classification model to the uncertainty of the output result from the classification model. However, the calculation result of the contribution may vary depending on the setting of the baseline data. Therefore, the prior art disclosed in Non-Patent Document 1 may not be clear about what technical meaning is indicated by the calculated contribution of each dimension. For example, in a case where model input drift occurs intermittently multiple times in a certain period, or in a case where model input drift occurs constantly in a certain period, even if the contribution of each dimension is calculated, it may not be clear what the calculated contribution value means.

[0006] In view of the above, in a case where the input data to a model has multiple dimensions, it may be one of the objects of the present disclosure to calculate the contribution of each dimension to the model output drift. In addition, it may be one of the objects of the present disclosure to calculate the contribution of each dimension according to what technical meaning is indicated. [Means for solving the problem]

[0007] In order to achieve at least one of the above objects, the present disclosure may have the following features, for example. One aspect of the present disclosure is an information processing system. The information processing system realizes a model for outputting information about a class according to an input or a feature obtained by a conversion process for the input. The information processing system includes a model output drift detection unit, a baseline data setting unit, and a contribution calculation unit. The model output drift detection unit determines whether an increase in model output drift, which indicates uncertainty in the model output, has occurred. The baseline data setting unit selects baseline data from the input to the model or the feature at a time before the time corresponding to the input to the model or the feature for which an increase in model output drift has been detected. The contribution calculation unit calculates the contribution of each dimension to the model output drift for multiple dimensions of the input to the model or the feature based on the baseline data. Effect of the Invention

[0008] As described above, the present disclosure is capable of calculating the contribution of each dimension of the input data to the model to the model output drift when an increase in the model output drift occurs, by using the functions of the model output drift detection unit, the baseline data setting unit, and the contribution calculation unit. Also, as described above, the present disclosure is capable of calculating the contribution of each dimension by the contribution calculation unit using the baseline data selected by the baseline data setting unit, and therefore is capable of calculating the contribution according to what technical meaning is indicated by the contribution of each dimension that is to be calculated, by selecting the baseline data.

[0009] A method or program for implementing the same processing as that implemented by the above-mentioned information processing system can also achieve the same effects as those achieved by the above-mentioned information processing system. Furthermore, in the form of a program, costs can be reduced in many cases. With a program, design changes regarding processing can be easily made. Other features that the present disclosure may have and the effects corresponding to these features will be disclosed in this specification, the claims, or the drawings. [Brief description of the drawings]

[0010] [Figure 1] 1 shows a functional configuration common to each of the embodiments of the present disclosure. [Diagram 2] 1 illustrates a system configuration according to an embodiment of the present disclosure. [Diagram 3] 1 illustrates a computer architecture for implementing an embodiment of the present disclosure. [Figure 4] 1 shows a functional configuration of a first embodiment. [Diagram 5] 2 shows a flowchart of the first embodiment. [Figure 6] The significance of selecting baseline data based on time period is shown. [Figure 7] 13 illustrates the significance of selecting baseline data based on predicted class. [Figure 8] 13 shows a flowchart for calculating the contribution of each dimension. [Figure 9] 1 shows an explanatory screen according to the first embodiment. [Figure 10] This shows the functional configuration for machine learning. [Figure 11] A flowchart for machine learning is shown. [Figure 12] 1 shows a functional configuration of a second embodiment. [Figure 13] 1 shows a flowchart of a second embodiment. [Figure 14] 11 shows an explanatory screen according to the second embodiment. [Figure 15] 13 shows a functional configuration of a third embodiment. [Figure 16] 13 shows a flowchart of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the embodiments described below do not limit the disclosure according to the claims, and all of the elements and combinations thereof described in the embodiments are not necessarily essential to the solution of the present disclosure. The following description and drawings are examples for explaining the present disclosure, and are omitted and simplified as appropriate for clarity of explanation. The present disclosure can be implemented in various other forms. Unless otherwise limited, each component may be singular or plural. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. For this reason, the present disclosure is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings. Each of the systems, devices, models, or units of the present disclosure may be integrated as a single piece of hardware, or may be divided into multiple parts that work together to perform their roles. Several systems, devices, models, or units may be integrated as hardware. Each of the systems, devices, or units may be realized by having a computer execute software (programs) (as in FIG. 3). Some of the functions of the system, device, model, or unit may be realized by hardware (e.g., hardwired logic or FPGA), and the remaining functions may be realized by executing software (programs). All of the functions of the system, device, model, or unit may be realized by hardware. Some or all of the steps of the processes shown in the flowcharts and the like described in the present disclosure may be realized by hardware. One or more systems, devices, models, or units of the present disclosure may be realized using one or more hardware resources. For this purpose, each of the systems, devices, models, or units of the present disclosure may be realized virtually. For example, a virtual computer or container technique may be used. The program of the present disclosure may be included in the concept that generally includes software that constitutes a specific information processing system or its operating method according to the purpose of use by cooperation between software and hardware resources. In other words, the program of the present disclosure is not limited to a specific type or form of program. In addition, the program may be initially recorded in a compressed format. The same reference numbers are used in multiple drawings to indicate similar things. In the drawings showing the flowcharts described below, rectangular boxes indicate processing steps, and hexagonal boxes indicate conditional branch steps. In the drawings showing the flowcharts, "step" is abbreviated as "S." Also, the screen aspects shown in the drawings described below are examples, and are not limited to these.

[0012] 1. Functional configuration common to each of the embodiments of the present disclosure Fig. 1 shows a functional configuration common to each of the embodiments of the present disclosure. Note that not all of the functional configurations shown in Fig. 1 are essential. FIG. 1 illustrates an information processing system 100 according to the present disclosure. The information processing system 100 realizes a model 101 for outputting information 113 about a class in response to input data 111 to the model. For example, the information processing system 100 may realize the model 101 by reading a model program 1021 and model parameters 1031 (to be described later) from a model database 406 (in the drawings, a database may be abbreviated as DB) (to be described later) and executing the model program 1021 while using the model parameters 1031. The input data 111 to the model may be, for example, data at each time in time series data obtained from some system. The input data 111 to the model is input to the model 101 as is, or the input data 111 to the model is subjected to a conversion process 105 to be converted into a feature 112, and the feature 112 is then input to the model 101. The input data 111 or the feature 112 input to the model 101 is information having multiple dimensions for one time. If the number of dimensions of the input data 111 or the feature 112 is m, the input data 111 or the feature 112 can be expressed in the form of a vector x as shown in the following formula. Note that i indicates the index of the dimension of the input data 111 or the feature 112, and T indicates the transposition of rows and columns. x = ( x1, ,x i ,···,x m ) T The class information 113 output from the model 101 may be a result of class (classification) determination inferred based on the input data 111 or the feature amount 112. For example, the class information 113 may be expressed as a probability of belonging to each class j when the class index is j: j If the number of classes to which the class can belong is n, the class-related information 113 can be expressed in the form of a vector y as shown in the following formula. Note that each component y of the vector y is j The sum of y is 1. j A normalization of may be performed. y = ( y1, ,y j ,···,y n ) T

[0013] The information processing system 100 includes a model output drift detection unit 102, a baseline data setting unit 103, and a contribution degree calculation unit 104. In Fig. 1, each of the functional units consisting of the model output drift detection unit 102, the baseline data setting unit 103, and the contribution degree calculation unit 104 is shown as if it were separate from the model 101, but in reality, the function of the model 101 can be realized in each of the model output drift detection unit 102, the baseline data setting unit 103, and the contribution degree calculation unit 104. The model output drift detection unit 102 judges whether an increase in model output drift has occurred, which indicates uncertainty 114 of the class-related information 113 that is the output of the model 101. For example, when the class-related information 113 that is the output of the model 101 is expressed in the form of the vector y described above, if the result of the determination of the class (classification) inferred according to the input data 111 or the feature amount 112 is correct and there is absolutely no uncertainty, the model output drift detection unit 102 judges whether an increase in model output drift has occurred, which indicates uncertainty 114 of the class-related information 113 that is the output of the model 101. j (If normalization is performed, y is calculated only for j, which is the index of the correct class. j = 1, and for other j, y j = 0. However, as the uncertainty of the output of model 101 increases, y for j, which is the index of the class that is the correct answer, j The value of y for other j becomes lower. j When the uncertainty of the output of model 101 becomes extremely high and it is impossible to determine the class (classification), the value of y for all j becomes j The value of approaches a uniform value (1 / n). The model output drift detection unit 102 detects an increase in uncertainty 114 from information 113 about the class (for example, information in the form of the above-mentioned vector y) which is the output of the model 101, and detects 115 an increase in model output drift. The baseline data setting unit 103 selects baseline data 117 from past inputs or features 116 at past times. Here, the "past times" may refer to past times viewed from the time associated with the input data 111 (which is an input to the model 101) or features 112 associated with the information 113 on the class (which is an output from the model 101) for which an increase in model output drift was detected 115. The "past inputs or features 116" may be the input data 111 or features 112 stored in a later-described operational database 407, and the "past inputs or features 116" may be read out from the operational database 407. As in the first embodiment, the second embodiment, or a modified example described later, the baseline selection method performed by the baseline data setting unit 103 may be determined depending on what technical meaning is indicated by the contribution degree of each dimension that is to be calculated. A specific baseline selection method will be described later. The contribution calculation unit 104 calculates the contribution 118 of each dimension to the model output drift for multiple dimensions of the input data 111 or the feature amount 112 based on the baseline data 117. As in the above example, if the input data 111 or the feature amount 112 corresponding to one time has m elements (dimensions) x i In the case where the vector x has the following form, the contribution calculation unit 104 calculates the contribution 118 for each dimension index i. Here, since the baseline data setting unit 103 selects baseline data 117 that is in line with the desired purpose, the contribution calculation unit 104 can calculate the contribution 118 for each dimension index i that is in line with the desired purpose.

[0014] The information processing system 100 according to the present disclosure has the above-described functional configuration, and therefore can provide the effects described in the above-described [Effects of the Invention].

[0015] 2. System configuration of the embodiment of the present disclosure FIG. 2 shows a system configuration of a system including an information processing system 100 according to the present disclosure and a sensor terminal or a computer that communicates information with the information processing system 100. 2, an information processing system 100, a sensor terminal 201, and a computer 202 are capable of communicating with each other via a communication network 203. Although one each of the information processing system 100, the sensor terminal 201, and the computer 202 is shown in Fig. 2, the numbers of each of the information processing systems 100, the sensor terminals 201, and the computers 202 can be determined arbitrarily. The information processing system 100 realizes the functional configuration shown in Fig. 1. That is, the information processing system 100 calculates the contribution 118 of each dimension of the input data 111 or the feature amount 112 to the model output drift based on the functions of the model output drift detection unit 102, the baseline data setting unit 103, and the contribution calculation unit 104. Both the sensor terminal 201 and the computer 202 can provide information that will become the input data 111 itself, or information that can become the input data 111 after being processed. The sensor terminal 201 may include, for example, a communication unit 211, a storage unit 212, a control unit 213, and a sensor unit 214. The sensor unit 214 may acquire sensor data from devices such as a temperature sensor, a magnetic sensor, a speed sensor, and a camera. The storage unit 212 may store the sensor data acquired by the sensor unit 214. The control unit 213 may control the sensor data stored in the storage unit 212 to be transmitted to the information processing system 100. The communication unit 211 may perform transmission and reception processing according to a communication protocol on the communication network 203 under the control of the control unit 213. The sensor data transmitted from the sensor terminal 201 is received by the information processing system 100. The information processing system 100 handles the received sensor data as input data 111 (after processing the data as necessary). The computer 202 may include, for example, a communication unit 221, a storage unit 222, and a control unit 223. The storage unit 222 stores data. The control unit 223 may control the data stored in the storage unit 222 to be transmitted to the information processing system 100. The communication unit 221 may perform transmission and reception processing in accordance with a communication protocol on the communication network 203 under the control of the control unit 223. The data transmitted from the computer 202 is received by the information processing system 100. The information processing system 100 handles the received data as input data 111 (after processing the data as necessary). 2, the information processing system 100 has the sensor terminal 201 and the computer 202 as information sources for the input data 111, but the information processing system 100 may use other information sources for the input data 111. The information processing system 100 may have an information source for the input data 111 inside. In addition to the information that becomes the input data 111 itself or the information that can become the input data 111 after being processed, the information processing system 100 may also acquire information regarding the labels 1012 (correct answer information regarding classes (classifications)) used in the machine learning described below from various information sources including the sensor terminal 201 and the computer 202.

[0016] 3. Computer Architecture for Implementing the Embodiments of the Disclosure FIG. 3 illustrates a computer architecture for implementing an embodiment of the present disclosure. To realize the information processing system 100, an information processing device (e.g., a CPU, which may be one or more microprocessors) 301, a storage device (e.g., a memory) 302, a non-volatile recording medium (e.g., a non-volatile memory (e.g., a flash memory), a non-volatile disk device) 303, an external recording medium drive (e.g., a disk drive) 304, a display or output device (e.g., a display, a printer, a speaker) 306, an input device (e.g., a mouse, a keyboard, an imaging device, a sensor, a touch panel, a pointing device) 307, a communication device (e.g., a communication device for wired communication, a communication device for wireless communication. This may be a network interface device (NIC) that controls communication with other systems, devices, or servers according to a predetermined protocol. In the present disclosure, the communication device may communicate with one or more sensor terminals, or one or more computers, via a communication network 203.) 308, and some or all of an external input / output port 309 may be interconnected by an interconnection unit (e.g., a bus, a crossbar switch) 310. A program 320a (e.g., a program for realizing the functional configuration according to the present disclosure) and various information may be recorded in the non-volatile recording medium 303. As the various information, for example, various tables 321 or various information 322 may be recorded in the non-volatile recording medium 303. Alternatively, a part or all of the above-mentioned program or various information may be acquired (accessed) from outside of FIG. The external recording medium drive 304 can connect to an external recording medium (for example, a portable recording disk (DVD, etc.), an IC card, an SD card, a non-volatile memory (for example, a flash memory), a portable hard disk) 305. Note that a program 320a (for example, a program for realizing the functional configuration according to the present disclosure) and the above-mentioned various information may be transferred and stored from this external recording medium 305 to the non-volatile recording medium 303 or the storage device 302. The external recording medium 305 may be used to record programs and data handled in the information processing system 100. The external recording medium drive 304 and the external recording medium 305 may be connected to the information processing system 100 via the communication network 203. The program 320a (e.g., a program for realizing the functional configuration related to the present disclosure) and the various information described above may be provided via the communication device 308, the external input / output port 309 or the input device 307, and stored in the non-volatile recording medium 303 or the memory device 302. In order for the architecture of FIG. 3 to function as the information processing system 100, each unit (including functional units and models) in the information processing system 100, or a part of each unit (to execute one or a series of processes (steps)), the program 320a may be loaded into the storage device 302 (for example, from the non-volatile recording medium 303). The loaded program is indicated by 320b in FIG. 3. The information processing device 301 may then execute the program 320b (using various information present in the non-volatile recording medium 303, etc., as necessary). By executing the program 320b, the function of the information processing system 100, each unit (including functional units and models) in the information processing system 100, or a part of each unit is realized (one or a series of processes (steps) is executed). At this time, various buffers 323 temporarily formed in the storage device 302 may also be used as appropriate.

[0017] 4. First embodiment of the present disclosure The first embodiment of the present disclosure will be described below. Note that it is not essential to realize all of the functional configurations described below and perform all of the processes, and an information processing system, method, or program may realize a part of the functional configurations described below and perform a part of the processes.

[0018] 4.1. Outline of Functional Configuration of First Embodiment of the Present Disclosure Fig. 4 shows a functional configuration of the first embodiment of the present disclosure. To realize the functional configuration shown in Fig. 4, for example, an information processing system 100 having the computer architecture shown in Fig. 3 described above may be used. Here, the functional configuration shown in Fig. 4 will be explained in outline. A detailed explanation will be given together with the contents of the processing in the section "4.2. Processing of the first embodiment of the present disclosure" described later. 1, the information processing system 100 calculates contributions 118 to the model output drift of each dimension of input data 111 to the model or feature amount 112 obtained by performing a conversion process on the input data 111, using the functions of the model output drift detection unit 102, the baseline data setting unit 103, and the contribution calculation unit 104. Here, the input data 111 to the model may be data at individual times in time-series data obtained from a certain system 499. The system 499 may be, for example, one in which a classification model is applied to data obtained from the system 499. Either the input data 111 or the feature quantity 112 has multiple dimensions corresponding to one time. Here, the number of dimensions is assumed to be m. For example, either the input data 111 or the feature quantity 112 has an element for index i of the dimension (element) x corresponding to one time. i The vector x may be in the form: Either or both of the input data 111 and the feature amount 112 are recorded in the operation database 407. At that time, time information corresponding to the input data 111 and the feature amount 112 may also be recorded in the operation database 407. Either or both of the input data 111 and the feature amount 112 recorded in the operation database 407 may be used by the baseline data setting unit 103 as past input or feature amount 116. The model 101 shown in Fig. 1 is realized inside the model output drift detection unit 102. The model 101 may also be realized inside the baseline data setting unit 103 or the contribution degree calculation unit 104. Fig. 4 shows a model database 406. This model database 406 records a model program 1021 and model parameters 1031, which will be described later. When the model output drift detection unit 102, the baseline data setting unit 103, or the contribution degree calculation unit 104 realizes the model 101, the model program 1021 and the model parameters 1031 are read from the model database 406, and the model program 1021 is executed while using the model parameters 1031. The model output drift detection unit 102 includes a model output calculation unit 421, an uncertainty calculation unit 422, and a model output drift determination unit 423. The baseline data setting unit 103 includes a reference time selection unit 431, a past model output acquisition unit 432, and a baseline data selection unit 433. The contribution calculation unit 104 includes an individual contribution calculation unit 441 and a set contribution calculation unit 442. The processing performed by each functional unit will be explained in order in the section "4.2. Processing of the first embodiment of the present disclosure" described later. The information processing system 100 may include a display output control unit 405. After the contributions 118 to the model output drift of the respective dimensions of the input data 111 or the feature amount 112 are obtained, the display output control unit 405 may control information on the obtained contributions 118 to be displayed or output on, for example, the display or output device 306. A specific display or output mode will be described later.

[0019] 4.2 Processing of the First Embodiment of the Present Disclosure Fig. 5 shows a flowchart of the process of the first embodiment of the present disclosure. Hereinafter, the process of the first embodiment of the present disclosure will be described mainly with reference to the flowchart shown in Fig. 5 and the functional configuration shown in Fig. 4. 5, model program 1021 and model parameters 1031 are read from model database 406. The destination to which this information regarding model 101 is read may be model output drift detection unit 102. In addition, when the baseline data setting unit 103 and contribution degree calculation unit 104 also realize the function of model 101, the destination to which this information regarding model 101 is read may include baseline data setting unit 103 and contribution degree calculation unit 104. 5, the information processing system 100 acquires input data 111 for the model. The input data 111 may be acquired from the system 499. Alternatively, the sensor terminal 201 or the computer 202 shown in FIG. 2 may be the source of the input data 111. In this step 502, when the input data 111 is acquired from outside the information processing system 100, the communication device 308 may be involved in acquiring the input data 111 from outside the information processing system 100. 5, the information processing system 100 may record the input data 111 in the operation database 407. Of the input data 111 and the feature amount 112, at least one having a dimension corresponding to the contribution degree 118 is recorded in the operation database 407. When recorded, the input data 111 is recorded in the operation database 407 together with time information. In step 504 in Fig. 5, a feature amount calculation unit 425 (illustrated in Fig. 10 described later, but omitted in Fig. 4) may calculate feature amounts 112 from the input data 111 by performing a conversion process 105. For example, when it is assumed that the input data 111 cannot be directly input to the model 101, or that the accuracy of class (classification) inference by the model 101 is not high if the input data 111 is directly input to the model 101, the feature amount 112 may be calculated in this manner. Note that the function of this conversion process 105 (feature amount calculation unit 425) may be treated as a part of the function of the model 101. 5, the information processing system 100 may record the feature amount 112 in the operation database 407. As described above, at least one of the input data 111 and the feature amount 112 is recorded in the operation database 407. When recorded, the feature amount 112 is recorded in the operation database 407 together with time information.

[0020] 5 (model output calculation step), the model output calculation unit 421 realizes the model 101 using the model program 1021 and the model parameters 1031, thereby obtaining class-related information 113 according to the input data 111 or the feature amount 112. As described with reference to FIG. 1, the input data 111 or the feature amount 112 may be expressed in the form of a vector x as shown in the following equation. x = ( x1, ,x i ,···,x m ) T As described with reference to FIG. 1, the class information 113 output from the model 101 may be expressed in the form of a vector y as shown in the following formula. Note that each component y of the vector y j (probability of membership in class j) should be 1, so that y j A normalization of may be performed. y = ( y1, ,y j ,···,y n ) T 5, the model output calculation unit 421 may store the class-related information 113 in the operation database 407. When this storage is performed, the past model output acquisition unit 432 described later can obtain past class-related information 435 without implementing the model 101.

[0021] In step 508 (uncertainty calculation step) of FIG. 5, the uncertainty calculation unit 422 calculates a model output drift degree 424 indicating the uncertainty of the output from the model, based on the class-related information 113. When the model output drift degree 424 is U, the uncertainty calculation unit 422 may determine an appropriate function to calculate U. When the class-related information 113 is in the form of the above-mentioned vector y, the uncertainty calculation unit 422 may calculate the model output drift degree U, for example, by using an entropy-form function shown in the following formula. In the following formula, the number of classes (classifications) is n, and the membership probability for class (classification) index j is y j Then, log n y j Let y be the base n. j Let Σ denote the sum from j=1 to n. U = -Σy j *log n y j (Σ is the sum from j=1 to n) According to the function defined in the above formula, if there is no uncertainty in the vector y (y for only one j), j = 1, and for other j, y j =0), U=0. On the other hand, according to the function defined above, if the uncertainty of vector y is so high that the class (classification) cannot be determined from vector y at all (y for all j), j =1 / n), then U=1. Alternatively, as a modified example, when the class-related information 113 is in the form of the above-mentioned vector y, the uncertainty calculation unit 422 may obtain the model output drift degree U by using, for example, the function shown in the following formula. In the following formula, max is the maximum value of y j It is said to show that. U = 1 - max{ y1, ,y j ,···,y n} According to the function defined in the above formula, if there is no uncertainty in the vector y (y for only one j), j = 1, and for other j, y j =0), U=0. On the other hand, according to the function defined above, if the uncertainty of vector y is so high that the class (classification) cannot be determined from vector y at all (y for all j), j =1 / n), then U=1-1 / n. The uncertainty calculation unit 422 may record the calculated model output drift degree 424 (U) in the operation database 407 in association with the corresponding input data 111 or feature amount 112 .

[0022] 5, the model output drift determination unit 423 determines whether the model output drift degree 424(U) satisfies a predetermined condition. For example, the model output drift determination unit 423 may determine whether the model output drift degree 424(U) exceeds a predetermined threshold value or is equal to or greater than the threshold value. For example, when the model output drift degree U is calculated using one of the above two equations as a function for calculating U, the value of U is generally equal to or greater than 0 and less than 1, so that a value of around 0.7 may be set as the threshold value. If the determination result of step 509 is positive, control is transferred to step 510 . If the determination result in step 509 is negative, the control proceeds to step 502. Thereafter, steps 502 to 509 are repeated for new input data 111. Note that the information processing system 100 may be in a standby state in step 502 until the new input data 111 arrives. 5, the model output drift determination unit 423 determines that an increase in model output drift has occurred, which indicates output uncertainty of the model 101. When an increase in model output drift is detected 115, the model output drift determination unit 423 may output a model output drift degree 424(U) at that time. The first embodiment can appropriately detect an increase in model output drift in the model 101 that determines the class (classification)-related information 113 by using the model output calculation unit 421, the uncertainty calculation unit 422, and the model output drift determination unit 423. Note that the steps performed by the model output drift detection unit 102 may be collectively called a model output drift detection step.

[0023] In step 511 (reference time selection step) of Fig. 5, the reference time selection unit 431 selects a time period that satisfies a predetermined condition. Here, the time period that satisfies the predetermined condition is selected from time periods that include a time period before a time corresponding to the input data 111 or feature amount 112 to the model in which an increase in model output drift has been detected. In this first embodiment, the reference time selection unit 431 may select one predetermined time period immediately before a time corresponding to the input data 111 or feature amount 112 to the model in which an increase in model output drift has been detected, as the time period that satisfies the predetermined condition. Examples of the selected time period are shown as 602 in Fig. 6 and 911 in Fig. 9, which will be described later. Then, in the same step 511 (reference time selection step), the reference time selection unit 431 selects input data 111 or feature amount 112 for the model at a time included in the selected time period. Here, the number of samples of the selected input data 111 or feature amount 112 is set to w. The reference time selection unit 431 may acquire the input data 111 or feature amount 112 included at a time included in the selected time period from the operation database 407 that accumulates the input data 111 or feature amount 112. In FIG. 4, the input data 111 or feature amount 112 accumulated in the operation database 407 is described as "past input, etc. 116 (past input or feature amount 116)", and the input data 111 or feature amount 112 acquired by the reference time selection unit 431 is described as "input or feature amount 434 of a specific time period".

[0024] In step 512 (past model output acquisition step) of FIG. 5, the past model output acquisition unit 432 acquires class-related information 113 ("past class-related information 435" in FIG. 4) corresponding to the input data 111 or feature 112 selected by the reference time selection unit 431 ("input or feature 434 for a specific time period" in FIG. 4). There are several methods for acquiring the past class information 435. For example, the past model output acquisition unit 432 may obtain the class information 113 (past class information 435) that is the output of the model 101 for each sample (input data 111 or feature amount 112) included in the input or feature amount 434 in a predetermined time period by implementing the model 101 using the model program 1021 and the model parameters 1031. Alternatively, if the operation database 407 accumulates not only the history of the time information, the input data 111, and the feature amount 112, but also the history of the class information 113 that was previously obtained as an output from the model 101, the past model output acquisition unit 432 may acquire the class information 113 (past class information 435) that corresponds to the time included in the time period that satisfies the predetermined condition selected in step 511 from the operation database 407.

[0025] In step 513 (baseline data selection step) of FIG. 5, the baseline data selection unit 433 selects one or more samples (input data 111 or feature 112) to become baseline data 117 from the group of samples of input data 111 or feature 112 selected by the reference time selection unit 431. When the baseline data selection unit 433 selects a sample (input data 111 or feature amount 112) to be the baseline data 117, the class information 113 for the input data 111 or feature amount 112 for which an increase in model output drift has been detected, and the past class information 435 acquired by the past model output acquisition unit 432 may be used. For example, the baseline data selection unit 433 may compare the assignment class C inferred from the class information 113 for the input data 111 or feature amount 112 for which an increase in model output drift has been detected, with the assignment class C inferred from the past class information 435 for each sample (input data 111 or feature amount 112) included in the sample group of the input data 111 or feature amount 112 selected by the reference time selection unit 431. The assignment class C inferred from the class information 113 (or the past class information 435) may be, for example, an assignment probability y j The class (classification) corresponding to the index j with the highest value of C may be determined. The class C thus obtained can be expressed by the following formula. C = arg max{ y1, ,y j ,···,y n} (arg max is y j j) where is the maximum The baseline data selection unit 433 may select, for example, samples (input data 111 or feature 112) having a value of the belonging class C that matches the value of the belonging class C inferred from the class-related information 113 for the input data 111 or feature 112 for which an increase in model output drift has been detected, as the baseline data 117. As described above, if the number of samples selected by the reference time selection unit 431 is w, the number of samples selected by the baseline data selection unit 433 is less than or equal to w. A group of samples (input data 111 or feature 112) selected in this manner may be called a baseline data set.

[0026] As described above, the baseline data setting unit 103 in the first embodiment selects, as baseline data 117, a sample (input data 111 or feature amount 112) that (1) belongs to a time period immediately before a time corresponding to the input data 111 or feature amount 112 for which an increase in model output drift has been detected, and (2) corresponds to a value of the same assignment class C as the value of the assignment class C inferred from information 113 on the class corresponding to the input data 111 or feature amount 112 for which an increase in model output drift has been detected. Note that the steps performed by the baseline data setting unit 103 may be collectively referred to as a baseline data setting step. The advantages of the baseline data setting unit 103 selecting the baseline data 117 using the above criteria (1) and (2) will be described.

[0027] FIG. 6 illustrates the advantage of selecting, as baseline data 117, a sample (input data 111 or feature 112) that belongs to a time period immediately preceding the time corresponding to the input data 111 or feature 112 for which an increase in model output drift has been detected (1). In FIG. 6, the input data 111 or feature 112 for which an increase in model output drift has been detected is described as “data 601 at the time of drift occurrence.” The input data 111 or feature 112, which is the data 601 at the time of drift occurrence, is expressed as a vector x 0 As described above, in the first embodiment, the reference time selection unit 431 selects one time period immediately before the time corresponding to the input data 111 or feature 112 for which an increase in model output drift has been detected. In FIG. 6, the selected time period is represented as "selected time period 602." In the first embodiment, one or more baseline data 117 are selected from a set of samples (inputs or features 434 in a predetermined time period) corresponding to the time included in the selected time period 602. In FIG. 6, the baseline data selected in this manner is represented as "baseline data 603 selected in the present disclosure." The baseline data 603 selected in the present disclosure is x B Here, it is assumed that a sample (input data 111 or feature 112) corresponding to a time not included in the selected time period 602 is randomly selected as baseline data. In FIG. 6, this randomly selected baseline data is represented as x R This is expressed as: Suppose sample x R From a time later than the time corresponding to sample x (a time not included in the selected time period 602), 0 There is a small model input drift (not detectable as an increase in model output drift) that occurs consistently up to the time corresponding to sample x B A large model input drift occurred after the time corresponding to sample x 0 Assume that we detect an increase in model output drift at the time corresponding to sample x R , x B and x 0 The model output drift degree U(x R ), U(x B ) and U(x 0) can have the relationship shown in the lower diagram of Figure 6. That is, U(x 0 ) to the value of U(x R ) is the small model input drift that continues from the time before the selected time period 602 and the sample x 0 This reflects both the large model input drift that occurred close to the time corresponding to U(x 0 ) to the value of U(x B ) does not reflect the small model input drifts that have continued from times prior to the selected time period 602 and is the difference between the sample x 0 This reflects a large model input drift that occurred at a time close to just before the time corresponding to the input data 111 or the feature amount 112 at which the increase in the model output drift was detected. This is represented as "change amount 605 due to detected drift" in Fig. 6. In other words, when an increase in the model output drift is detected by the method of selecting the baseline data 117 in the first embodiment, the information processing system 100 can eliminate information indicating a small model input drift that has continued from before as much as possible and obtain information indicating the mode of the model input drift that has newly occurred at a time relatively close to the time corresponding to the input data 111 or the feature amount 112 at which the increase in the model output drift was detected.

[0028] FIG. 7 illustrates the advantage of selecting as baseline data 117 a sample (input data 111 or feature 112) corresponding to the same value of assigned class C as the value of assigned class C inferred from information 113 about the class corresponding to input data 111 or feature 112 for which an increase in model output drift has been detected above (2). In FIG. 7, the input data 111 or feature 112 for which an increase in model output drift was detected is described as "data 601 at the time of drift occurrence." The input data 111 or feature 112, which is the data 601 at the time of drift occurrence, is expressed as a vector x 0As described above, in the first embodiment, the baseline data selection unit 433 selects, as the baseline data 117, a sample (input data 111 or feature amount 112) corresponding to the same value of the attribute class C as the value of the attribute class C inferred from the information 113 on the class corresponding to the input data 111 or feature amount 112 in which an increase in model output drift has been detected. In FIG. 7, the baseline data selected based on the value of the attribute class C is represented as "baseline data 603 selected in the present disclosure." The baseline data 603 selected in the present disclosure is represented as x B Here, the sample (input data 111 or feature 112) is randomly selected without considering the value of the class C, resulting in a vector x 0 Assume that a sample (input data 111 or feature 112) corresponding to a value of the attribute class C different from the value of the attribute class C corresponding to x is selected as the baseline data. In FIG. 7, this randomly selected baseline data is x r This is expressed as: We assume that even if the same factor is involved, the model output drift degree U will differ if the value of the attribute class C is different. In this case, sample x r , x B and x 0 The model output drift degree U(x r ), U(x B ) and U(x 0 ) can have the relationship shown in the lower diagram of Figure 7. That is, U(x 0 ) to the value of U(x r ) is the difference between sample x r From the time corresponding to the sample x 0 In addition to the minutes that indicate the drift that started at the time between the time corresponding to sample x r and sample x 0 This also reflects the effect of the difference in the value of the attribute class C between the two classes. This is shown in Figure 7 as "change 706 including the fluctuation of the uncertainty between classes". On the other hand, U(x 0 ) to the value of U(x B) does not reflect the effect of different values ​​of the attribute class C, and the difference between the sample x B From the time corresponding to the sample x 0 This reflects the amount of change due to the detected drift 605 in Fig. 7. In other words, when an increase in the model output drift is detected by the method of selecting the baseline data 117 in the first embodiment, the information processing system 100 can acquire information indicating the state of the drift, which eliminates the difference in the value of the model output drift degree U due to the difference in the belonging class C as much as possible.

[0029] In step 514 (individual contribution calculation step) of FIG. 5, the individual contribution calculation unit 441 calculates the baseline data 117 (vector x B ), the input data 111 or the feature quantity 112 (vector x B ) and the input data 111 or features 112 (vector x 0 ) to the model output drift. The individual contribution calculation unit 441 calculates the contribution of each dimension to the model output drift in the relationship between the vector x 0 and vector x B The difference of the model output drift degree U based on 0 )-U(x B ) to the vector, the contribution of each dimension of the vector to the vector i is expressed as the contribution of each dimension I i The individual contribution calculation unit 441 may use the Integrated gradients method or a method derived from the Integrated gradients method disclosed in Non-Patent Document 1. The individual contribution calculation unit 441 calculates a vector x B The vector x 0 The contribution of each dimension i, IG, as a path integral with the coordinates indicated by i Such a formula for the path integral is expressed as (B) of step 809 in FIG. 8. That is, the individual contribution calculation unit 441 calculates the vector xB The vector x 0 The i-th element x of the vector x of the model output drift degree U along the path whose end point is the coordinate indicated by i Partial derivative by ∂U / ∂x i By integrating, the contribution of dimension i, IG i Here, we can calculate the vector x B The vector x 0 The path with the coordinates indicated by is expressed as a vector x B The vector x 0 The vector x B In the coordinates shown by, α=0, and the vector x 0 If α=1 in the coordinates indicated by x, then the formula (B) in step 809 in FIG. 8 can be expressed as the formula (C). B The vector x 0 The contribution of dimension i, IG, is calculated by dividing the straight line path ending at the coordinates indicated by, into s equal parts, finding the approximation of the integral of the s intervals, and then adding up these approximations. i The contribution of dimension i obtained by the above approximation calculation, IG i The formula expressing the approximation of is represented as (A) in step 809 of FIG.

[0030] FIG. 8 shows that the individual contribution calculation unit 441 calculates the baseline data 117 (vector x B ), the input data 111 or the feature quantity 112 (vector x B ) and the input data 111 or features 112 (vector x 0 ) contribution of each dimension i to the model output drift, IG i 13 shows a flowchart of a process for calculating an approximation value of 8, the individual contribution calculation unit 441 selects one baseline data 117 from the set of baseline data 117 selected by the baseline data selection unit 433. Here, the baseline data 117 is represented by an m-dimensional vector x B It is expressed as follows. Let i be the index of the dimension, and x B is expressed as follows: x B = ( x1 B ,···,x i B ,···,x m B ) T In addition, the input data 111 or features 112 for which an increase in model output drift has been detected are represented as an m-dimensional vector x 0 It is expressed as x 0 is expressed as follows: x 0 = ( x1 0 ,···,x i 0 ,···,x m 0 ) T In step 802 in FIG. 8, the individual contribution calculation unit 441 prepares a variable tmp for storing the value of the interval integral that constitutes the path integral, and initializes the value of the variable tmp to zero. In step 803 in FIG. 8, individual contribution calculation section 441 prepares an integer variable k as an index of the section that constitutes the route, and initializes the value of integer variable k to zero. In step 804 of FIG. 8, the individual contribution calculation unit 441 calculates the interval x B +(k / s)×(x 0 -x B ) from the starting point x B +((k+1) / s)×(x 0 -x B ) at any point x in the interval of the end points indicated by i This point x can be any point included in the interval. For example, x B +(k / s)×(x 0 -xB ) begins at the partial differential ∂U / ∂x i The point x to be searched for may be x B +((k+1 / 2) / s)×(x 0 -x B ) is the midpoint of the interval indicated by the partial differential ∂U / ∂x i may be set as the point x for which the partial differential ∂U / ∂x is calculated. The individual contribution calculation unit 441 may use, for example, a method such as Stochastic Gradient Descent (SGD) or Adaptive moment (Adam) as a method for calculating the partial differential. Alternatively, the individual contribution calculation unit 441 realizes the model 101 using the model program 1021 and the model parameters 1031, and calculates, for each point indicated by each x, an n-dimensional vector y which is information 113 on the class corresponding to x+Δx indicated by a group of points in the vicinity of the point, and then calculates the model output drift degree U, and calculates the partial differential ∂U / ∂x at the point indicated by the x. i It may be an approximation of In step 805 of FIG. 8, the individual contribution calculation unit 441 calculates the partial differential ∂U / ∂x i Add the value of to the variable tmp. In step 806 in FIG. 8, individual contribution calculation section 441 adds 1 to the value of integer variable k. 8, the individual contribution calculation unit 441 judges whether the value of the integer variable k is smaller than the number of divisions s of the path for the path integration. If the judgment in step 807 is positive, the control returns to step 804, and the individual contribution calculation unit 441 calculates the partial differential ∂U / ∂x i If the determination in step 807 is negative, the partial differential ∂U / ∂x is calculated for all sections included in the path for the path integral. i After calculating the value of ∂U / ∂x, i Since the sum of the values ​​of is stored in the variable tmp, control proceeds to step 808. In step 808 of FIG. 8, the individual contribution calculation unit 441 calculates the variables tmp and (x i 0 -x i B) / s. In step 809 of FIG. 8, the individual contribution calculation unit 441 converts the multiplication result of step 808 into the vector x B and vector x 0 The contribution of dimension i to the model output drift in the relationship i In step 804, x B +(k / s)×(x 0 -x B ) begins at the partial differential ∂U / ∂x i When the point x for which the equation is to be calculated is set as the point x, the multiplication result in step 808 is expressed by the equation (A) in step 809. The value calculated by the equation (A) is the contribution of the dimension i to the model output drift, IG i This is an approximation of the value expressed by the formula (B) and the formula (C).

[0031] In step 515 (set contribution calculation step) of FIG. 5, the set contribution calculation unit 442 calculates the baseline data 117 (vector x B ) for each of the dimensions i, the contribution (individual contribution) IG i Based on the above, the contribution (set contribution) of each dimension i to the model output drift corresponding to the set of baseline data 117 is calculated. B There are various statistical processes that can be used to calculate the collective contribution from the individual contribution of each of the items. The statistical processes may be any appropriate method, such as a process for calculating the average value, a process for calculating the median value, a process for calculating the mode value, or a process for performing various statistical processes after removing outliers. As described above, when a plurality of samples are set as baseline data 117 and set contribution calculation unit 442 calculates the set contribution using the individual contribution calculated for each of baseline data 117, it is possible to suppress the influence of outlier samples and fluctuations between samples on the contribution. Note that the steps performed by contribution calculation unit 104 may be collectively referred to as a contribution calculation step.

[0032] 5 (display output control step), the display output control unit 405 controls information on the contribution 118 of each dimension to the model output drift to be displayed or output, for example, on the display or output device 306. The information on the contribution 118 of each dimension to the model output drift to be displayed or output may be the calculated contribution 118 itself, or may be information derived from the contribution 118 or information for explaining the contribution 118. FIG. 9 shows an example of an explanatory screen 900 that is displayed or output by, for example, the display or output device 306 under the control of the display output control unit 405 in the first embodiment. The explanation screen 900 may have a contribution table 901 and a text explanation field 902. The contribution table 901 may include multiple rows indicating the contribution 118 to the model output drift for each dimension i, a row indicating the model output drift degree 424 corresponding to the time or time period, and a row indicating the time or time period. In the contribution table 901, the row indicating the time or time period indicates the input data 111 or the feature amount 112 (vector x 0 ), and a column for a first time period 911 which is a time period including a time before the drift occurrence time 910 and which is selected by the reference time selection unit 431. A row showing the model output drift degree 424 in the contribution table 901 may have a column for the value of the model output drift degree 424(U) corresponding to the drift occurrence time 910, and a column for the model output drift degree 424(U) corresponding to the first time period 911. In order to obtain the model output drift degree 424 corresponding to the first time zone 911, for example, the display output control unit 405 or the contribution degree calculation unit 104 may calculate the model output drift degree 424 of each of the baseline data 117 included in the set of baseline data 117 corresponding to the time included in the first time zone 911, by using the class information 113 (past class information 435) recorded in the operation database 407 in step 507. Alternatively, after the uncertainty calculation unit 422 calculates the model output drift degree 424 in step 508, the calculated model output drift degree 424 may be stored in the operation database 407, and the display output control unit 405 or the contribution degree calculation unit 104 may read out the value of the model output drift degree 424 from the operation database 407. Then, the display output control unit 405 or the contribution calculation unit 104 may perform appropriate statistical processing or the like on each model output drift degree 424(U) of the baseline data 117 to obtain the model output drift degree 424(U) corresponding to the first time zone 911. In the contribution table 901, a number of rows showing the contribution 118 to the model output drift for each dimension i include baseline data 117 (x B ) and the input data 111 or feature quantity 112 (x 0 ) contribution of each dimension i to the model output drift, IG i In the example of FIG. 9, the model output drift degree 424 (U(x 0 )) is 0.80, and the model output drift degree 424 (U(x B The set of i) is 0.30, and the difference ΔU is 0.50. And the contribution of dimension i, I i (where the values ​​of the aggregate contribution are IG1 = -0.05, IG2 = +0.35, IG3 = +0.05, and IG4 = +0.15. Here, IG iThe sum of IG1, IG2, IG3 and IG4 in Figure 7 is almost equal to ΔU. (IG i There may be some discrepancies between the sum over and ΔU.) The contribution degree list 901 as described above shows the contribution degree I G from any dimension i to the increase in the model output drift degree 424 detected at the drift occurrence time 910. i It is possible to present numerical information on whether the The text description field 902 can present information about the model output drift in the form of a sentence using natural language. In the example of FIG. 9, the display output control unit 405 may present, in the text description field 902, time information corresponding to the input data 111 or feature 112 for which an increase in model output drift was detected, as a sentence such as "Increased drift was detected at '2023 / 05 / 31 3:00'." The display output control unit 405 may present, in the text description field 902, the degree of contribution of each dimension i to the model output drift, I i A description of the event may be presented as "A large drift occurred in dimension 2 at 3:00 on 31 / 05 / 2023." or "A small drift occurred in dimension 4 at 3:00 on 31 / 05 / 2023." Note that the term "large drift" corresponds to a value of +0.35 for IG2, and the term "small drift" corresponds to a value of +0.15 for IG4. The text explanation column 902 as described above helps those with little knowledge of the numerical information in the contribution table 901 to understand the nature of the model output drift.

[0033] For example, the first embodiment of the present disclosure described above can be utilized in an overall configuration in which time-series data obtained from a certain system is input to a model, information on classes (classifications) is output from the model, and a control system that controls the certain system performs control using the information on classes (classifications). In the above assumed overall configuration, if the uncertainty of the output results from the model (model output drift) increases and the inference accuracy of the model deteriorates, control of the certain system may be hindered. Therefore, in the above assumed overall configuration, when an increase in model output drift occurs during operation of the system and model, it is desirable to identify the cause of the increase in model output drift as a clue for taking measures to improve the inference accuracy of the model and eliminate the model output drift (for example, re-executing machine learning using new learning data or re-verifying the structure of the model). Therefore, by utilizing the first embodiment of the present disclosure, it is possible to calculate the contribution degree I G of each dimension i of the input data 111 or the feature amount 112 to the model output drift in accordance with a desired purpose. i This is expected to help identify the cause of the increase in model output drift.

[0034] 5. Machine Learning for the Models in this Disclosure In the following, machine learning for constructing the model 101 will be described. In order to realize the model 101, for example, some information processing system executes the model program 1021 while using the model parameters 1031. Machine learning means, for example, optimizing (learning) the model parameters 1031 by applying learning data to the model 101. In the following, supervised learning will be described as machine learning for constructing the model 101 for outputting information related to classes (classification). In addition, as the supervised learning, for example, a method such as Random Forest or Light Gradient Boosting Machine (Light GBM) may be used. In addition, the machine learning may be unsupervised learning (for example, unsupervised learning related to clustering).

[0035] 5.1. Overview of Machine Learning Functionality 10 shows the functional configuration of an information processing system 1000 that performs machine learning to build a model 101. The information processing system 1000 that performs machine learning may be the same as or different from the information processing system 100 in the first embodiment. Here, an overview of the functional configuration of the information processing system 1000 that performs machine learning will be described, and the processes performed by each of the functional units will be described later in the section "5.2. Machine learning processing." 10, the model database 406 records a model program 1021 and model parameters 1031. When machine learning is executed, the model program 1021 and the model parameters 1031 are read from the model database 406. Then, the model output calculation unit 421 executes the model program 1021 while using the model parameters 1031, thereby realizing the model 101. 10, an information processing system 1000 that performs machine learning has a training-time database 1002 or is able to access the training-time database 1002. The training-time database 1002 stores training data consisting of data to be used as input to the model 101 and correct answer information output from the model 101 when supervised machine learning is performed. Functional units realized by the information processing system 1000 performing machine learning may include a label assignment unit 1001, a learning control unit 1003, a feature amount calculation unit 425, a model output calculation unit 421, an error calculation unit 1004 (loss function calculation unit 1004), and a parameter adjustment unit 1005. Of these, the label assignment unit 1001 may be realized in the information processing system 1000 when performing a process of accumulating learning data in a learning-time database 1002. The learning control unit 1003, the feature amount calculation unit 425, the model output calculation unit 421, the error calculation unit 1004 (loss function calculation unit 1004), and the parameter adjustment unit 1005 may be realized in the information processing system 1000 when performing a learning process to perform an optimization (learning) process of the model parameters 1031. As in the first embodiment, these functional units may be constructed by executing a program, or may be realized by hardware resources.

[0036] 2. Machine Learning Processing Fig. 11 shows a flowchart of machine learning for constructing a model 101. The process shown in the flowchart in Fig. 11 will be described below with reference to the functional configuration of an information processing system 1000 performing machine learning shown in Fig. 10. Note that, in the flowchart in Fig. 11, steps 1101 to 1103 relate to accumulating learning data in a learning-time database 1002, while steps 1104 to 1111 relate to performing a learning process to optimize (learn) model parameters 1031, and there may be a time gap between these two.

[0037] 11 , in order to accumulate learning data in the learning-time database 1002, first, in step 1101, the information processing system 1000 performing machine learning acquires data 1011 equivalent to the input data 111 to be accumulated in the learning-time database 1002. The information processing system 1000 performing machine learning may acquire the data 1011 from, for example, a system 1010. When the data 1011 is acquired from outside the information processing system 1000 performing machine learning, a device equivalent to the communication device 308 included in the information processing system 1000 performing machine learning may be used. 11, the label assignment unit 1001 associates a label 1012, which is correct answer information of a class (classification) corresponding to the data 1011, with the data 1011. Here, the label assignment unit 1001 may set the correct answer information of a class (classification) manually specified by an operator or the like as the label 1012. Alternatively, the label assignment unit 1001 may determine the correct answer information of a class (classification) using a sensor or the like that can be used when creating learning data but is not usually used when operating the model 101, and may set the determined correct answer information of the class (classification) as the label 1012. In step 1103 of FIG. 11, the information processing system 1000 performing machine learning records the data 1011 and the label 1012 in the learning-time database 1002 while associating them with each other. The above steps 1101 to 1103 may be repeatedly executed every time a process for storing training data in the training-time database 1002 is performed. That is, the training-time database 1002 usually contains a plurality of pairs of data 1011 and labels 1012.

[0038] 11, when executing the learning process to perform the optimization (learning) process of the model parameters 1031, first, in step 1104 of Fig. 11, the information processing system 1000 performing machine learning acquires input data 111 to the model (derived from the above-mentioned data 1011) and labels 1013 (derived from the above-mentioned labels 1012) that are associated with each other, from the learning-time database 1002. There may be a time gap between the timing of acquiring the input data 111 and the timing of acquiring the labels 1013. 11, the feature amount calculation unit 425 may calculate the feature amount 112 from the input data 111 by performing the conversion process 105. For example, when it is not possible to input the input data 111 to the model 101 as it is, or when it is assumed that the accuracy of class (classification) inference by the model 101 is not high if the input data 111 is input to the model 101 as it is, the feature amount 112 may be calculated in this manner. Note that the function of this conversion process 105 (feature amount calculation unit 425) may be treated as a part of the function of the model 101. 11, the model output calculation unit 421 realizes the model 101 using the model program 1021 and the model parameters 1031, thereby obtaining class-related information 113 according to the input data 111 or the feature amount 112. As described with reference to FIG. 1, the input data 111 or the feature amount 112 may be expressed in the form of a vector x as shown in the following equation. x = ( x1, ,x i ,···,x m ) T As described with reference to FIG. 1, the class information 113 output from the model 101 may be expressed in the form of a vector y as shown in the following formula. Note that each component y of the vector y j (probability of membership in class j) should be 1, so that y j A normalization of may be performed. y = ( y1, ,y j ,···,y n ) T In step 1107 of FIG. 11, the error calculation unit 1004 (loss function calculation unit 1004) calculates what is called an error or loss function using the class-related information 113 (vector y) obtained by the model output calculation unit 421 and the label 1013 obtained from the learning-time database 1002. In calculating the error or loss function in the error calculation unit 1004, the label 1013 is calculated by dividing the z jThe value of is 1, and the z corresponding to the other index j is j It may be treated as an n-dimensional vector z given by the following equation, where the value of is 0. z = ( z1, ,z j ,...,z n ) T (Only the correct answer is j j =1, otherwise 0) The error or loss function may be, for example, a function indicating the cross-entropy. In step 1108 of FIG. 11, the learning control unit 1003 judges whether or not the end condition of the learning process is satisfied. For example, the learning control unit 1003 may judge that the end condition of the learning process is satisfied when the number of iterations Q, which is the number of times a series of processes from step 1104 to step 1107 is executed, exceeds the upper limit value ThH. Alternatively, the learning control unit 1003 may judge that the end condition of the learning process is satisfied, for example, even if the number of iterations Q does not exceed the upper limit value ThH, if the number of iterations Q exceeds the lower limit value ThL and the error or loss function (Loss function) calculated in the most recent step 1107 satisfies the convergence condition. If it is judged that the end condition of the learning process is satisfied in step 1108, the control proceeds to step 1111. If it is not judged that the end condition of the learning process is satisfied in step 1108, the control proceeds to step 1109. In step 1109 in Fig. 11, the parameter adjustment unit 1005 updates the values ​​of the model parameters 1031 so that the value or absolute value of the error or loss function calculated in step 1107 becomes smaller. In Fig. 10, the updated model parameters are represented as 1032. As a method for updating the model parameters 1031 (1032), for example, an optimization algorithm such as Stochastic Gradient Descent (SGD), Momentum SGD, Adaptive Gradient Algorithm (Adagrad), RMSprop, AdaDelta, Adaptive moment (Adam), Adamax, etc. may be used. Each time the value of the model parameter 1031 (1032) is updated in step 1109, the parameter adjustment unit 1005 may store the updated model parameter 1032 in the model database 406, or the process of storing the updated model parameter 1032 in the model database 406 may be performed in step 1111 described later only after it is determined that the end condition of the learning process is satisfied. 11, the learning control unit 1003 adds 1 to the number of iterations Q, which is the number of times the series of processes from step 1104 to step 1107 are executed (increments the number of iterations Q). Note that the number of iterations Q may be initialized to 1 when the learning process starts. After step 1110, control returns to step 1104, where new learning data (input data 111 and labels 1013) are read from the learning-time database 1002, and a series of processes from step 1104 onwards are carried out. Note that when step 1106 is next executed, the model parameters 1032 updated in the most recent step 1109 are made new model parameters 1031 and applied to the model 101. After it is determined in step 1108 described above that the condition for terminating the learning process is met, in step 1111 of FIG. 11 , if there are updated model parameters 1032 that have not yet been reflected in the model database 406, the learning control unit 1003 or the parameter adjustment unit 1005 stores the model parameters 1032 in the model database 06.

[0039] 6. Second embodiment of the present disclosure The second embodiment of the present disclosure will be described below. It is not essential to realize all of the functional configurations described below and perform all of the processing, and an information processing system, method, or program for realizing part of the functional configurations described below and performing part of the processing may be used. In addition, the description or illustration in the drawings of parts that may have the same functional configuration as the first embodiment or perform the same processing will be omitted as appropriate. The model 101 in the second embodiment of the present disclosure may also be constructed by machine learning described using FIGS. 10 and 11. In short, the second embodiment of the present disclosure detects an increase in model output drift by detecting input data 111 or features 112 (vector x 0 ) Select multiple time periods before the time corresponding to the time period i, and calculate the contribution of each combination of time period and dimension i to the model output drift, IG i (obtaining time series information on contribution degree).

[0040] 6-1. Outline of Functional Configuration of Second Embodiment of the Present Disclosure Fig. 12 shows a functional configuration of the second embodiment of the present disclosure. To realize the functional configuration shown in Fig. 12, for example, an information processing system having the computer architecture shown in Fig. 3 described above may be used. In the following, the information processing system in the second embodiment is indicated by the reference number 1200 in order to distinguish it from the information processing system 100 in the first embodiment. Here, the functional configuration shown in Fig. 12 will be explained briefly. A detailed explanation will be given together with the contents of the processing in the section "6-2. Processing of the second embodiment of the present disclosure". The second embodiment of the present disclosure is different from the first embodiment in the manner in which the reference time selection unit 431 selects a time period. In the first embodiment, the reference time selection unit 431 selects the input data 111 or the feature amount 112 (vector x 0In contrast, in the second embodiment described below, the reference time selection unit 431 selects a time period immediately before the time corresponding to the vector x 0 In selecting a time period before the time corresponding to the dimension i, multiple time periods are selected. Accordingly, the contribution degree I i is also obtained for each time period to which the set of baseline data 117 belongs. In this way, in the second embodiment of the present disclosure, for multiple time periods and multiple dimensions i, the contribution degree I i (Contribution time series information is obtained). Therefore, in the second embodiment of the present disclosure, the vector x 0 When we detect an increase in model output drift for vector x 0 We can examine the drift behavior of each dimension i by going back to multiple time periods in the past for the time for i. For example, a small model input drift may have occurred for a certain dimension for a certain period of time, or a small model input drift may have occurred multiple times for a certain dimension, but an increase in model output drift may not be detected. Even in a complex situation where such a small model input drift continues, and then the model input drift for the certain dimension has recently become cumulatively large and large, or a large model input drift has occurred for another dimension, causing an increase in model output drift to be detected, the second embodiment of the present disclosure can estimate or grasp the history and timing (e.g., start time or start time zone) of drift occurrence for each dimension i. In Fig. 12, the functional configuration and processing similar to those of the first embodiment shown in Fig. 4 may be described in a simplified manner. For example, the functional configuration and processing related to the model output drift detection unit 102 are similar between the first embodiment and the second embodiment, so that the description related to the model output drift detection unit 102 is simplified in Fig. 12.

[0041] 6.2 Processing of the Second Embodiment of the Present Disclosure Fig. 13 shows a flowchart of the process of the second embodiment of the present disclosure. Hereinafter, the process of the flowchart shown in Fig. 13 will be described mainly with reference to the functional configuration shown in Fig. 12. The second embodiment of the present disclosure may also perform the same processes as those in the first embodiment, from step 501 to step 510. A description of the processes in these steps will be omitted. In the second embodiment, after step 510, the control is transferred to step 1311.

[0042] In step 1311 of FIG. 13, the reference time selection unit 431 selects multiple time periods that satisfy a predetermined condition. Here, the multiple time periods that satisfy the predetermined condition are input data 111 or feature amount 112 (vector x 0 In the second embodiment, the reference time selection unit 431 selects a time period including a time before the time corresponding to the vector x 0 A plurality of time periods immediately before the time corresponding to the time may be selected as a plurality of time periods that satisfy the predetermined condition. Examples of the selected plurality of time periods are shown as 1411, 1412, 1413, and 1414 in FIG. 14 described later. Then, in the same step 1311, the reference time selection unit 431 selects input data 111 or feature amount 112 for the model at a time included in the selected multiple time periods. Here, the number of samples of the input data 111 or feature amount 112 selected for each time period is w. The reference time selection unit 431 may acquire the input data 111 or feature amount 112 included at a time included in the selected multiple time periods from the operation database 407 that accumulates the input data 111 or feature amount 112. In FIG. 12, the input data 111 or feature amount 112 accumulated in the operation database 407 is described as "past input or feature amount 116", and the input data 111 or feature amount 112 acquired by the reference time selection unit 431 is described as "input or feature amount 434 of a specific time period (434-1, 434-2, ...)" for each time period. Note that, although two time periods are illustrated in FIG. 12, the number of time periods to be selected may be three or more. FIG. 14 described below shows an example in which the number of time slots to be selected is four.

[0043] The process performed by the past model output acquisition unit 432 in step 1312 in Fig. 5 is similar to the process performed in step 512 in the first embodiment. However, in this second embodiment, for a plurality of selected time periods, since there is a "predetermined time period input or feature amount 434 (434-1, 434-2, ...)" for each time period, the past model output acquisition unit 432 performs processing for each time period to obtain "past class-related information 435 (435-1, 435-2, ...)" for each time period.

[0044] The process performed by the baseline data selection unit 433 in step 1313 of FIG. 5 is similar to the process performed in step 513 in the first embodiment. However, in this second embodiment, for the multiple selected time periods, there is a "predetermined time period input or feature amount 434 (434-1, 434-2, . . .)" for each time period, and there is "past class information 435 (435-1, 435-2, . . .)" for each time period. Therefore, the baseline data selection unit 433 performs a process for each time period to obtain "baseline data 117 (117-1, 117-2, . . .)" for each time period. The baseline data selection unit 433 performs a comparison process of the values ​​of the belonging class C by using the input data 111 or feature amount 112 (vector x 0 ) and the values ​​of the class C for the input data 111 or the feature amount 112 at times included in the selected multiple time periods.

[0045] The process performed by the individual contribution calculation unit 441 in step 1314 in Fig. 5 and the process performed by the set contribution calculation unit 442 in step 1315 in Fig. 5 are similar to the processes performed in steps 514 and 515 in the first embodiment. However, in this second embodiment, for the multiple selected time periods, since "baseline data 117 (117-1, 117-2, ...)" exists for each of the time periods, the individual contribution calculation unit 441 and the set contribution calculation unit 442 perform processing for each of the time periods to obtain "respective contributions 118 (118-1, 118-2, ...) of the dimensions" for each of the time periods. In other words, what is calculated by the set contribution calculation unit 442 is contribution time-series information.

[0046] In step 1315 of FIG. 13, the display output control unit 405 calculates the contribution degree I iThe information to be displayed or output is, for example, the calculated contribution degree 118 (IG i ) itself, or may be information derived from the contribution degree 118 or information for explaining the contribution degree 118. 14 shows an example of an explanation screen 1400 in the second embodiment, which is displayed or output on, for example, the display or output device 306. The explanation screen 1400 may have a drift occurrence time-based contribution degree list table 1403 and a past time zone-based contribution degree list table 1404.

[0047] The drift occurrence time base contribution list 1403 is a set of contributions I G for each combination of time period and dimension i calculated by the set contribution calculation unit 442. i The contribution degree list 1403 based on the drift occurrence time in the second embodiment is similar to the contribution degree list 901 in the first embodiment (FIG. 9) except that there are a plurality of time periods (four in FIG. 14). That is, the contribution degree list 901 in the first embodiment (FIG. 9) has a column for the drift occurrence time 910 and a column for the first time period 911, whereas the contribution degree list 1403 based on the drift occurrence time in the second embodiment (FIG. 14) has columns for the second time period 1412, the third time period 1413, and the fourth time period 1414 in addition to the column for the drift occurrence time 1410 and the column for the first time period 1411 (in the example of FIG. 14, the number of selected time periods is four). The time period immediately preceding the drift occurrence time 1410 may be designated as a first time period 1411, the time period immediately preceding the first time period 1411 may be designated as a second time period 1412, the time period immediately preceding the second time period 1412 may be designated as a third time period 1413, and the time period immediately preceding the third time period 1413 may be designated as a fourth time period 1414. Contribution of each dimension i to the model output drift for each column of time periods (IG i) is calculated in relation to the time period and the drift occurrence time 1410. For example, in a case where the value of the model output drift degree U for the drift occurrence time 1410 is 0.80, the value of the model output drift degree U for the second time period 1412 is 0.30, and the value of the difference ΔU between the two model output drift degrees is 0.50, the value of the contribution degree IG1 of the dimension 1 related to the relationship between the drift occurrence time 1410 and the second time period 1412 will be 0.00, the value of the contribution degree IG2 of the dimension 2 will be +0.35, the value of the contribution degree IG3 of the dimension 3 will be 0.00, and the value of the contribution degree IG4 of the dimension 4 will be +0.15, and the sum of these contribution degrees may be 0.50. In other words, the sum of the contribution degrees of each dimension will be close to the value of ΔU.

[0048] The display output control unit 405 calculates the contribution degree 118 (IG i ), and information on the model output drift degree 424(U) for each time period or drift occurrence time 1410 may be processed to generate information to be included in the past time period reference contribution degree list 1404, and then the past time period reference contribution degree list 1404 may be controlled to be displayed or output on the explanation screen 1400. Fig. 14 shows an example in which the past time period reference contribution degree list 1404 is generated using information on the contribution degree IGi in the drift occurrence time reference contribution degree list 1403 with the fourth time period 1414 as the reference. The past time zone reference contribution list 1404 shows the contribution of each dimension i related to the relationship between a predetermined reference time zone (fourth time zone 1414 in FIG. 14) and each time zone or drift occurrence time 1410. For example, in a case where the value of the model output drift degree U for the first time zone 1411 is 0.55, the value of the model output drift degree U for the fourth time zone 1414 (which is the reference time zone) is 0.00, and the value of the difference ΔU between the model output drift degrees between the two is 0.55, the contribution of dimension 1 related to the relationship between the first time zone 1411 and the fourth time zone 1414 (which is the reference time zone) will be 0.00, the contribution of dimension 2 will be +0.35, the contribution of dimension 3 will be +0.05, and the contribution of dimension 4 will be +0.15, and the sum of these contributions may be 0.55. In other words, the sum of the contributions of each dimension is close to the value of ΔU. The contribution value of each dimension in the above example is displayed or output in the column for the first time period 1411 in the contribution list table 1404 based on past time periods. Let t be an index indicating each of the time periods or the drift occurrence time 1410, ∞ be an index indicating a predetermined reference time period (the fourth time period 1414 in FIG. 14), and i be an index of a dimension. The contribution degree displayed or output in the contribution degree list 1403 based on the drift occurrence time period (that is, the contribution degree calculated by the set contribution degree calculation unit 442) is expressed as IG. i,t (However, for the index t indicating the drift occurrence time 1410, IG i,t The value of is set to 0.) The contribution degree displayed in the past time zone based contribution degree list 1404 is calculated by IGM i,t Then, the display output control unit 405 calculates IGM i,t Request. IGM i,t =IG i,∞ -IG i,t For example, the IG for the combination of the fourth time zone 1414 and dimension 2 2,∞ The value of is +0.50, and the IG for the combination of the first time zone 1411 and dimension 2 2,tis +0.15, and the difference is +0.35. Therefore, the display output control unit 405 calculates the IGM 2、t The value is set to +0.35.

[0049] The drift occurrence time reference contribution table 1403 and the past time period reference contribution table 1404 as described above can present the time series transition of the contribution of each dimension i to the model output drift over multiple time periods. The drift occurrence time reference contribution list 1403 presents the contribution to the model output drift of each dimension i related to the relationship between each time period and the drift occurrence time 1410, and therefore has the same effect as the first embodiment, and can also present information on the model input drift of each dimension i that occurred from a time period that is not immediately before the drift occurrence time 1410 to the drift occurrence time 1410. If a time period with a small value or absolute value of the model output drift degree 424(U) is selected as a time period (fourth time period 1414 in FIG. 14) that is a predetermined reference, the contribution degree list 1404 of the past time period reference 1404 can provide information similar to the absolute contribution degree of each dimension i to the model output drift degree 424(U) at each time period or drift occurrence time 1410. Therefore, even in a complex situation where, for example, small model input drift related to a certain dimension i continues, and then the model input drift related to the certain dimension recently accumulates and becomes large, or a large model input drift occurs related to another dimension, and an increase in model output drift is detected, the history of drift occurrence and occurrence timing (for example, start time and start time period) for each dimension i can be estimated or understood to some extent by the information indicated by the contribution degree list 1404 of the past time period reference 1404. In the example of Figure 14, from the information in the contribution list 1404 of the past time zone reference 1404, it can be estimated or understood that although a small scale model input drift has been continuing for dimension 3 since the third time zone 1413, the model input drift of dimension 3 alone is not enough to detect an increase in model output drift, and that model input drift has occurred for dimensions 2 and 4 since the second time zone, with the model input drift of dimension 2 gradually increasing to become large, and the model input drift of dimension 4 gradually increasing to become medium.

[0050] 7. Third embodiment of the present disclosure The third embodiment of the present disclosure will be described below. It is not essential to realize all of the functional configurations described below and perform all of the processing, and an information processing system, method, or program for realizing part of the functional configurations described below and performing part of the processing may be used. In addition, the description or illustration in the drawings of parts that may have the same functional configuration as the first embodiment or perform the same processing will be omitted as appropriate. The model 101 in the third embodiment of the present disclosure may also be constructed by machine learning described using FIGS. 10 and 11. In short, the third embodiment of the present disclosure accumulates historical information regarding past increases in model output drift, and when an increase in model output drift is detected, a search process is performed to identify past increases in model output drift that are similar (have a high degree of similarity) to the behavior of the increase in model output drift, and information regarding the identified past increases in model output drift is provided.

[0051] 7-1. Outline of functional configuration of the third embodiment of the present disclosure FIG. 15 shows a functional configuration of the third embodiment of the present disclosure. To realize the functional configuration shown in FIG. 15, for example, an information processing system having the computer architecture shown in FIG. 3 may be used. In the following, the information processing system in the third embodiment is indicated by the reference number 1500 in order to distinguish it from the information processing system 100 in the first embodiment. Here, the functional configuration shown in FIG. 12 will be explained briefly. A detailed explanation will be given together with the contents of the processing in the section "7-2. Processing of the third embodiment of the present disclosure". In a third embodiment of the present disclosure, upon detection of an increase in model output drift, the contribution 118 (IG i The functional configuration and processing up to the calculation of (a) may be the same as those in the first or second embodiment. Therefore, in FIG. 15, the model output drift detection unit 102, the baseline data setting unit 103, and the contribution degree calculation unit 104 are simply shown. The information processing system 1500 may have a drift history database 1501 and a similar drift search unit 1502 in addition to the functional units and the like that the first or second embodiment has. Each time an increase in model output drift is detected, the information processing system 1500 records a record 1510 in the drift history database 1501 to accumulate information about the increase in model output drift. The record 1510 may include, for example, a time (a time associated with the input data 111 or the feature 112; reference number in the record item is 1514), input data 111 to the model (reference number in the record item is 1511), feature 112 (reference number in the record item is 1512), information about the class 113 (reference number in the record item is 1513), and each contribution degree 118 of dimension i (IG i , reference number 1518 in the record section), and factors of increase in model output drift (reference number 1519 in the record section). Record 1510 may also include information on model output drift degree 424(U). For factor 1519, which is one of the items in record 1510, factor input unit 1503 may record information in record 1510. For example, when an operator or the like manually identifies information on a cause of an increase in model output drift and inputs the information, or when information on a cause of an increase in model output drift is identified based on some automated processing, factor input unit 1503 may record the identified information in factor 1519 of record 1510 corresponding to the increase in model output drift. When an increase in model output drift is detected, the similar drift search unit 1502 performs a search process to identify past increases in model output drift that are similar (have a high degree of similarity) to the behavior of the increase in model output drift, and presents the records related to the identified increases in model output drift as similar records 1520. The display output control unit 405 controls the display or output device 306 to display or output information contained in the presented similar record 1520, for example. The drift history database 1501 and the as-operation database 407 may be integrated into one integrated database. Although the as-operation database 407 in the first or second embodiment does not necessarily have the items of the contribution degree 1518 and the factor 1519 among the items that the record 1510 in the third embodiment may have, the records 1510 in the drift history database 1501 and the records in the as-operation database 407 have similar formats, such integration is possible.

[0052] 7.2 Processing of the Third Embodiment of the Present Disclosure Fig. 16 shows a flowchart of the process of the third embodiment of the present disclosure. Hereinafter, the process of the flowchart shown in Fig. 16 will be described mainly with reference to the functional configuration shown in Fig. 15. The third embodiment of the present disclosure may also perform the same processes as those in the first embodiment, from step 501 to step 516. A description of the processes in these steps will be omitted. In the third embodiment, after step 516, the control is transferred to step 1617. 16, the similar drift search unit 1502 calculates the distance between the information held by each record 1510 stored in the drift history database 1501 and information related to the most recent increase in model output drift. The similar drift search unit 1502 calculates, for example, the distance between the input data 111 related to the most recent increase in model output drift and the input data 1511 recorded in the record 1510, the distance between the feature amount 112 related to the most recent increase in model output drift and the feature amount 1512 recorded in the record 1510, the distance between the information related to the class 113 related to the most recent increase in model output drift and the information related to the class 1513 recorded in the record 1510, the contribution degree 118 (IG i) and each contribution degree 1518 of the dimension i recorded in the record 1510. The similarity drift search unit 1502 may calculate the distance by using, for example, a method such as a mean squared error that is used to calculate the distance between data having a plurality of variables (dimensions, elements). 16, the similar drift search unit 1502 selects one or more records 1510 based on the distance information calculated in step 1617, and sets the selected records 1510 as similar records 1520. The similar drift search unit 1502 may select a predetermined number of records 1510 as similar records 1520 in ascending order of distance (highest degree of similarity) from information relating to the most recent increase in model output drift. Alternatively, the similar drift search unit 1502 may set a record 1510 as a similar record 1520 if the distance from the information relating to the most recent increase in model output drift is smaller than a predetermined threshold or equal to or less than the threshold. The steps performed by the similar drift search unit 1502 may be collectively called a similar drift search step.

[0053] In step 1619 of FIG. 16, the display output control unit 405 controls the display or output device 306 to display or output information about the record 1510 selected as the similar record 1520, for example. If the time 1514 in the record 1510 is included as information to be displayed or output, the information processing system 1500 can provide the administrator, etc. with information on the time when an increase in model output drift similar to (highly similar to) the most recent increase in model output drift occurred in the past. This allows the administrator, etc. to know the time when something similar to the most recent increase in model output drift will occur and the tendency of the frequency of occurrence. If the time when something similar to the most recent increase in model output drift will occur and the tendency of the frequency of occurrence become clear, it is expected that the administrator, etc. will be able to easily make decisions when taking measures such as re-learning the model 101 (such as determining whether or not measures are necessary and the urgency of taking measures). If the factors 1519 in the record 1510 are included as information to be displayed or output, the information processing system 1500 can provide the operator, etc. with information about what factors were recognized as causing a past increase in model output drift when a similar (highly similar) increase in model output drift occurred in the past to the most recent increase in model output drift. This is expected to make it easier for the operator, etc. to take measures to eliminate the model output drift. If the information to be displayed or output includes one or more of the input data 1511, the feature amount 1512, the information about the class 1513, and the contribution degree 1518 in the record 1510, it can be provided to the manager, etc. as information about a past increase in model output drift similar (highly similar) to the most recent increase in model output drift. The manager, etc. can use the provided information to perform various analyses.

[0054] 8.Other (variations) The present disclosure is not limited to the above-described embodiment, but includes various modifications. A part of the configuration or processing of the embodiment may be replaced with the configuration or processing of another conceivable embodiment. The configuration or processing of the embodiment may be added to the configuration or processing of another conceivable embodiment. For example, the present disclosure may have the following modified embodiments.

[0055] (α) Modifications to the baseline data selection method In the above embodiment, the reference time selection unit 431 selects one or more time periods, and selects past input data or feature values ​​116 corresponding to the time included in the selected time period as input data or feature values ​​434 for the specified time period. Also, in the above embodiment, the baseline data selection unit 433 selects, from the input data or feature values ​​434 for the specified time period, data associated with a value of the attribution class C equal to a value of the attribution class C corresponding to the most recent increase in model output drift, and sets the data as the baseline data 117. That is, the above embodiment employs both a filtering method based on time periods and a filtering method based on the value of the attribution class C as a method for selecting the baseline data 117. The effects of the above filtering methods are as described with reference to Figs. 6 and 7. However, the method of selecting the baseline data is not limited to the method in the above embodiment. For example, only one of the filtering method based on time of day and the filtering method based on the value of belonging class C may be used. When the filtering method based on time of day is used and the filtering method based on the value of belonging class C is not used, the baseline data selection unit 433 will use all of the inputs or features 434 in a predetermined time period as they are, as baseline data 117. When the filtering method based on the value of belonging class C is used and the filtering method based on time of day is not used, the reference time selection unit 431 will provide the past input data or features 116 held by the operation time database 407 to the past model output acquisition unit 432 and the baseline data selection unit 433 as they are. Alternatively, a specific selection method in the filtering method based on time zones or the filtering method based on the value of the belonging class C may be changed. For example, the filtering method based on time zones may be changed so that the reference time selection unit 431 selects a predetermined number of most recent input data 111 or feature values ​​112 instead of selecting a predetermined time zone. Also, for example, the filtering method based on the belonging class C may be changed so that the baseline data selection unit 433 selects past input data 111 or feature values ​​112 corresponding to the belonging class C that matches or is similar to the belonging class C corresponding to the most recent increase in the model output drift instead of selecting past input data 111 or feature values ​​112 corresponding to the value of the belonging class C that matches the value of the belonging class C corresponding to the most recent increase in the model output drift. The above modified example allows calculation of the contribution degree with more flexible settings depending on what technical meaning is indicated and what contribution degree of each dimension is to be calculated.

[0056] (β) Acquisition of baseline uncertainty in the contribution calculation unit 104 In the above embodiment, the contribution calculation unit 104 includes an individual contribution calculation unit 441 and a set contribution calculation unit 442 . Here, the contribution calculation unit 104 may further include a baseline uncertainty acquisition unit. The baseline uncertainty acquisition unit is for acquiring a model output drift degree 424(U) (past model output drift degree) for each of the baseline data 117 selected by the baseline data selection unit 433. i Depending on the calculation method, the individual contribution calculation unit 441 may use information on the past model output drift degree acquired by the baseline uncertainty acquisition unit. In addition, in order to display or output the model output drift degree on the explanation screen 900 or 1400, the display output control unit 405 may use information on the past model output drift degree acquired by the baseline uncertainty acquisition unit. The baseline uncertainty acquisition unit may obtain the class-related information 113 corresponding to each of the baseline data 117 by implementing the model 101, and may further calculate the model output drift degree based on the class-related information 113. Alternatively, if the past class-related information 435 or the past model output drift degree information is accumulated in the operation database 407, the baseline uncertainty acquisition unit may read out the past class-related information 435 or the past model output drift degree information from the operation database 407, and acquire the past model output drift degree corresponding to each of the baseline data 117 (by calculating the model output drift degree based on the past class information 435, as necessary). The above-mentioned modified examples can correspond to modified examples of the calculation method in the individual contribution degree calculation unit 441. The above-mentioned modified examples can also provide information used by the display output control unit 405.

[0057] (γ) Display output variation In the above embodiment, the explanation screen 900 in FIG. 9 and the explanation screen 1400 in FIG. 14 show the contribution degree I i 9 presents the value of as numerical information itself. Also, the explanation screen 900 in Fig. 9 displays or outputs a sentence in a natural language that shows an explanation regarding the model output drift in a text explanation field 902. Alternatively or in addition to the above, the display output control unit 405 displays the contribution degree I i A color plot image colored according to the magnitude of the value of and the contribution of each dimension i, IG i The display output control unit 405 may control the display or output device 306 to display or output a monochrome image having a tone density according to the magnitude of the value of . The display output control unit 405 may also control the display or output device 306 to output a voice reading a sentence in a natural language that explains the model output drift. The above-described modified examples make it possible to present information in a form that is easy for users of the information processing system of the present disclosure, such as managers, to use.

[0058] (δ) Modification of contribution time series information formation In the second embodiment described above, a plurality of time periods are selected, and the contribution degree is calculated for each combination of time period and dimension i, thereby obtaining contribution degree time-series information. Instead, in the modified example, the reference time selection unit 431 selects a relatively wide time period (including the multiple time periods selected in the second embodiment), and then does not perform statistical processing (to obtain a representative value for each time period) by the set contribution calculation unit 442, but calculates the contribution I i may be treated as the respective contribution degrees 118 of the dimension i as they are to obtain the contribution degree time series information. The above modified example makes it possible to obtain contribution time-series information without setting a plurality of time periods.

[0059] (ε) Updating model parameters for multiple input data units in machine learning In the above embodiment, in the iterative process from step 1104 to step 1110 in FIG. 11, the error (loss function) is calculated for each pair of input data 111 and label 1013 used for model training, and then the model parameters are updated. However, in a modified example related to machine learning, in the iterative process from step 1104 to step 1110 in FIG. 11, the model parameters may be updated after calculating an overall error (loss function) for a group of pairs treated collectively, either for multiple pairs of input data 111 and labels 1013, or for all pairs of input data 111 and labels 1013 used in the learning process. The above-described modified example can reduce the frequency of updating the model parameters, and can perform learning processing in accordance with standard characteristics indicated by the set of input data 111 and labels 1013 used in machine learning.

[0060] The technical matters described in each of the embodiments of the present disclosure and the modified examples of the embodiments described above can be combined as appropriate as long as no technical contradiction occurs.

Claims

1. An information processing system, the information processing system realizes a model for outputting information related to a class in response to an input or a feature obtained by a conversion process of the input; The information processing system includes a model output drift detection unit, a baseline data setting unit, and a contribution degree calculation unit, The model output drift detection unit determines whether an increase in model output drift, which indicates uncertainty in an output of the model, has occurred; the baseline data setting unit selects baseline data from an input to the model or the feature amount at a time before a time corresponding to the input to the model or the feature amount that is a target for which an increase in the model output drift has been detected; The contribution calculation unit calculates a contribution of each of a plurality of dimensions of the input to the model or the feature quantities to the model output drift based on the baseline data.

2. 2. The information processing system according to claim 1, The model output drift detection unit includes a model output calculation unit, an uncertainty calculation unit, and a model output drift determination unit, the model output calculation unit realizes the model to obtain information about a class in response to an input to the model or the feature amount; The uncertainty calculation unit calculates a model output drift degree indicating uncertainty of an output from the model based on information about the class, The model output drift determination unit determines that an increase in the model output drift has occurred when the degree of model output drift satisfies a predetermined condition.

3. 2. The information processing system according to claim 1, The baseline data setting unit includes a reference time selection unit, a past model output acquisition unit, and a baseline data selection unit, the reference time selection unit selects an input to the model or the feature amount at a time included in a time period satisfying a predetermined condition, from among times prior to a time corresponding to the input to the model or the feature amount for which an increase in the model output drift has been detected; the past model output acquisition unit obtains information about a class in response to the input to the model or the feature amount selected by the reference time selection unit by implementing the model, or obtains information about a class from a history of information about the class obtained in the past for the input to the model or the feature amount selected by the reference time selection unit, an information processing system, wherein the baseline data selection unit selects the baseline data from among the inputs to the model or the features selected by the reference time selection unit, based on information regarding a class for the input to the model or the features for which an increase in the model output drift has been detected, and information regarding the class determined by the past model output acquisition unit.

4. 2. The information processing system according to claim 1, The contribution calculation unit includes an individual contribution calculation unit and a group contribution calculation unit, the individual contribution calculation unit calculates, for each of the baseline data, a contribution of each of the dimensions based on an input to the model or the feature amount for which an increase in the model output drift is detected and the baseline data; The set contribution calculation unit calculates the contribution of each of the dimensions corresponding to the set of baseline data based on the contribution of each of the dimensions corresponding to each of the baseline data.

5. 2. The information processing system according to claim 1, The information processing system includes a display output control unit, The display output control unit controls information regarding the contribution degree of each of the dimensions to be displayed or output.

6. 2. The information processing system according to claim 1, the baseline data setting unit selects baseline data for each of a plurality of time periods prior to a time corresponding to an input to the model or the feature value for which an increase in the model output drift has been detected, from the input to the model or the feature value at a time included in the time period; The contribution degree calculation unit calculates, for each of the time periods, a contribution degree of each of the dimensions based on the baseline data belonging to the time period; The contribution degree calculation unit obtains contribution degree time-series information having information regarding the contribution degree corresponding to a combination of the time period and the dimension.

7. 2. The information processing system according to claim 1, the information processing system includes a drift history database and a similar drift search unit, the drift history database has records corresponding to increases in the model output drift detected by the model output drift detection unit in the past; The record has some or all of the following information related to an increase in the model output drift: time, an input to the model, a feature obtained by a conversion process on the input, information on a class which is an output from the model, the contribution of each of the dimensions, and a cause of the increase in the model output drift; an information processing system, wherein, when the model output drift detection unit detects an increase in the model output drift, the similar drift search unit performs a search process to compare some or all of the input to the model, features obtained by a conversion process on the input, information on the class that is the output from the model, and information on the contribution of each of the dimensions, which are related to the detected increase in model output drift, with corresponding information in records held by the drift history database, and identifies records that are highly similar in terms of the compared information.

8. A method executed by an information processing system, comprising: the information processing system realizes a model for outputting information related to a class in response to an input or a feature obtained by a conversion process of the input; The method includes a model output drift detection step, a baseline data setting step, and a contribution degree calculation step, The model output drift detection step is a step of determining whether an increase in model output drift, which indicates uncertainty in the output of the model, has occurred; The baseline data setting step is a step of selecting baseline data from an input to the model or the feature amount at a time before a time corresponding to the input to the model or the feature amount that is a target for which an increase in the model output drift has been detected, The method, wherein the contribution calculation step is a step of calculating a contribution of each of a plurality of dimensions of the input to the model or the feature quantities to the model output drift based on the baseline data.

9. 9. The method of claim 8, The model output drift detection step includes a model output calculation step, an uncertainty calculation step, and a model output drift determination step; the model output calculation step is a step of calculating information about a class by implementing the model, in response to an input to the model or the feature quantity; The uncertainty calculation step is a step of calculating a model output drift measure indicating uncertainty of an output from the model based on information about the class; The method, wherein the model output drift determining step is a step of determining that an increase in the model output drift has occurred when the degree of the model output drift satisfies a predetermined condition.

10. 9. The method of claim 8, The baseline data setting step includes a reference time selection step, a past model output acquisition step, and a baseline data selection step, the reference time selection step is a step of selecting an input to the model or the feature quantity at a time included in a time period satisfying a predetermined condition, among times before a time corresponding to the input to the model or the feature quantity that is a target for which an increase in the model output drift has been detected; the past model output acquisition step is a step of obtaining information about a class in accordance with the input to the model or the feature value selected in the reference time selection step by implementing the model, or obtaining information about the class from a history of information about the class obtained in the past for the input to the model or the feature value selected in the reference time selection step; The baseline data selection step is a step of selecting the baseline data from among the inputs to the model or the features selected in the reference time selection step, based on information regarding a class for the input to the model or the features for which an increase in the model output drift has been detected, and information regarding the class obtained in the past model output acquisition step.

11. 9. The method of claim 8, The contribution degree calculation step includes an individual contribution degree calculation step and a group contribution degree calculation step, the individual contribution calculation step is a step of calculating a contribution of each of the dimensions based on an input to the model or the feature amount for which an increase in the model output drift is detected, for each of the baseline data, and the baseline data; The method, wherein the set contribution calculation step is a step of calculating the contribution of each of the dimensions corresponding to the set of baseline data based on the contribution of each of the dimensions corresponding to each of the baseline data.

12. 9. The method of claim 8, The method includes a display output control step, A method, wherein the display output control step is a step of controlling information regarding the contribution of each of the dimensions to be displayed or output.

13. 9. The method of claim 8, the baseline data setting step is a step of selecting, for each of a plurality of time periods prior to a time corresponding to an input to the model or the feature value for which an increase in the model output drift has been detected, baseline data from the input to the model or the feature value at a time included in the time period; The contribution degree calculation step is a step of calculating, for each of the time periods, a contribution degree of each of the dimensions based on the baseline data belonging to the time period; The method, wherein the contribution calculation step is a step of obtaining contribution time-series information having information regarding the contribution corresponding to a combination of the time period and the dimension.

14. 9. The method of claim 8, the information processing system includes a drift history database; The method comprises a similarity drift search step, the drift history database has records corresponding to increases in the model output drift detected in the past by the model output drift detection step; The record has some or all of the following information related to an increase in the model output drift: time, an input to the model, a feature obtained by a conversion process on the input, information on a class which is an output from the model, the contribution of each of the dimensions, and a cause of the increase in the model output drift; The similar drift search step is a step of performing a search process, when an increase in model output drift is detected by the model output drift detection step, to compare some or all of the input to the model, features obtained by a conversion process on the input, information on the class that is the output from the model, and information on the contribution of each of the dimensions that are related to the detected increase in model output drift, with corresponding information in records held in the drift history database, and identifying records that are highly similar in terms of the compared information.

15. A program to be executed in an information processing system, the information processing system realizes a model for outputting information related to a class in response to an input or a feature obtained by a conversion process of the input; the program is for causing the information processing system to execute a model output drift detection step, a baseline data setting step, and a contribution degree calculation step, The model output drift detection step is a step of determining whether an increase in model output drift, which indicates uncertainty in the output of the model, has occurred; The baseline data setting step is a step of selecting baseline data from an input to the model or the feature amount at a time before a time corresponding to the input to the model or the feature amount that is a target for which an increase in the model output drift has been detected, The contribution calculation step is a step of calculating, based on the baseline data, a contribution of each of a plurality of dimensions of the input to the model or the feature quantities to the model output drift.

Images