Information processing apparatus, information processing method, and program

Abstract

To improve a system by detecting accuracy deterioration of inference in the system for executing inference of a combination optimization problem using a learned model.SOLUTION: An information processing apparatus includes an acquisition unit configured to acquire input data including a condition in a combinatorial optimization problem, an analysis unit configured to acquire a determination index for evaluating an optimization result based on the optimization result derived by inputting the acquired input data to a learned model that has executed machine learning for deriving the optimization result satisfying the condition based on learning data, and an evaluation unit configured to evaluate the optimization result derived by the learned model based on the acquired determination index and a set determination criterion of the determination index. And an updating unit configured to update at least one of the learned model and the determination criterion.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In the field of data science, research and development has been carried out on systems that perform situation-specific predictions (hereinafter simply referred to as "predictions") based on statistics, etc., using trained models that have been trained based on concepts of the target, such as collected training data and labels of true or false assigned to prediction results, as well as systems that perform inference for combinatorial optimization problems that derive optimal plans by satisfying constraints from among a large number of options. Furthermore, machine learning approaches are becoming increasingly common, such as solving combinatorial optimization problems using reinforcement learning or deep learning, or determining parameters using training data.

[0003] After a trained model is generated by performing pre-learning using training data, etc., if there are changes over time in the data input to the trained model or the concept of the event to be predicted, the accuracy of the predictions and inferences output by the trained model may deteriorate.In systems that perform predictions and inferences using trained models, a technology has been proposed that detects deterioration in the accuracy of predictions and inferences made by the trained model by comparing the distributions of training data and input data, or prediction results with events that have actually occurred. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2021 / 079444 Summary of the Invention [Problem to be solved by the invention]

[0005] Inference for combinatorial optimization problems using a trained model differs from regression or classification using a trained model in that it outputs a plan (approximate solution) that satisfies constraints, so it does not necessarily match the events that have actually occurred, and it has not always been possible to detect deterioration in inference accuracy based on precision, accuracy rate, etc. The present invention aims to detect deterioration in inference accuracy in a system that performs inference for combinatorial optimization problems using a trained model and improve the system. [Means for solving the problem]

[0006] The information processing device according to the present invention is characterized by having an acquisition means for acquiring input data including conditions in a combinatorial optimization problem; an analysis means for acquiring a judgment index for evaluating the optimization result based on the optimization result derived by inputting the acquired input data into a trained model that has undergone machine learning to derive an optimization result that satisfies the conditions based on the training data; an evaluation means for evaluating the optimization result derived by the trained model based on the acquired judgment index and a judgment criterion for the judgment index that has been set; and an update means for updating at least one of the trained model and the judgment criterion according to the evaluation result of the optimization result by the evaluation means. [Effects of the Invention]

[0007] According to the present invention, in a system that performs inference for combinatorial optimization problems using a trained model, it is possible to detect deterioration in the accuracy of inference and improve the system. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 1 illustrates an example of a hardware configuration of an information processing device. [Figure 3] FIG. 1 illustrates an example of a functional configuration of an information processing system. [Figure 4] 10 is a flowchart illustrating an example of processing by the information processing device. [Figure 5] FIG. 10 is a diagram illustrating an example of processing by an information processing device. [Figure 6] FIG. 10 is a diagram illustrating the accuracy evaluation of inference using a trained model. [Figure 7] FIG. 10 is a diagram illustrating the assignment of false negative labels. [Figure 8] FIG. 10 is a diagram illustrating the assignment of false positive labels. [Figure 9] FIG. 10 is a diagram illustrating the selection of a trained model. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] 1 is a diagram showing an example of the configuration of an information processing system 100 according to one embodiment of the present invention. The information processing system 100 according to this embodiment performs inference on a combinatorial optimization problem using a trained model (hereinafter also referred to as a "mathematical optimization model") that has undergone machine learning to derive an optimization result that satisfies a condition based on training data. The information processing system 100 inputs input data into the mathematical optimization model and performs inference on the combinatorial optimization problem, thereby obtaining an optimization result that corresponds to the input data.

[0011] The information processing system 100 in this embodiment includes a server device 101 and a terminal device 102. The server device 101 and the terminal device 102 are connected to each other so as to be able to communicate with each other via a network NW. Note that the type of the network NW is not particularly limited as long as it is able to connect the server device 101 and the terminal device 102 to each other so as to be able to communicate with each other. Furthermore, although the example shown in FIG. 1 shows one terminal device 102, the number of terminal devices 102 in the information processing system 100 is not particularly limited, and multiple terminal devices 102 may be provided.

[0012] The server device 101 uses a mathematical optimization model to perform inference on a combinatorial optimization problem based on input data and derives an optimization result corresponding to the input data. The server device 101 also calculates and acquires a judgment index (evaluation index) for evaluating the optimization result based on the derived optimization result, and uses the acquired judgment index (evaluation index) to make a pass / fail judgment on the derived optimization result. The server device 101 also updates the mathematical optimization model that performs inference on the combinatorial optimization problem and updates judgment criteria (such as a threshold value used to evaluate the optimization result) according to the obtained judgment result.

[0013] The terminal device 102 is a schematic representation of a device that serves as an input / output interface for users (including a system operator) of the information processing system 100 to use the server device 101. For example, the terminal device 102 may be used as an input / output interface for receiving various instructions from users to the server device 101, and for providing optimization results derived by the server device 101 to users.

[0014] 2 is a diagram showing an example of the hardware configuration of an information processing device 200 that can be used as the server device 101 and the terminal device 102 in the information processing system 100 shown in FIG. 1. The information processing device 200 has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, and a network I / F 207. The information processing device 200 may also have at least one of an output device 205 and an input device 206. The CPU 201, the ROM 202, the RAM 203, the auxiliary storage device 204, the output device 205, the input device 206, and the network I / F 207 are communicatively connected via a system bus 208.

[0015] The CPU (Central Processing Unit) 201 is a central processing unit that controls various operations of the information processing device 200. For example, the CPU 201 may control the operation of the entire information processing device 200. The ROM (Read Only Memory) 202 stores control programs, boot programs, etc. that can be executed by the CPU 201. The RAM (Random Access Memory) 203 is the main storage memory of the CPU 201, and is used as a work area or a temporary storage area for expanding various programs.

[0016] The auxiliary storage device 204 stores various data, various programs, etc. The auxiliary storage device 204 is realized by a storage device capable of temporarily or permanently storing various data, such as a non-volatile memory represented by an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0017] The output device 205 is a device that outputs various types of information and is used to present various types of information to a user. For example, the output device 205 is realized by a display device such as a display. The output device 205 may present information to a user by displaying various types of display information. As another example, the output device 205 may be realized by an audio output device that outputs sounds such as voices and electronic sounds. In this case, the output device 205 may present information to a user by outputting sounds such as voices and electronic sounds. Furthermore, the device used as the output device 205 may be changed as appropriate depending on the medium used to present information to a user.

[0018] The input device 206 is used to receive various instructions from the user. For example, the input device 206 may include an input device such as a mouse, a keyboard, or a touch panel. As another example, the input device 206 may include a sound collection device such as a microphone to collect voices uttered by the user. In this case, the collected voices may be subjected to various analytical processes such as acoustic analysis and natural language processing, and the contents of the voices may be recognized as instructions from the user. Furthermore, the device used as the input device 206 may be changed as appropriate depending on the method for recognizing instructions from the user. Furthermore, multiple types of devices may be used as the input device 206.

[0019] The network I / F 207 is used for communication with external devices etc. via a network. The device used as the network I / F 207 may be changed as appropriate depending on the type of communication path and the communication method to be applied.

[0020] The CPU 201 loads a program stored in the ROM 202 or the auxiliary storage device 204 into the RAM 203 and executes the program, thereby realizing the functions of the devices shown in Fig. 3 and the processes shown in Fig. 4. The program for the information processing device 200 may be provided to the information processing device 200 by a recording medium such as a CD-ROM, or may be downloaded via a network, etc. When the program for the information processing device 200 is provided by a recording medium, the program recorded on the recording medium is installed in the auxiliary storage device 204 by inserting the recording medium into a predetermined drive device.

[0021] 2 is merely an example and does not necessarily limit the hardware configuration of the information processing device 200 in this embodiment. As a specific example, some components such as the output device 205 and the input device 206 may not be included. As another example, components according to the functions realized by the information processing device 200 may be added as appropriate.

[0022] Fig. 3 is a diagram showing an example of the functional configuration of the information processing system 100 according to this embodiment. In Fig. 3, the same components as those shown in Fig. 1 are denoted by the same reference numerals.

[0023] The following describes the server device 101. The server device 101 includes a communication unit 301, a control unit 302, an input / output control unit 303, an acquisition unit 304, an inference unit 305, a management unit 306, an analysis unit 307, an evaluation unit 308, an update processing unit 309, and a storage unit 310.

[0024] The communication unit 301 is a communication interface that enables each component of the server device 101 to transmit and receive information to and from other devices such as the terminal device 102 via the network NW. The communication unit 301 can be realized by, for example, the network I / F 207. In the following description, when each component of the server device 101 transmits and receives information to and from other devices, it is assumed that the information is transmitted and received via the communication unit 301 unless otherwise specified.

[0025] The control unit 302 controls the various components of the server device 101. The input / output control unit 303 executes various processes related to presenting various types of information to the user and accepting information input (e.g., instructions, etc.) from the user. For example, the input / output control unit 303 may execute processes related to presenting a UI (User Interface) and accepting input via the UI. This enables the server device 101 to recognize instructions from the user and present the results of processing in accordance with those instructions to the user.

[0026] The acquisition unit 304 acquires input data related to the combinatorial optimization problem to be solved. The input data includes the problem to be optimized (such as various planning problems) and conditions (constraints on the combinatorial optimization problem).

[0027] The inference unit 305 uses a trained model (mathematical optimization model) that has undergone machine learning to derive an optimization result that satisfies a condition based on training data to perform inference on the combinatorial optimization problem based on the input data acquired by the acquisition unit 304, and derives an optimization result. The inference unit 305 outputs the derived optimization result and information related to the execution of inference using the mathematical optimization model (for example, memory usage in the system, number of parallel processes, input / output file sizes, etc.) as intermediate outputs.

[0028] The management unit 306 manages various types of data. For example, the management unit 306 manages the optimization results and intermediate outputs output by the inference unit 305, and the judgment index (evaluation index) output by the analysis unit 307. The management unit 306 also manages judgment criteria (thresholds, etc.) used to evaluate the optimization results, trained models (mathematical optimization models) that perform inference for combinatorial optimization problems, verification data related to the trained models (mathematical optimization models), and the like.

[0029] The analysis unit 307 analyzes the optimization results and intermediate output output from the inference unit 305 to obtain a judgment index (evaluation index) for evaluating the optimization results. The analysis unit 307 calculates the judgment index (evaluation index), for example, from the optimization results and intermediate output output from the inference unit 305. For example, the analysis unit 307 reads the optimization results output in a CSV file or the like, processes the data, and obtains the judgment index (evaluation index). The analysis unit 307 may also analyze the training data and the input data acquired by the acquisition unit 304 to obtain statistical information on the data. Here, the statistical information is, for example, information for detecting changes in the data by statistically comparing the training data and the input data.

[0030] The evaluation unit 308 evaluates the optimization result derived by the inference unit 305 based on the judgment index (evaluation index) acquired by the analysis unit 307. The evaluation unit 308 evaluates the optimization result by, for example, comparing the judgment index (evaluation index) acquired by the analysis unit 307 with a set judgment criterion, and determines whether the optimization result passes or fails. This makes it possible to evaluate the accuracy of inference using a trained model (mathematical optimization model), and to detect deterioration in the accuracy of inference. The evaluation unit 308 may also detect changes in the data by comparing statistical information of the training data and input data acquired by the analysis unit 307.

[0031] The update processing unit 309 performs an update process as appropriate in accordance with the evaluation result of the optimization result by the evaluation unit 308, and updates the trained model (mathematical optimization model) used for inference on the combinatorial optimization problem by the inference unit 305 or the information (decision index, decision criterion) used for evaluating the optimization result. Note that, in the update process, the update processing unit 309 may update both the trained model (mathematical optimization model) and the information (decision index, decision criterion) used for evaluating the optimization result. Furthermore, the update processing unit 309 may perform an update process based on instructions received from a user, and update the trained model (mathematical optimization model) and the information (decision index, decision criterion) used for evaluating the optimization result. For example, when the update processing unit 309 receives an error in the evaluation result of the optimization result by the evaluation unit 308, it may perform an update process to update the trained model (mathematical optimization model) and the information (decision index, decision criterion) used for evaluating the optimization result.

[0032] The storage unit 310 schematically illustrates a storage area for storing various data, various programs, and the like. For example, the storage unit 310 may store data and programs for each component of the server device 101 to execute processing. The storage unit 310 may also store data (code and parameter values) related to a trained model (mathematical optimization model) that performs inference on a combinatorial optimization problem, training data used in machine learning of the trained model (mathematical optimization model), and the like. The storage unit 310 may also store input data related to the combinatorial optimization problem acquired by the acquisition unit 304, determination criteria (threshold values, etc.) used to evaluate the optimization results, and the like. The storage unit 310 may also store optimization results and intermediate outputs output by the inference unit 305, determination indices (evaluation indices) acquired by the analysis unit 307, and the like.

[0033] Next, a description will be given of the terminal device 102. The terminal device 102 includes a communication unit 321, a control unit 322, an input / output control unit 323, and a storage unit 324.

[0034] The communication unit 321 is a communication interface that enables each component of the terminal device 102 to transmit and receive information to and from other devices such as the server device 101 via the network NW. The communication unit 321 can be realized, for example, by the network I / F 207. In the following description, when each component of the terminal device 102 transmits and receives information to and from other devices, it is assumed that the information is transmitted and received via the communication unit 321 unless otherwise specified.

[0035] The control unit 322 is responsible for controlling each component of the terminal device 102. The input / output control unit 323 executes various processes related to presenting various information to the user and accepting information input (e.g., instructions, etc.) from the user. For example, the input / output control unit 323 may execute processes related to presenting a UI and accepting input via the UI. This enables the terminal device 102 to recognize instructions from the user and present the results of processing in accordance with those instructions to the user.

[0036] The storage unit 324 schematically shows a storage area for storing various data, various programs, etc. For example, the storage unit 324 may store data and programs for each component of the terminal device 102 to execute processing.

[0037] Fig. 4 is a flowchart showing an example of processing of the information processing system 100 in this embodiment. Fig. 4 shows an example of updating information (decision index, decision criterion) related to evaluation of the optimization result in accordance with the evaluation result of the optimization result during system operation. Each process in the flowchart shown in Fig. 4 is executed by, for example, the server device 101 of the information processing system 100.

[0038] In step S401, the acquisition unit 304 acquires input data related to the combinatorial optimization problem to be solved, and the analysis unit 307 analyzes the input data acquired by the acquisition unit 304 to acquire statistical information related to the input data.

[0039] In step S402, the evaluation unit 308 determines whether the input data falls within the range of the training data based on the input data and its statistical information acquired in step S401. For example, the evaluation unit 308 determines whether the input data falls within the range of the training data using statistical information on the input data and the training data. If the evaluation unit 308 determines that the input data falls within the range of the training data (YES), the process of step S404 is executed. On the other hand, if the evaluation unit 308 determines that the input data does not fall within the range of the training data (NO), the process of step S403 is executed. For example, in the case of a production planning problem, if the input data includes a product to be produced that was not included in the training data, the evaluation unit 308 determines that the input data does not fall within the range of the training data.

[0040] In step S403, the control unit 302 records the input data acquired in step S401 in the storage unit 310 as abnormal data.

[0041] In step S404, the inference unit 305 performs inference using a mathematical optimization model on the combinatorial optimization problem based on the input data acquired in step S401, and derives an optimization result. The inference unit 305 then outputs the derived optimization result and information related to the execution of inference using the mathematical optimization model as intermediate output. The information output as intermediate output may include, for example, system-related information such as memory usage, the number of parallel processes (number of containers), processing time, and input / output file sizes.

[0042] In step S405, the analysis unit 307 analyzes the optimization results and intermediate output obtained in the process in step S404 to obtain judgment indices (evaluation indices) for evaluating the optimization results. For example, in the case of a production planning problem, the analysis unit 307 obtains information such as yield, cost, delivery date achievement rate, presence or absence of violations of manufacturing conditions, processing time, etc. from the optimization results and intermediate output as judgment indices (evaluation indices).

[0043] In step S406, the evaluation unit 308 performs a determination process regarding the evaluation of the optimization result based on the determination index (evaluation index) acquired in step S405. The evaluation unit 308 makes a determination regarding the evaluation of the optimization result by comparing the determination index (evaluation index) acquired in step S405 with a set determination criterion (threshold value, etc.). For example, in the case of a production planning problem, the evaluation unit 308 determines whether the lower limit of the yield or the upper limit of the processing time has been exceeded, whether a violation has occurred in the delivery deadline or manufacturing conditions, etc.

[0044] In step S407, the control unit 302 records in the storage unit 310 the optimization results and intermediate outputs obtained in the process in step S404, the judgment index (evaluation index) acquired in step S405, and the like.

[0045] In step S408, the evaluation unit 308 determines whether the judgment index (evaluation index) acquired based on the optimization result in step S405 falls within the set judgment criteria as a result of the judgment process in step S406. That is, the evaluation unit 308 performs inference using a mathematical optimization model and performs a pass / fail judgment on the derived optimization result. If the evaluation unit 308 determines that the judgment index (evaluation index) acquired based on the optimization result falls within the judgment criteria (YES), the derived optimization result is determined as a fail, and the process of step S412 is executed. On the other hand, if the evaluation unit 308 determines that the judgment index (evaluation index) acquired based on the optimization result falls within the judgment criteria (NO), the derived optimization result is determined as a pass, and the process of step S409 is executed. For example, in the production planning problem described above, if the yield is below the set lower limit or the processing time exceeds the set upper limit, the evaluation unit 308 determines that the judgment index (evaluation index) does not fall within the judgment criteria, and the derived optimization result is determined as a fail.

[0046] In step S409, the control unit 302 determines whether the data in this series of processes is a false negative. If the control unit 302 determines that it is not a false negative (True Negative) (NO), the data in this series of processes is treated as negative data, and the process shown in Fig. 4 is terminated. On the other hand, if the control unit 302 determines that it is a false negative (YES), the process of step S410 is executed.

[0047] In step S410, the control unit 302 assigns a false negative label to the data in the current series of processes and records it in the storage unit 310 as false negative data. In step S411, the update processing unit 309 performs an update process to update the information (judgment index, judgment criteria) used to evaluate the optimization results. Since the false negative is determined because an abnormality is not detected using the set judgment criteria, the update processing unit 309 updates the type of judgment index to be acquired and the set judgment criteria. For example, the update processing unit 309 may acquire false negative data for a specific period or range and update the judgment criteria so that an abnormality is detected. Furthermore, for example, the update processing unit 309 may identify the judgment index that caused the false negative using the false negative data and negative data, and add the identified judgment index to the judgment indexes used to evaluate the optimization results. After executing the process of step S411, the process shown in FIG. 4 ends. The process of labeling false negatives and the like will be described later with reference to FIG.

[0048] In step S412, the control unit 302 records the data in this series of processes in the storage unit 310 as abnormal data.

[0049] In step S413, the control unit 302 determines whether the data in this series of processes is a false positive. If the control unit 302 determines that it is not a false positive (True Positive) (NO), the data in this series of processes is considered to be positive data, and the processing shown in Fig. 4 is terminated. On the other hand, if the control unit 302 determines that it is a false positive (YES), the processing of step S414 is executed.

[0050] In step S414, the control unit 302 assigns a false positive label to the data in this series of processes and records it in the storage unit 310 as false positive data. In step S415, the update processing unit 309 performs an update process to update the information (decision index, decision criterion) used to evaluate the optimization results. A false positive is determined because the data is not abnormal from a human perspective but is detected as abnormal by the set decision criterion. Therefore, the update processing unit 309 updates the type of decision index to be acquired and the set decision criterion. For example, the update processing unit 309 may acquire false positive data for a specific period or range and delete a decision index or relax the decision criterion so that the data is not detected as abnormal. Furthermore, for example, the update processing unit 309 may identify a decision index that caused a false positive using the false positive data and the positive data, and delete the identified decision index from the decision indexes used to evaluate the optimization results. After deleting a certain decision index from the decision indexes used to evaluate the optimization results, the trained model may be re-trained using the false positive data and the positive data. If the false positive data results in a negative decision, the deleted decision index may be included again in the decision indexes used to evaluate the optimization results. Furthermore, for example, the update processing unit 309 may derive a threshold value recommended for the determination index using the false positive data and the most recent positive data, and update the determination criteria with the derived threshold value. After executing the process of step S415, the process shown in FIG. 4 ends. The processing related to the above-mentioned labeling of false positives will be described later with reference to FIG.

[0051] In this way, input data is input into a trained model (mathematical optimization model), and an evaluation index for evaluating the optimization result is obtained from the derived optimization result and information (intermediate output) on the derivation of the optimization result, and the derived optimization result is evaluated by comparing the evaluation index with the set judgment criterion. This makes it possible to detect deterioration in the accuracy of inference by the trained model based on the evaluation result of the optimization result, and by updating the information (judgment index, judgment criterion) used to evaluate the optimization result, the accuracy of the inference can be made appropriate and the system can be improved. Furthermore, if an error in the evaluation result of the optimization result is received, the information (judgment index, judgment criterion) used to evaluate the optimization result can be updated in accordance with the error, thereby appropriately maintaining the accuracy of inference by the trained model and improving the system.

[0052] Fig. 5 is a flowchart showing an example of processing by the information processing system 100 in this embodiment. Selection and updating of a trained model (mathematical optimization model) used in the information processing system 100 will be described with reference to Fig. 5. Fig. 5 shows the process divided into a learning time 510 in which machine learning of a model (e.g., an AI model) is performed, a verification time 520 in which the model to be used is determined, and an inference time 530 in which the model is actually used and processing is performed.

[0053] During learning 510, the server device 101 performs learning processing 512 on the model using learning data 511, and performs machine learning to obtain an optimization result that satisfies the conditions. Then, the server device 101 stores the trained model generated by the learning processing 512 in a trained model registry 513. Note that in this embodiment, the trained model registry 513 can store multiple trained models that have different codes and / or parameters.

[0054] During validation 520 before deployment, the server device 101 uses validation data 521 to perform inference on a combinatorial optimization problem based on the validation data 521 for each trained model 522 stored in the trained model registry 513, thereby deriving an optimization result. Next, similar to the process shown in FIG. 4 , the server device 101 performs index calculation processing 523 to acquire a judgment index (evaluation index) for evaluating the optimization result based on the optimization result and intermediate output, and index determination processing 524 to perform a judgment process related to the evaluation of the optimization result based on the acquired judgment index (evaluation index). Then, the server device 101 performs a model determination process to determine the trained model to be deployed based on the processing results of the index calculation processing 523 and the index determination processing 524. For example, the server device 101 may set priorities for the judgment index (evaluation index) acquired in the index calculation processing 523 and the judgment result in the index determination processing 524, compare the judgment index (evaluation index) and the judgment results in order of priority, and select the best trained model to be deployed. 9, for example, the priorities of the judgment indexes (evaluation indexes) and the judgment results may be set in the order of "violation of user conditions," "violation of manufacturing equipment," "production time," and "amount," and the server device 101 may compare the information in each trained model with that of model C to determine the model to deploy. In this example, Model A and Model B show better results than Model C for "violation of user conditions" and "violation of manufacturing equipment," and Model A shows better results than Model B for "production time," so the server device 101 decides to adopt Model A as the model to deploy.

[0055] During inference 530, when the model is actually used and processed, the server device 101 inputs input data 531 into the trained model determined and deployed in model determination process 525 during verification 520, executes inference on a combinatorial optimization problem based on the input data 531, and derives an optimization result. Next, the server device 101 performs index calculation process 533 and index determination process 534, which are similar to the index calculation process 523 and index determination process 524 during verification 520, acquires a determination index (evaluation index) for evaluating the optimization result, and performs determination processing regarding the evaluation of the optimization result. Next, the server device 101 performs alert output process 535, and if it is determined in the index determination process 534 that the determination index (evaluation index) does not fall within the determination criteria, outputs an alert to the user or the like, indicating that an unguaranteed result has been obtained and confirmation is required. In addition, the server device 101 records the input data etc. when the alert was output as abnormal case data 540, performs verification data update processing 550 based on the data 540 to update the verification data 521, and performs learning data update processing 560 based on the data 540 to update the learning data 511.

[0056] In this way, by performing the same index calculation process and index determination process at verification time 520 when the model to be used is determined and at inference time 530 when the model is actually used and processed, it is possible to detect abnormal cases that did not exist at verification time 520 and detect deterioration in the accuracy of inference using the trained model for deriving optimization results. Then, by performing re-learning and re-verification using training data 511 and verification data 521 updated to reflect data 540 that has become an abnormal case, and by determining and updating a new trained model to be deployed, it is possible to improve the accuracy of inference using the trained model and improve the system.

[0057] 6 is a diagram illustrating processing related to the evaluation of the accuracy of inference by a trained model (mathematical optimization model) in this embodiment. The mathematical optimization model 601 is a trained model that has undergone machine learning to obtain an optimization result that satisfies a condition based on training data 602. When input data 603 related to a combinatorial optimization problem to be solved is input, the mathematical optimization model 601 executes inference on the combinatorial optimization problem based on the input data 603, and derives an optimization result 605 corresponding to the input data 603. In addition, information related to the execution of inference by the mathematical optimization model 601 is output as intermediate output 606.

[0058] The analysis unit 307 executes an index calculation process 607 using the optimization result 605 and intermediate output 606, and derives a judgment index (evaluation index) 608 for evaluating the optimization result based on the optimization result 605 and intermediate output 606. The derived judgment index 608 is used in an index judgment process 609 by the evaluation unit 308, and is also saved as index history data 610. In the index judgment process 609, a judgment is made regarding the evaluation of the optimization result based on the judgment index 608 and a set and saved judgment criterion 611, and the judgment result is output as pass / fail 612 of the optimization result.

[0059] It is also possible to store the input data 603 as the most recent data 604, and statistically compare the learning data 602 with the most recent data 604 to detect changes in the data, known as data drift.

[0060] Processing related to the assignment of false negative labels will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining the assignment of false negative labels. In FIG. 7, the same elements as those shown in FIG. 6 are assigned the same reference numerals. In this embodiment, the control unit 302 of the server device 101 records data related to the inferred combinatorial optimization problem as normal data 702 or abnormal data 703 based on the optimization result pass / fail 612 indicating the judgment result regarding the evaluation of the optimization result. That is, the control unit 302 records the data related to the inferred combinatorial optimization problem as normal data 702 if the optimization result pass / fail 612 indicates a pass judgment, and records the data as abnormal data 703 if the optimization result pass / fail 612 indicates a fail judgment.

[0061] However, for example, if the optimization result pass / fail 612 is judged to be a pass, but the user judges the output optimization result 605 to be abnormal from a human perspective, it is possible to notify the server device 101 of the occurrence of an abnormality. For example, in the case of a production planning problem, this would be the case when a product is produced using equipment that a human would not normally select. When the server device 101 receives an abnormality notification from the user (701), the control unit 302 assigns a false negative label to the data related to the combinatorial optimization problem for which the optimization result 605 was output, and sets it as false negative data.

[0062] Furthermore, for example, when an anomaly is detected by the anomaly data detection process 705 using the index history data 610 and the validation data 704 for a normal case, the control unit 302 assigns a false negative label to the data related to the combinatorial optimization problem for which the optimization result 605 is output, and sets it as false negative data. Here, the anomaly data detection process 705 may apply a general anomaly detection algorithm such as One-Class SVM (Support Vector Machine) or ABOD (Angle Based Outlier Detection).

[0063] The process of assigning a false positive label will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining the assignment of a false positive label. As described above, if the pass / fail judgment of the optimization result indicates a fail judgment, the control unit 302 of the server device 101 records the data related to the inferred combinatorial optimization problem as abnormal data 801.

[0064] The server device 101 provides the abnormal data 801 to the user 802 together with the judgment index (evaluation index) acquired from the abnormal data (811). If the user determines that the abnormal data 801 and judgment index (evaluation index) provided by the server device 101 should not be regarded as abnormal data, the user notifies the server device 101 that the data will be regarded as false positive data (812). For example, in the case of a production planning problem, this applies when it becomes possible to produce a product with equipment that was previously unable to produce it, or when costs increase due to quality improvements, etc.

[0065] When the server device 101 receives this notification, the control unit 302 assigns a false positive label to the data related to the combinatorial optimization problem and sets it as false positive data 803. The control unit 302 also modifies (813) the processing related to the evaluation of the optimization results based on the false positive data 803, and updates the index calculation logic 804 and the judgment criteria 805 used to obtain the judgment index (evaluation index). For example, the control unit 302 uses the false positive data 803 to relax the judgment criteria so that false positives are judged as negatives. The control unit 302 then reevaluates the abnormal data 801 using the modified index calculation logic 804 and the judgment criteria 805 (814).

[0066] According to this embodiment, an evaluation index for evaluating the optimization result is obtained based on the optimization result derived by inputting input data into a trained model (mathematical optimization model) and information (intermediate output) related to the derivation of the optimization result, and the derived optimization result is evaluated by comparing it with a set judgment criterion. This makes it possible to detect deterioration in the accuracy of inference by the trained model based on the evaluation result of the optimization result, and by updating the trained model (mathematical optimization model) and the information (judgment index, judgment criterion) used to evaluate the optimization result, it becomes possible to improve the accuracy of inference and improve the system.

[0067] In the above-described embodiment, the trained model (mathematical optimization model) and the information (decision index, decision criterion) used to evaluate the optimization results are updated based on the evaluation results of the optimization results and the false-negative data and false-positive data set by accepting errors therein. However, the trained model (mathematical optimization model) and the information (decision index, decision criterion) used to evaluate the optimization results may be updated based on the evaluation results of the optimization results without performing the process of setting false-negative data and false-positive data. In this case, the update processing unit 309 may update the trained model (mathematical optimization model) and the information (decision index, decision criterion) used to evaluate the optimization results when, for example, the evaluation unit 308 determines that the decision index (evaluation index) acquired based on the optimization results does not fall within the set decision criterion.

[0068] It should be noted that the above-described embodiments are merely examples of specific embodiments of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0069] 100 Information Processing Systems 101 Server device 102 Terminal Equipment 201 CPU 202 ROM 203 RAM 204 Auxiliary storage device 205 Output Device 206 Input Device 207 Network I / F 301, 321 Communications Department 302, 322 Control section 303, 323 Input / output control section 304 Acquisition Department 305 Reasoning Department 306 Management Department 307 Analysis Department 308 Evaluation Department 309 Update processing section 310, 324 Storage section

Claims

1. an acquisition means for acquiring input data including conditions for a combinatorial optimization problem; an analysis means for inputting the acquired input data into a trained model that has undergone machine learning to derive an optimization result that satisfies a condition based on training data, and acquiring a judgment index for evaluating the optimization result based on the derived optimization result; evaluation means for evaluating the optimization result derived by the trained model based on the acquired judgment index and a judgment criterion for the judgment index that has been set; an update means for updating at least one of the trained model and the judgment criterion according to an evaluation result of the optimization result by the evaluation means.

2. 2. The information processing device according to claim 1, wherein the update means updates at least one of the trained model and the judgment criterion when an error in the evaluation result of the optimization result by the evaluation means is received.

3. The information processing device according to claim 1 or 2, characterized in that, when the update means receives an error in the evaluation result of the optimization result by the evaluation means, it identifies the judgment index that caused the error based on the acquired judgment index and adds it to the judgment index in the evaluation of the optimization result or deletes it from the judgment index.

4. 4. The information processing device according to claim 3, wherein the update means, after deleting the judgment index that caused the error from the judgment index in the evaluation of the optimization result, includes the deleted judgment index in the judgment index in the evaluation of the optimization result when it determines that the evaluation result of the optimization result derived by inputting the input data that accepted the error in the evaluation result of the optimization result into a re-trained trained model is correct.

5. 3. The information processing apparatus according to claim 1, wherein the update means, when receiving an error in the evaluation result of the optimization result by the evaluation means, updates the determination criteria so that the error does not occur.

6. setting a priority order for the acquired judgment indexes and the judgment results regarding the judgment indexes by the evaluation means; 3. The information processing apparatus according to claim 1, wherein the judgment index and the judgment result are compared in the order of priority to determine the trained model to be used for deriving the optimization result.

7. 3. The information processing device according to claim 1, wherein the update means updates the trained model when the evaluation means determines that the acquired judgment index is not within the judgment criterion.

8. 3. The information processing apparatus according to claim 1, wherein the analysis means acquires the judgment index based on the derived optimization result and information related to the derivation of the optimization result.

9. An information processing method executed by an information processing device, an acquisition step of acquiring input data including conditions for a combinatorial optimization problem; an analysis step of inputting the acquired input data into a trained model that has undergone machine learning to derive an optimization result that satisfies a condition based on training data, and acquiring a judgment index for evaluating the optimization result based on the derived optimization result; an evaluation step of evaluating the optimization result derived by the trained model based on the acquired judgment index and a judgment criterion for the judgment index that has been set; an update step of updating at least one of the trained model and the judgment criterion according to the evaluation result of the optimization result in the evaluation step.

10. The computer of the information processing device an acquisition step of acquiring input data including conditions for a combinatorial optimization problem; an analysis step of inputting the acquired input data into a trained model that has undergone machine learning to derive an optimization result that satisfies a condition based on training data, and acquiring a judgment index for evaluating the optimization result based on the derived optimization result; an evaluation step of evaluating the optimization result derived by the trained model based on the acquired judgment index and a judgment criterion for the judgment index that has been set; and an update step of updating at least one of the trained model and the judgment criterion depending on the evaluation result of the optimization result in the evaluation step.

Images