Information processing device, method, and program

Abstract

It is possible to set an appropriate likelihood threshold in task processing using a trained model. [Solution] A program to be executed on a computer having a processor and a memory, the program causing the processor to execute the following steps: accepting input of data to be the subject of a specified task; a first trained model that outputs a result of the task when the data is input; determining the result of the task using the input data; a second trained model that outputs a likelihood of the result of the task according to the first trained model by inputting the input data and the result of the task; determining the likelihood using the input data and the result of the task; and outputting the result of the task and the likelihood.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a method, and a program. [Background technology]

[0002] In recent years, there has been a demand for improving the accuracy of tasks processed by trained models. For example, when a task involves character recognition from image data, there is a demand for technology that can extract text data from form data with high accuracy using an OCR model. In an OCR model, the likelihood (confidence) of each of multiple candidates for an extracted word is calculated, and the word with the highest likelihood is used as the recognition result.

[0003] In order to improve such recognition by OCR, there is a technology in which, for example, a document is scanned, image data converted from the image is set to be read, an OCR function is provided, character information displayed in the read items of the image data is read by OCR to generate text data, the image data and text data of the read items are displayed in parallel, the likelihood of the text data is calculated, and the display mode of the displayed text data is changed (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-175037 A Summary of the Invention [Problem to be solved by the invention]

[0005] When performing task processing automatically, it is necessary to set a threshold for the likelihood of the task processing result in advance. However, there are cases where the task processing result is correct despite the low likelihood, or the task processing result is incorrect despite the high likelihood.

[0006] For example, in a trained OCR model used for character recognition, there are cases in which the content is correctly recognized despite a low likelihood in some inferences, and cases in which the content is misidentified despite a high likelihood. This has resulted in a problem in that it is not possible to set an appropriate likelihood threshold when processing tasks using a trained model.

[0007] An object of the present disclosure is to provide a technology that can set an appropriate likelihood threshold in task processing using a trained model. [Means for solving the problem]

[0008] In order to solve the above problems, a program according to one embodiment of the present disclosure includes: A program to be executed by a computer having a processor and a memory, the program causing the processor to execute the following steps: accepting input of data to be the subject of a specified task; a first trained model that outputs a result of the task when the data is input; determining the result of the task using the input data; a second trained model that outputs a likelihood of the result of the task according to the first trained model by inputting the input data and the result of the task; determining the likelihood using the input data and the result of the task; and outputting the result of the task and the likelihood. Effect of the Invention

[0009] According to the present disclosure, it is possible to set an appropriate likelihood threshold in task processing using a trained model. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a system 1. [Diagram 2] FIG. 2 is a diagram showing the functional configuration of the information processing device 10. As shown in FIG. [Diagram 3]FIG. 3 is a diagram showing the data structure of the learning data DB 121. As shown in FIG. [Figure 4] FIG. 4 is a diagram showing the data structure of the model DB 122. As shown in FIG. [Diagram 5] FIG. 5 is a diagram showing the functional configuration of the user terminal 20. As shown in FIG. [Figure 6] FIG. 6 is a diagram showing an example of a processing flow in the system 1. [Figure 7] FIG. 7 is a diagram showing an example of a processing flow in the system 1. [Figure 8] FIG. 8 is a diagram showing an example of a processing flow in the system 1. [Figure 9] FIG. 9 is a diagram showing an example of a processing flow in the system 1. [Figure 10] FIG. 10 is an example of a screen displayed on the user terminal 20. [Figure 11] FIG. 11 is a diagram showing a basic hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In all the drawings explaining the embodiment, the same reference numerals are given to common components, and repeated explanations are omitted. Note that the following embodiment does not unduly limit the contents of the present disclosure described in the claims. In addition, not all of the components shown in the embodiment are essential components of the present disclosure. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly.

[0012] In the following description, a "processor" refers to one or more processors. The at least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). The at least one processor may be a single-core or multi-core.

[0013] Furthermore, the at least one processor may be a processor in the broad sense, such as a hardware circuit (for example, a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC)) that performs part or all of the processing.

[0014] In the following explanation, information that gives an output for an input may be described using an expression such as "xxx table", but this information may be data of any structure or a learning model such as a neural network that generates an output for an input. Therefore, the "xxx table" may be called "xxx information".

[0015] Furthermore, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0016] In addition, in the following explanation, the processing may be described with the "program" as the subject, but since the program is executed by a processor to perform a specified processing step by appropriately using a memory unit and / or an interface unit, etc., the subject of the processing may be the processor (or a device such as a controller having the processor).

[0017] The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0018] In the following description, identification numbers are used as identification information for various objects, but other types of identification information (for example, identifiers including alphabetic characters or symbols) may be used instead.

[0019] In addition, in the following description, when describing elements of the same type without distinguishing between them, reference signs (or common signs among the reference signs) may be used, and when describing elements of the same type with distinction between them, the identification numbers (or reference signs) of the elements may be used.

[0020] In the following description, the control lines and information lines are those that are considered necessary for the description, and not all control lines and information lines in the product are necessarily shown. All components may be connected to each other.

[0021] Each information processing device is configured by a computer equipped with a calculation device and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the information processing device 10 and the user terminal 20, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0022] (First embodiment) <Configuration> A description will be given of the configuration of an information processing system according to the first embodiment. Fig. 1 is a block diagram illustrating the configuration of an information processing system according to the first embodiment.

[0023] 1, the system 1 includes an information processing device 10 and a user terminal 20. The information processing device 10 and the user terminal 20 are communicatively connected via a network 80. The information processing device 10 is a device that exhibits a function of learning the trained model of the present disclosure and a function of performing a predetermined task processing using the trained model. The user terminal 20 is an information processing device operated by a user.

[0024] The information processing device 10 is realized by a desktop PC (Personal Computer), a laptop PC, or the like.

[0025] As shown in FIG. 1, the information processing device 10 includes a communication IF 12, an input / output IF 13, a memory 15, and a storage 16.

[0026] The communication IF 12 is an interface for inputting and outputting signals so that the information processing device 10 can communicate with an external device. The input / output IF 13 functions as an interface with an input device for receiving an input operation from a user and an output device for presenting information to a user. The input / output IF 13 is for temporarily storing a program and data processed by the program, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The memory 15 is a storage device for saving data, such as a flash memory or a HDD (Hard Disc Drive). The storage 16 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, etc.

[0027] The user terminal 20 is realized by a mobile terminal such as a smartphone or tablet compatible with a mobile communication system, a stationary PC (Personal Computer), a laptop PC, or the like.

[0028] As shown in FIG. 1, the user terminal 20 includes a communication IF 22, an input / output IF 23, a memory 25, and a storage .

[0029] The communication IF 22 is an interface for inputting and outputting signals so that the user terminal 20 can communicate with an external device. The input / output IF 23 functions as an interface with an input device for receiving input operations from a user and an output device for presenting information to a user. The input / output IF 23 is for temporarily storing programs and data processed by the programs, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The memory 25 is a storage device for saving data, such as a flash memory or a HDD (Hard Disc Drive). The storage 26 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, etc.

[0030] <Functional configuration of information processing device 10> 2 is a diagram showing a functional configuration of the information processing device 10. As shown in FIG. 2, the information processing device 10 fulfills the functions of a communication unit 101, a storage unit 102, and a control unit 103.

[0031] The communication unit 101 performs processing for communicating with an external device.

[0032] The storage unit 102 stores data and programs to be used.

[0033] Specifically, the storage unit 102 stores a learning data DB (database) 121, a model DB 122, and the like.

[0034] The data structure used in the first embodiment will be described below. The data structures of the learning data DB 121 and the model DB 122 stored in the information processing device 10 shown in Fig. 3 and Fig. 4 will be described below. Fig. 3 and Fig. 4 are diagrams showing the data structures of the respective databases.

[0035] The learning data DB 121 is a database that holds user information. The learning data DB 121 includes an item “ID”, an item “data”, an item “task”, an item “processing result”, an item “likelihood”, and an item “correct label”.

[0036] The item "ID" is information for identifying data.

[0037] The item "data" is data used for the task.

[0038] The item "task" is information indicating which task the data is used for.

[0039] The item "processing result" is the result of processing the data.

[0040] The item "likelihood" is the likelihood of a processing result. The likelihood is, for example, the confidence that the processing result of a task is correct, the confidence that it is within an acceptable range, etc.

[0041] The item "correct label" is information indicating whether the processing result is correct or not, a correct processing result, etc.

[0042] The model DB 122 is a database that holds a model for executing a predetermined task, and includes an item "ID", an item "model", an item "task", an item "version", and an item "update date and time".

[0043] The item "ID" is information for identifying a model.

[0044] The item "model" is data on the model.

[0045] The item "task" is information indicating which task the model is used for.

[0046] The item "version" is information about the version of the model that was updated through learning.

[0047] The item "Update date and time" is the date and time when the model was updated through learning.

[0048] The control unit 103 exerts functions shown as various modules by the processor of the information processing device 10 performing processing according to a program. The control unit 103 includes a reception control unit 131, a transmission control unit 132, a learning unit 133, a display unit 134, a task processing unit 135, and the like.

[0049] The reception control unit 131 controls the processing in which the information processing device 10 transmits and receives signals to and from external devices in accordance with a communication protocol.

[0050] The transmission control unit 132 controls the process in which the information processing device 10 transmits a signal to an external device in accordance with a communication protocol.

[0051] The learning unit 133 learns a first trained model and a second trained model. The first trained model is a model that outputs the results of a task when data that is the subject of the task is input.

[0052] The second trained model is a model that outputs the likelihood of the task result by inputting the data input to the first trained model and the task result.

[0053] Here, we will explain the reason for adopting the second trained model. The first trained model generally cannot achieve 100% accuracy in task processing. In order to improve the accuracy of the task processing, for example, it is possible to adopt an approach that processes the same task or a first trained model that is different from the one used by the development company. Here, the different first trained models are referred to as the first engine and the second engine, respectively.

[0054] In this case, the recognition results of the independent first and second engines may or may not be the same for the input data. When the recognition results are not the same, one of them may be correct or may be incorrect, resulting in low reliability and the need for manual checking. On the other hand, when the recognition results are the same, both may be correct or may be incorrect. For this reason, even when two different first trained models are used, there are cases where manual checking is required since there is a possibility of an incorrect answer. In addition, this may result in cases where reliability cannot be guaranteed despite the high cost, such as when the two first trained models, the first engine and the second engine, are systems of different companies.

[0055] The technology disclosed herein uses one first trained model instead of two first trained models as described above. The technology disclosed herein then evaluates the reliability of the result of the first trained model using the second trained model. Specifically, the second trained model correctly evaluates the processing result in the first trained model by outputting the likelihood of the result of the task obtained by the first trained model.

[0056] By using this second trained model, it is possible to obtain a likelihood that correctly determines the reliability of the task result itself by the first trained model. This allows the user to appropriately set the likelihood threshold for the task result. For example, by appropriately setting the likelihood threshold, it is possible to ensure sufficient reliability while eliminating the need for human checking and subjecting the result to subsequent automatic processing. In addition, it is possible to reduce the cost of building a system that uses multiple first trained models and the cost of manual work.

[0057] These models can be any AI model, such as a neural network, a machine learning model such as SVM, or a language model (LLM (Large Language Model), SLM (Small Language Model)).

[0058] The learning unit 133 acquires a model corresponding to a given task (excluding a task for evaluating the result of the task of the first trained model) from the model DB 122. When there are multiple applicable models, the learning unit 133 may be configured to select, for example, the latest model. In addition, the learning unit 133 acquires data to be used for the task and, if necessary, a correct answer label from the training data DB 121. This is because the correct answer label may not be necessary depending on the task.

[0059] The learning unit 133 learns the first trained model using the acquired data and the correct label. Any learning method can be adopted depending on the model, task, etc. The learning unit 133 stores the updated first trained model in the model DB 122.

[0060] Furthermore, the learning unit 133 acquires a model (the second trained model before learning, or the trained second trained model) corresponding to a task for which an evaluation is performed on the result of the task of the first trained model from the model DB 122. When there are multiple corresponding models, the learning unit 133 may be configured to select, for example, the latest model. Furthermore, the learning unit 133 acquires, from the training data DB 121, data used for the task for which the evaluation is performed, a processing result, and a correct answer label.

[0061] The learning unit 133 uses the data used for the task, the processing result, and the correct answer label to learn the second trained model so as to output the likelihood for the result of the task by inputting the data to be input to the first trained model and the result of the task. Any method of learning may be adopted depending on the model, task, etc. The learning unit 133 stores the second trained model updated by learning in the model DB 122.

[0062] The learning unit 133 may be configured to automatically re-learn when an arbitrary condition is satisfied, using data obtained by performing task processing by the task processing unit 135 described later. The arbitrary condition may be, for example, that the task processing unit 135 has been executed a predetermined number of times.

[0063] Here, an example will be described in which the task is to read a document on which characters are handwritten using an image scanner or the like and to perform OCR (Optical Character Recognition) processing (character recognition processing).

[0064] In this case, the first trained model is a model that has been trained to output character recognition results when image data to be subjected to character recognition is input.

[0065] In addition, the second trained model is a model that receives image data and a character recognition result as input and outputs a likelihood for the character recognition result.

[0066] Furthermore, the data stored in the training data DB121 used for training the first trained model includes at least image data.

[0067] In addition, the data stored in the training data DB121 used for training the second trained model includes at least the result of character recognition.

[0068] The task is not limited to character recognition processing. Any task that requires the likelihood of a processing result to be expressed may be adopted. For example, image recognition, vibration recognition, meter indicator detection, anomaly detection, etc. may be adopted as the task. The first trained model may be designed so that the input data and output data are adapted to each task. For example, if the task is image recognition, the input data for the first trained model is image data, the output data that is the processing result of the task is the image recognition result (for example, detecting a person, recognizing a type of animal, detecting a vehicle, etc.), and the likelihood is a value indicating the reliability of the recognition result. Similarly, the second trained model may be designed according to the task. For example, if the task is image recognition, the input data for the second trained model is the image data and the image recognition result of the first trained model, and the output data is the likelihood for the recognition result of the image.

[0069] The display unit 134 causes the user terminal 20 to display various screens.

[0070] Specifically, the display unit 134 displays a predetermined interface on the user terminal 20. The predetermined interface is an interface that accepts input of input data for the user to perform task processing, an interface that displays the result of task processing, an interface that displays likelihood, etc. The display unit 134 transmits data related to the screen to be displayed (e.g., the result of the task, the likelihood, etc.) to the user terminal 20, thereby causing the user terminal 20 to display it.

[0071] The task processing unit 135 performs task processing using the first trained model for input data input by a user, and calculates the likelihood using the second trained model for the task processing result of the first trained model.

[0072] Specifically, the task processing unit 135 first acquires input data. The input data is acquired by being transmitted to the information processing device 10 from an interface that accepts input of input data displayed on the user terminal 20.

[0073] The task processing unit 135 obtains a result of the task by inputting the input data to the first trained model.

[0074] The task processing unit 135 inputs the input data and the task result into the second trained model, thereby obtaining the likelihood for the task result.

[0075] Then, the task processing unit 135 passes the result of the task processing and the likelihood to the display unit 134, thereby causing the user terminal 20 to display the processing result.

[0076] <Functional Configuration of User Terminal 20> Fig. 5 is a diagram showing the functional configuration of the user terminal 20. As shown in Fig. 5, the user terminal 20 fulfills the functions of a communication unit 201, a storage unit 202, and a control unit 203.

[0077] The communication unit 201 performs processing for the user terminal 20 to communicate with an external device.

[0078] The storage unit 202 stores data and programs used by the user terminal 20.

[0079] The control unit 203 performs functions shown as various modules by the processor of the user terminal 20 performing processes according to the programs. The control unit 203 performs functions as a reception control unit 231, a transmission control unit 232, and a display unit 233.

[0080] The reception control unit 231 controls the processing in which the user terminal 20 transmits and receives signals from external devices in accordance with a communication protocol.

[0081] The transmission control unit 232 controls the process in which the user terminal 20 transmits a signal to an external device in accordance with a communication protocol.

[0082] The display unit 233 displays the information received from the information processing device 10. Specifically, the display unit 233 displays a screen displaying various interfaces received from the information processing device 10 on the output device 24 of the user terminal 20.

[0083] The display unit 233 causes the transmission control unit 232 to transmit the input data to the information processing device 10 in response to input of input data by a user to an interface that accepts input of the input data for performing task processing.

[0084] <Operation> An operation of the information processing system according to the first embodiment will be described below. Fig. 6 is a flowchart illustrating a learning process of the first trained model.

[0085] In step S601, the learning unit 133 acquires data to be used for the task from the learning data DB 121, and a correct label as necessary.

[0086] In step S603, the learning unit 133 learns a first trained model using the acquired data, the processing result, and the correct label.

[0087] In step S605, the learning unit 133 stores the updated first trained model in the model DB 122.

[0088] FIG. 7 is a flowchart illustrating a learning process of the second trained model in the first embodiment.

[0089] In step S701, the learning unit 133 acquires, from the learning data DB 121, data to be used for a task of evaluating the result of the task of the first trained model, a processing result, and a correct answer label.

[0090] In step S703, the learning unit 133 uses the data used in a task to evaluate the result of the task of the first trained model, the processing result, and the correct answer label to input data to be input into the first trained model and the result of the task, thereby training the second trained model to output the likelihood for the result of the task.

[0091] In step S705, the learning unit 133 stores the updated second trained model in the model DB 122.

[0092] FIG. 8 is a flowchart illustrating task processing according to the first embodiment.

[0093] In step S801, the display unit 233 of the user terminal 20 displays a screen.

[0094] In step S803, the user terminal 20 accepts input of data.

[0095] In step S805, the user terminal 20 transmits the acquired data to the information processing device 10.

[0096] In step S807, the information processing device 10 receives data from the user terminal 20.

[0097] In step S809, the task processing unit 135 of the information processing device 10 obtains a result of character recognition by inputting data into the first trained model.

[0098] In step S811, the task processing unit 135 of the information processing device 10 inputs the data and the character recognition result to the second trained model, thereby obtaining the likelihood for the task result according to the first trained model.

[0099] In step S813, the information processing device 10 transmits the task result and the likelihood to the user terminal 20.

[0100] In step S815, the user terminal 20 receives the task result and the likelihood.

[0101] In step S815, the user terminal 20 displays the task result and the likelihood.

[0102] <Summary> As described above, the technology disclosed herein accepts input of data to be the subject of a predetermined task, and when the data is input, the first trained model outputs the result of the task, and obtains the result of the task using the input data. The technology disclosed herein inputs the input data and the result of the task, and outputs the likelihood using the input data and the result of the task to obtain the likelihood, and outputs the result of the task and the likelihood. According to the technology disclosed herein, the likelihood is obtained by the second trained model for the result of the task by the first trained model. This makes it possible to evaluate the accuracy of the task by the first trained model. Therefore, it is possible to set an appropriate likelihood threshold in the processing of the task using the first trained model based on the likelihood and the result of the task.

[0103] (Modification) Although several embodiments of the present disclosure have been described above, these embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are within the scope of the invention and its equivalents as described in the claims, as well as the scope and spirit of the invention.

[0104] (Variation 1) In addition, in the above embodiment, an example was described in which the second trained model inputs the data input to the first trained model and the result of the task by the first trained model, and outputs the likelihood of the result of the task by the first trained model, but this is not limited to this. For example, the second trained model can be configured to output the likelihood of the result of the task by the first trained model by inputting features related to the task of the input data and the result of the task.

[0105] In this case, the learning unit 133 may extract features related to the task from the acquired data and learn the second trained model. The learning unit 133 may acquire features related to the task stored in advance in the training data DB 121. The training data DB 121 may include context data related to the task in addition to input data for the target task. The context data is, for example, the features of the input data, data obtained by performing any conversion on the input data, and data to be referred to when performing processing related to the task. For example, if the task is to perform character recognition from image data, the context data is, for example, the features of the image data, the features of a part of the image data, the features of the other image data when the image data is a part of other image data, and image data in which the image quality, brightness, etc. are changed. The features of the image data may be calculated, for example, from the contents written and the importance of the items by natural language analysis. The part of the image data may be, for example, a part of the image data related to an item to be read by character recognition. The features of the part of the image data may be, for example, the features of an image of an item unit of a form, and may be calculated from the image quality of the image, the relative position in the entire form, and character information. In addition, when the task is character recognition, the first trained model may output character recognition results for the entire image data, or may output character recognition results for each item contained in the image data.

[0106] Furthermore, the task processing unit 135 may extract features related to the task from the input data. Then, the task processing unit 135 inputs the features related to the task of the input data and the result of the task to the second trained model, thereby obtaining the likelihood of the result of the task according to the first trained model.

[0107] By inputting features related to the task in the input data and the result of the task by the first trained model, the second trained model can take into account the context of the task, thereby making it possible to accurately calculate the likelihood of the result of the task by the first trained model.

[0108] The second trained model may output the likelihood of the result of the task by the first trained model by inputting the input data, context data related to the task of the input data, and the result of the task. The context data may be a feature of the input data, or other context data other than the feature of the input data. In this case, the task processing unit 135 inputs the input data, the context data related to the task of the input data, and the result of the task into the second trained model to obtain the likelihood of the result of the task by the first trained model.

[0109] The first trained model will express a uniform likelihood regardless of the context of the task. The likelihood threshold needs to be set individually according to the importance of the task. However, in the past, it was not possible to flexibly adjust the likelihood threshold of the first trained model by taking into account the importance of the task. As a result, it was not possible to set an appropriate likelihood threshold. When using a second trained model that inputs such context data, the input data, the context data related to the task of the input data, and the result of the task by the first trained model are input, so that the context of the task can be taken into consideration in addition to the input data. Therefore, in the technology disclosed herein, the second trained model can evaluate the result of the task by the first trained model by taking into account the context of the task, and can accurately calculate the likelihood. This makes it possible to set an appropriate likelihood threshold.

[0110] In addition, the second trained model may be configured to output a likelihood for the result of the task by the first trained model by inputting input data, features related to the task of the input data, context data other than the features related to the task of the input data, and the result of the task.

[0111] (Variation 2) In addition, the first trained model may output a first likelihood, which is the likelihood for the result of the task of the first trained model itself.

[0112] In this case, the second trained model receives the data input to the first trained model, the result of the task by the first trained model, and the first likelihood output by the first trained model, and outputs a second likelihood, which is the likelihood for the result of the task by the first trained model.

[0113] In this case, the learning unit 133 uses the data used for the task, the processing result, the likelihood, and the correct answer label to input the data to be input into the first trained model, the result of the task, and the first likelihood, and learns the second trained model to output the likelihood for the result of the task.

[0114] In addition, the task processing unit 135 inputs the input data, the task result, and the first likelihood into a second trained model, thereby obtaining a second likelihood for the task result from the first trained model.

[0115] As a result, the second trained model evaluates the result of the task by the first trained model by taking into account the first likelihood obtained by the first trained model itself. This makes it possible to accurately calculate the likelihood (second likelihood) for the result of the task by the first trained model.

[0116] Note that Modification 1 and Modification 2 may be combined. Specifically, the second trained model may be configured to output the second likelihood by inputting features related to the task of the data, the result of the task, and the first likelihood, or may be configured to output the second likelihood by inputting data input to the first trained model, features related to the task of the data, the result of the task, and the first likelihood.

[0117] (Variation 3) In addition, when the first trained model outputs a first likelihood, which is a likelihood for the result of a task of the first trained model itself, the second trained model may be configured to output a score for correcting the first likelihood for the result of the task by the first trained model by inputting the result of the task by the first trained model and the first likelihood output by the first trained model.

[0118] In this case, the learning unit 133 uses the processing result, the likelihood, and the correct label to learn the second trained model so as to output a score for correcting the likelihood by inputting the task result and the likelihood.

[0119] Furthermore, the task processing unit 135 performs task processing using the first trained model for input data input by the user, and obtains a score for correcting the likelihood using the second trained model for the task processing result of the first trained model. Specifically, the task processing unit 135 obtains the task result and the first likelihood for the task result by inputting the input data to the first trained model. The task processing unit 135 obtains the score for correcting the first likelihood by inputting the task processing result and the first likelihood to the second trained model.

[0120] The task processing unit 135 obtains a value calculated using the likelihood and the score. Depending on the nature of the score output by the second trained model, the task processing unit 135 obtains, for example, a sum obtained by adding the likelihood and the score, or a value obtained by a statistical calculation method (weighted average, variance, etc.) of the likelihood and the score. When the sum is used, the task processing unit 135 obtains the ratio of the likelihood to the score in the sum.

[0121] The display unit 134 may display the sum of the likelihood and the score in an interface that displays the likelihood and the score, or may display the sum of the likelihood and the score in a manner that allows the ratio of the likelihood to the score in the sum to be understood.

[0122] The operation of this modified example will be described below. Fig. 9 is a flowchart illustrating task processing. Note that Fig. 9 illustrates an example in which the task is character recognition.

[0123] In step S903, the user terminal 20 accepts input of image data obtained by reading a form using a scanner device or the like through user operation.

[0124] In step S905, the user terminal 20 transmits the acquired image data to the information processing device 10.

[0125] In step S 907 , the information processing device 10 receives the image data from the user terminal 20 .

[0126] In step S909, the task processing unit 135 of the information processing device 10 inputs the image data to the first trained model to obtain a character recognition result and a likelihood for the character recognition result.

[0127] In step S911, the task processing unit 135 of the information processing device 10 inputs the result of character recognition and the likelihood to the second trained model, thereby obtaining a score for correcting the likelihood.

[0128] In step S913, the task processing unit 135 of the information processing device 10 calculates a sum by adding the likelihood and the score. The task processing unit 135 also calculates the ratio of the likelihood to the score in the sum.

[0129] In step S915, the information processing device 10 transmits the character recognition result, the likelihood, the score, the additional value, and the ratio to the user terminal 20.

[0130] In step S917, the user terminal 20 receives the character recognition result, the likelihood, the score, the additional value, and the ratio.

[0131] In step S919, the user terminal 20 displays the character recognition result, the likelihood, the score, the sum, and the ratio.

[0132] An example of a screen used in the information processing system of this modified example will be described below. Fig. 10 is a diagram showing an example of a screen displayed on user terminal 20 of this modified example.

[0133] A description will be given of examples of screens used in the information processing system of the first embodiment. Fig. 10 is a diagram showing an example of a screen displayed on the user terminal 20 of this modified example.

[0134] The screen in FIG. 10 includes a display unit 1001, a display unit 1002, and a display unit 1003.

[0135] The display unit 1001 is an interface that accepts input of input data for performing task processing by a user. For example, when the task is character recognition, the display unit 1001 is configured to allow input of image data obtained by reading a form using a scanner device or the like.

[0136] The display unit 1002 is an interface that displays the results of task processing. For example, when the task processing is character recognition, the display unit 902 displays the results of character recognition.

[0137] The display unit 1003 is an interface that displays the likelihood. Specifically, the display unit 1003 displays the likelihood and the score. As shown in FIG. 10, the display unit 1003 displays the added value of the likelihood and the score as the likelihood. *The display unit 1003 displays the likelihood and the score so that the ratio of the likelihood and the score can be understood. For example, when 0.00 to 1.00 is adopted as the likelihood, the display unit 1003 may display the added value of 0.84 as the likelihood in an easily understandable manner using a bar graph, as shown in FIG. 10. In addition, the likelihood (shaded part in the example of FIG. 10) and the score (black part in the example of FIG. 10) may be displayed in different colors or shapes so that the ratio of the likelihood obtained by the first trained model (0.65 in the example of FIG. 10) and the score obtained by the second trained model (0.19 in the example of FIG. 10) can be understood. By expressing in this way, it is possible to make it possible for a human being to intuitively determine which likelihood should be set based on not only the likelihood represented only by the first trained model but also the corrected likelihood.

[0138] According to the technology of this modification, a first trained model accepts input of data to be the subject of a predetermined task, and when the data to be the subject of the task is input, the result of the task and the likelihood for the result of the task are output, and the result of the task and the likelihood are obtained using the input data. The technology of the present disclosure inputs the result of the task and the likelihood, and outputs a score for correcting the likelihood, and uses the result of the task processing and the likelihood to obtain the score, and outputs the result of the task, the likelihood, and the score. According to the technology of the present disclosure, it is possible to set an appropriate likelihood threshold in task processing using a trained model. Specifically, the technology of the present disclosure displays the likelihood for the processing result of the first trained model in a corrected form. The second trained model corrects the likelihood for the processing result of the first trained model, making it easier for the user to set the likelihood threshold than when the judgment is made only by the first trained model. In addition, after the user sets the likelihood threshold, the effect of being able to automatically entrust task processing is also obtained.

[0139] In addition, the technology disclosed herein may be such that the task is character recognition, the input data is image data to be character recognized, and the likelihood is a likelihood for a result of the character recognition. According to the technology disclosed herein, in a character recognition process using a trained model, a likelihood for a result of the character recognition and a score for correcting the likelihood are output. According to the technology disclosed herein, it is possible to set an appropriate likelihood threshold value.

[0140] In addition, the technology disclosed herein outputs a value obtained by adding the likelihood and the score together with the result of character recognition so that the ratio of the likelihood and the score in the added value can be known. This allows the user to visually check the ratio between the likelihood for the processing result of the first trained model and the score for correcting the likelihood. According to the technology disclosed herein, the user can visually check the ratio between the likelihood in the first trained model and the corrected score, so that an appropriate likelihood threshold can be set.

[0141] The second trained model may be configured to output the second likelihood by inputting the input data, context data related to the task of the input data, the result of the task, and the first likelihood. The context data may be a feature of the input data, context data other than the feature of the input data, etc. In this case, the task processing unit 135 inputs the input data, the context data related to the task of the input data, the result of the task, and the first likelihood to the second trained model, thereby obtaining the likelihood for the result of the task by the first trained model. When such a second trained model is used, by inputting input data, context data related to the task of the input data, the result of the task using the first trained model, and the first likelihood, it is possible to take into account the context of the task in addition to the input data, the result of the task, and the first likelihood, thereby making it possible to accurately determine the likelihood for the result of the task using the first trained model. In addition, the second trained model may be configured to output a likelihood for the result of the task according to the first trained model by inputting input data, features related to the task of the input data, context data other than the features related to the task of the input data, the result of the task, and the first likelihood.

[0142] (Variation 4) In the above embodiment, the second trained model outputs the likelihood for the output result of the task of the first trained model itself, but the present invention is not limited to this. The second trained model can be configured to output the likelihood for a predetermined process using the output result of the first trained model.

[0143] Specifically, the second trained model can be a model that outputs the likelihood of a specified process using the results of a task by inputting input data, the results of a task using the first trained model, and context data related to the processing of the task.

[0144] In this case, the task processing unit 135 may be configured to execute a predetermined process using the result of the task based on the first trained model, the likelihood, and the likelihood based on the second trained model.

[0145] For example, it is possible that the task of the first trained model is character recognition and the predetermined process is a registration process of a predetermined form. In this case, the second trained model uses data used in the predetermined process (feature amounts of the predetermined form, items to be registered for the predetermined form, data format, other data to be referenced when performing the registration process, etc.) as context data.

[0146] In this case, the learning unit 133 learns the second trained model to output the likelihood of a predetermined process using the result of the task by inputting the input data, the result of the task by the first trained model, and context data related to the processing of the task. The learning unit 133 trains the second trained model using, for example, the result and likelihood of the task, the context data related to the processing of the task, and the correct answer label.

[0147] In this case, the task processing unit 135 may be configured to execute the registration process of the form using the input data, the result of character recognition by the first trained model, and the likelihood by the second trained model.

[0148] The combination of the task of the first trained model and the predetermined processing is not limited to the combination of character recognition and document registration processing. If the task is character recognition, other combinations are also possible. The predetermined processing can be, for example, a quality control process, a manufacturing record process, or the like, using all or part of a report, design document, RFP, requirements specification document, basic design document, organizational chart, profile of person in charge, work situation, or the like as input data. If the task is image recognition, the predetermined processing can be obstacle avoidance in autonomous driving, face recognition, or the like. In this way, the second trained model of the present disclosure can be used to derive the likelihood for various combinations of tasks and processing. By setting a likelihood threshold to be set for the processing based on the likelihood output by the second trained model according to the modified example, the predetermined processing can be automated with high accuracy.

[0149] (Variation 5) It can also be used when the first trained model is a generative AI. For example, an SLM can be used for the first trained model. For example, assume that the task of the SLM is to extract characters for a predetermined item included in image data of a predetermined form. In this case, the task processing unit 135 generates a first prompt including an instruction to extract characters for the predetermined item from the image data. The task processing unit 135 inputs the first prompt to the first trained model to obtain a result of extracting the characters.

[0150] Moreover, the learning unit 133 may use a learning method suitable for SLM as a learning method for the first trained model. Moreover, the second trained model may be trained so as to output the likelihood for the result of extracting the character by inputting image data and the character obtained by the first trained model.

[0151] Since SLM has fewer parameters than LLM, it can reduce computational resources and improve task processing speed compared to when LLM is used, but the accuracy may be inferior. As described above, by using SLM as the first trained model and correcting the likelihood using the second trained model according to the technology of the present disclosure, it is possible to ensure the accuracy of the task while reducing computational resources and improving the processing speed compared to when only LLM is used. Therefore, by using SLM as the first trained model and then calculating the likelihood using the second trained model, it is possible to evaluate the first trained model and appropriately set the likelihood threshold, rather than performing character recognition using only LLM.

[0152] In this case, the task processing unit 135 may be configured to generate a second prompt that instructs the extraction of characters for a predetermined item included in the image data of a predetermined form from the input data input to the first trained model according to the likelihood. The task processing unit 135 generates the second prompt, for example, when the likelihood is smaller than a predetermined threshold or is not within a reliable range. In this case, the task processing unit 135 inputs the second prompt to the LLM to obtain the result of extracting the characters. As a result of the processing, the task processing unit 135 outputs the result by the first trained model and the result by using the LLM, or outputs the result by using the LLM. This makes it possible to ensure the accuracy of task processing using the LLM even when the likelihood by the SLM is low.

[0153] Furthermore, the task processing unit 135 may be configured to acquire an evaluation result from a user when outputting a result using the first trained model and a result using the LLM as processing results. In this case, the task processing unit 135 stores the input data, the task processing result, the likelihood, the score, and the acquired evaluation result in the training data DB 121. The training unit 133 retrains the first trained model at any timing using the input data, the task processing result, the likelihood, the score, and the acquired evaluation result.

[0154] With this configuration, the accuracy of SLM can be improved to approach that of LLM, while requiring fewer computational resources than LLM and improving processing speed.

[0155] (Other variations) In each of the above modified examples, context data that cannot be extracted from the input data can also be used. In this case, a configuration may be adopted in which the context data that cannot be extracted from the input data is received from the user terminal 20 and accepted, or the context data is stored in advance in the learning data DB 121. The context data that cannot be extracted from the input data is, for example, a feature value for the entire data when the input data is a part of the entire data (a part of a plurality of data, a part of image data, etc.), or data obtained by performing a conversion process on the entire data. In addition, the entire data itself may be received from the user terminal 20 and may be stored in advance in the learning data DB 121. In this case, a configuration may be adopted in which the task processing unit 135 extracts the context data from the entire data.

[0156] <Basic computer hardware configuration> 11 is a block diagram showing a basic hardware configuration of a computer 1400. The computer 1400 includes at least a processor 1401, a main memory device 1402, an auxiliary memory device 1403, a communication IF 1404 (interface), and an input / output IF 1405. These are electrically connected to each other by a communication bus 1406.

[0157] The processor 1401 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0158] The main memory device 1402 is for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0159] The auxiliary storage device 1403 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0160] The communication IF 1404 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet via a specified access point (e.g., Wi-Fi (registered trademark)), etc. In the case of wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of wired connection, the network also includes a network that is directly connected by a USB (Universal Serial Bus) cable or the like.

[0161] It should be noted that the computer 1400 can be virtually realized by distributing all or part of each hardware configuration among multiple computers 1400 and connecting them to each other via a network. In this way, the computer 1400 is a concept that includes not only a computer 1400 housed in a single housing or case, but also a virtualized computer system.

[0162] <Basic Functional Configuration of Computer 1400> A description will now be given of the functional configuration of a computer realized by the basic hardware configuration (FIG. 10) of the computer 1400. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0163] The functional units of the computer 1400 can also be realized by distributing all or part of the functional units among a plurality of computers 1400 connected to each other via a network. The computer 1400 is a concept that includes not only a single computer 1400 but also a virtualized computer system.

[0164] The control unit is realized by the processor 1401 reading out various programs stored in the auxiliary storage device 1403, expanding the programs in the main storage device 1402, and executing processes according to the programs. The control unit can realize functional units that perform various information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0165] The storage unit is realized by a main storage device 1402 and an auxiliary storage device 1403. The storage unit stores data, various programs, and various databases. Furthermore, the processor 1401 can secure a storage area corresponding to the storage unit in the main storage device 1402 or the auxiliary storage device 1403 in accordance with a program. Furthermore, the control unit can cause the processor 1401 to execute processing for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0166] The term database refers to a relational database, which is used to manage sets of data called masters and tables in a tabular format structurally defined by rows and columns, by associating them with each other. In a database, a table is called a table or master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Usually, a column that serves as a primary key for uniquely identifying a record is set in each table and each master, but setting a primary key in a column is not essential. The control unit can cause the processor 1301 to add, delete, or update records in a specific table or master stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0167] In addition, the database and master in this disclosure may include any data structure (such as a list, a dictionary, an associative array, or an object) in which information is structurally defined. The data structure also includes data that can be considered as a data structure by combining data with a function, class, method, or the like written in any programming language.

[0168] The communication unit is realized by the communication IF 1404. The communication unit realizes a function of communicating with other computers 1400 via a network. The communication unit can receive information transmitted from other computers 1400 and input the information to the control unit. The control unit can cause the processor 1401 to execute information processing on the received information according to various programs. In addition, the communication unit can transmit information output from the control unit to other computers 1400.

[0169] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0170] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Java (registered trademark), and the like.

[0171] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.

[0172] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general purpose processors, application specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the functions described. Processors include transistors and other circuits and are considered to be circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in a memory. In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed in this specification or any hardware known to be programmed to realize or perform the described functions. If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0173] Although several embodiments of the present disclosure have been described above, these embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are within the scope of the invention and its equivalents as described in the claims, as well as the scope and spirit of the invention.

[0174] (Additional Note) The matters described in the above embodiments will be supplemented below.

[0175] (Appendix 1) A program for causing a computer having a processor and a memory to execute the program, the program causing the processor to: A step (S807) of accepting input of data to be subjected to a predetermined task; A step (S809) of obtaining a result of the task using a first trained model that outputs a result of the task when the data is input and the input data; A second trained model that outputs a likelihood for the result of the task by the first trained model by inputting the input data and the result of the task, and a step (S811) of calculating the likelihood using the input data and the result of the task; outputting the result of the task and the likelihood (S815); A program that executes the following.

[0176] (Appendix 2) the second trained model outputs a likelihood of the result of the task by the first trained model by inputting a feature related to the task of the data and a result of the task; In the step of calculating the likelihood, the likelihood is calculated using the second trained model, the feature amount, and a result of the task. The program described in (Appendix 1).

[0177] (Appendix 3) In the step of receiving an input, an input of the data and context data relating to the task that cannot be extracted from the data is received; The second trained model receives the input context data and a result of the task, and outputs a likelihood of the result of the task by the first trained model; In the step of calculating the likelihood, the likelihood is calculated using the second trained model, the input context data, and a result of the task. The program described in (Appendix 1).

[0178] (Appendix 4) the second trained model receives the data, a feature related to the task of the input data, and a result of the task, and outputs a likelihood of the result of the task by the first trained model; In the step of calculating the likelihood, the likelihood is calculated using the second trained model, the input data, the feature amount, and a result of the task. The program described in (Appendix 1).

[0179] (Appendix 5) In the step of receiving an input, an input of the data and context data relating to the task that cannot be extracted from the data is received; The second trained model receives the data, the input context data, and the result of the task, and outputs a likelihood of the result of the task by the first trained model; In the step of calculating the likelihood, the likelihood is calculated using the second trained model, the input data, the input context data, and a result of the task. The program described in (Appendix 1).

[0180] (Appendix 6) When the first trained model receives the data, the first trained model outputs a result of the task and a first likelihood for the result of the task; the second trained model inputs the data, the result of the task, and the first likelihood output by the first trained model, and outputs a second likelihood for the result of the task; In the step of obtaining a result of the task, the result of the task and the first likelihood are obtained using the first trained model and the input data; In the step of calculating the likelihood, the second likelihood is calculated using the second trained model, the input data, a result of the task, and a first likelihood output by the first trained model; In the outputting step, the result of the task and the second likelihood are outputted. The program described in (Appendix 1).

[0181] (Appendix 7) the second trained model outputs the second likelihood by inputting a feature related to the task of the data, a result of the task, and the first likelihood; In the step of calculating the likelihood, the second likelihood is calculated using the second trained model, the feature amount, a result of the task, and the first likelihood. (Appendix 6) The program described in.

[0182] (Appendix 8) In the step of receiving an input, an input of the data and context data relating to the task that cannot be extracted from the data is received; the second trained model outputs the second likelihood by inputting the input context data, the result of the task, and the first likelihood; In the step of calculating the likelihood, the second likelihood is calculated using the second trained model, the input context data, a result of the task, and the first likelihood. (Appendix 6) The program described in.

[0183] (Appendix 9) the second trained model outputs the second likelihood by inputting the data, a feature of the data related to the task, a result of the task, and the first likelihood; In the step of calculating the likelihood, the second likelihood is calculated using the second trained model, the input data, the feature amount, a result of the task, and the first likelihood. (Appendix 6) The program described in.

[0184] (Appendix 10) In the step of receiving an input, an input of the data and context data relating to the task that cannot be extracted from the data is received; the second trained model outputs the second likelihood by inputting the data, the input context data, a result of the task, and the first likelihood; In the step of calculating the likelihood, the second likelihood is calculated using the second trained model, the input data, the input context data, a result of the task, and the first likelihood. (Appendix 6) The program described in.

[0185] (Appendix 11) A program for causing a computer having a processor and a memory to execute the program, the program causing the processor to: A step (S907) of accepting input of data to be subjected to a predetermined task; a step (S909) of obtaining a result of the task and the first likelihood by using a first trained model that outputs a result of the task and a first likelihood for the result of the task when the data is input, and the input data; a second trained model that outputs a score for correcting the first likelihood by inputting the data, the result of the task, and the first likelihood output by the first trained model; and a step (S911) of calculating the score using the input data, the result of the task, and the first likelihood; outputting the task result, the first likelihood, and the score (S913); A program that executes the following.

[0186] (Appendix 12) In the outputting step, a value calculated using the first likelihood and the score is output as a likelihood for a result of the task together with the result of the task. (Appendix 11) The program described in.

[0187] (Appendix 13) In the outputting step, a value obtained by adding the first likelihood and the score is output as a likelihood for the result of the task, together with the result of the task, so that a ratio between the likelihood and the score in the added value can be known. (Appendix 11) The program described in.

[0188] (Appendix 14) the task is character recognition, the data is image data to be subjected to character recognition, The likelihood is a likelihood for a result of the character recognition. The program described in (Appendix 1).

[0189] (Appendix 15) An information processing device including a processor and a memory, the processor comprising: A step (S807) of accepting input of data to be subjected to a predetermined task; A step (S809) of obtaining a result of the task using a first trained model that outputs a result of the task when the data is input and the input data; A second trained model that outputs a likelihood for the result of the task by the first trained model by inputting the input data and the result of the task, and a step (S811) of calculating the likelihood using the input data and the result of the task; outputting the result of the task and the likelihood (S815); An information processing device that executes the above.

[0190] (Appendix 16) 1. A computer-implemented method comprising a processor and a memory, the processor comprising: A step (S807) of accepting input of data to be subjected to a predetermined task; A step (S809) of obtaining a result of the task using a first trained model that outputs a result of the task when the data is input and the input data; A second trained model that outputs a likelihood for the result of the task by the first trained model by inputting the input data and the result of the task, and a step (S811) of calculating the likelihood using the input data and the result of the task; outputting the result of the task and the likelihood (S815); How to do it. [Explanation of symbols]

[0191] 1 Information processing system, 10 Information processing device, 12 Communication IF, 13 Input / output IF, 15 Memory, 16 Storage, 20 User terminal, 22 Communication IF, 23 Input / output IF, 24 Output device, 25 Memory, 26 Storage, 80 Network, 101 Communication unit, 102 Memory unit, 103 Control unit, 121 Learning data DB, 122 Model DB, 131 Reception control unit, 132 Transmission control unit, 133 Learning unit, 134 Display unit, 135 Task processing unit, 201 Communication unit, 202 Memory unit, 203 Control unit, 231 Reception control unit, 232 Transmission control unit, 233 Display unit, 234 Registration unit, 235 Notification unit, 236 Display unit, 1400 Computer, 1401 Processor, 1402 Main memory unit, 1403 Auxiliary memory unit, 1404 Communication IF, 1405 Input / Output IF, 1406 communication bus.

Claims

1. A program for causing a computer having a processor and a memory to execute the program, the program causing the processor to: accepting input of data to be subjected to a predetermined task; a step of obtaining a result of the task and the first likelihood by using a first trained model that outputs a result of the task and a first likelihood for the result of the task when the data is input, and the input data; a second trained model that receives the input data, the result of the task, and the first likelihood output by the first trained model, and outputs a second likelihood for the result of the task by the first trained model; and a step of calculating the second likelihood using the input data, the result of the task, and the first likelihood output by the first trained model; outputting the result of the task and the second likelihood; A program that executes the following.

2. the second trained model outputs the second likelihood by inputting a feature related to the task of the data, a result of the task, and the first likelihood; In the step of calculating the second likelihood, the second likelihood is calculated using the second trained model, the feature amount, a result of the task, and the first likelihood. The program according to claim 1.

3. In the step of receiving an input, an input of the data and context data relating to the task that cannot be extracted from the data is received; The second trained model outputs the second likelihood by inputting the input context data, the result of the task, and the first likelihood; In the step of calculating the second likelihood, the second likelihood is calculated using the second trained model, the input context data, a result of the task, and the first likelihood. The program according to claim 1.

4. the second trained model outputs the second likelihood by inputting the data, a feature of the data related to the task, a result of the task, and the first likelihood; In the step of calculating the second likelihood, the second likelihood is calculated using the second trained model, the input data, the feature amount, a result of the task, and the first likelihood. The program according to claim 1.

5. In the step of receiving an input, an input of the data and context data relating to the task that cannot be extracted from the data is received; The second trained model outputs the second likelihood by inputting the data, the input context data, a result of the task, and the first likelihood; In the step of calculating the second likelihood, the second likelihood is calculated using the second trained model, the input data, the input context data, a result of the task, and the first likelihood. The program according to claim 1.

6. A program for causing a computer having a processor and a memory to execute the program, the program causing the processor to: accepting input of data to be subjected to a predetermined task; a step of obtaining a result of the task and the first likelihood by using a first trained model that outputs a result of the task and a first likelihood for the result of the task when the data is input, and the input data; a second trained model that outputs a score for correcting the first likelihood by inputting a result of the task and the first likelihood output by the first trained model, and a step of calculating the score using the result of the task and the first likelihood; outputting the task result, the first likelihood, and the score; A program that executes the following.

7. In the outputting step, a value calculated using the first likelihood and the score is output as a likelihood for a result of the task together with the result of the task. The program according to claim 6.

8. In the outputting step, a value obtained by adding the first likelihood and the score is output as a likelihood for the result of the task, together with the result of the task, so that a ratio between the likelihood and the score in the added value can be known. The program according to claim 6.

9. the task is character recognition; the data is image data to be subjected to character recognition, the first likelihood and the second likelihood are likelihoods for a result of the character recognition. The program according to claim 1.

10. An information processing device including a processor and a memory, the processor comprising: accepting input of data to be subjected to a predetermined task; a step of obtaining a result of the task and the first likelihood by using a first trained model that outputs a result of the task and a first likelihood for the result of the task when the data is input, and the input data; a second trained model that receives the input data, the result of the task, and the first likelihood output by the first trained model, and outputs a second likelihood for the result of the task by the first trained model; and a step of calculating the second likelihood using the input data, the result of the task, and the first likelihood output by the first trained model; outputting the result of the task and the second likelihood; An information processing device that executes the above.

11. 1. A computer-implemented method comprising a processor and a memory, the processor comprising: accepting input of data to be subjected to a predetermined task; a step of obtaining a result of the task and the first likelihood by using a first trained model that outputs a result of the task and a first likelihood for the result of the task when the data is input, and the input data; a second trained model that receives the input data, the result of the task, and the first likelihood output by the first trained model, and outputs a second likelihood for the result of the task by the first trained model; and a step of calculating the second likelihood using the input data, the result of the task, and the first likelihood output by the first trained model; outputting the result of the task and the second likelihood; How to do it.

Images