Information processing apparatus, information processing method, information processing system, and program

Abstract

To select a learned model appropriate for a task from among a plurality of learned models.SOLUTION: An information processing apparatus processes a predetermined task by inference using a learned model. The information processing apparatus includes: acquisition means for acquiring a pair of input data on the predetermined task and correct answer data corresponding to the input data as feedback information; inference means for performing inference for the predetermined task by inputting the input data constituting the pair related to the acquired feedback information to each of a plurality of learned models prepared in advance corresponding to the predetermined task; and identification means for identifying a learned model suitable for processing the predetermined task from among the plurality of learned models on the basis of a plurality of inference results obtained by the inference and the correct answer data constituting the pair with the input data input to each of the plurality of learned models.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a technology for processing tasks on a cloud. [Background technology]

[0002] There is a technology that processes a predetermined task, such as a face detection task for detecting a face in an image, by using a trained model existing on a cloud. This type of technology includes a technology that processes a task by using a trained model dedicated to a user that is constructed by additionally training the trained model with custom data specific to the user.

[0003] Regarding building a user-specific trained model, Patent Document 1 discloses a technique for training a custom machine-learned model based at least in part on custom data associated with a computer application. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special table 2021-521505 publication Summary of the Invention [Problem to be solved by the invention]

[0005] However, there are multiple trained models used in a certain task due to differences in versions, generation of variants of trained models dedicated to a user by additionally training custom data specific to the user, etc. When multiple trained models exist in this way, it is difficult for a user to select an appropriate trained model for the task to be processed from the multiple trained models. In this respect, the method of Patent Document 1 may also have a similar problem.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and provides a technology for selecting an appropriate trained model for a task from among multiple trained models based on differences in versions, subspecies, etc. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present disclosure is an information processing device that processes a specified task by inference using a learned model, and is characterized in having: an acquisition means for acquiring a pair of input data for the specified task and supervised data corresponding to the input data as feedback information; an inference means for inputting the input data that constitutes the pair related to the acquired feedback information into each of a plurality of pre-prepared learned models corresponding to the specified task, and performing inference for the specified task; and an identification means for identifying a learned model suitable for processing the specified task from the plurality of learned models based on a plurality of inference results obtained by the inference and the supervised data that constitutes a pair with the input data input to each of the plurality of learned models. Effect of the Invention

[0008] According to the technology disclosed herein, it is possible to select a trained model that is appropriate for a task from among multiple trained models. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device on a virtual server side. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of a client-side information processing device. [Figure 4] 1 is a block diagram illustrating an example of a functional configuration of an information processing system. [Diagram 5] 4 is a flowchart showing a flow of processing executed by the information processing system. [Figure 6]FIG. 11 is a diagram showing an example of a UI screen displayed by a display unit. [Figure 7] FIG. 4 is a diagram illustrating an example of data managed by a data management unit. [Figure 8] 13 is a flowchart showing a detailed flow of an inference process. [Figure 9] 11 is a flowchart showing the flow of a process for automatically selecting an appropriate combination of a model and a custom reasoner. [Figure 10] 1 is a block diagram illustrating an example of a functional configuration of an information processing system. [Figure 11] 13 is a flowchart showing the flow of a model learning process. [Figure 12] 13 is a flowchart showing the flow of a learning process of a custom inference device. [Figure 13] 13 is a flowchart showing the flow of a model organizing process. [Figure 14] 13 is a flowchart showing the flow of a custom reasoner organization process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments for carrying out the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Note that the same configurations will be described with the same reference numerals. Also, each process (step) in the flow chart will be denoted with an "S" at the beginning.

[0011] <<Embodiment 1>> In this embodiment, a feedback from a user on a client terminal on an inference result obtained on a cloud server is stored on the server and utilized to automatically select an appropriate model for a task from among a plurality of models. Note that the task is a task used by a user.

[0012] Tasks that can be handled in this embodiment include, for example, a face detection / object detection task in which an image is input, an entity extraction task in which a document image is input, and a voice recognition task in which a voice is input. The "face detection / object detection task" is a task in which, if a face or a specific object is included in the image represented by the image data, a bounding box that surrounds the region of interest is inferred. In other words, the "face detection / object detection task" can also be said to be a task in which a region of interest is inferred in an image represented by input image data. The "document image entity extraction task" is a task in which text data obtained by performing character recognition processing (OCR processing) on ​​a document image is acquired. Then, if the acquired text data includes a specific entity such as a company name, a person's name, or a place in the document image, the text is inferred. In other words, the "document image entity extraction task" can also be said to be a task in which text corresponding to a specific item is inferred from text obtained by performing character recognition processing on an input document image. The "voice recognition task" is a task in which, using a voice signal obtained by converting speech data as input, the contents of the speech are transcribed into text and output as text data. In other words, the above-mentioned "voice recognition task" can be said to be a task of inferring text data from input voice data. However, the types of tasks to which the present embodiment is applied are not limited to the above-mentioned tasks.

[0013] <Information Processing System> FIG. 1 is a diagram showing a configuration example of an information processing system according to the present embodiment. The information processing system 10 of the present embodiment includes an information processing device (hereinafter referred to as a first information processing device) 11 on the virtual server side, which is the cloud server, and an information processing device (hereinafter referred to as a second information processing device) 12 on the client side, which is the client side terminal. The second information processing device 12 accepts an input 13 of input data by a user to the second information processing device 12, and transmits the accepted input data to the first information processing device 11. The first information processing device 11 receives the input data, performs inference processing using a predetermined model, and outputs the obtained inference result. The first information processing device 11 transmits the inference result to the second information processing device 12. Then, the second information processing device 12 transmits the confirmation result as feedback information to the first information processing device 11, after the inference result is confirmed 14 by the user. The first information processing device 11 holds the feedback information. There are a plurality of second information processing devices 12, which communicate with the first information processing device 11 and perform necessary operations according to the respective tasks and business environments. The number of second information processing devices 12 included in the information processing system 10 is not limited to three, and may be less than three or more than three.

[0014] <Hardware configuration of first information processing device> 2 is a diagram showing an example of a hardware configuration of the first information processing device 11. The first information processing device 11 is a server device configured on a cloud. The first information processing device 11 has a CPU H201, a ROM H202, a RAM H203, an interface H204, and a system bus H205.

[0015] The CPU H201 is a central processing unit that controls various devices connected to the system bus H205. The ROM (Read Only Memory) H202 is a read-only memory that stores computer programs and data that do not require modification, such as a BIOS (Basic Input / Output System) program and a boot program. The RAM (Random Access Memory) H203 has a storage area for temporarily storing data received from the outside. The RAM H203 is also used as a storage area used when the CPU H201 executes various processes (for example, a storage area used when executing image processing).

[0016] The interface H204 is an interface for communicating with an external device via a network. The interface H204 is not particularly limited, and may have a network interface function such as Ethernet (registered trademark), or a serial bus interface function such as USB (Universal Serial Bus). Furthermore, it is sufficient that the interface H204 is capable of connecting to each other, whether wired or wireless.

[0017] <Hardware configuration of the second information processing device> 3 is a diagram showing an example of a hardware configuration of the second information processing device 12. The second information processing device 12 is a terminal device (information terminal) on which a user views and operates a screen, such as a PC (personal computer) or a tablet terminal.

[0018] The second information processing device 12 includes a CPU H301, a ROM H302, a RAM H303, an external memory H304, an input unit H305, a display device H306, an interface H307, and a system bus H308.

[0019] The CPU H301 is a central processing unit, and controls various devices connected to the system bus H308. The ROM H302 is a read-only memory, and stores computer programs and data that do not require modification, such as a BIOS program and a boot program. The RAM H303 has a storage area for temporarily storing data received from the outside. The RAM H303 is also used as a storage area (for example, a storage area used when performing image processing) used when the CPU H301 executes various processes. The external memory H304 stores programs processed by the second information processing device 12, data received from the outside, and the like. The input unit H305 is a touch panel, keyboard, mouse, or robot controller, and performs processing related to input of information, etc. The display device H306 outputs the calculation results of the second information processing device 12 to the display device according to instructions from the CPU H301. The display device may be any type of device, such as a liquid crystal display device, a projector, or an LED indicator, as long as it is capable of displaying the above calculation results on a screen according to instructions from the CPU H301. The interface H307 is an interface for communicating with an external device via a network. The interface H307 is not particularly limited, and may have a function of a network interface such as Ethernet, or a function of a serial bus interface such as USB. Furthermore, it is sufficient if the interface H307 can be connected to each other regardless of whether it is wired or wireless.

[0020] <Functional configuration of information processing system> Fig. 4 is a block diagram showing an example of a functional configuration of the information processing system 10. A transmitting unit 422 in the second information processing device 12 and a receiving unit 401 in the first information processing device 11 communicate information via a network. A transmitting unit 408 in the first information processing device 11 and a receiving unit 423 in the second information processing device 12 communicate information via a network. However, the functional configuration of the information processing system shown in Fig. 4 is an example and does not limit the scope of application of the technology of the present disclosure. The second information processing device 12 may take on part of the functions of the first information processing device 11.

[0021] The first information processing device 11 has a receiving unit 401, a setting unit 402, a model management unit 403, a model inference unit 404, a custom inference unit 405, an inference determination unit 406, an evaluation unit 407, a transmitting unit 408, a data management unit 409, a storage unit 410, and a model selection unit 411.

[0022] The receiving unit 401 receives information from the second information processing device 12 via the network. The received information is, for example, information on input data, information on settings when performing inference, feedback information, and instructions for the first information processing device 11. The receiving unit 401, the transmitting unit 408, the transmitting unit 422, and the receiving unit 423 all convert data into an appropriate format before transmission, and execute data transmission after conversion. After reception, the data is converted into data in a format that is easy to use at the output destination. The received information is output to the setting unit 402 or the data management unit 409 according to the instruction.

[0023] The setting unit 402 performs settings related to the model based on the received information. Specifically, the setting unit 402 organizes setting information related to the model and custom inference device used for inference, and outputs the information to the model management unit 403.

[0024] The model management unit 403 receives setting information related to the model and the custom inference device, and prepares for inference according to the settings indicated in the received setting information. That is, the model management unit 403 selects a model and an inference device to be used based on the settings indicated in the received setting information from among a plurality of models and a plurality of custom inference devices prepared in advance corresponding to a specific task. If the selected model is not loaded, the selected but unloaded model is loaded. Also, if a custom inference device to be used exists, it is prepared. Then, an inference execution command is output to the model inference unit 404 using the specific model, which is a command to execute inference.

[0025] The model inference unit 404 receives an inference execution command and target input data, and executes inference processing. The model used in the inference processing of the model inference unit 404 is, for example, a deep learning model such as CNN, RNN, or Transformers. In other words, the model used in the inference processing can be said to be a trained model (first trained model) that has been machine-learned in advance. The inference result obtained by the model inference unit 404 is output to the custom inference unit 405 and the inference determination unit 406. Details of the inference processing will be described with reference to FIG. 8.

[0026] The custom inference unit 405 receives an inference execution command, target input data, and an inference result by the model inference unit 404, and executes an inference process using a custom inference device. The custom inference device used in the inference process of the custom inference unit 405 can be said to be, for example, a trained model (second trained model) that has been machine-learned in advance with less training data compared to the first trained model. The inference result obtained by the custom inference unit 405 is output to the inference determination unit 406.

[0027] The inference determination unit 406 obtains an inference result from at least one of the model inference unit 404 and the custom inference unit 405, and determines an inference result for the target input data. The inference result determined by the inference determination unit 406 is output to the evaluation unit 407.

[0028] The evaluation unit 407 compiles inference results corresponding to the predetermined input data input from the inference determination unit 406 and outputs them to the data management unit 409. In addition, the evaluation unit 407 compares a plurality of correct answer data with a GT label that corresponds to a plurality of input data and forms a pair with the plurality of input data with a plurality of inference results for the plurality of input data to obtain a correct answer rate and calculates and acquires an evaluation index. The GT label is information indicating a correct answer (Ground Truth) that is assigned to data. Note that a combination (pair) of the input data and the correct answer data that corresponds to the input data and is assigned the GT label is held in advance as an evaluation dataset. That is, the evaluation dataset has a plurality of pairs of input data and the corresponding correct answer data. The evaluation index indicates the degree to which the model inference unit 404 is suitable for using the model (first trained model) in the task to be processed. In addition, the evaluation index indicates the degree to which the custom inference unit 405 is suitable for using a custom inference device (second trained model). Those results are output to the transmission unit 408 and the model selection unit 411 according to the settings.

[0029] The transmitting unit 408 outputs the inference result and the notification acquired from the evaluating unit 407 to the receiving unit 423. As a result, the inference result and the notification acquired from the evaluating unit 407 are transmitted to the second information processing device 12.

[0030] The data management unit 409 acquires the following information for the input data acquired from the receiving unit 401. That is, the data management unit 409 acquires a pair of input data of a predetermined task and supervised data corresponding to the input data as feedback information. The data management unit 409 also acquires an inference result acquired from the evaluation unit 407, the inference result acquired from at least one of the model inference unit 404 and the custom inference unit 405. The data management unit 409 then manages the acquired inference result by linking it to the input data. The input data to which the acquired inference result is linked is held in the holding unit 410. When inference is executed on the input data, the data management unit 409 outputs the input data to the model inference unit 404. Details of the data managed by the data management unit 409 will be described later.

[0031] The storage unit 410 stores the input data and its related information in association with each other. When the data in the storage unit 410 is to be used or deleted in the first information processing device 11, the data is processed through the data management unit 409.

[0032] The model selection unit 411 selects a combination (pair) of a model and a custom inference device from among combinations of a model used in the model inference unit 404 and a custom inference device used in the custom inference unit 405, based on an evaluation index for each combination. The evaluation index for each combination is acquired from the evaluation unit 407. Information indicating the selected combination is then output to the setting unit 402.

[0033] The second information processing device 12 also includes a data acquisition unit 421 , a transmission unit 422 , a reception unit 423 , a display unit 424 , a feedback acquisition unit 425 , and an authentication unit 426 .

[0034] The data acquisition unit 421 acquires input data to be inferred by the first information processing device 11. The data acquisition unit 421 also acquires information such as various settings and execution commands for target tasks. The acquired data and information are output to the transmission unit 422.

[0035] The transmission unit 422 outputs the input data and information input from the data acquisition unit 421 to the reception unit 401. In addition, the transmission unit 422 outputs feedback information input from a feedback acquisition unit 425, which will be described in detail later, to the reception unit 401. As a result, the input data and information input from the data acquisition unit 421 and the feedback information input from the feedback acquisition unit 425 are transmitted to the first information processing device 11.

[0036] The receiving unit 423 receives the transmission data transmitted from the transmitting unit 408 of the first information processing device 11. The received data includes, for example, an inference result and a notification. The received data is output to the display unit 424.

[0037] The display unit 424 displays the inference result and the notification input from the receiving unit 423. The display unit 424 performs the display using a display device such as a display monitor, a head mounted display, a touch panel, a projector, etc. Such a display allows the user to check the inference result and the notification.

[0038] The feedback acquisition unit 425 acquires user feedback on the inference result displayed by the display unit 424. That is, the feedback acquisition unit 425 acquires, as feedback information, information indicating a user's evaluation result on the inference result displayed by the display unit 424. Details of the feedback (feedback information) will be described later. The acquired feedback information is output to the transmission unit 422.

[0039] The authentication unit 426 manages users who use the information processing system 10 by user ID, and performs authentication by the user ID. The authentication unit 426 may manage a plurality of user IDs by linking them to a group ID that manages them on a group basis, and perform authentication by the group ID. The group ID may be set, for example, on a workplace basis or on a company basis. When a user uses the information processing system 10 to transmit or receive data or information to or from the first information processing device 11, the authentication unit 426 performs authentication on the first information processing device 11 via the transmission unit 422, and the user is then able to use the information processing system 10.

[0040] <Processing procedure in information processing system> Fig. 5 is a flowchart showing a procedure of a process in which an information processing system including the second information processing device 12 and the first information processing device 11 acquires and holds feedback information on an inference result. The process shown in the flowchart of Fig. 5 is started, for example, when input data for a predetermined task to be processed is input. However, the information processing system does not necessarily have to perform all the steps described in this flowchart.

[0041] As a preparation for carrying out this flowchart, the second information processing device 12 initializes the system. That is, the second information processing device 12 reads a program from the external memory H304 and makes the second information processing device 12 operable. Similarly, the first information processing device 11 is made operable. The second information processing device 12 and the first information processing device 11 are assumed to be in a state in which they can communicate with each other using the transmitting unit 422, the receiving unit 401, the transmitting unit 408, and the receiving unit 423.

[0042] Furthermore, the user can use the information processing system after being authenticated by the user ID or group ID. By assigning a unique ID to the user, the setting unit 402 in the first information processing device 11 can automatically acquire settings for each user and execute the target task with user-specific settings, or a custom inference device can be set uniquely for the user. Furthermore, by linking the user ID to the group ID, the user corresponding to the user ID linked to the same group ID can be set to execute the target task with workplace-specific settings. These setting information are stored in the first information processing device 11. By using the user ID or group ID, the user can freely access the first information processing device 11 using any communication terminal such as a communication terminal at home, a communication terminal at work, or a communication terminal at a shared office as the second information processing device 12 without being tied to a location. Furthermore, the target task can be processed using individually customized settings linked to the user's personal ID or the group ID at work.

[0043] In S501, the data acquisition unit 421 of the second information processing device 12 acquires input data. The input data is, for example, data such as an image, a document image, text, and voice. For example, when a face detection / object detection task is executed, image data that may include a face or object to be detected is acquired as input data. When an entity extraction task of a document image is executed, text data obtained by OCR processing of the document image and a document image are acquired as input data. When a voice recognition task is executed, vector data representing a spoken voice signal is acquired as input data. However, the type of input data is not limited to these, and may be any multidimensional vector data or tensor data. In addition, the data acquisition unit 421 acquires setting information related to the target task. The setting information related to the target task includes, for example, a target task for performing a predetermined process on the input data and parameters at that time. The above parameters affect the estimation result when a model is used, and are for changing and tuning the model according to the data set. The above parameters are used, for example, to exclude a predetermined number of pixels outside the frame from the processing target. In the document image, a setting value for associating the number of characters that can be handled with the input is used. However, these setting information may be the setting information held in the first information processing device 11 as the setting information associated with the user ID. The acquired data and setting information are subjected to data conversion by the data acquisition unit 421 to compress and encrypt the data into a format suitable for transmission. The converted data is then output to the transmission unit 422.

[0044] In S502, the transmitting unit 422 transmits the data received from the data acquiring unit 421. In S503, the receiving unit 401 receives the data transmitted by the transmitting unit 422 in S502. The receiving unit 401 then decrypts and decompresses the data to convert it to data in the original format. The input data and setting information converted to the original format are output to the setting unit 402.

[0045] In S504, the setting unit 402 sets the target task according to the setting information received in S503. Specifically, the setting unit 402 sets the processing task for the input data, and when the target task is selected, the selected model and custom inference device are set according to the user ID or group ID. Then, the model management unit 403 sets the model inference unit 404 and the custom inference unit 405 to an executable state based on the setting information and execution parameters. These execution parameters are also executed with values ​​linked to the user ID or group ID. However, when executing by changing specific parameters as setting information, the setting is performed according to the received setting information.

[0046] In S505, the model inference unit 404, the custom inference unit 405, and the inference determination unit 406 execute inference processing of the target task on the input data. Details of the inference processing will be described later. The obtained inference result is output to the evaluation unit 407.

[0047] In S506, the evaluation unit 407 organizes the inference results. When there are results inferred by the model inference unit 404 and results inferred by the custom inference unit 405, the evaluation unit 407 distinguishes between them and outputs them to the data management unit 409. The evaluation unit 407 also converts the inference results and the input data into a format suitable for display that is easy for the user to understand, based on the inference results and the input data, and outputs the converted result data to the transmission unit 408. For example, a format suitable for display in a face detection / object detection task is an image in which a detected bounding box is superimposed on an input image. For example, a format suitable for display in an entity extraction task of a document image is an image in which the text of an entity extracted on a document image is highlighted and superimposed together with its label (such as a company name or a person's name), and the extracted text information. For example, a format suitable for display in a speech recognition task is a video in which a voice signal itself is associated with text information converted from the voice, and subtitle text is displayed in accordance with the timing of the voice being emitted, and the converted text information.

[0048] In S507, the transmission unit 408 transmits the data received from the evaluation unit 407. In S508, the reception unit 423 receives the data transmitted by the transmission unit 408 in S507. The received data is output to the display unit 424.

[0049] In S509, the display unit 424 displays the inference result, and if the inference result is incorrect, a message is displayed to prompt the user to input the correct answer.

[0050] (UI screen) FIG. 6 is a diagram showing an example of a UI screen displayed by the display unit 424. FIG. 6 shows an example of presenting an inference result to a user. Here, the process in S509 will be described using an example of an entity extraction task for a document image. The UI screen 600 shows a display of a screen shown to a user. In the UI screen 600, a document image 610 to be inferred is displayed in the left half area. An area 611 surrounding "XYZ Corporation" with a dashed line shows the extraction position of the entity resulting from the inference. In addition, an area 612 surrounding "Suzuki Hajime" with a dashed line shows the extraction position of the entity resulting from the inference. Furthermore, in the UI screen 600, the right half area includes character string display areas 621 and 623 for displaying character strings of the inferred entity extraction result, a message display area 625 for displaying a message to the user, and a confirmation button 626. The character string display area 621 includes an edit button 622 for editing the entity extraction position, which is the area 611 surrounding "XYZ Corporation" with a dashed line corresponding to the company name. That is, the area 611 and the character string display area 621 correspond to the extraction result of the company name. The character string display area 623 includes an edit button 624 for editing the entity extraction position, which is the area 612 surrounding "Suzuki Hajime" corresponding to the person's name with a dashed line. That is, the area 612 and the character string display area 623 correspond to the extraction result of the person's name. The message displayed in the message display area 625 is, for example, a message that prompts the user to input a correct answer if the inference result is correct or not, or if it is incorrect. In FIG. 6, the message "If the inference result below is incorrect, please correct it correctly. If the inference result is correct or if you have finished correcting it, press the confirm button." is displayed in the message display area 625. The confirm button 626 is a button that is pressed when the inference result is correct or when the user has finished correcting the inference result. In the example of FIG. 6, it can be seen from the area 612 and the character string display area 623 that although "XYZ Corporation" has been correctly extracted as the company name, only "Suzuki Hajime" has been extracted from the person's name "Suzuki Ichiro".Therefore, the user presses the edit button 624, expands the entity extraction position, which is the area 612 surrounding “Suzuki Hajime” with a dashed line, to the right, corrects the extracted character string in the character string display area 623 to “Suzuki Ichiro”, and then presses the confirm button 626.

[0051] In S510, the feedback acquisition unit 425 acquires feedback information from the user. The feedback information is information including a pair of input data and correct data for the input data. If the inference result is correct, the user inputs that the inference result is correct, and the inference result is registered in the feedback information as correct data. If the inference result is incorrect, the user is prompted to correct the inference result to a correct one, and the corrected inference result after the inference result is corrected by the user is registered in the feedback information as correct data. For example, the feedback from the user when the answer is incorrect in a face detection / object detection task is to specify a bounding box surrounding the detection target for the input image. For example, the feedback from the user when the answer is incorrect in an entity extraction task of a document image is to specify an area in which the text of the entity to be detected exists with a bounding box for the document image, and input the text. For example, the feedback from the user when the answer is incorrect in a voice recognition task is to rewrite the erroneous text part with the correct text for the text converted from the input voice. The acquired feedback information is output to the transmission unit 422.

[0052] In S511 , the transmission unit 422 transmits the feedback information input from the feedback acquisition unit 425 .

[0053] In S512, the receiving unit 401 receives the data transmitted by the transmitting unit 422 in S511. If a user ID and a group ID are set in the received data, the receiving unit 401 links the received data with the IDs of both parties (user ID and group ID) and outputs the data with the IDs of both parties linked to the data management unit 409. In this way, by transmitting and receiving data between the first information processing device 11 and the second information processing device 12, reactions and confirmations from the user are exchanged. In the flowcharts shown hereinafter, such simple exchanges with the user will be explained without describing the transmission and reception processes.

[0054] In S513, the data management unit 409 manages the correct answer data obtained from the feedback information in association with the input data, the user ID, and the group ID. Furthermore, the data management unit 409 manages the input data associated with the correct answer data by setting a flag indicating that the input data is assessable data. The assessable flag may be set automatically and randomly, or the user may separately select data to be used as assessment data and set a flag. The method of using this assessable data will be described in the flowchart of automatically reselecting an appropriate model in the first embodiment.

[0055] (Examples of data managed by the Data Management Department) Fig. 7 is a diagram showing an example of data managed by the data management unit 409. Fig. 7(a) shows data for managing inference results for input data, and Fig. 7(b) shows data for managing attributes of the input data.

[0056] The inference result management data 710 includes data corresponding to the following items. That is, it includes data corresponding to each item of an input data ID 711, a correct answer data 712, an inference result 713 of an inference determination unit, an inference result 714 of a custom inference unit, a custom inference device ID 715, an inference result 716 of a model inference unit, and a model ID 717. For example, the "000001" in the first line of the input data ID 711 is linked to the "correct answer data 1", "inference result D1", "inference result C1", "C00001", "inference result M1", and "M00001" for each of the above-mentioned items. The "000001" in the second line, the "000002" in the third line, and the "000003" in the fourth line of the input data ID 711 are also linked to the data of each of the above-mentioned items, similar to the "000001" in the first line. In other words, the inference result management data 710 can be said to be composed of a sequence of input data, correct answer data, inference result of the inference determination unit, inference result of the custom inference unit, custom inference device ID, inference result of the model inference unit, and model ID. Furthermore, the inference result management data 710 may manage data corresponding to items other than those shown in FIG. 7(a).

[0057] The attribute management data 720 includes data corresponding to each of the items of the input data ID 721, the acquisition date and time 722, the user ID 723, the group ID 724, and the evaluation possible flag 725. For example, the "000001" in the first line of the input data ID 721 is linked to "2022 / 4 / 27 10:30:41", "U00001", "G00001", and "On" for each of the above-mentioned items. The "000002" in the second line, the "000003" in the third line, and the "000004" in the fourth line of the input data ID 721 are also linked to the data of each of the above-mentioned items, similar to the "000001" in the first line. That is, the attribute management data 720 can be said to be composed of columns of the input data, the user ID, the group ID, and the evaluation possible flag. In the evaluation possible flag 725, "On" indicates that it is an evaluation target, and "On" indicates that it is not an evaluation target. The attribute management data 720 may manage data corresponding to items other than those shown in FIG. 7(b).

[0058] In S514, the storage unit 410 stores the input data and the correct answer data corresponding to the input data. By repeatedly executing the process shown in the flowchart of Fig. 5, the storage unit 410 accumulates a plurality of pairs of input data and correct answer data corresponding to the input data, the number of times the process has been repeatedly executed. When the process of S514 is completed, the process shown in Fig. 5 is terminated.

[0059] Through the above processing, an inference result for the input data and feedback information from the user regarding the inference result can be obtained, and the obtained inference result and the feedback information corresponding to this inference result can be linked and stored.

[0060] <Details of inference process> FIG. 8 is a flowchart showing a detailed flow of the inference process (S505).

[0061] In S801, the model inference unit 404 executes an inference process on the input data. In the inference process by the model inference unit 404, a deep learning model is used. For example, a convolutional neural network (CNN) may be used as the deep learning model. In addition, for example, a recurrent neural network (RNN), Transformers, etc. may be used as the deep learning model. A trained model is generated by performing supervised learning in advance on such a model (deep learning model) using correct answer data. Then, an inference result is obtained by processing the input data using the trained model. In addition, based on the acquired inference result, a process is performed to obtain an F-score (F value), which is one of the evaluation indices for a binary classification task and is a harmonic mean value of two values, the precision rate and the recall rate. For example, in a model for a face detection / object detection task, a process is performed to identify a region that seems to be a face or a region that seems to be a specific object from an input image, and to obtain a bounding box and its F-score (F value). For example, in a model for an entity extraction task of a document image, the following process is performed. That is, the process solves the multi-class classification problem of entity labels from text obtained by OCR (optical character recognition) processing of document images, and obtains the entity labels and F-scores (F-values) to be extracted for each unit of a document, such as a word or phrase, called a token. For example, models for speech recognition tasks include those that infer text strings from emitted speech signals using acoustic models, language models, and pronunciation dictionaries, and those that infer text strings directly from speech features using end-to-end speech recognition. The process also involves obtaining the F-scores (F-values).

[0062] In S802, the custom inference unit 405 judges whether or not to perform inference using a custom inference device based on at least one of the setting information, the input data, and the inference result by the model inference unit 404. For example, if a custom inference device is not set in the setting information acquired in S503, inference using the custom inference device is not performed. Also, if the F-score (confidence) of the inference result by the model inference unit 404 exceeds a predetermined threshold (first threshold), the inference result by the model inference unit 404 is deemed sufficient, and inference using the custom inference device is not performed. Conversely, inference using the custom inference device is performed in any of the following cases. That is, if a custom inference device is set in the setting information acquired in S503 and the F-score (confidence) of the inference result by the model inference unit 404 is equal to or less than the above-mentioned predetermined threshold (first threshold), inference using the custom inference device is performed. Alternatively, inference using the custom inference device is performed when the input data is deemed to be suitable for inference using a custom inference device, such as when the input data is deemed to have a high similarity to the data (input data) used in creating the custom inference device. The above-mentioned predetermined threshold (first threshold) used in S802 is a value that can be set arbitrarily. If a determination result is obtained that inference using a custom inference device is to be performed (YES in S802), the process proceeds to S803. If a determination result is obtained that inference using a custom inference device is not to be performed (NO in S802), the process proceeds to S804.

[0063] In S803, the custom inference unit 405 executes an inference process on the input data. The details of the inference by the custom inference device will be described. The inference process by the custom inference unit 405 is less versatile than the inference process by the model inference unit 404 and can be performed only under specific conditions, but has the characteristic that the amount of calculation required for the inference process is relatively small and the memory usage is relatively small. Therefore, the inference process by the custom inference unit 405 is performed conditionally, for example, only when the F score (confidence) of the inference result by the model inference unit 404 is equal to or less than the above-mentioned predetermined threshold (first threshold) and there is a problem with the inference result by the model inference unit 404. Therefore, an inference result different from that by the model inference unit 404 can be obtained in a complementary manner. In addition, the custom inference unit 405 does not need to be limited to only one custom inference device for the input data, and multiple custom inference devices may be used. When multiple inference results are obtained by multiple custom inference devices, all of the multiple inference results are output to the inference determination unit 406. For example, the custom inference unit 405 in a face detection / object detection task cuts out a foreground image of an object to be detected and registers it as a template, and performs inference by performing template matching. Furthermore, the degree of match during template matching can be acquired as a confidence score. For example, the custom inference unit 405 in a document image entity extraction task registers a document image and an appearance area of ​​an entity to be extracted in the document in advance. Then, during inference, a registered document image identical to the document image of the input data is searched for, and text corresponding to the entity appearance area of ​​the registered document image is extracted as an entity. Furthermore, the degree of match with the registered document image can be acquired as a confidence score. For example, the custom inference unit 405 in a speech recognition task uses MFCC (Mel Frequency Cepstrum Coefficients) to make a speech signal into a 12-dimensional MFCC feature, and features a 13-dimensional feature vector obtained by adding one dimension of the logarithmic energy of the speech signal. However, it is also possible to further add first- and second-order time derivatives of the 13-dimensional feature to make it into a 39-dimensional feature.The feature vector and the text string are linked for each word or phrase, and in the time series region where a feature vector with high similarity is obtained in the input speech signal, the text string linked to the feature vector is used as the inference result to obtain an inference result. Also, the average similarity with the input speech signal can be obtained as a confidence score.

[0064] In S804, the inference determination unit 406 acquires the inference result by the model inference unit 404 and the inference result by the custom inference unit 405, and determines one inference result for the input data. If an inference result by the custom inference device exists via S803, and the reliability score of the inference result by the custom inference device is equal to or greater than a predetermined threshold (second threshold), the inference determination unit 406 determines the result by the custom inference device as the inference result. On the other hand, if the reliability score of the inference result by the custom inference device is less than the above-mentioned predetermined threshold (second threshold), or if the process reaches the process of S804 without going through the process of S803, the inference determination unit 406 determines the result by the model inference unit 404 as the inference result. Note that the above-mentioned predetermined threshold (second threshold) used in S804 is a threshold different from the above-mentioned first threshold, and is a value that can be set arbitrarily. When the process of S804 is completed, the inference process shown in FIG. 8 is terminated.

[0065] Through the above processing, the model inference unit 404, the custom inference unit 405, and the inference determination unit 406 can be used to obtain an inference result for the input data.

[0066] <Automatic model selection process> 9 is a flowchart showing the procedure of the process of automatically selecting a model. By executing the automatic model selection process, an appropriate combination (pair) is automatically selected from among a plurality of models and custom reasoners.

[0067] The timing for starting the model selection may be, for example, when the first information processing device 11 has sufficient surplus computational resources. Alternatively, it may be when a trigger signal for reselecting a model is received from the second information processing device 12. The trigger signal for reselecting a model may be issued by a user operation, or may be set to issue a trigger signal at a timing when the frequency of use is low, such as after business hours, depending on the usage status of the second information processing device 12. In addition, since it is difficult to select a model in a situation where a sufficient number of evaluation data described later is not collected, a sufficient number of evaluation data may be one of the conditions for starting the model selection. In other words, the model selection may be started when a predetermined number or more of evaluation data are collected and the trigger signal is received. The predetermined number can be set arbitrarily. Then, S901 is started.

[0068] In S901, the setting unit 402 performs settings for selecting a model. Specifically, the setting unit 402 sends information on a combination of a model to be evaluated and a custom inference device and its setting parameters to the model management unit 403. The setting unit 402 also sends setting information on evaluation data to be used for evaluation to the data management unit 409.

[0069] In S902, the model management unit 403 lists combinations of the model of the model inference unit 404 and the custom inference unit of the custom inference unit 405. However, it is not necessary to list all combinations, including the case where only the model inference unit 404 is used without using the custom inference unit 405. That is, in the case where the combinations of the model of the model inference unit 404 and the custom inference unit of the custom inference unit 405 are limited in the information of the setting parameters sent from the setting unit 402, it is not necessary to list all combinations. For example, models, custom inference units, and combinations thereof that clearly had relatively low inference accuracy in the evaluation of past model selection may be excluded. In addition, when a user specifies that a specific model, custom inference unit, or combination thereof is to be excluded, the specified model, custom inference unit, and combination thereof are excluded. Each of the combinations listed in this way is hereinafter referred to as an inference model. From the group of inference models listed in this way, the model selection unit 411 selects an inference model appropriate for the user in a process described in detail below.

[0070] In S903, the data management unit 409 extracts a plurality of pairs of input data and correct answer data corresponding to the input data from the storage unit 410 as an evaluation data set used for evaluating the inference model, based on the setting information of the evaluation data sent from the setting unit 402. The pair of input data and correct answer data corresponding to the input data is extracted from among pairs of input data and correct answer data with a flag indicating that the data is evaluable. In this way, by managing the evaluable data, the evaluation data and the learning data can be managed separately, and it is possible to prevent the evaluation data from being mixed in the learning data. The data extracted as the evaluation data set may be all the retained data with a flag set, or the retained data obtained by filtering the retained data with a flag set may be used. As a method of filtering, the data may be limited within the range of the period during which the data was acquired, or may be limited by the user ID that inputted the data or the group ID to which the user belongs. Alternatively, the data may be limited by the attributes of the data. For example, in filtering data in face detection and object detection tasks, the data may be limited by gender, the presence or absence of glasses, the presence or absence of a mask, a specific object category, the size of the image, and the like. For example, data filtering in an entity extraction task may be limited by the presence or absence of a specific entity label, the presence or absence of a specific character string, the aspect ratio of a document, etc. For example, data filtering in a speech recognition task may be limited by the presence or absence of a specific recognition character string, the pitch, speed, length, etc. of the voice. The extracted pairs of input data and correct answer data are held in the data management unit 409 as an evaluation dataset for automatically selecting a model. In other words, the number of pairs extracted as the evaluation dataset can be set to any number as long as it is more than one.

[0071] In S904, the model management unit 403 prepares to perform inference on the inference models one by one in sequence.

[0072] In S905, the model management unit 403 uses the prepared inference model for the evaluation data set to perform the inference process described with reference to Fig. 8. Then, the evaluation unit 407 stores the inference results linked to each inference model.

[0073] In S906, the process returns to S904 and is repeated until the inference process of S905 is performed for all of the inference model groups. When inference results have been obtained for all of the inference model groups for the evaluation dataset, the process proceeds to S907.

[0074] In S907, the evaluation unit 407 compares the inference results of each inference model for the evaluation dataset with each correct answer data. If the inference result is similar to the correct answer data, the evaluation unit 407 calculates and obtains an evaluation index as a correct answer. If the inference result is not similar to the correct answer data, the evaluation unit 407 calculates and obtains an evaluation index as an incorrect answer. The evaluation unit 407 sends the evaluation index of the evaluation dataset for each inference model to the model selection unit 411.

[0075] In S908, the model selection unit 411 compares the evaluation indexes of the inference models and selects the optimal inference model. Basically, the model selection unit 411 selects an inference model with a high evaluation index, but selects an inference model according to the inference model selection criteria set by the user. For example, the user preregisters data that should be correctly answered (pairs of input data and corresponding correct answer data) in the evaluation data set, and an inference model that is likely to correctly answer data similar to the data can be selected. For example, when switching the inference model according to some input trigger, multiple inference models are selected according to the model switching. For example, the upper limit model size and memory usage are set according to the machine specifications of the first information processing device 11, which is the environment in which the inference model operates, and a model that meets these conditions is selected. In this way, by allowing the user to set various conditions and select a model, it is possible to flexibly set a model that is preferable for each user. Information on the inference model selected by the model selection unit 411 is sent to the setting unit 402.

[0076] In S909, the setting unit 402 changes the setting to use the selected inference model, so that the model in the model inference unit 404 and the custom inferencer in the custom inference unit 405 are updated.

[0077] By the above process, even when a new model or custom inference tool is created, it is possible to update the model and custom inference tool to one that is more appropriate for the user from among a plurality of models and custom inference tools.

[0078] In the above-mentioned configuration, the data management unit 409 acquires and holds the user's input content for the inference result as feedback information. As a result, even if the user does not consciously prepare an evaluation dataset, the input data and the correct answer data corresponding to the input data are linked as pairs by simply using the system, and an evaluation dataset including multiple pairs can be automatically generated. Furthermore, by using the obtained evaluation dataset, the model selection unit 411 selects an appropriate inference model according to the user's usage situation, and thus customization can be performed individually for each user.

[0079] (Modification 1 of the first embodiment) In S506 in FIG. 5, an example has been described in which the evaluation unit 407 converts the inference result into a format suitable for display that is easy for the user to understand, but the present invention is not limited to this. For example, the evaluation unit 407 performs only a process of outputting the inference result and the input data to the transmission unit 408. Then, a conversion unit (not shown) in the second information processing device 12 may convert the input data and the inference result received by the reception unit 423 into data in a format suitable for display that is easy for the user to understand, and output the converted data to the display unit 424 for display. In this way, by processing and converting the display data in the second information processing device 12, conversion can be performed without worrying about communication capacity even if the amount of data after data conversion is relatively large. In addition, interactive conversion can be easily performed.

[0080] (Modification 2 of the first embodiment) In S903 in FIG. 9, an example has been described in which the data management unit 409 extracts data from a pair of input data and correct answer data corresponding to the input data as the evaluation data set, but the present invention is not limited thereto. For example, the data expansion unit (not shown) performs data expansion on a pair of input data and correct answer data corresponding to the input data to generate a plurality of pairs of input data and correct answer data corresponding to the input data. Then, the generated plurality of pairs of input data and correct answer data may be used as the evaluation data set. In the data expansion in the face detection / object detection task, for example, noise may be added to the image, the image may be enlarged or reduced, the image may be affine transformed, or the image may be color-converted. In the data expansion in the entity extraction task of the document image, for example, the entity string may be replaced with another string, the string may be moved in parallel, moved randomly, or enlarged or reduced to change the position, or a string other than the entity may be replaced with a synonym. In the data expansion in the speech recognition task, for example, noise may be added to the voice signal, the strength or speed of the voice signal may be enlarged or reduced, or the time series of the voice signal may be replaced or deleted in units of sentences or phrases. In this way, by data augmenting the evaluation dataset, the robustness of the model selected can be improved when performing a process to automatically select an appropriate combination of models and custom reasoners.

[0081] (Modification 3 of embodiment 1) In S903 in FIG. 9, an example has been described in which the data management unit 409 extracts only user IDs or group IDs from pairs of input data and correct answer data corresponding to the input data, but the present invention is not limited to this. For example, the data management unit 409 may manage whether a data set is made public or private to other users or groups for each user ID or group ID by a pre-setting. Depending on information indicated in a whitelist or blacklist, it may be set whether a data set is made public or private only for a specific user ID or group ID. In this way, since not all data sets are visible but a mechanism for making the data sets private exists, it becomes possible to use highly confidential data sets when using this information processing system.

[0082] <<Embodiment 2>> In this embodiment, an aspect of providing a more appropriate inference model for a user by learning and updating an inference model on a cloud server will be described. The inference model consists of two types: a model and a custom inference machine. Basically, inference by a model is used, but inference by a custom inference machine is used as a complement when inference by a model does not obtain the result desired by the user. In order to perform complementary inference, the custom inference machine responds in a short time based on feedback information from the user regarding an error in the inference result and reflects it in the inference result. A relatively large amount of data is used for learning a model, and learning also requires a relatively large amount of time and computational resources, and there is a tendency that a relatively large number of diverse data sets can be correctly inferred during inference. On the other hand, a relatively small amount of data set is used for learning a custom inference machine, and learning does not require a large amount of computational resources in a relatively short time, and there is a tendency that correct inference can be performed only for limited problems. A model is one model that is shared and used by many users, whereas a custom inference machine is one that is shared and used by a small number of users, such as each user or each group. By using these two types of inference models, the model can be used to base up inference, and the custom inference machine can be used to respond to individual data sets for each user. In this embodiment, in order to realize such inference, a model and a custom inference machine are trained.

[0083] <Functional configuration of information processing system> 10 is a block diagram showing an example of a functional configuration of the information processing system of this embodiment. The first information processing device 21 has a receiving unit 1001, a setting unit 1002, a model management unit 1003, a model inference unit 1004, a custom inference unit 1005, an inference determination unit 1006, an evaluation unit 1007, a transmission unit 1008, a data management unit 1009, and a storage unit 1010. The first information processing device 21 further has a model selection unit 1011, a model learning unit 1012, and a customization unit 1013. The second information processing device 22 has a data acquisition unit 1021, a transmission unit 1022, a receiving unit 1023, a display unit 1024, a feedback acquisition unit 1025, and an authentication unit 1026. The functional units 1001 to 1011 are the same as the functional units 401 to 411, and the functional units 1021 to 1026 are the same as the functional units 421 to 426, and therefore the description thereof will be omitted. The model learning unit 1012 and the customization unit 1013, which are different from the first embodiment, will be described.

[0084] The model learning unit 1012 receives learning data consisting of pairs of input data and supervised data from the data management unit 1009 and the storage unit 1010, and generates a model by performing learning. The generated model is deployed so that it can be used by the model inference unit 1004. Furthermore, the generated model is registered as a new model in the model management unit 1003. Details of the model learning process will be described later with reference to the figures.

[0085] The customization unit 1013 receives learning data consisting of pairs of input data and supervised data from the data management unit 1009 and the storage unit 1010, and generates a custom inference device by performing learning. The generated custom inference device is deployed so that it can be used by the custom inference unit 1005. Furthermore, the generated custom inference device is registered as a new custom inference device in the model management unit 1003. Details of the learning process of the custom inference device will be described later with reference to the figures.

[0086] <Model learning process> A processing procedure in the information processing system of this embodiment will be described. For the sake of explanation, an example will be described in which a model is learned assuming that it is shared within the same group, and a custom inference device is learned assuming that it is set individually for each user. FIG. 11 is a flowchart showing a process flow for learning a model available in the model inference unit 1004 in the first information processing device 21. The process contents shown in the flowchart of FIG. 11 are started, for example, when an administrator of the information processing system transmits a trigger signal for learning a model from the second information processing device 22, and the first information processing device 21 receives the trigger signal. Then, S1101 is started.

[0087] In S1101, the setting unit 1002 receives a trigger signal and acquires a group ID of a target for model learning from the authentication information of the transmission source. Then, the acquired target group ID is output to the data management unit 1009. Furthermore, in order to learn the model, the setting unit 1002 issues a command to the model learning unit 1012 to instruct the model learning unit 1012 to perform initialization settings for starting learning. Upon receiving this command, the model learning unit 1012 performs initialization settings. In this way, the model learning unit 1012 performs initialization settings, thereby preparing for learning.

[0088] In S1102, the data management unit 1009 receives the group ID output by the setting unit 1002 in S1101, and extracts a dataset of the target group ID. Furthermore, the data management unit 1009 checks whether or not there is an evaluable flag as described in the first embodiment, and extracts a dataset of the desired group ID and for which no evaluable flag is set. Information on the extracted dataset is output to the model learning unit 1012.

[0089] In S1103, the model learning unit 1012 acquires the number of data included in the dataset from the information about the dataset extracted by the data management unit 1009 in S1102. Then, the number of data included in the dataset is compared with a predetermined threshold (third threshold) corresponding to the minimum number of data required for learning. If the determination result indicates that the number of data included in the dataset is equal to or greater than the predetermined threshold (third threshold) (YES in S1103), the process proceeds to S1104. On the other hand, if the determination result indicates that the number of data included in the dataset is less than the predetermined threshold (third threshold) (NO in S1103), the following process is performed. That is, information that the number of data included in the dataset is insufficient and learning is not possible is sent to the second information processing device 22 through the transmission unit 1008, and is presented to the user via the display unit 1024. Note that the predetermined threshold (third threshold) used in S1103 is a value that can be set arbitrarily. Then, the learning process shown in FIG. 11 is completed.

[0090] In S1104, the model learning unit 1012 acquires a learning dataset from the data management unit 1009 and the storage unit 1010 based on the information on the dataset extracted in S1103. The learning dataset is composed of a plurality of pieces of data, each of which is a combination of input data and correct answer data corresponding to the input data, and the model is learned by machine learning of the deep learning model. Then, the model learning unit 1012 assigns a new identifier such as a name to the learned model and transmits it to the model inference unit 1004. Here, an example of learning in the first information processing device 21 having the model inference unit 1004 and the like has been described, but this is not limited to this. The learning dataset may be sent to an information processing device (not shown) of a learning server, learning may be performed in the information processing device of the learning server, and the model obtained by learning may be received from the information processing device of the learning server. In this way, by using the information processing device of the learning server, processing can be distributed. In other words, processing that requires huge computer resources such as deep learning is executed in the information processing device of the learning server, which is relatively large in scale. Processes that do not require many computer resources, such as inference and data management, are executed by the relatively small-scale first information processing device 21. This makes it possible to keep the required specifications of the first information processing device 21 relatively low.

[0091] In S1105, the model inference unit 1004 receives the model newly learned in S1104 and arranges it in a state where model inference is possible. In addition, the setting unit 203 registers the identifier of the newly learned model and makes it selectable as a model. When the process of S1105 is completed, the learning process shown in FIG. 11 is completed.

[0092] Through the above process, a model can be trained using the stored dataset. Since training is performed using a dataset that contains data accumulated according to usage by the user or the organization to which the user belongs, a model that is more likely to solve problems specific to the user or organization can be generated.

[0093] <Custom inference machine learning process> Fig. 12 is a flowchart showing the flow of a process for learning a custom inference device that can be used by the custom inference unit 1005. The process contents shown in the flowchart of Fig. 12 start at the timing when a trigger signal for learning the custom inference device is received. The trigger signal is issued, for example, when the feedback acquisition unit 1025 acquires a certain number of pieces of data with incorrect inference results and correct answer data. Alternatively, for example, a user may transmit a trigger signal for learning the custom inference device from the second information processing device 22 at an arbitrary timing, and the process may start at the timing when the first information processing device 21 receives the trigger signal.

[0094] In S1201, the setting unit 1002 receives a trigger signal and acquires a user ID for training the custom inference device based on the authentication information of the sender or the user who added the data. The acquired user ID for training is then output to the data management unit 1009. The setting unit 1002 also issues a command to the customization unit 1013 to instruct the customization unit 10 to perform initialization settings for starting training in order to train the custom inference device. Upon receiving this command, the customization unit 10 performs the initialization settings. In this way, the customization unit 1013 performs the initialization settings, thereby preparing for training.

[0095] In S1202, the data management unit 1009 receives the user ID output by the setting unit 1002 in S1201, and extracts a data set of the target user ID. Furthermore, the data management unit 1009 checks whether or not there is an evaluable flag as described in the first embodiment, and extracts a data set of the desired user ID and for which no evaluable flag is set. Information on the extracted data set group is output to the model inference unit 1004.

[0096] In S1203, the model inference unit 1004 executes an inference process on the input data of the data set group extracted by the data management unit 1009 in S1202. The inference process executed in S1203 is the same as the inference process shown in FIG. 8 of the first embodiment, and detailed description thereof will be omitted. Note that, in FIG. 8, an example of inference using both a model and a custom inference device has been described, but this is not limiting. For example, inference using a custom inference device may not be performed, and only the inference result of the model may be output to the inference determination unit 1006 as the inference result. The obtained inference result and the supervised data of the data set group are output to the evaluation unit 1007.

[0097] In S1204, the evaluation unit 1007 receives the inference result and the correct answer data corresponding to the inference result, and performs a correct / incorrect judgment for each of the multiple data constituting the dataset. If the inference result and the correct answer data are the same, the data is judged to be correct. If the inference result and the correct answer data are different, the data is judged to be incorrect. Based on these judgment results, each of the multiple data constituting the dataset is classified as correct or incorrect. The result of classifying each of the multiple data constituting the dataset as correct or incorrect is output to the data management unit 1009.

[0098] In S1205, the data management unit 1009 receives a set of data consisting of a plurality of data whose inference results have been classified as correct or incorrect, and extracts a set of data to be used for training the custom inference device. Specifically, the data management unit 1009 extracts data classified as incorrect from the received set of data as a training data set. Information on the data extracted as the training data set is output to the customization unit 1013.

[0099] In S1206, the customization unit 1013 obtains the number of data from the information on the learning dataset output in S1205, and compares it with a predetermined threshold (fourth threshold) which is the minimum number of data required for learning. If the determination result indicates that the number of data constituting the learning dataset is equal to or greater than the predetermined threshold (fourth threshold) (YES in S1206), the process proceeds to S1207. On the other hand, if the determination result indicates that the number of data constituting the learning dataset is less than the predetermined threshold (fourth threshold) (NO in S1206), the following process is performed. That is, information indicating that the number of data constituting the dataset is insufficient and learning is not possible is sent to the second information processing device 22 through the transmission unit 1008, and is presented to the user through the display unit 1024. Note that the predetermined threshold (fourth threshold) used in S1206 is a value that can be set arbitrarily. Then, the learning process shown in FIG. 12 is completed.

[0100] In S1207, the customization unit 1013 acquires a data set from the data management unit 1009 and the storage unit 1010 based on the information on the learning data set extracted in S1205, and uses the acquired data set to train the custom inference device. The acquired data set is composed of a plurality of data, each of which is a combination (pair) of input data and correct answer data corresponding to the input data. The customization unit 1013 assigns a new identifier, such as a name, to the trained custom inference device and transmits it to the custom inference unit 1005. In training the custom inference device, the input data or a part of the input data is converted into a feature and linked to the correct answer data. A pair of the feature and the correct answer data is registered as a dictionary. During inference, the input data to be inferred or a part of the input data is converted into a feature, and a feature with a high similarity to the feature in the dictionary is searched for. If the similarity with the maximum similarity is equal to or greater than a predetermined threshold (fifth threshold), the correct answer data linked to the feature is returned as an inference result. On the other hand, if the similarity is less than the above-mentioned predetermined threshold (fifth threshold), the inference result by the custom inference device is processed as not being present. Note that the above-mentioned predetermined threshold (fifth threshold) used in searching for features with high similarity is a value that can be set arbitrarily.

[0101] In S1208, the custom inference unit 1005 receives the custom inference unit newly trained in S1207, and deploys it in a state where custom inference is possible. In addition, the setting unit 203 registers the identifier of the newly trained custom inference unit, and makes it selectable as a custom inference unit. When the process of S1208 is completed, the learning process shown in FIG. 12 is completed.

[0102] Through the above process, the stored data set can be utilized to train the custom inference device. By performing training using data that was incorrectly answered by the existing model inference unit 1004 or the custom inference unit 1005, the inference accuracy can be improved in a complementary manner.

[0103] <Model sorting process> FIG. 13 is a flowchart showing the flow of the process of organizing a model. The process shown in the flowchart of FIG. 13 starts when a trigger signal for organizing a model is received. The timing of issuing the trigger signal is, for example, the time when the process of selecting a model (S908) described in the first embodiment using FIG. 9 is completed. However, for a user immediately after introducing this information processing system, learning of a model or a custom inference device based on the data of that user or group is not often performed, and the number of items to be organized is relatively small. Therefore, the condition for issuing one trigger signal may be that the number of models or custom inference devices to be organized exceeds a certain number. The certain number is a value that can be set arbitrarily.

[0104] 9 in the first embodiment, the model management unit 1003 manages the selected model in association with its user ID and group ID. In this way, each model is associated with the user ID and group ID that use it.

[0105] In this flowchart, after the organizing process is started, processing is performed from S1301 for all models that are currently available and managed by the model management unit 1003, and the processing is repeated until the organizing process is completed for all models. In the following explanation, while the loop processing is repeatedly performed for each model, the model that is the target of the loop is called the target model.

[0106] In S1301, the model management unit 1003 checks the number of uses of the target model. Specifically, the model management unit 1003 counts the number of user IDs and group IDs linked to the target model to obtain and check the number of uses of the target model.

[0107] In S1302, the model management unit 1003 branches to the next process according to the number of uses of the target model confirmed in S1301. That is, if the number of uses is determined to be 0, the process proceeds to S1308. If the number of uses is determined to be less than a predetermined threshold (sixth threshold) T, the process proceeds to S1303. If the number of uses is determined to be equal to or greater than the predetermined threshold (sixth threshold) T, the number of users of the target model is large, so no changes are made and the sorting process shown in FIG. 13 is terminated. The predetermined threshold (sixth threshold) T is set so that the process does not proceed to S1303 without there being another model with a higher number of uses than the target model. The predetermined threshold (sixth threshold) T is set, for example, to the value of the number of uses of the model in the top 10% when the number of uses of each model except for the model with a number of uses of 0 is arranged in order of least to most.

[0108] In S1303, the model management unit 1003 selects a replacement candidate for replacing the target model with another model as a replacement candidate model. The replacement candidate model is a model that has the most correct answers for the data for which the target model answered correctly in the evaluation data set, among models that are used more frequently than the target model. However, if there are multiple models that have the same number of correct answers for the data for which the target model answered correctly, the model that has the most correct answers for the data for which the target model answered incorrectly is selected as the replacement candidate model.

[0109] In S1304, the model management unit 1003 checks whether the replacement candidate model is upwardly compatible with the target model. The condition for the replacement candidate model to be upwardly compatible with the target model is that all data for which the target model has given a correct answer in the evaluation data set is correct. If a confirmation result is obtained that the replacement candidate model is upwardly compatible with the target model (YES in S1304), the process proceeds to S1307. If a confirmation result is obtained that the replacement candidate model is not upwardly compatible with the target model (NO in S1304), the following process is performed. That is, the information of the data set that is the cause of the non-upward compatibility, that is, the data set in which the target model gave a correct answer but the replacement candidate model gave an incorrect answer, is output to the customization unit 1013. Then, the process proceeds to S1305. However, the condition for upward compatibility is not limited to that all data for which the target model has given a correct answer in the evaluation data set is correct. For example, the evaluation data set may be divided into a data set that must be correct for a user who uses the model and a data set that does not need to be correct, and if all data sets that should be correct are correct, it may be considered as upwardly compatible.

[0110] In S1305, the customization unit 1013 receives information on the data sets for which the replacement candidate models were incorrect, and trains the custom inference device based on these data sets. The custom inference device is trained by executing the processes of S1206 and S1207 in the flowchart of FIG. 12. Furthermore, the model inference unit 1004, the custom inference unit 1005, and the inference determination unit 1006 perform inference on the evaluation data set using a combination of the replacement candidate models and the trained custom inference device. The inference process performed in S1305 executes the inference process shown in FIG. 8 of the first embodiment. Then, the evaluation unit 1007 performs evaluation on the evaluation data set, and outputs the evaluation result to the model management unit 1003.

[0111] In S1306, the model management unit 1003 receives the evaluation result output in S1305 and checks whether the target model can be replaced by a combination of the replacement candidate model and the trained custom inference device. Specifically, if the received evaluation result is upwardly compatible with the evaluation result of the target model, it is determined that the target model can be replaced. The conditions for upward compatibility are as described above in S1304. If the determination result indicates that the target model can be replaced (YES in S1306), the process proceeds to S1307. If the determination result indicates that the target model cannot be replaced (NO in S1306), the number of users who use the target model is small, but there is no model that can be replaced, so the target model is left as it is, and the sorting process shown in FIG. 13 is completed.

[0112] In S1307, the model management unit 1003 replaces the target model with a replacement candidate model (and a combination of a custom inference device, hereinafter referred to as a replacement candidate model). Specifically, the user ID and group ID linked to the target model are deleted, and instead, the user ID and group ID linked to the target model are linked to the replacement candidate model. Also, if the model and custom inference device selected according to the user ID and group ID managed by the setting unit 1002 are the target model, the model and custom inference device selected according to the user ID and group ID are replaced with the replacement candidate model.

[0113] In S1308, the model management unit 1003 organizes the target model. Specifically, the target model is compressed to reduce the memory capacity relatively, and then stored in an archive unit (not shown). The compressed model stored in the archive unit can be used again as the original model by decompressing it. Here, the method of compressing and storing the model has been described as organizing the model, but the method is not limited to this. For example, the model may be deleted so that the target model cannot be used again. However, when deleting a model, a deletion flag may be attached to the target model, and the target model may be temporarily moved to a trash folder that holds data to be deleted while being linked to information on the user ID and group ID linked to the target model. The target model that has been moved to the trash folder cannot be used by the model inference unit 1004. However, if the user wants to use it again as the target model, the model management unit 1003 may move it back from the trash folder to a specified folder that manages the model. When the model is restored in this way, the user ID and group ID linked to the target model can be selectively restored. In addition, after a certain period of time has passed since the model was moved to the trash folder, the target model is completely deleted. The certain period of time can be set to any desired period of time.

[0114] In S1309, the model management unit 1003 stores the target model in the archive unit in the model information related to the managed models, and changes the setting so that it is clear that the model is not immediately usable. Alternatively, when a model is deleted, the identifier of the target model may be deleted from the managed model information.

[0115] The above process allows the models to be organized. By organizing the models, even if a large number of models are generated by model learning, it is possible to prevent the storage capacity of the first information processing device 21 from becoming congested due to unnecessary models. In particular, in S1303, by reselecting replacement candidate models for models whose usage count is less than a predetermined threshold (sixth threshold) T, it is possible to actively replace models that are used less with models that are used more, and organize the models that are used less. This allows the storage capacity of the first information processing device 21 to be used more efficiently.

[0116] <Custom reasoner cleanup process> FIG. 14 is a flowchart showing the procedure of the process of organizing the custom inference devices. The process shown in the flowchart of FIG. 14 starts when a trigger signal for organizing the custom inference devices is received. The timing of issuing the trigger signal is, for example, the time when the process of organizing the models (S1308) described with reference to FIG. 13 is completed. However, for a user immediately after the introduction of this information processing system, learning of models and custom inference devices based on the data of that user or group is not often performed, and there are relatively few targets for organizing. Therefore, the condition for issuing one trigger signal may be that the number of models and custom inference devices to be organized exceeds a certain number. The certain number is a value that can be set arbitrarily.

[0117] 9 in the first embodiment, the custom inference devices in the inference model selected by the model management unit 1003 are also managed by linking them with the user ID and group ID. In this way, each custom inference device is linked with the user ID and group ID that will use it.

[0118] Furthermore, in this flowchart, after the reorganization process is started, the process from S1401 is performed for all custom inference devices that are currently available and are managed by the model management unit 1003, and the process is repeated until the reorganization process is completed for all custom inference devices. In the following explanation, as the loop process is repeatedly performed for each custom inference device, the custom inference device that is the target of the loop will be referred to as the target custom inference device.

[0119] In S1401, the model management unit 1003 checks the number of uses of the target custom reasoner. Specifically, the model management unit 1003 obtains and checks the number of uses of the target custom reasoner by counting the number of user IDs and group IDs linked to the target custom reasoner.

[0120] In S1402, the model management unit 1003 branches to the next process depending on the number of uses of the target custom inference device confirmed in S1401. That is, the model management unit 1003 determines whether the number of uses of the target custom inference device is 0. If the determination result indicates that the number of uses of the target custom inference device is 0 (YES in S1402), the process proceeds to S1403. If the determination result indicates that the number of uses of the target custom inference device is not 0 (NO in S1402), since there is a user of the target custom inference device, the model management unit 1003 makes no changes and ends the organization process shown in FIG.

[0121] In S1403, the model management unit 1003 organizes the target custom inference device. Specifically, the target custom inference device is deleted so that the target custom inference device cannot be used. However, when deleting a custom inference device, a deletion flag may be attached to the target custom inference device, and the target custom inference device may be temporarily moved to the trash folder while being linked to the information of the user ID and group ID linked to the target custom inference device. The target custom inference device that has been moved to the trash folder cannot be used by the custom inference unit 1005. However, if the user wants to use it again as the target custom inference device, the model management unit 1003 may move it back from the trash folder to a predetermined folder that manages custom inference devices. When the custom inference device is restored in this way, the user ID and group ID linked to the target custom inference device can be selectively restored. Furthermore, when a certain period of time has passed after being moved to the trash folder, the target custom inference device is completely deleted. Note that the certain period of time can be set to any time.

[0122] In S1404, the model management unit 1003 deletes the identifier of the target custom reasoner from the custom reasoner information related to the managed custom reasoners.

[0123] By carrying out the above process, the custom reasoners can be organized. By organizing the custom reasoners, the number of combinations of models and custom reasoners can be reduced, and the time required for the above-mentioned model selection process can be relatively shortened.

[0124] As described above, according to this embodiment, a part of a data set including data accumulated while a user uses this information processing system can be used to learn and generate a new inference model. The learned inference model is learned based on the data used by the user, so that a more customized inference result can be provided to the user. In addition, by managing the inference model and the data set by linking them with a user ID or a group ID, it is possible to customize the data of an organization, not only for use by an individual user, but also for use by an organization such as a company. In addition, by linking the inference model with a user ID or a group ID and comparing the inference models with each other for the evaluation data set, it is possible to compress or delete the inference model. As a result, an inference model is generated by learning, but when the inference models continue to increase, the capacity of the server becomes tight, and it is possible to organize the appropriate inference model.

[0125] (Modification 1 of the second embodiment) Although the embodiment of sharing and learning a model among groups has been described using FIG. 11, the present invention is not limited thereto. For example, a model may be shared among multiple groups to perform learning with a larger data set. In that case, an administrator capable of notifying multiple groups notifies the learning plan of the model, transmits a trigger signal to start learning, and performs learning according to the flowchart of FIG. 11. In S1102, a data set that matches any of multiple group IDs is extracted. As a result, since the model is learned using more data sets than in the case of one group, the stability of learning and the versatility of the model can be improved. In addition, for example, learning may be performed for each user on a model. In that case, a trigger signal to start learning by the user is transmitted, and learning is performed according to the flowchart of FIG. 11. In S1102, a data set that matches the target user ID is extracted. As a result, since learning is performed based only on the data set used by the target user, a highly unique model customized for the user can be learned.

[0126] (Modification 2 of the second embodiment) Although the embodiment in which the custom inference machine is trained for each user has been described with reference to FIG. 12, the present invention is not limited to this. For example, the custom inference machine may be shared within the same group, and training may be performed with a larger data set. In this case, in S1202, a data set having a desired group ID and no evaluable flag is extracted. In this way, by sharing the custom inference machine within the same group, for example, if the same organization within a company is registered as a group, the data sets of each user (employee) within the organization can be integrated to train the custom inference machine for the organization's unique tasks. This makes it possible to collect data for customization to the organization's unique tasks more efficiently, and as a result, it is expected that the accuracy of the custom inference machine will be improved.

[0127] (Modification 3 of the second embodiment) Although the embodiment of learning the custom inference device after a certain number of data or more are collected and a data set is obtained, the embodiment is not limited to this. For example, the threshold value of the certain number may be set to 1, and when even one incorrect data is obtained, the customization unit 1013 may execute learning of the custom inference device. In this case, if there is no change in the configuration of the inference model between the time when the inference result is output and the time when feedback from the user is obtained, the processes from S1202 to S1204 can be omitted. In other words, it is self-evident that the one incorrect data is already incorrect in the current inference model, and it can be used as a data set including data classified as incorrect in S1205. Then, in S1206 and thereafter, the inference result corrected by the customization unit 1013 through feedback is registered in the dictionary as correct data by obtaining and linking the feature amount, and learning is completed. In this way, when the current inference result is incorrect, by immediately learning the custom inference device, it is possible to obtain an inference result that immediately responds to the error and does not repeat the same error.

[0128] (Fourth Modification of the Second Embodiment) Although the trigger for starting model learning has been described with reference to FIG. 11 as being issued by a system administrator, the present invention is not limited to this. For example, a trigger may be issued when the number of feedbacks from users exceeds a certain number. In this way, the scale of model learning can be kept within a certain range, so learning can be stabilized by learning with a proven number of data, and the time required for learning can be easily estimated. For example, a trigger may be issued when similar feedback is obtained from another user using the same model or another user in the same group. In this way, model learning can be performed before similar feedback arrives from other users to improve inference accuracy, and it is possible to prevent erroneous inference results from being output to other users.

[0129] (Other embodiments) In the first embodiment, the data management unit 409 manages the input data acquired by the receiving unit 401 in association with the corresponding correct answer data, but the present invention is not limited thereto. For example, when the first information processing device 11 is started as a service, it is necessary to learn a model to be set in the model inference unit 404 in advance using initial learning data. This initial learning data is prepared by the service provider and used for learning the initial model. This learning data may be stored in the storage unit 410 and managed by the data management unit 409. In this way, by the data management unit 409 managing the initial learning data, the data management unit 1009 can extract the initial learning data in S1102 of FIG. 11 in the second embodiment, and it becomes easy to make the number of data equal to or greater than the threshold in S1103 of FIG. 11. As a result, the user can start using the service and learn the model at an early stage.

[0130] The present disclosure can also be realized by executing the following process. That is, the present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or various storage media, and having one or more processors in a computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0131] The disclosure of this embodiment includes the following configuration examples. (Configuration 1) An information processing device that processes a predetermined task by inference using a trained model, an acquisition means for acquiring a pair of input data for the predetermined task and correct answer data corresponding to the input data as feedback information; an inference means for inputting the input data constituting the pair related to the acquired feedback information into each of a plurality of trained models prepared in advance corresponding to the predetermined task, and performing inference for the predetermined task; A specification means for specifying a trained model suitable for processing the predetermined task from among the trained models based on a plurality of inference results obtained by the inference and the correct answer data that constitutes a pair with the input data input to each of the trained models; 13. An information processing device comprising:

[0132] (Configuration 2) The acquiring means presents an inference result obtained by inputting input data of the predetermined task into the trained model to a user, and acquires the input data input into the trained model and the supervised answer data paired with the input data as the feedback information from the user. 2. The information processing device according to configuration 1.

[0133] (Configuration 3) The method further includes a storage unit for storing the input data and the correct answer data that constitute the pair related to the feedback information in association with each other, The identification means identifies the trained model suitable for processing the predetermined task based on a plurality of inference results obtained by the inference when the input data held by the holding means is input to each of the plurality of trained models, and based on the supervised answer data that forms a pair with the input data input to each of the plurality of trained models. 3. The information processing device according to configuration 2.

[0134] (Configuration 4) The identification means identifies a trained model suitable for processing the predetermined task according to an evaluation index obtained by inputting input data of the predetermined task into each of the plurality of trained models and comparing a plurality of inference results obtained by the inference with the supervised data that forms a pair with the input data input into each of the plurality of trained models. 2. The information processing device according to configuration 1.

[0135] (Configuration 5) The inference means includes: A first inference means for inputting the input data into a first trained model that has been machine-learned in advance among the plurality of trained models, and performing inference for the predetermined task; a second inference means for inputting the input data into a second trained model among the plurality of trained models, the second trained model being machine-learned in advance with less training data than the first trained model, and performing inference for the predetermined task; having The identification means identifies a combination of the first trained model and the second trained model based on the evaluation index obtained by inputting input data of the predetermined task into the first trained model and comparing an inference result obtained by the inference with the supervised data paired with the input data input into the first trained model, and the evaluation index obtained by inputting input data of the predetermined task into the second trained model and comparing a plurality of inference results obtained by the inference with the supervised data paired with the input data input into the second trained model. 5. The information processing device according to configuration 4.

[0136] (Configuration 6) When a condition for identifying a combination of the first trained model and the second trained model is set in advance, the identification means identifies the combination of the first trained model and the second trained model in accordance with the preset condition. 6. The information processing device according to configuration 5.

[0137] (Configuration 7) When a pair of the input data to be corrected and the correct answer data is set in advance as the condition, the identification means identifies a combination of the first trained model and the second trained model having a relatively high evaluation index for the pair of the input data to be corrected and the correct answer data. 7. The information processing device according to configuration 6.

[0138] (Configuration 8) The identification means, when a reliability calculated from an inference result obtained by inputting input data of the predetermined task into the first trained model and performing the inference exceeds a first threshold, identifies the first trained model as a trained model suitable for processing the predetermined task. 6. The information processing device according to configuration 5.

[0139] (Configuration 9) The identification means, when a reliability calculated from the inference result obtained by inputting input data of the predetermined task into the first trained model and performing the inference is equal to or less than a first threshold, and a reliability calculated from the inference result obtained by inputting input data of the predetermined task into the second trained model and performing the inference is greater than a second threshold different from the first threshold, identifies the second trained model as a trained model suitable for processing the predetermined task. 9. The information processing device according to configuration 5 or 8.

[0140] (Configuration 10) The identification means, when a reliability calculated from the inference result obtained by inputting input data of the predetermined task into the first trained model and performing the inference is equal to or less than a first threshold, but a reliability calculated from the inference result obtained by inputting input data of the predetermined task into the second trained model and performing the inference is equal to or less than a second threshold different from the first threshold, identifies the first trained model as a trained model suitable for processing the predetermined task. 10. The information processing device according to configuration 5 or 9.

[0141] (Configuration 11) The present invention further includes a first learning means for learning the trained model based on the correct answer data that is paired with the input data when an inference result obtained by inputting the input data into the trained model identified by the identification means differs from the correct answer data that is paired with the input data input into the trained model. 11. The information processing device according to any one of configurations 1 to 10.

[0142] (Configuration 12) Further comprising a second learning means for learning the first trained model, When a new first trained model is generated by the second learning means, The identification means is performing an evaluation based on a combination of the input data and the correct answer data that forms a pair with the input data; Identifying a combination of the first trained model and the second trained model based on an evaluation result. 11. The information processing device according to any one of configurations 5 to 10.

[0143] (Configuration 13) The second learning means learns the first trained model when the number of pieces of feedback information is equal to or greater than a certain number. 13. The information processing device according to configuration 12.

[0144] (Configuration 14) The second learning means, when a user other than the user who uses the information processing device acquires feedback information on a result obtained for the input data in the information processing device, learns the first trained model based on correct answer data constituting the pair related to the feedback information of the other user. 14. The information processing device according to configuration 13.

[0145] (Configuration 15) The second learning means, when the other user belongs to the same group as the user who uses the information processing device, learns the first trained model based on the correct answer data constituting the pair related to the feedback information of the other user. 15. The information processing device according to configuration 14.

[0146] (Configuration 16) Further comprising a management means for managing the first trained model and the second trained model. 16. An information processing device according to any one of configurations 5 to 10 and 12 to 15.

[0147] (Configuration 17) The management means compresses or deletes models that are used by zero users from among the first trained model and the second trained model that are managed by the management means to organize the models. 17. The information processing device according to configuration 16.

[0148] (Configuration 18) The management means extracts a model that shows an evaluation result of superior compatibility for a pair of the input data and the correct answer data from among the first trained models managed by the management means and replaces the model used by a relatively small number of users with the extracted model. 18. The information processing device according to configuration 16 or 17.

[0149] (Configuration 19) The method further includes setting means for setting the first trained model indicated by the specified combination among the first trained models managed by the management means to be used by the first inference means, and setting the second trained model indicated by the specified combination among the second trained models managed by the management means to be used by the second inference means. 19. The information processing device according to any one of configurations 16 to 18.

[0150] (Configuration 20) The storage means stores the input data in association with at least one of an authentication ID for authenticating a user and a group ID for authenticating users on a group basis. 4. The information processing device according to configuration 3.

[0151] (Configuration 21) The predetermined task is a task of inferring a region of interest in an image represented by input image data. 21. The information processing device according to any one of configurations 1 to 20.

[0152] (Configuration 22) The predetermined task is a task of inferring text corresponding to a specific item from text obtained by performing character recognition processing on an input document image. 21. The information processing device according to any one of configurations 1 to 20.

[0153] (Configuration 23) The predetermined task is a task of inferring text data from input voice data. 21. The information processing device according to any one of configurations 1 to 20.

[0154] (Configuration 24) An information processing system comprising: an information processing device according to any one of configurations 1 to 23; and an information terminal communicably connected to the information processing device via a network, The information terminal includes: A data acquisition means for acquiring the input data; a display means for inputting the input data acquired by the data acquisition means to the inference means and displaying an inference result obtained by the inference; a feedback acquisition means for acquiring feedback from a user on the displayed inference result as the feedback information; An information processing system comprising:

[0155] (Configuration 25) The information terminal further includes an authentication means for authenticating a user ID that identifies a user. 25. The information processing system according to configuration 24.

[0156] (Configuration 26) The authentication means authenticates a group to which the user belongs by a group ID that identifies the group to which the user belongs. 26. An information processing system according to configuration 25.

[0157] (Configuration 27) An information processing method for processing a predetermined task by inference using a trained model, comprising: an acquisition step of acquiring a pair of input data for the predetermined task and correct answer data corresponding to the input data as feedback information; an inference step of inputting the input data constituting the pair related to the acquired feedback information into each of a plurality of trained models prepared in advance corresponding to the predetermined task, and performing inference for the predetermined task; A step of identifying a trained model suitable for processing the predetermined task from among the trained models based on a plurality of inference results obtained by the inference and the correct answer data that forms a pair with the input data input to each of the trained models; 13. An information processing method comprising:

[0158] (Configuration 28) A program for causing a computer to function as each of the means of an information processing device described in any one of configurations 1 to 23. [Explanation of symbols]

[0159] 11: First information processing device 401: Receiving unit 404: Model inference section 405: Custom Inference 411: Model selection section

Claims

1. A management means for managing multiple trained models, An inference means that performs inference using one or more models from the aforementioned plurality of trained models, An acquisition means for acquiring user modifications to the inference results obtained by the aforementioned inference, A means for identifying, based on the user's modifications, a setting for performing inference using one or more of the aforementioned multiple trained models, A setting means for making the settings for performing the inference based on the identified settings, An information processing device characterized by having the following features.

2. In inference using a predetermined model included in the plurality of trained models, the inference means From the registered data, search for data similar to the entered data. The data associated with the similar data found through the search is output as the inference result of the inference. The information processing apparatus according to feature 1.

3. The inference means obtains the similarity between the input data and the registered data in the search, and determines similar data based on the obtained similarity. The information processing apparatus according to feature 2.

4. Recording means for recording identification information of a trained model used in the inference performed based on the specified settings. The information processing apparatus according to claim 1, further comprising the features.

5. Notification means for notifying a client terminal of the inference result obtained by the inference. The information processing apparatus according to claim 1, further comprising the features.

6. The acquisition means acquires as feedback information a pair of input data used in the inference and user modifications to the inference result obtained by the inference using the input data, The identification means inputs the input data constituting the pair relating to the acquired feedback information into each of the plurality of trained models and performs inference, and identifies a trained model suitable for the inference from among the plurality of trained models based on the plurality of inference results obtained by the inference and the modifications that constitute a pair with the input data input into each of the plurality of trained models. The information processing apparatus according to feature 1.

7. The system further includes a holding means for linking and holding the input data and the modification that constitute the pair relating to the feedback information, The identification means inputs the input data held by the holding means into each of the plurality of trained models to perform inference, and identifies a trained model suitable for the inference based on the plurality of inference results obtained by the inference and the modifications that form pairs with the input data input into each of the plurality of trained models. The information processing apparatus according to feature 6.

8. The identification means inputs the input data into each of the plurality of trained models and performs inference, and identifies a trained model suitable for the inference according to an evaluation index obtained by comparing the plurality of inference results obtained by the inference with the modifications that form a pair with the input data input to each of the plurality of trained models. The information processing apparatus according to feature 6.

9. The aforementioned inference means is A first inference means that performs inference by inputting input data into a first pre-trained model, which is one of the multiple pre-trained models that has been pre-machine-trained, and A second inference means performs inference by inputting input data into a second pre-trained model, which is pre-trained with less training data than the first pre-trained model among the multiple pre-trained models, The information processing apparatus according to claim 1, characterized by having the following features.

10. The identifying means identifies a combination of the first trained model and the second trained model based on an evaluation index obtained by comparing the inference results obtained by inputting the input data into the first trained model with the modifications that form a pair with the input data input into the first trained model, and an evaluation index obtained by comparing a plurality of inference results obtained by inputting the input data into the second trained model with the modifications that form a pair with the input data input into the second trained model. The information processing apparatus according to feature 9.

11. If conditions for identifying a combination of the first trained model and the second trained model are set in advance, the identifying means identifies the combination of the first trained model and the second trained model according to the set conditions. The information processing apparatus according to feature 9.

12. If, as a condition, pairs of input data and corrections to be corrected are pre-defined, the identification means identifies a combination of the first trained model and the second trained model that have relatively high evaluation metrics for the pairs of input data and corrections to be corrected. The information processing apparatus according to feature 11.

13. The identification means identifies the first trained model as a trained model suitable for the inference if the confidence level obtained from the inference result obtained by inputting the input data into the first trained model exceeds a first threshold. The information processing apparatus according to feature 9.

14. The identifying means identifies the second trained model as a trained model suitable for the inference if the confidence level obtained from the inference result obtained by inputting the input data into the first trained model is less than or equal to a first threshold, and the confidence level obtained from the inference result obtained by inputting the input data into the second trained model is greater than a second threshold that is different from the first threshold. The information processing apparatus according to feature 9.

15. The identification means identifies the first trained model as a trained model suitable for the inference if the confidence level obtained from the inference result obtained by inputting the input data into the first trained model is less than or equal to a first threshold, but the confidence level obtained from the inference result obtained by inputting the input data into the second trained model is less than or equal to a second threshold that is different from the first threshold. The information processing apparatus according to feature 9.

16. If the inference result obtained by inputting the input data into the trained model identified by the identification means differs from the modification that forms a pair with the input data input into the trained model, the system further comprises a first learning means that trains the trained model based on the modification that forms a pair with the input data. The information processing apparatus according to feature 6.

17. The system further comprises a second learning means for training the first pre-trained model, When a new first trained model is generated by the second learning means, The aforementioned specifying means is, An evaluation is performed based on the combination of the input data and the modification that forms a pair with the input data. Based on the evaluation results, identify the combination of the first trained model and the second trained model. The information processing apparatus according to feature 9.

18. The acquisition means acquires as feedback information a pair of input data used in the inference and user modifications to the inference result obtained by the inference using the input data, The second learning means performs training on the first trained model if the number of feedback pieces of information is greater than or equal to a certain number. The information processing apparatus according to feature 17.

19. The second learning means, when another user different from the user using the information processing device obtains feedback information on the results obtained for the input data using the information processing device, performs learning on the first trained model based on the modifications that constitute the pair relating to the feedback information of the other user. The information processing apparatus according to feature 18.

20. The second learning means, when the other user belongs to the same group as the user using the information processing device, performs the first trained model training based on the modifications that constitute the pair relating to the other user's feedback information. The information processing apparatus according to feature 19.

21. The management means compresses or deletes models from among the multiple trained models under management that have zero users, thereby organizing them. The information processing apparatus according to feature 1.

22. The management means extracts models from among the trained models under management that have a relatively small number of users, models that show a superior evaluation result for the input data and the modification pair, and replaces the models with relatively few users with these models. The information processing apparatus according to feature 6.

23. The system further includes a setting means for setting the first inference means to use the first trained model represented by the specified combination from among the first trained models managed by the management means, and the second inference means to use the second trained model represented by the specified combination from among the second trained models managed by the management means. The information processing apparatus according to feature 9.

24. The storage means stores the input data in association with at least one of the following: an authentication ID for authenticating a user, or a group ID for authenticating users on a group basis. The information processing apparatus according to feature 7.

25. The aforementioned inference is a process that infers the region of interest in the image represented by the input image data. The information processing apparatus according to feature 1.

26. The aforementioned inference is a process that infers text corresponding to a specific item from the text obtained by performing character recognition processing on an input document image. The information processing apparatus according to feature 1.

27. The aforementioned inference is a process that infers text data from input audio data. The information processing apparatus according to feature 1.

28. An information processing system comprising an information processing device according to claim 1 and an information terminal that is communicably connected to the information processing device via a network, The acquisition means acquires, as feedback information, a pair of input data used in the inference and user modifications to the inference result obtained by the inference using the input data. The aforementioned information terminal is A data acquisition means for acquiring the aforementioned input data, A display means that inputs the input data acquired by the data acquisition means to the inference means and displays the inference result obtained by the inference, A feedback acquisition means for acquiring user feedback on the displayed inference results as feedback information, An information processing system characterized by having the following features.

29. The aforementioned information terminal further includes authentication means for authenticating a user ID that identifies a user. The information processing system according to feature 28.

30. The aforementioned authentication means authenticates the group to which the user belongs using a group ID that identifies the group to which the user belongs. The information processing system according to feature 29.

31. A management process for managing multiple trained models, An inference step which performs inference using one or more models from the aforementioned plurality of trained models, A step of obtaining user modifications to the inference results obtained by the aforementioned inference, A selection step in which the settings for performing inference using one or more of the aforementioned multiple trained models are identified based on the user's modifications, A setting step of setting up the settings for performing the inference based on the identified settings, An information processing method characterized by having the following features.

32. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

Images