Information processing apparatus, information processing system, information processing method, and program

Abstract

To solve the problem in which it is difficult for a user to select a learned model suitable for the purpose of use.SOLUTION: An information processing apparatus comprises: management means that manages database information in which model information on a learned model and at least one of good-at objects that a user is good at and poor-at objects that the user is poor at, from among processing objects to be processed by the learned model, are linked to each other; and generation means that, on the basis of the database information, generates display control information for displaying the model information and information on at least one of the good-at objects and the poor-at objects linked to the model information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] In recent years, systems using machine learning technology have been put into practical use in a variety of fields. The trained models used in these systems are not only created by companies and organizations, but can also be created by ordinary users for their own purposes, resulting in a huge number of machine learning models.

[0003] There are also internet-based services that allow general users to publish and obtain machine learning models they have created, and general users who use these services must select and obtain a model that is suitable for their own purposes from the vast number of publicly available trained models.

[0004] For example, Patent Document 1 proposes a method for assisting the user in selecting a trained model by displaying the outputs of multiple trained models and the success / failure judgment results to the user in a tabular format. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-204970 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the conventional technology disclosed in the above-mentioned patent document only displays the output of multiple trained models and the success / failure judgment results in a tabular format, which poses a problem that it is difficult for users to select a trained model that is suitable for their application from the many publicly available machine learning models.

[0007] Therefore, the present invention provides an information processing device, an information processing system, an information processing method, and a program that assist users in selecting a trained model using machine learning, allowing users to easily select a trained model that is suitable for their application. [Means for solving the problem]

[0008] In order to solve this problem, for example, an information processing device of the present invention has the following arrangement: A management means for managing database information in which model information about a trained model and at least one of a good target and a poor target among processing targets processed by the trained model are linked; A generation means for generating display control information for displaying the model information and information relating to at least one of the good target and the bad target linked to the model information based on the database information; Equipped with. [Effects of the Invention]

[0009] According to the present invention, an information processing device, an information processing system, an information processing method, and a program are provided that assist users in selecting a trained model using machine learning, allowing users to easily select a trained model that is suitable for their application. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a hardware configuration diagram of a server-side information processing apparatus in the information processing system according to the embodiment. [Figure 2] FIG. 2 is a hardware configuration diagram of a client-side information processing apparatus in the information processing system according to the embodiment. [Figure 3] FIG. 1 is a diagram illustrating a configuration of an information processing system according to an embodiment. [Figure 4] FIG. 1 is a block diagram showing an example of the configuration of an information processing system according to a first embodiment. [Figure 5]FIG. 10 is a flowchart showing the procedure of a display process for displaying a list of good images and bad images linked to model information. [Figure 6] FIG. 10 is a diagram showing an example of database information for managing good images and bad images in association with model information. [Figure 7] FIG. 10 is a diagram showing an example of a list display of model information and good images and bad images linked to the model information. [Figure 8] FIG. 10 is a flowchart showing the procedure of a model evaluation process. [Figure 9] FIG. 10 is a block diagram showing an example of the configuration of an information processing system according to a second embodiment. [Figure 10] FIG. 10 is a flowchart showing the procedure of a display process for displaying a list of model information linked in a database to a favorite image that is the subject of a search instruction. [Figure 11] FIG. 10 is a diagram showing database information for managing good images and bad images linked to model information, and database information for managing photography information. [Figure 12] FIG. 10 is a diagram showing an example of a list display of favorite images. [Figure 13] FIG. 10 is a diagram showing an example of a list display of model information. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] An information processing system according to an embodiment will be described below. The hardware configurations of a server-side information processing device and a client-side information processing device included in the information processing system according to the embodiment will be described with reference to Figs. 1 and 2. The server may be a virtual server provided in a cloud or the like. The information processing device is an example of a computer.

[0013] FIG. 1 is a hardware configuration diagram of an information processing device 10 on the server side of an information processing system according to an embodiment.

[0014] The information processing device 10 is realized by, for example, a cloud server. The information processing device 10 has a CPU 1101, a ROM 1102, a RAM 1103, an I / F 1104, and a system bus 1105. The CPU 1101, the ROM 1102, the RAM 1103, and the I / F 1104 are connected via the system bus 1105 so as to be able to send and receive data to and from each other.

[0015] The CPU 1101 is an abbreviation for Central Processing Unit and is an arithmetic processing device. The CPU 1101 controls the ROM 1102, RAM 1103, I / F 1104, and various devices connected to a system bus 1105. The information processing device 10 may include other processors such as an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), and a QPU (Quantum Processing Unit) instead of or in addition to the CPU 1101. Some or all of the functions of the information processing device 10, which will be described later, are realized by one or more processors including the CPU 1101 reading out programs stored in the ROM 1102, etc., loading the programs into the RAM 1103, and executing them. Some or all of the functions of the information processing device 10 may also be realized by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0016] ROM 1102 is an abbreviation for Read Only Memory and is a non-volatile storage device. ROM 1102 stores various programs such as a BIOS (Basic Input / Output System) program and a boot program. Instead of or in addition to ROM 1102, information processing device 10 may have a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0017] The RAM 1103 is an abbreviation for Random Access Memory. The RAM 1103 is a memory that can read and write data at high speed, and is used as the main storage device of the CPU 1101. For example, the RAM 1103 functions as a work area when the CPU 1101 executes a program.

[0018] The I / F 1104 functions as an interface for communication and with external devices. For example, the I / F 1104 functions as a communication interface for communicating information via a network. The communication interface may be Ethernet (registered trademark), USB, serial communication, wireless communication, or the like, and is not particularly limited to a particular type.

[0019] FIG. 2 is a hardware configuration diagram of the client-side information processing device 11 of the information processing system according to the embodiment.

[0020] The information processing device 11 is a terminal device on which a user views and operates a screen. The information processing device 11 may be, for example, a PC (personal computer) or a tablet terminal. The information processing device 11 has a CPU 1111, a ROM 1112, an input device 1113, a display device 1114, a RAM 1115, a hard disk 1116, a media drive 1117, an I / F 1118, and a system bus 1119. The CPU 1111, the ROM 1112, the input device 1113, the display device 1114, the RAM 1115, the hard disk 1116, the media drive 1117, and the I / F 1118 are connected via the system bus 1119 so as to be able to transmit and receive data to and from each other.

[0021] The CPU 1111 is an arithmetic processing unit. The CPU 1111 controls the ROM 1112 to the I / F 1118 and various devices connected to a system bus 1119. The information processing device 11 may have other processors such as an MPU, a GPU, or a QPU instead of or in addition to the CPU 1111. Some or all of the functions of the information processing device 11, which will be described later, are realized by one or more processors including the CPU 1111 reading out a program stored in either the ROM 1112 or the hard disk 1116, expanding the program in the RAM 1115, and executing the program. Some or all of the functions of the information processing device 11 may also be realized by one or more circuits such as an ASIC and a PLD including an FPGA.

[0022] The ROM 1112 is a non-volatile storage device that stores a BIOS (Basic Input Output System) program, a boot program, and the like.

[0023] The input device 1113 receives information input by a user, etc. The input device 1113 is, for example, a touch panel, a keyboard, a mouse, or a robot controller.

[0024] The display device 1114 displays the calculation results of the information processing device 11 and images transmitted from the server-side information processing device 10 in accordance with instructions from the CPU 1111. The display device 1114 is not particularly limited to a particular type as long as it is a device that displays images and information, such as a liquid crystal display device, a projector, or an LED indicator.

[0025] The RAM 1115 is a memory that can read and write data at high speed, and is used as the main storage device of the CPU 1111. For example, the RAM 1115 functions as a work area when the CPU 1111 executes a program.

[0026] The hard disk 1116 is, for example, a nonvolatile large-capacity storage device and is used as storage. The hard disk 1116 stores applications, data such as image data, libraries, etc. The hard disk 1116 reads or writes data based on instructions from the CPU 1111.

[0027] The media drive 1117 is connected to a removable storage medium. The media drive 1117 writes, for example, data stored in the hard disk 1116 to the connected storage medium. By writing data to the storage medium, the media drive 1117 transfers the data to an external device, for example, a digital still camera, a computer such as a PC, or a tablet terminal.

[0028] The I / F 1118 is a communication interface that realizes information communication with an external device via a network. The I / F 1118 realizes information communication with, for example, the server-side information processing device 10. The communication interface may be Ethernet, USB (Universal Serial Bus), serial communication, wireless communication, or the like, and is not particularly limited to a specific type of communication.

[0029] 3 is a configuration diagram of an information processing system according to an embodiment. The information processing system according to this embodiment includes a server-side information processing device 10 and a client-side information processing device 11. The server-side information processing device 10 communicates with one or more client-side information processing devices 11, and transmits and receives data and controls display information.

[0030] The terms used in the description of this embodiment are defined as follows. A "good image" is an example of a good target, and is an evaluation image in the evaluation data used to evaluate a trained model that is determined to have high detection accuracy when using the trained model. "Weak image" is an example of a weak target, and is an evaluation image in the evaluation data used to evaluate the trained model that is determined to have low detection accuracy using the trained model. "Common evaluation data" refers to common evaluation data used when evaluating all trained models managed by the information processing system of this embodiment. The common evaluation data includes evaluation images used for evaluation of each task and detection accuracy information associated with each evaluation image.

[0031] (First embodiment) The information processing system of the first embodiment evaluates trained models using machine learning on a cloud server, and associates images that the trained model is good at and images that it is not good at calculated from the evaluation results with model information and displays them on a client-side terminal, thereby supporting a user's task of selecting a trained model. As an example of a task handled in this embodiment, an object detection task for detecting an object contained in an image input to a model will be used for explanation. The object detection task is a task in which, using image data as input, if a specific object is captured in the image, the task infers a bounding box that surrounds the area of ​​the object. However, the types of tasks that can be handled in this embodiment are not limited to this example. An image is an example of a processing target processed by a trained model.

[0032] Fig. 4 is a block diagram showing an example of functions of the information processing system of the first embodiment. Fig. 4 is an example of a functional configuration example, and does not limit the scope of application of this embodiment. The server-side information processing device 10 includes a server receiving unit 101, a model evaluation unit 102, a determination unit 103, a management unit 104, a data holding unit 105, a generation unit 106, and a server transmission unit 107. The client-side information processing device 11 includes a model information acquisition unit 111, a client transmission unit 112, a client receiving unit 113, and a display unit 114.

[0033] The client transmitting unit 112 and the server receiving unit 101, and the server transmitting unit 107 and the client receiving unit 113 communicate information via a network. The server transmitting unit 107 and the client transmitting unit 112 convert data into a format suitable for transmission and then transmit the data. The server receiving unit 101 and the client receiving unit 113 convert the received data into data in a format that is easy to use at the output destination and output it.

[0034] The server receiving unit 101 receives information from the client-side information processing device 11 via a network. The server receiving unit 101 receives, for example, model information of a trained model. The received model information is output to the model evaluation unit 102 and the management unit 104.

[0035] The model evaluation unit 102 receives the model information, performs evaluation of the trained model indicated by the model information, and generates a model evaluation result. Details of the model evaluation will be explained later using a flowchart showing the processing procedure for displaying a list of good images and bad images linked to the model information. The model evaluation unit 102 outputs the model evaluation result to the determination unit 103.

[0036] The determination unit 103 determines the favorable images and the unfavorable images based on the model evaluation results. Details of the process of determining the favorable images and the unfavorable images will be explained later using a flowchart showing the processing procedure for displaying a list of favorable images and unfavorable images linked to model information. The determination unit 103 outputs information on the determined favorable images and unfavorable images to the management unit 104.

[0037] The management unit 104 generates and updates database information for linking and managing the received model information with the determined good images and bad images. The management unit 104 outputs the model information, good images, and bad images to a data storage unit 105 in order to store data such as images included in the database information as entities. The management unit 104 outputs the database information it manages to a generation unit 106.

[0038] The data storage unit 105 receives model information, good images, and poor images from the management unit 104 and stores them based on database information. The data storage unit 105 stores evaluation images before they are evaluated as good images or poor images, and correct answer information for the evaluation images. Within the server-side information processing device 10, the use and deletion of data such as model information, good images, poor images, evaluation images, and correct answer information stored in the data storage unit 105 is performed via the management unit 104.

[0039] The generating unit 106 generates, based on the database information, display control information for display on the client-side information processing device 11. The generating unit 106 outputs the generated display control information to the server transmitting unit 107.

[0040] The server transmitting unit 107 transmits the display control information received from the generating unit 106 to the client receiving unit 113 of the client-side information processing device 11 via the network.

[0041] The model information acquisition unit 111 acquires model information of a model trained by machine learning. The model information acquisition unit 111 outputs the acquired model information to the client transmission unit 112.

[0042] The client transmitting unit 112 transmits the model information received from the model information acquiring unit 111 to the server receiving unit 101 of the server-side information processing device 10 via the network.

[0043] The client receiving unit 113 receives information from the server-side information processing device 10 via the network. The received information is display control information. The received display control information is output to the display unit 114.

[0044] The display unit 114 displays an image to the user based on the display control information received from the client receiving unit 113. The display unit 114 displays the image using a display device 1114 capable of displaying images, such as a display monitor, a head-mounted display, a touch panel, or a projector.

[0045] Next, the processing procedure in this embodiment will be described. In the following description, each process (step) will be numbered with an S at the beginning.

[0046] 5 is a flowchart showing the steps of a display process in which an information processing system consisting of a client-side information processing device 11 and a server-side information processing device 10 displays a list of good images and bad images linked to model information. The process shown in the flowchart in FIG. 5 starts, for example, when model information to be evaluated is input to the client-side information processing device 11. However, the information processing system does not necessarily have to perform all of the steps described in this flowchart, and the order of the processes may be changed.

[0047] As preparation for implementing this flowchart, the client-side information processing device 11 initializes the system. For example, the CPU 1111 reads a program from the ROM 1112 or the hard disk 1116, and puts the information processing device 11 into an operable state. In the server-side information processing device 10, the CPU 1101 reads a program from the ROM 1102, and puts the information processing device 10 into an operable state. As a result, the client-side information processing device 11 and the server-side information processing device 10 are able to communicate with each other using the client transmitting unit 112, the server receiving unit 101, the server transmitting unit 107, and the client receiving unit 113.

[0048] In S1001, the model information acquisition unit 111 of the client-side information processing device 11 acquires model information registered on the hard disk 1116 or the like. The model information includes, for example, entity data such as parameters of the trained model, the user name who registered the model, the date the trained model was registered, and a task name indicating the name or type of task in which the trained model is used. However, the type of model information to be acquired is not limited to this, and may be entity data of the model and any information related to the model. The model information acquisition unit 111 outputs the model information to the client transmission unit 112, and the process proceeds to S1002s.

[0049] In S1002s, the client transmitting unit 112 performs data conversion on the acquired model information. The client transmitting unit 112 performs data conversion processing, for example, compressing and encrypting the model information into a format suitable for transmission. The client transmitting unit 112 transmits the data-converted model information to the server receiving unit 101 of the server-side information processing device 10.

[0050] In S1002r, the server receiving unit 101 of the server-side information processing device 10 performs conversion processing such as data decoding and decompression on the received model information to convert the model information into its original format. The server receiving unit 101 outputs the model information to the model evaluation unit 102 and management unit 104, and the process proceeds to S1003.

[0051] In S1003, the model evaluation unit 102 executes a model evaluation process using the model information and common evaluation data received from the server receiving unit 101. As a result of executing the model evaluation process, the model evaluation unit 102 calculates the inference reliability and inference judgment result of the trained model for all evaluation images. Details of the model evaluation process will be described later. The model evaluation unit 102 outputs the inference reliability and inference judgment result of the trained model corresponding to all the obtained evaluation images to the determination unit 103, and proceeds to S1004.

[0052] In S1004, the determination unit 103 determines one or more good images and poor images from the evaluation images based on the inference reliability and the inference judgment result corresponding to all evaluation images received from the model evaluation unit 102. The determination unit 103 determines the good images by extracting evaluation images determined to have been detected successfully in the inference judgment result, and determining at least one or more of the extracted evaluation images in descending order of inference reliability as good images. The determination unit 103 determines the poor images by extracting evaluation images determined to have been detected unsuccessfully in the inference judgment result, and determining at least one or more of the extracted images in descending order of inference reliability as poor images. The determination unit 103 outputs information on the obtained one or more good images and poor images to the management unit 104, and proceeds to S1005.

[0053] In S1005, the management unit 104 executes a database management process. For example, the management unit 104 receives model information from the server receiving unit 101 and receives information on one or more good images and bad images from the determination unit 103. The management unit 104 generates or updates database information for managing the model information in association with the good images and bad images. The management unit 104 stores the model information, good images, and bad images based on the database information, and outputs the entities of the model information, good images, and bad images to the data storage unit 105. The management unit 104 outputs the database information it manages to the generation unit 106.

[0054] 6 is a diagram showing an example of database information that manages good images and bad images in association with model information. The database information will be described below with reference to FIG.

[0055] Model ID 201 is identification information for identifying a registered trained model. Model registrant 202 holds the name of the user who inputs and registers the trained model into the model information acquisition unit 111. Registration date 203 holds the date when the trained model was input and registered into the model information acquisition unit 111. Task name 204 holds the name or type of the task in which the trained model input into the model information acquisition unit 111 is used.

[0056] The favorite images 205 and 206 hold the file names of one or more favorite images in order of the detection accuracy (inference reliability) determined to be high among the one or more favorite images received from the determination unit 103. Note that, as an example here, the favorite images 205 and 206 hold the file names of the top two favorite images, but there is no restriction on the number of images that can be held, and the number of file names of favorite images is not limited to this example.

[0057] The poorly-rated images 207 and 208 hold the file names of one or more poorly-rated images in the order determined to have the lowest detection accuracy (inference reliability) among the one or more poorly-rated images received from the determination unit 103. Note that, as an example here, the poorly-rated images 207 and 208 hold the file names of the top two poorly-rated images, but there is no restriction on the number of images to be held, and the number of file names of the poorly-rated images is not limited to this example.

[0058] Model data 209 to 212 are examples of data related to models managed in database information. Each of model data 209 to 212 includes a model ID 201 and data from a model registrant 202 to a poor image 208 associated with the model ID 201.

[0059] In S1006, the generation unit 106 acquires data necessary for the display unit 114 to display images based on the database information received from the management unit 104, and generates display control information. The generation unit 106 extracts, for example, data indicating multiple pieces of model information stored in the data storage unit 105, one good image associated with the model information and determined to have the highest detection accuracy, and one poor image associated with the model information and determined to have the lowest detection accuracy, from the database information, and acquires the data along with the corresponding good image and poor image data. Hereinafter, the acquired data will be referred to as acquired data. Based on the acquired data, the generation unit 106 generates display control information related to display content and display settings, including, for example, each piece of model information, the display positions of the good image and poor image associated with the model information, window size, and display text indicating the content of the model information. However, the types of acquired data and display control information generated by the generation unit 106 are not limited to these and may be any information that can be managed using database information. The generation unit 106 outputs the acquired data and the generated display control information to the server transmission unit 107, and the process proceeds to S1007s.

[0060] In S1007s, the server transmitting unit 107 compresses and encrypts the acquired data and display control information received from the generating unit 106 to convert the data, and then transmits the data to the client receiving unit 113 of the information processing device 11 on the client side.

[0061] In S1007r, the client receiving unit 113 receives the acquired data and the display control information. The client receiving unit 113 decodes and decompresses the received acquired data and display control information to convert them into the original format. The client receiving unit 113 outputs the converted acquired data and the display control information to the display unit 114, and the process proceeds to S1008.

[0062] In S1008, the display unit 114 displays a list of multiple model information and good images and bad images linked to the model information on the display device 1114 based on the acquired data and display control information received from the client receiving unit 113.

[0063] 7 is an example of a list display including model information and good images and bad images linked to the model information. The list display will be described below with reference to FIG.

[0064] Model information 301 to 303 indicate model information such as the task name of the task in which the model is used, the model registrant of each model, and the registration date of the model.

[0065] Favorite image 304 to favorite image 306 display one image that is associated with each model information and that is determined to have the highest detection accuracy.

[0066] Poorly Qualified Image 307 to Poorly Qualified Image 309 display one image that is associated with each piece of model information and that is determined to have the lowest detection accuracy.

[0067] Note that, as an example, an example is shown here in which three pieces of model information and one good image and one bad image linked to each piece of model information are displayed, but there are no restrictions on display control such as the number of model information, good images, and bad images to be displayed, and the display position, and the format of the images to be displayed is not limited to the example above.

[0068] When the display unit 114 has executed the list display, the display process ends.

[0069] Through the above processing, a list of model information and good images and bad images linked to the model information can be displayed.

[0070] Next, details of the model evaluation process in S1003 will be explained using Fig. 8. Fig. 8 is a flowchart showing the steps of the model evaluation process.

[0071] In S2001, the model evaluation unit 102 acquires one evaluation image to be evaluated from among multiple evaluation images in the common evaluation data managed by the data storage unit 105, and the correct answer information corresponding to that evaluation image. Note that the correct answer information shown here is, for example, information defined by the user in advance when performing supervised learning, such as a bounding box that encloses an object region to be detected in an input image in an object detection task. Note that, although the definition of the correct answer information is shown here as an example, the definition of the correct answer information is not limited to this example. After acquiring the correct answer information, the model evaluation unit 102 proceeds to S2002.

[0072] In S2002, the model evaluation unit 102 performs inference processing on the evaluation image to be evaluated. The model used for inference may be a machine learning model such as a convolutional neural network (CNN) or transformers. The model processes the evaluation image to be evaluated to obtain a feature map, which is an intermediate output of the inference processing, and an inference result. The inference result shown here is, for example, a bounding box obtained by a model for an object detection task by identifying an area in the input image that is likely to be the object to be detected. The feature map, which is an intermediate output, is, for example, a two-dimensional map in which high values ​​are recorded at two-dimensional coordinates corresponding to an area that is likely to be a specific object during inference. Note that, while the bounding box, which is the inference result, and the feature map, which is the intermediate output, are shown as output results here as an example, the inference result and the intermediate output are not limited to this example. When the model evaluation unit 102 completes the inference processing, the process proceeds to S2003.

[0073] In S2003, the model evaluation unit 102 calculates the inference reliability of the evaluation image to be evaluated based on the feature map, which is an intermediate output output during inference. For example, in an object detection task, a model identifies an area likely to be a specific object from an input image and detects a bounding box. If the numerical value of the feature map area, which is an intermediate output referenced during the detection, is high, a high inference reliability is calculated; conversely, if the numerical value is low, a low reliability is calculated. Note that, although the inference reliability calculation method is shown here as an example, the inference reliability calculation method is not limited to this example. After calculating the inference reliability, the model evaluation unit 102 proceeds to S2004.

[0074] In S2004, the model evaluation unit 102 compares the inference result of the evaluation image with the ground truth information and determines whether the evaluation image is successfully detected or not based on the similarity between the inference result and the ground truth information. For example, in an object detection task, a predetermined threshold is set in advance for the deviation amount and size difference between the center positions of the bounding boxes of the inference result and the ground truth information. If the deviation amount and size difference exceed the threshold, the similarity is low and the detection is not successful. If the deviation amount and size difference are within the threshold, the similarity is high and the detection is successful. Note that while the inference result determination method is shown here as an example, the inference result determination method is not limited to this example. After determining the inference result, the model evaluation unit 102 generates an inference determination result and proceeds to S2005.

[0075] In S2005, the model evaluation unit 102 determines whether model evaluation has been performed using all evaluation images in the common evaluation data. If the model evaluation unit 102 determines that there are evaluation images for which model evaluation has not been performed, the process proceeds to S2001. On the other hand, if the model evaluation unit 102 determines that model evaluation has been completed for all evaluation images, the model evaluation process ends.

[0076] Through the above processing, the model evaluation unit 102 can obtain the inference reliability and inference judgment results of the trained model for all evaluation images in the common evaluation data.

[0077] According to the present embodiment having the above-described configuration, model information of the trained model is displayed along with good images and poor images linked to the model information. This allows the present embodiment to assist users in selecting trained models from the vast number of publicly available trained models, allowing the user to select a trained model suitable for their application. In particular, by displaying good images and poor images, the present embodiment can provide an environment in which even users who are not familiar with trained models can visually select an appropriate trained model, compared to when the accuracy rate of the trained model is displayed.

[0078] In this embodiment, by displaying both the good image and the bad image together with the model information, the user can compare the two images, making it easier for the user to intuitively understand what the model is good at and what it is not good at. Specifically, if only the good image 306 shown in FIG. 7 is displayed and the bad image 309 is not displayed, it is difficult for the user to determine whether the model is good at detecting cars (here, cars), whether it is good at bright images, or whether it is good at detecting a single object. However, in this embodiment, by displaying the good image 306 and the bad image 309, the user can intuitively understand that the trained model of the model information 303 is good at detecting images with a single object.

[0079] In this embodiment, the evaluation of the trained model by machine learning is performed on the server side, thereby reducing the processing load on the client side and assisting in the selection of users who will use the client-side information processing device 11.

[0080] In this embodiment, the trained model is evaluated using common evaluation data, thereby reducing the variation in the evaluation of multiple trained models.

[0081] (Modification 1 of the first embodiment) In S1004 of FIG. 5, the determination unit 103 determines one or more good images and poor images included in the evaluation images based on the inference reliability and inference judgment results corresponding to all evaluation images received from the model evaluation unit 102. However, this is not limited to this. For example, the determination unit 103 obtains inference results corresponding to all evaluation images from the model evaluation unit 102. The determination unit 103 may then compare the inference results with the correct answer information for all evaluation images in the common evaluation data to determine one or more good images in descending order of similarity, and determine one or more poor images in descending order of similarity. Note that the method of calculating similarity in the object detection task may be, for example, a method of calculating similarity based on the ratio of the deviation amount and size difference between the center positions of the bounding boxes of the inference result and the correct answer information. While the similarity calculation method is shown here as an example, the method of determining good images and poor images is not limited to this method.

[0082] (Second embodiment) The information processing system of the second embodiment assists a user in selecting a trained model through machine learning by displaying a list of model information managed in association with specific information of a target selected on a client terminal from among one type of specific information managed in database information, such as good images, bad images, and shooting information, displayed on a client terminal. Here, an object detection task involving image input will be used as an example of a task handled in this embodiment. However, the types of tasks that can be handled in this embodiment are not limited to this example. The hardware configuration of the information processing system of the second embodiment is substantially the same as the example configuration of the first embodiment shown in FIGS. 1 and 2.

[0083] 9 is a block diagram showing an example of the configuration of an information processing system according to the second embodiment. The server-side information processing device 40 includes a server receiving unit 401, a model evaluation unit 402, a determination unit 403, a management unit 404, a data storage unit 405, a generation unit 406, a server transmission unit 407, and a shooting information acquisition unit 408. The client-side information processing device 41 includes a model information acquisition unit 411, a client transmission unit 412, a client reception unit 413, a display unit 414, and an instruction acceptance unit 415.

[0084] Of the functional configurations of the information processing system according to the second embodiment, the description of the functional configurations that are the same as the configuration example of the first embodiment shown in Figure 3 will be omitted, and only the server receiving unit 401, management unit 404, data storage unit 405, generation unit 406, shooting information acquisition unit 408, and instruction reception unit 415, which have different functions, will be described.

[0085] The server receiving unit 401 receives information from the client transmitting unit 412 of the client-side information processing device 41 via the network. The server receiving unit 401 receives, for example, model information and user instruction information. The server receiving unit 401 receives data that has been converted into an appropriate format before transmission. The server receiving unit 401 converts the received data into data in a format that is easy to use at the output destination and outputs it. The server receiving unit 401 outputs the data-converted model information to the model evaluation unit 402 and management unit 404, and outputs the data-converted user instruction information to the generation unit 406.

[0086] The management unit 404 receives model information from the server receiving unit 401, receives good images and bad images from the determination unit 403, and receives shooting information of all evaluation images stored in the data storage unit 405. The management unit 404 generates and updates database information for managing the good images, bad images, and shooting information by linking them to the model information. The management unit 404 outputs the model information, good images, and bad images to the data storage unit 405 in order to store the data included in the database information as entities. The management unit 404 outputs the database information it manages to the generation unit 406.

[0087] The data holding unit 405 receives model information, good images, and bad images from the management unit 404, and receives shooting information from the shooting information acquisition unit 408, and holds the information based on the database information. Within the server-side information processing device 40, the use and deletion of data such as information and images held by the data holding unit 405 is carried out through the management unit 404.

[0088] The generation unit 406 generates display control information for displaying a list of favorite images from which the user can select a favorite image. The generation unit 406 also generates display control information for displaying a list of model information linked to the favorite image selected by the user. Both list displays will be described later. The generation unit 406 outputs the generated display control information to the server transmission unit 407.

[0089] The photographing information acquisition unit 408 acquires photographing information recorded in all evaluation images stored in the data storage unit 405. Note that the photographing information acquisition unit 408 may acquire photographing information from a photographing device that captured the evaluation images, such as a digital camera. The photographing information acquisition unit 408 stores the photographing information as a data entity, and outputs the photographing information corresponding to all evaluation images to the data storage unit 405. The photographing information is, for example, the type of camera and lens used for capturing the images, and mode information at the time of capturing the images. However, the type of photographing information acquired by the photographing information acquisition unit 408 is not limited to this, and may be any information related to capturing the images.

[0090] The instruction receiving unit 415 receives instruction information from the user in the client-side information processing device 41. The received instruction information is a model information search instruction to search for model information based on specific information linked to model information such as favorite images, and an instruction to end the list display of favorite images, model information, etc. The instruction receiving unit 415 outputs the received model information search instruction and list display end instruction to the client transmitting unit 412.

[0091] Next, the processing procedure in this embodiment will be described. In the following description, each process (step) will be prefixed with an S and the step number will be indicated. In order to clarify the differences from the processing in FIG. 5 described in the first embodiment, the same processing as that described in FIG. 5 will be described by the corresponding step and the description will be omitted.

[0092] 10 is a flowchart of a display process in which an information processing system having a client-side information processing device 41 and a server-side information processing device 40 displays a list of model information linked by database information to a favorite image that is the subject of a search instruction. The process shown in the flowchart in FIG. 10 starts, for example, when model information to be processed is input. However, the information processing system does not necessarily have to perform all of the steps described in this flowchart.

[0093] As preparation for implementing this flowchart, the client-side information processing device 41 initializes the system. For example, the CPU 1111 reads a program from the ROM 1112 or the hard disk 1116, and puts the information processing device 41 into an operable state. In the server-side information processing device 40, the CPU 1101 reads a program from the ROM 1102, and puts the information processing device 40 into an operable state. As a result, the client-side information processing device 41 and the server-side information processing device 40 are able to communicate with each other using the client transmitting unit 412, the server receiving unit 401, the server transmitting unit 407, and the client receiving unit 413.

[0094] The processing of S3001, S3002, S3003, and S3004 is the same as the processing of S1001, S1002, S1003, and S1004.

[0095] In S3005, the photographing information acquisition unit 408 acquires the photographing information recorded in all evaluation images in the common evaluation data stored in the data storage unit 405. The photographing information shown here includes, for example, the type of camera used for shooting, the type of lens, and mode information at the time of shooting. In order to store the photographing information as a data entity, the photographing information acquisition unit 408 outputs the photographing information corresponding to all evaluation images to the data storage unit 405, and the process proceeds to S3006.

[0096] In S3006, the management unit 404 executes database management processing. For example, the management unit 404 receives model information from the server receiving unit 401, receives information on one or more good images and bad images from the determination unit 403, and receives shooting information corresponding to all evaluation images stored in the data storage unit 405. The management unit 404 generates or updates database information for managing the model information by linking the good images, bad images, and shooting information. The management unit 404 generates or updates database information for managing the shooting information by linking the shooting information to the evaluation images. The management unit 404 outputs the entities of the model information, good images, and bad images to the data storage unit 405 in order to store the model information, good images, and bad images based on the database information. The management unit 404 outputs the database information it manages to the generation unit 406.

[0097] Fig. 11 shows an example of database information generated by the management unit 404. Fig. 11(A) shows an example of database information that manages good images and bad images in association with model information. The database information will be described below with reference to Fig. 11(A).

[0098] Model ID 501 is identification information for identifying a registered model. Model registrant 502 holds the name of the user who inputs and registers the model in model information acquisition unit 411. Registration date 503 holds the date when the model information was input and registered in model information acquisition unit 411. Task name 504 holds the name or type of the task in which the model input in model information acquisition unit 411 is used.

[0099] The favorite image 505 holds the file names of multiple favorite images in order of the detection accuracy determined to be high among the one or more favorite images received from the determination unit 403. Note that, as an example, the favorite image 505 holds the file name of the top-level favorite image here, but there is no restriction on the number of file names of images that can be held, and the present invention is not limited to this example.

[0100] The poor image 506 holds file names of a plurality of poor images in the order determined to have the lowest detection accuracy among the one or more poor images received from the determination unit 403. Note that, as an example here, the poor image 506 holds the file name of one poor image at the top (lowest detection accuracy), but there is no restriction on the number of file names of images to be held, and the present invention is not limited to this example.

[0101] Model data 507 to 510 are examples of data related to models managed by database information. Each of model data 507 to 510 includes a model ID 501 and data from a model registrant 502 to a poor image 506 associated with the model ID 501.

[0102] Fig. 11(B) is an example of database information of shooting information. The database information of shooting information will be described with reference to Fig. 11(B).

[0103] Evaluation image 511 holds the file name of the evaluation image included in the common evaluation data. Camera type 512 holds the type of camera used when the evaluation image was taken. Lens type 513 holds the type of lens used when the evaluation image was taken. Shooting mode 514 holds the mode setting when the evaluation image was taken.

[0104] Shooting information 515 to 518 are examples of shooting information managed by the database information of shooting information. Each of shooting information 515 to 518 includes an evaluation image 511 and data from camera type 512 to shooting mode 514 associated with the evaluation image 511. Note that the database information of shooting information may link the shooting information with at least one of good images and bad images.

[0105] In S3007, the generation unit 406 acquires data such as image data required for display by the display unit 414 based on the database information received from the management unit 404, and generates display control information for displaying a list of favorite images. The generation unit 406 acquires, for example, multiple favorite images managed by the management unit 404 using database information and stored in the data storage unit 405. The generation unit 406 generates display control information related to display content and display settings, including the display position, window size, and display text of each acquired favorite image. However, the types of data acquired and display control information generated by the generation unit 406 are not limited to these, and may be any information that can be managed using database information. The generation unit 406 outputs the acquired data such as the favorite image (hereinafter, "acquired data") and the generated display control information to the server transmission unit 407, and proceeds to S3008s.

[0106] In S3008s, the server transmitting unit 407 converts the display control information based on the acquired data and the database information, and then transmits the converted data to the client receiving unit 413 of the information processing device 10.

[0107] In S3008r, the client receiving unit 113 receives the display control information. The client receiving unit 413 converts the received display control information into data and outputs the converted data to the display unit 414.

[0108] In S3009, the display unit 414 displays a list of a plurality of favorite images on the display device 1114 based on the acquired data and display control information received from the client receiving unit 413.

[0109] 12 shows an example of a list of favorite images displayed by the display unit 114 using the display control information. Hereinafter, the list of favorite images will be described with reference to FIG.

[0110] Favorite images 601 to 609 displayed in the list are favorite images included in the database information managed by the management unit 404 in the information processing device 40 on the server side.

[0111] Note that, as an example, an example of displaying a list of nine favorite images is shown here, but there are no restrictions on the display control such as the images to be displayed in the list, the number of images, and the display position of the images, and the list display is not limited to this example.

[0112] In S3010, the instruction receiving unit 415 determines whether a search instruction for model information has been acquired. For example, the instruction receiving unit 415 determines whether a search instruction for model information has been acquired from the user based on whether the user selected any of the favorite images in the list of favorite images displayed by the display unit 414. Specifically, with favorite images 601 to 609 displayed as a list, the user touches a touch panel, which is an example of the input device 1113. The instruction receiving unit 415 detects the coordinates touched by the user and determines whether a favorite image is displayed at those coordinates. If a favorite image is displayed at those coordinates, the instruction receiving unit 415 determines that the favorite image has been selected and that a search instruction has been acquired. The search instruction here is an instruction to search the database information for one or more pieces of model information linked to the favorite image selected by the user and display a list of the model information as the search results. The search instruction received by the instruction receiving unit 415 is not limited to this, and may be any instruction information such as an instruction to search for model information using different specific information linked to database information (poor images, shooting information, etc.). When the instruction receiving unit 415 receives a search instruction, it outputs the search instruction to the client transmission unit 412 and proceeds to S3011s. On the other hand, when the instruction receiving unit 415 does not receive a search instruction, it proceeds to S3012.

[0113] In S3011s, the client transmitting unit 412 converts the search instruction including information on the favorite image selected by the user into data, and then transmits the data to the server receiving unit 401 on the server side.

[0114] In S3012, the instruction receiving unit 415 determines whether or not an instruction to end display has been received from the user. If the instruction receiving unit 415 determines that an instruction to end display has not been received, it repeats S3010. On the other hand, if the instruction receiving unit 415 determines that an instruction to end display has been received, it stops the display on the display unit 414 and ends the process.

[0115] In S3011r, the server receiving unit 401 converts the received search instruction into data and outputs it to the generating unit 406.

[0116] In S3013, the generation unit 406 acquires data such as a favorite image and generates display control information necessary for display on the display unit 414, based on the search instruction received via the server receiving unit 401 and the database information received from the management unit 404. The display control information generated by the generation unit 406 is control information related to display content and display settings, such as multiple pieces of model information linked to the favorite image selected by the user and included in the search instruction, the display position of the favorite image and model information, window size, and display text indicating the content of the model information. The generation unit 406 outputs the generated display control information to the server transmitting unit 407, and the process proceeds to S3014s.

[0117] In S3014s, the server transmitting unit 407 converts the received display control information into data and then transmits it to the client receiving unit 413 on the client side.

[0118] In S3014r, the client receiving unit 413 converts the display control information received from the server transmitting unit 407 into data, outputs it to the display unit 414, and the process proceeds to S3015.

[0119] In S3015, the display unit 414 uses the display control information generated based on the user's search instruction and the database information to cause the display device 1114 to display a list of model information.

[0120] 13 shows an example of a list display of model information displayed by display unit 414 using the display control information. The following description will be made with reference to FIG.

[0121] Favorite image 701 is the favorite image to be searched for that has been received from the user by instruction receiving unit 415. Model information 702 to 707 displayed in a list is model information linked to the favorite image to be searched for in the database information. Note that, as an example, an example is shown here in which one favorite image and a list of six pieces of model information linked to that favorite image are displayed, but there are no restrictions on display control such as the number of images to be displayed, the number of pieces of model information, and the display position, and the display format is not limited to this example.

[0122] With the above-described configuration, according to this embodiment, the user can be assisted in the selection of a trained model by displaying a list of model information managed in association with the target specific information selected on the client-side terminal from among one type of specific information managed in database information such as good images, bad images, and shooting information, which are displayed in a list on the client-side terminal.

[0123] (Modification 1 of the second embodiment) 10, a list of good images managed in the database information is displayed on the display unit 414, and a list of model information linked to the good images is displayed based on the search instruction received by the instruction receiving unit 415. However, the images to be displayed are not limited to this. For example, the generation unit 406 may display a list of poor images managed in the database information, and generate display control information for displaying a list of model information linked to the poor images to be searched for as good images based on the search instruction indicating the poor images received by the instruction receiving unit 415.

[0124] In this case, it is possible to search for models that excel at scenes that previously failed to be detected, which is expected to improve the search performance for model information.

[0125] (Modification 2 of the second embodiment) 10, a list of favorite images managed in database information is displayed on the display unit 414, and a list of model information linked to the favorite images is displayed based on a search instruction received by the instruction receiving unit 415. However, the images to be displayed are not limited to this. For example, the generation unit 406 may display a list of model information managed in database information, and generate display control information for displaying favorite images and poor images managed in association with model information to be searched, based on a search instruction received by the instruction receiving unit 415.

[0126] In this case, the user can easily check the good images and bad images that represent the characteristics of the model information based on the user's instructions, which is expected to improve the search performance for model information.

[0127] (Modification 3 of the second embodiment) 10, a list of favorite images managed in database information is displayed on the display unit 414, and a list of model information linked to the favorite images is displayed based on a search instruction received by the instruction receiving unit 415. However, this is not limited to this. For example, the display unit 414 may display a list of shooting information managed in database information, and based on a search instruction received by the instruction receiving unit 415, a list of model information managed in association with the shooting information to be searched may be displayed on the display unit 414. For example, the generation unit 406 may generate display control information for displaying a list of model information in which the evaluation image linked to the shooting information indicated by the search instruction is the favorite image.

[0128] In this case, model information suited to one's own needs can be searched for according to the photographic equipment and photographic mode used, and therefore improvement in model information search performance can be expected.

[0129] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0130] In the first and second embodiments, an information processing system including a server-side information processing device and one or more client-side information processing devices capable of communicating with the server-side information processing device has been described, but the information processing system is not limited to this example. For example, the information processing system may be an embodiment in which a single information processing device performs operations from model information acquisition to database management, model information display, search, and list display. Furthermore, the information processing system may be realized by three or more information processing devices.

[0131] In the above-described embodiment, an example was described in which the management unit 104, 404 manages database information in which both good images and poor images are linked to model information. However, the management unit may also manage database information in which at least one of good images and poor images is linked to model information. For example, if the management unit manages only good images in the database information, the generation unit may display the good images together with a list of model information. This allows the information processing system to present to the user what types of images each trained model is good at, thereby assisting the user in selecting a trained model. If the management unit manages only poor images in the database information, the generation unit may display the poor images together with a list of model information. This allows the information processing system to present to the user what types of images each trained model is poor at, thereby preventing the user from selecting the wrong trained model.

[0132] In the above-described embodiments, a trained model based on machine learning for processing images has been described as the target, but the processing target is not limited to images. For example, the above-described embodiments may be applied to a trained model based on machine learning for processing audio or text data. In the case of a machine learning model that processes audio, the determination unit may determine good audio as a good target and poor audio as a poor target based on the good frequency and dB (decibels) indicating the loudness of the good sound, and the generation unit may generate display control information that displays a list of information about the frequency or loudness in text or the like, and provide the list display to the user. Furthermore, the information processing system may output an example of audio selected by the user as sound in the list display of information about audio.

[0133] In the above embodiment, the generating unit has been described as displaying a list of model information, but the display format of the model information is not limited to a list display. For example, the generating unit may generate display control information so that a set including one piece of model information and a good image and a bad image linked to the model information is displayed on one screen, and the sets of model information, good image, and bad image are sequentially switched and displayed in response to a switching instruction from the user.

[0134] In the above-described embodiment, the generation unit has been described with an example of display control information in which the good image and the bad image are displayed as information about the good image and the bad image, but the display control information is not limited to this example. For example, the generation unit may convert information about the good image and the bad image into text, rather than the good image and the bad image themselves. For example, the generation unit may generate display control information such that a "bright image" is displayed as information about the good image and a "dark image" is displayed as information about the bad image.

[0135] In the above-described embodiment, no explanation was given of what to do when a user selects a trained model. However, for example, the display unit may display a message warning that an image that the user is attempting to process is similar to an image that the trained model selected by the user is not good at processing.

[0136] In the above embodiment, an example has been described in which a trained model and a favorite image are displayed and a trained model is selected, but the display method for selection is not limited to this. For example, the display unit may display multiple trained models, prompt the user to select a trained model, and then display at least one of the favorite image and the favorite image associated with the trained model, and ask the user whether the trained model is correct.

[0137] The disclosure of this specification includes the following information processing device, information processing system, information processing method, and program. (Item 1) A management means for managing database information in which model information about a trained model and at least one of a good target and a poor target among processing targets processed by the trained model are linked; A generation means for generating display control information for displaying the model information and information relating to at least one of the good target and the bad target linked to the model information based on the database information; An information processing device comprising: (Item 2) an evaluation means for acquiring the model information, evaluating the trained model, and generating an evaluation result; 2. The information processing device according to item 1, further comprising: (Item 3) The evaluation means evaluates the trained model using common evaluation data including a common evaluation target to generate the evaluation result. 3. The information processing device according to item 2, (Item 4) a determination means for determining the good object and the bad object based on the inference reliability indicated by the evaluation result and an inference determination result obtained by determining the inference result; Item 4. The information processing device according to item 2 or 3, further comprising: (Item 5) A determination means for determining the good subject and the bad subject based on the similarity obtained by comparing the inference result indicated by the evaluation result with correct answer information. Item 5. The information processing device according to any one of items 2 to 4, further comprising: (Item 6) The processing target is a captured image, The management means manages database information of the model information, the good images as the good subjects, the bad images as the bad subjects, and photography information linking at least one of the photographed images with photography information at the time the images were photographed. 6. The information processing device according to any one of items 1 to 5, (Item 7) The generating means generates display control information for displaying at least one of the good subject and the bad subject and a list of the model information. 7. The information processing device according to any one of items 1 to 6, (Item 8) the management means manages, in the database information, specific information that is linked to the model information and is information related to the processing target; The generating means generates display control information for displaying at least a part of the specific information and a list of the model information. 8. The information processing device according to any one of items 1 to 7, wherein: (Item 9) The management means searches for the model information associated with the specialty object selected by the user, The generating means generates display control information for displaying a list of the searched model information together with the target object. 9. The information processing device according to any one of items 1 to 8, wherein: (Item 10) The management means searches for the model information linked to the weak subject selected by the user as the strong subject, The generating means generates display control information for displaying a list of the searched model information together with the target object. 10. The information processing device according to any one of items 1 to 9, (Item 11) the management means searches for the model information based on specific information selected by a user from specific information that is linked to the model information and is information related to the processing target; The generating means generates display control information for displaying a list of the retrieved model information together with the specific information. 11. The information processing device according to any one of items 1 to 10, (Item 12) The generating means generates display control information for displaying a list of a plurality of pieces of model information and the good targets and the bad targets linked to each of the plurality of pieces of model information. 12. The information processing device according to any one of items 1 to 11, (Item 13) the information processing device according to item 1; a display means for displaying, on a display device, at least one of the good target and the bad target and a list of the model information based on the display control information; a model information acquisition means for acquiring the model information; An information processing system comprising: (Item 14) A management process for managing database information in which model information about the trained model and at least one of a good target and a poor target among processing targets processed by the trained model are linked; A generation process for generating display control information for displaying the model information and information relating to at least one of the good target and the bad target linked to the model information based on the database information; An information processing method comprising: (Item 15) A program for causing a computer to function as each means of the information processing device according to any one of items 1 to 12.

[0138] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0139] 10, 11, 40, 41... Information processing device, 102, 402... Model evaluation unit, 103, 403... Decision unit, 104, 404... Management unit, 106, 406... Generation unit, 301-303, 702-707... Model information, 304-306, 601-609, 701... Good images, 307-309... Bad images, 515-518... Shooting information.

Claims

1. A management means for managing database information in which model information about a trained model and at least one of a good target and a poor target among processing targets processed by the trained model are linked; A generation means for generating display control information for displaying the model information and information relating to at least one of the good target and the bad target linked to the model information based on the database information; An information processing device comprising:

2. an evaluation means for acquiring the model information, evaluating the trained model, and generating an evaluation result; 2. The information processing apparatus according to claim 1, further comprising:

3. The evaluation means evaluates the trained model using common evaluation data including a common evaluation target to generate the evaluation result.

3. The information processing apparatus according to claim 2, wherein:

4. a determination means for determining the good object and the bad object based on the inference reliability indicated by the evaluation result and an inference determination result obtained by determining the inference result; 3. The information processing apparatus according to claim 2, further comprising:

5. A determination means for determining the good subject and the bad subject based on the similarity obtained by comparing the inference result indicated by the evaluation result with correct answer information.

3. The information processing apparatus according to claim 2, further comprising:

6. The processing target is a captured image, The management means manages database information of the model information, the good images as the good subjects, the bad images as the bad subjects, and photography information linking at least one of the photographed images with photography information at the time the images were photographed.

2. The information processing apparatus according to claim 1, wherein:

7. The generating means generates display control information for displaying at least one of the good subject and the bad subject and a list of the model information.

2. The information processing apparatus according to claim 1, wherein:

8. the management means manages, in the database information, specific information that is linked to the model information and is information related to the processing target; The generating means generates display control information for displaying at least a part of the specific information and a list of the model information.

2. The information processing apparatus according to claim 1, wherein:

9. The management means searches for the model information associated with the specialty object selected by the user, The generating means generates display control information for displaying a list of the searched model information together with the target object.

2. The information processing apparatus according to claim 1, wherein:

10. The management means searches for the model information linked to the weak subject selected by the user as the strong subject, The generating means generates display control information for displaying a list of the searched model information together with the target object.

2. The information processing apparatus according to claim 1, wherein:

11. the management means searches for the model information based on specific information selected by a user from specific information that is linked to the model information and is information related to the processing target; The generating means generates display control information for displaying a list of the retrieved model information together with the specific information.

2. The information processing apparatus according to claim 1, wherein:

12. The generating means generates display control information for displaying a list of a plurality of pieces of model information and the good targets and the bad targets linked to each of the plurality of pieces of model information.

2. The information processing apparatus according to claim 1, wherein:

13. The information processing device according to claim 1 ; a display means for displaying, on a display device, at least one of the good target and the bad target and a list of the model information based on the display control information; a model information acquisition means for acquiring the model information; An information processing system comprising:

14. A management process for managing database information in which model information about the trained model and at least one of a good target and a poor target among processing targets processed by the trained model are linked; A generation process for generating display control information for displaying the model information and information relating to at least one of the good target and the bad target linked to the model information based on the database information; An information processing method comprising:

15. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12.

Images