Information processing device, information processing method, and program

Abstract

This technology provides users with the ability to easily select machine learning models suitable for their specific applications. [Solution] The information processing device includes an acquisition means for acquiring model information, which is information relating to a machine learning model; a reference means for acquiring related model information, which indicates machine learning models related to the acquired model information, by referring to a database that shows the relationships between a plurality of machine learning models; and a generation means for generating display control information for displaying a screen containing information on recommended machine learning models based on the related model information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Systems using machine learning model technologies have been put into practical use in various fields, and the number of available machine learning models has become enormous. In such a situation, it has become difficult for users to appropriately select a machine learning model suitable for their desired use. Therefore, there is a need for a technology to assist in the selection of machine learning models.

[0003] For example, in Patent Document 1, a method for assisting the selection work of machine learning models is proposed by displaying the outputs of a plurality of machine learning models and the success or failure determination results in a tabular format to the user.

[0004] In addition, Patent Document 2 discloses a technique for preparing information including learning results as model versions for each task.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the prior art disclosed in Patent Document 1, since only the outputs of a plurality of learning models and the success or failure determination results are displayed in a tabular format, it is difficult to select a machine learning model suitable for the user's use among the huge number of publicly available machine learning models. There was also the same problem in the technology of Patent Document 2.

[0007] Therefore, the present invention provides a technology that allows users to easily select a machine learning model suitable for their application. [Means for solving the problem]

[0008] To solve this problem, for example, the information processing apparatus of the present invention has the following configuration. That is, A means of obtaining model information, which is information about a machine learning model, A reference means that obtains related model information indicating machine learning models related to the acquired model information by referring to a database that shows the relationships between multiple machine learning models, A generation means for generating display control information for displaying a screen containing information on a recommended machine learning model based on the aforementioned related model information, It is equipped with. [Effects of the Invention]

[0009] According to the present invention, a technology is provided that allows users to easily select a machine learning model suitable for their application. [Brief explanation of the drawing]

[0010] [Figure 1] Hardware configuration diagram of the server-side information processing device of the information processing system according to the embodiment. [Figure 2] Hardware configuration diagram of the client-side information processing device of the information processing system according to the embodiment. [Figure 3] Overall configuration diagram of the information processing system according to the embodiment. [Figure 4] A block diagram showing an example of the functional configuration of the information processing system according to the first embodiment. [Figure 5] A flowchart of the related model recommendation process for recommending related machine learning models in the first embodiment. [Figure 6] A diagram illustrating the screen displayed when selecting a machine learning model in the first embodiment. [Figure 7]Diagram of a database for managing related information between machine learning models of the first embodiment. [Figure 8] Flowchart of related model determination processing for determining related models of the first embodiment. [Figure 9] Block diagram showing an example of the functional configuration of an information processing system of the second embodiment. [Figure 10] Flowchart of related model recommendation processing for recommending related machine learning models of the second embodiment. [Figure 11A] Flowchart of related model determination processing for determining related models of a modification example of the second embodiment. [Figure 11B] Database of the average usage period of machine learning models of a modification example of the second embodiment. [Figure 11C] Flowchart of related model recommendation processing for recommending related machine learning models of a modification example of the second embodiment. [Figure 12] Block diagram showing an example of the functional configuration of an information processing system of the third embodiment. [Figure 13] Flowchart of related model recommendation processing for recommending related machine learning models of the third embodiment. [Figure 14] Block diagram showing an example of the functional configuration of an information processing system of the fourth embodiment. [Figure 15A] Flowchart of related model recommendation processing for recommending related machine learning models on the client side of the fourth embodiment. [Figure 15B] Flowchart of related model recommendation processing for recommending related machine learning models on the virtual server side of the fourth embodiment. [Figure 16] Diagram showing an example of a screen for inputting failure information of the fourth embodiment. [Figure 17] Table of related machine learning models, detection rates, and false detection rates of the fourth embodiment.

Modes for Carrying Out the Invention

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] The information processing system (also known as a machine learning model selection support system) according to the embodiment will be described in detail below. The information processing system according to the embodiment includes an information processing device on the virtual server side and one or more client-side information processing devices that are connected to the information processing device on the virtual server side via a network. The virtual server may be a physical server, or a server consisting of one or more computers.

[0013] (Description of the hardware configuration of the information processing system) Before describing this embodiment, the hardware configuration of the information processing device on which the information processing system shown in each embodiment is implemented will be described using Figures 1 and 2.

[0014] (Description of virtual server hardware configuration) Figure 1 is a hardware configuration diagram of the information processing device 10 on the virtual server side of the information processing system in the embodiment.

[0015] The information processing device 10 may be configured on, for example, a cloud server. The information processing device 10 may be, for example, a computer or a virtual computer. The information processing device 10 has a CPU 101, ROM 102, RAM 103, communication I / F 104, hard disk 105, and a system bus 106. The CPU 101, ROM 102, RAM 103, communication I / F 104, and hard disk 105 are connected to each other via the system bus 106 so that data can be sent and received from each other.

[0016] CPU101 stands for Central Processing Unit and is a processor. For example, CPU101 controls the entire information processing device 10. For example, CPU101 controls various devices connected to the system bus 106. CPU101 reads computer programs (also called programs) stored in ROM102 or hard disk 105, loads them into RAM103, and executes them to realize various functions and perform various processes.

[0017] The information processing device 10 may have other processors such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and QPU (Quantum Processing Unit) in place of or in addition to the CPU 101. Furthermore, the information processing device 10 may have multiple processors of the same type, each performing a different function.

[0018] Some or all of the functions of the information processing device 10 may be implemented by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0019] ROM102 stands for Read Only Memory and is a non-volatile memory. ROM102 stores the BIOS (Basic Input / Output System) program and the boot program.

[0020] RAM103 stands for Random Access Memory and is a high-speed read and write memory. RAM103 is used as the main memory of the CPU101 and functions as a working area when a program is executed. For example, when a program is executed, RAM103 stores data such as the program and parameters necessary for program execution, which are read from the hard disk 105 or other sources.

[0021] The communication interface 104 is a communication interface that communicates information with external devices such as the client-side information processing device 11, which will be described later, via the network. The communication standard for the communication interface 104 may be Ethernet (registered trademark), USB (Universal Serial Bus), serial communication, or wireless communication, and the type of communication is not limited.

[0022] The hard disk 105 is a non-volatile storage device. The hard disk 105 is used to store and read application programs, data necessary for program execution, libraries, and data to be processed by the program. The information processing device 10 may have other storage devices, such as an SSD (Solid State Drive), instead of or in addition to the hard disk 105.

[0023] (Description of the client's hardware configuration) Figure 2 is a hardware configuration diagram of the client-side information processing device 11 of the information processing system in this embodiment.

[0024] The information processing device 11 is a terminal device on which a user can view and operate a screen. The information processing device 11 may be a computer such as a PC (personal computer) or a tablet terminal. The information processing device 11 has a CPU 111, ROM 112, input device 113, display device 114, RAM 115, hard disk 116, media drive 117, communication I / F 118, and system bus 119. The CPU 111, ROM 112, input device 113, display device 114, RAM 115, hard disk 116, media drive 117, and communication I / F 118 are connected to each other via the system bus 119 so that data can be sent and received from each other.

[0025] The CPU 111 controls various devices connected to the system bus 119. The CPU 111 reads computer programs (also called programs) stored in the ROM 112 or hard disk 116, loads them into the RAM 115, and executes them to realize various functions and perform various processes. The information processing device 11 may have other processors such as an MPU, GPU, NPU, and QPU in place of or in addition to the CPU 111. The information processing device 11 may also have multiple processors of the same type, with each processor realizing a different function. Some or all of the functions of the information processing device 11 may be realized by one or more circuits such as an ASIC and a PLD including an FPGA.

[0026] ROM112 stores the BIOS program, boot program, and other similar programs.

[0027] The input device 113 performs processing related to the input of information, receives information such as instructions from the user, and outputs it to the CPU 111. The input device 113 may be, for example, a touch panel, keyboard, mouse, or robot controller.

[0028] The display device 114 displays the calculation results of the information processing device 11 and display information transmitted from the information processing device 10 on the virtual server side, in accordance with instructions from the CPU 111. The display device 114 may be a liquid crystal display device, a projector, an LED indicator, etc., and is not limited to any particular type.

[0029] RAM115 is used as the main memory of CPU111 and functions as a working area when a program is running.

[0030] The hard disk 116 is used for storing and reading application programs, data necessary for program execution, libraries, and data processed by the program.

[0031] The media drive 117 enables the writing of data from the hard disk 116 to a removable storage medium. The media drive 117 also enables the transfer of the written data to an external digital still camera, PC, or tablet device.

[0032] The communication interface 118 communicates with external devices via a network. For example, the communication interface 118 communicates with the information processing device 10 on the virtual server side. The communication standard of the communication interface 118 may be Ethernet, USB, serial communication, wireless communication, etc., and the type of communication is not limited.

[0033] (Description of the overall structure of the information processing system) Figure 3 is an overall configuration diagram of the information processing system according to the embodiment. As shown in Figure 3, the information processing system includes a virtual server-side information processing device 10 and one or more client-side information processing devices 11. The virtual server-side information processing device 10 is communicated with one or more client-side information processing devices 11 for sending and receiving data. The information processing device 10 controls the sending and receiving of data with the information processing device 11, and the display information displayed on the information processing device 11.

[0034] (First embodiment) The information processing system in the first embodiment assists the user in selecting a machine learning model. In the information processing system, the information processing device 10 on the virtual server side maintains the degree of association between machine learning models as a database on the cloud server and extracts machine learning models related to the machine learning model selected by the user. The information processing device 11 on the client side receives and displays the information of these related machine learning models, thereby assisting the user in selecting a machine learning model.

[0035] This embodiment explains the machine learning model's inference and evaluation tasks using an image as input, specifically an object detection task. The object detection task takes image data as input and, if a specific object is present in the image, infers a bounding box surrounding that object. The model is designed to handle a variety of tasks, including object detection, region estimation and division, and classification tasks such as classifying subjects like people and cars. Regarding object detection inference and evaluation methods, for example, neural network-based techniques exist. For details on learning methods for object detection using neural networks, see "Tian et al. FCOS: Fully Convolutional One-Stage Object Detection, arXiv 2019".

[0036] (Description of the block diagram showing the system configuration of the first embodiment) Figure 4 is a block diagram showing an example of the functional configuration of the information processing system in the first embodiment. The functional configurations of the information processing devices 10 and 11 in the first embodiment will be described with reference to Figure 4.

[0037] The virtual server-side information processing device 10 includes a model information receiving unit 401, a database reference unit 402, a data holding unit 403, a database management unit 404, a display control information generation unit 405, and a display control information transmission unit 406. The processor, including the CPU 101, may implement some or all of the functions of the model information receiving unit 401, the database reference unit 402, the data holding unit 403, the database management unit 404, the display control information generation unit 405, and the display control information transmission unit 406 by executing a program recorded on the hard disk 105 or the like.

[0038] The client-side information processing device 11 includes a model information acquisition unit 411, a model information transmission unit 412, a display control information receiving unit 413, and a display unit 414. The processor, including the CPU 111, may implement some or all of the functions of the model information acquisition unit 411, the model information transmission unit 412, the display control information receiving unit 413, and the display unit 414 by executing a program recorded on the hard disk 116 or the like.

[0039] The model information transmission unit 412 and the model information receiving unit 401, and the display control information transmission unit 406 and the display control information receiving unit 413 may communicate information via a network. Figure 4 is an example of a functional configuration and does not limit the scope of application of this embodiment.

[0040] The model information receiving unit 401 is an example of an acquisition method, and acquires information by receiving it from the client-side information processing device 11 via the network. The model information receiving unit 401, for example, receives and acquires model information, which is information relating to a machine learning model. This model information may be the model information of a machine learning model selected by the user for the purpose of modifying the machine learning model. The model information receiving unit 401 may convert the model information into data in a format that is easy to use at the output destination. The model information receiving unit 401 outputs the converted model information to the database reference unit 402.

[0041] The database reference unit 402 refers to the database held in the data holding unit 403 and, based on the received model information, refers to a database of related information that shows the relationships between multiple machine learning models. From the database of related information between machine learning models that has been acquired, the database reference unit 402 selects machine learning models that are highly related to the machine learning model selected by the user as machine learning models to recommend to the user. The database reference unit 402 acquires model information (also called related model information) of the selected related machine learning models and outputs it to the display control information generation unit 405.

[0042] The data storage unit 403 holds a database containing model information and related information between machine learning models. When the data storage unit 403 is used or deleted within the information processing device 10 on the virtual server side, the data is processed through the database management unit 404.

[0043] The database management unit 404 manages a database containing information such as relationship information indicating the relationships between machine learning models. When a new machine learning model is registered in the model storage unit, the database management unit 404 receives the information and updates the model information and relationship information between models in the database stored in the data storage unit 403. For example, the database management unit 404 may manage the database based on a degree of relevance indicating the degree of relationship between machine learning models. The model storage unit may be implemented by a hard disk 105, ROM 102, and RAM 103, etc.

[0044] The display control information generation unit 405 is an example of a generation means and generates display control information for displaying the machine learning model recommended by the client-side information processing device 11, based on related model information obtained from the database reference unit 402 and information on machine learning models contained in the database. The display control information generation unit 405 may also determine the order in which the machine learning models are displayed based on the degree of relevance between the machine learning models. The display control information includes, for example, information such as the name, model ID, purpose, and performance of the related machine learning models, the display position of this information, and control information related to screen display settings such as the window size. The display control information generation unit 405 outputs the generated display control information to the display control information transmission unit 406.

[0045] The display control information transmission unit 406 receives the display control information and transmits it to the client-side information processing device 11. The display control information transmission unit 406 may also convert the display control information to an appropriate format before transmission.

[0046] The model information acquisition unit 411 acquires model information selected by the user for the purpose of modifying the machine learning model. Model information is, for example, a model ID used to identify the machine learning model held in the data storage unit 403. When the model information acquisition unit 411 downloads a machine learning model, it saves the model information to the RAM 115. The download of the machine learning model and model information is transmitted and received from the server-side data storage unit 403 via a model transmission unit (not shown) and a model reception unit (not shown). The model ID is information used to identify a machine learning model and corresponds one-to-one with the machine learning model. The model information acquisition unit 411 may acquire model information selected by the user by operating an input device 113 such as a touch panel, keyboard, mouse, and robot controller, based on the model information displayed on the display device 114. The model information acquisition unit 411 outputs the acquired model information to the model information transmission unit 412.

[0047] The model information transmission unit 412 transmits the received model information to the model information receiving unit 401. The model information transmission unit 412 may convert the model information to an appropriate format before transmitting it.

[0048] The display control information receiving unit 413 receives information from the server-side information processing device 10 via the network. The display control information receiving unit 413 receives display control information, for example, from the display control information transmitting unit 406. The display control information receiving unit 413 converts the received display control information into data in a format that is easy to use at the output destination and outputs it to the display unit 414.

[0049] The display unit 414 displays a user display screen on the display device 114 based on the display control information received from the display control information receiving unit 413. The display unit 414 displays the information using, for example, a display monitor, a head-mounted display, a touch panel, or a projector. The display unit 414 displays information such as the name, model ID, purpose, and performance of the associated machine learning model at the display position indicated by the display settings of the display device 114, based on the display settings included in the display control information.

[0050] (Explanation of the flowchart showing the processing steps for model information and related model displays) Next, the processing procedure for the related model recommendation process in this embodiment will be described. In the following description, each step will be preceded by an S.

[0051] Figure 5 is a flowchart of the related model recommendation process for recommending related machine learning models in the first embodiment. The related model recommendation process is a flowchart of the processing procedure in which an information processing system having a client-side information processing device 11 and a virtual server-side information processing device 10 displays a list of machine learning models related to the machine learning model selected by the user, based on the model information. The processing content shown in the flowchart of Figure 5 starts, for example, when a machine learning model is selected by the user being processed. However, the information processing system does not necessarily have to perform all the steps described in this flowchart, and the order may be changed as appropriate.

[0052] As preparation for implementing this flowchart, the information processing devices 10 and 11 perform system initialization. Specifically, the CPU 111 of the client-side information processing device 11 reads a program from the ROM 112 to make the client-side information processing device 11 operational. Similarly, the CPU 101 of the virtual server-side information processing device 10 reads a program from the ROM 102 to make the server-side information processing device 10 operational. Then, the client-side information processing device 11 and the virtual server-side information processing device 10 are made able to communicate with each other using the model information transmission unit 412, the model information reception unit 401, the display control information transmission unit 406, and the display control information reception unit 413.

[0053] In S1001, the model information acquisition unit 411 acquires model information for a machine learning model selected by the user for the purpose of changing the machine learning model. For example, the display unit 414 displays one or more machine learning models on the display device 114, and the user selects a machine learning model to change from these models, and the model information acquisition unit 411 acquires model information for that machine learning model.

[0054] Figure 6 illustrates the screen displayed when selecting a machine learning model. Figure 6(a) is a schematic diagram of the machine learning model selection display screen that the user refers to when changing a machine learning model. The machine learning models 601, 602, 603, and 604 displayed on the display screen in Figure 6(a) are machine learning models held in the information processing device 11 and are machine learning models that the user can select. The machine learning models 601, 602, 603, and 604 may be machine learning models with different detection targets, such as people and animals. The machine learning model 601 enclosed in a thick border is the machine learning model selected by the user by operating the input device 113. The model information acquisition unit 411 acquires, for example, the model ID of the machine learning model 601 selected by the user as model information. The model ID is information for identifying a machine learning model and is model-specific information that does not overlap. However, the type of model information acquired by the model information acquisition unit 411 is not limited to the model ID, but may be any information related to the machine learning model. The model information acquisition unit 411 outputs the acquired model information to the model information transmission unit 412.

[0055] In S1002s, the model information transmission unit 412 transmits the model information received from the model information acquisition unit 411 to the model information receiving unit 401. The model information transmission unit 412 may transmit the acquired model information after converting the data into a format suitable for transmission by compression, encryption, etc.

[0056] In S1002r, the model information receiving unit 401 receives model information and outputs it to the database reference unit 402. If the received model information has been converted, the model information receiving unit 401 may perform data conversion such as data decoding and decompression on the model information to convert it back to its original format before outputting it to the database reference unit 402.

[0057] In S1003, the database reference unit 402 refers to a database containing relationship information between models held in the data holding unit 403 to search for machine learning models related to the machine learning model indicated by the model information received from the model information receiving unit 401. The database reference unit 402 outputs the model information (also called related model information) of the related machine learning models obtained as a result of the search to the display control information generation unit 405. Note that if there are many related machine learning models, for example, more than a predetermined number, the database reference unit 402 may search for only that number of machine learning models with a high degree of relevance as the search result.

[0058] Figure 7 shows an example of a database for managing related information between machine learning models. The following explanation will use Figure 7.

[0059] Figure 7(a) shows a database 200 containing related information between machine learning models. The database 200 holds a model ID for identifying registered machine learning models, a model name associated with the model ID, and the model IDs of related machine learning models. The data storage unit 403 may hold at least the database 200.

[0060] Figure 7(b) shows database 201, which represents the degree of relevance between machine learning models. Relevance is an example of related information, indicating, for example, the degree of relationship between machine learning models. Database 201 is a matrix arrangement of relevance between machine learning models calculated by the database management unit 404. Machine learning models indicated by hatching that exceed a predetermined threshold in database 201 are stored in database 200 as related machine learning models. Here, the threshold is set to 0.5, but the threshold may be changed as appropriate.

[0061] Next, we will explain the database based on the ancestral relationships of machine learning models. Here, ancestral relationships refer to the relationships between multiple machine learning models that share a common initial model, traced back to the machine learning model before it was trained, for example, when a machine learning model has been updated one or more times through repeated training.

[0062] Figure 7(c) shows a database 202 corresponding to the parent-child relationships between machine learning models. Here, the parent-child relationship refers to the relationship between two models when a trained machine learning model is used as the initial model and further trained. The initial model is the parent, and the model further trained from the initial model is the child. If a parent-child relationship exists between models, the value of the parent-child relationship is set to 1. On the other hand, if there is no parent-child relationship, the value of the parent-child relationship is set to 0. As an example of related information, if the parent-child relationship is unknown, the value of the parent-child relationship is set to 0. The presence or absence of a parent-child relationship may be obtained from a database showing the relationships between models (not shown). Parent-child relationship information may be obtained by storing the parent-child relationships in a database (not shown) using the model management means described in Patent Document 2, and referencing it by model ID. Here, the value of the parent-child relationship is set to 0 or 1, but for example, the degree of relevance may be set according to the number of iterations of training the machine learning model. For example, the number of iterations of training the machine learning model may be obtained from the model management means, and the degree of relevance may be set higher when the number of iterations is small.

[0063] Figure 7(d) shows a database 203 corresponding to the sibling relationships between machine learning models. Here, a sibling relationship refers to the relationship between two models when they are trained using a pre-trained machine learning model as the initial model, provided that the initial model is the same. Furthermore, a sibling relationship may also be defined as a relationship between machine learning models where the initial model is not only the parent, but also the parent of the parent is the same. If there is a sibling relationship between machine learning models, the sibling relationship value, which is an example of related information, is set to 1. On the other hand, if there is no sibling relationship between machine learning models, the sibling relationship value is set to 0. If the sibling relationship is unknown, the sibling relationship value is set to 0. The presence or absence of a sibling relationship may be obtained from a database showing the relationships between models (not shown). Sibling relationship information may be stored in a database (not shown) using the management means of the model management means described in Patent Document 2, and obtained by referencing it by model ID. Here, an example is shown where the sibling relationship value is set to 0 or 1, but for example, the sibling relationship value may be set according to the number of generations between machine learning models. Specifically, the sibling relationship value may be set higher when the number of generations between machine learning models is small. The number of generations may be, for example, the number of generations traced back to a common machine learning model. In this case, the reciprocal of the distance between generations of machine learning models may be used. Specifically, the sibling relationship value may be set to 1 / 2 if the parents are the same, and the sibling relationship value may be set to 1 / 4 if the parents' parents are the same. Parent-child relationships and sibling relationships can also be described as having the same ancestors as machine learning models, and in this embodiment, the degree of relevance may be set higher when the ancestors are the same.

[0064] Figure 7(e) shows a database 204 representing the ratio (hereinafter also referred to as the usage ratio) of the training datasets used by each machine learning model that were the same across models. In other words, the database 204 includes as related information a ratio set according to whether at least a portion of the training datasets used to train the machine learning models are the same or not. For example, the usage ratio is set to a value between 0 and 1, with the usage ratio set to 1 if all training datasets are the same, the usage ratio set to 0 if they are all different, and the usage ratio set to a value in between if only a portion of the dataset is used. If the usage ratio is unknown, the ratio is set to 0. Information on the usage ratio of training datasets may be obtained by storing information on the training datasets used in a database (not shown) using the training condition management means described in Patent Document 2, and referencing it by model ID. The usage ratio of the same dataset may be calculated, for example, as the ratio of the number of common images to the total number of training images for images, or as the ratio of the number of rows of common data to the total number of rows of training data for table data. However, the method of calculating the usage ratio of the dataset is not limited to this, as long as it represents the ratio of the data.

[0065] Figure 7(f) shows a database 205 for each machine learning model, representing the degree of relationship between machine learning models and the degree of relationship according to information set by the model creator. Here, a relationship between machine learning models refers to, for example, a successor model that performs the same task but does not have at least one parent-child or sibling relationship. For example, the information set by the model creator may be information indicating whether or not there is a relationship between machine learning models. There are various types of machine learning model training methods, initial models, and training datasets. Even if a model creator develops a successor model using a different training dataset with an initial model that does not have at least one parent-child or sibling relationship, it may still be recommended as a related model. If there is a relationship between machine learning models, the degree value is set to 1; if there is no relationship, the degree value is set to 0. The degree value is also called the author-set flag. In this embodiment, the model creator is free to register related models; however, to prevent the indiscriminate registration and display of unrelated models, the number of models that can be set may be limited. Alternatively, the creator may pay a fee to increase the relevance level, making the machine learning model more likely to be selected. The payment of money here is just one example of a predetermined condition.

[0066] In S1004, the display control information generation unit 405 generates display control information necessary for the server-side display unit 414, based on the database, to display on the display device 114. For example, the display control information generation unit 405 obtains data by referring to the model information received from the database reference unit 402 and the database, and generates display control information. The display control information generation unit 405 may obtain data such as model information of highly related machine learning models and generate display control information that sets a display order for a predetermined number of machine learning models to be displayed in order of relevance. The display control information generation unit 405 may, for example, generate display control information related to display settings such as the display position, window size, and display text of each model information and the model name associated with that model information. However, the types of data obtained and display control information generated by the display control information generation unit 405 are not limited to these, and may be any information that can be managed in the database. The display control information generation unit 405 outputs the obtained data and generated display control information to the display control information transmission unit 406.

[0067] In S1005s, the display control information transmission unit 406 transmits the display control information received from the display control information generation unit 405 and the acquired data to the client-side display control information receiving unit 413. The display control information transmission unit 406 may transmit the display control information and acquired data after converting the data into a format suitable for transmission by compression, encryption, etc.

[0068] In S1005r, the display control information receiving unit 413 receives the display control information and the acquired data. When the display control information receiving unit 413 receives the converted data, it may decode and decompress the display control information and the acquired data to convert them back to their original format. The display control information receiving unit 413 outputs the acquired data and the display control information to the display unit 414.

[0069] In S1006, the display unit 414 displays a display screen containing a list of multiple model information on the display device 114 based on the acquired data and display control information received from the display control information receiving unit 413. Figure 6(b) is a schematic diagram of the display screen containing model information of candidate machine learning models. In the display screen of Figure 6(b), machine learning model 605 is the machine learning model selected by the user. Machine learning models 606 and 607 are candidate models displayed based on their relevance. For example, if the machine learning model 605 selected by the user is a model that detects people, then other machine learning models 606 and 607 that detect people are selected and displayed as candidate models.

[0070] As a result, the information processing system terminates its processing.

[0071] As described above, this embodiment can select and present to the user a machine learning model related to the machine learning model selected by the user. This makes it easier for the user to select a machine learning model.

[0072] This embodiment can display a list of recommended models linked to model information.

[0073] Figure 8 is a flowchart of the related model determination process for determining related models in the first embodiment. In the related model determination process, the database management unit calculates the degree of relevance between machine learning models, determines the related models, and saves or updates the database. The database management unit starts the flowchart in Figure 8, for example, when a new machine learning model is added to the model storage unit (not shown).

[0074] In S2001, the database management unit 404 updates the parent-child relationship database 202 according to the parent-child relationships between machine learning models.

[0075] In S2002, the database management unit 404 updates the sibling relationship database 203 according to the sibling relationships between machine learning models.

[0076] In S2003, the database management unit 404 updates the training data database 204 according to the ratio of machine learning models that used the same training dataset.

[0077] In S2004, the database management unit 404 updates the database 205 of creator-defined degrees according to information on related machine learning models and author-defined flags indicating the degree set by the model creator.

[0078] In S2005, the database management unit 404 updates the database 201, which shows the relevance, by multiplying each database by a correction coefficient and adding the resulting values. The correction coefficient here is a coefficient that adjusts which databases between models the server administrator gives more weight to when determining the relevance. For example, the correction coefficient for the parent-child relationship database 202 is 0.5, the correction coefficient for the sibling relationship database 203 is 0.2, the correction coefficient for the training dataset database 204 is 0.5, and the correction coefficient for the degree database 205 is 1. The database management unit 404 adds the value obtained by multiplying each database by the correction coefficient. For example, the relevance showing the relationship between the machine learning model with model ID "0001" and the machine learning model with model ID "0003" is (0.5) + (0) + (0.25) + (0) = (0.75).

[0079] In S2006, the database management unit 404 refers to the database 201, which shows the degree of relevance, and compares the corrected degree of relevance stored in the database 201 with a pre-set threshold. Subsequently, the database management unit 404 determines machine learning models with a degree of relevance equal to or greater than the threshold as related models and updates the database 200, which represents the relationship information between machine learning models. Here, the threshold is set to 0.5, and the shaded areas in the database indicate locations where the degree is greater than or equal to the threshold. For example, the machine learning models related to the machine learning model with model ID "0003" are the machine learning models with model IDs "0000" and "0001". The pre-set threshold may be used by the server administrator to set the recommendation amount based on relevance. The database management unit 404 may also determine machine learning models with a degree of relevance equal to or less than the threshold as related models.

[0080] As a result, the database management unit 404 terminates the related model determination process.

[0081] In this embodiment, when the user selects a machine learning model to use on the client-side information processing device 11, a highly relevant model is recommended. However, when a new machine learning model related to a virtual server is registered, a display screen may be shown recommending the new machine learning model if it is related to the machine learning model the user is currently using.

[0082] As described above, this embodiment selects and recommends machine learning models that are highly relevant to the model currently used by the user from among the publicly available machine learning models, thereby providing an information processing system that allows users to easily select a machine learning model suitable for their application.

[0083] This embodiment selects relevant machine learning models based on a degree of relevance set according to information such as ancestral relationships (parent-child and sibling relationships), training data, and a degree set by the creator. As a result, this embodiment can select relevant machine learning models more appropriately.

[0084] (Second embodiment) In this embodiment, model information selected based on the user's model usage statistics is used to recommend machine learning models that are highly relevant to the selected machine learning model.

[0085] Figure 9 is a block diagram showing an example of the functional configuration of an information processing system according to the second embodiment. The virtual server-side information processing device 10 includes a statistical information receiving unit 409, a database reference unit 402, a data holding unit 403, a database management unit 404, a display control information generation unit 405, a display control information transmission unit 406, and a statistical information processing unit 407. The client-side information processing device 11 includes an inference unit 415, a statistical information acquisition unit 416, a statistical information transmission unit 418, a display control information receiving unit 413, and a display unit 414. The second embodiment will be described focusing on configurations that differ from the first embodiment.

[0086] The inference unit 415 performs inference on an image obtained by an imaging device (not shown) in the information processing device 11 for the object detection task of a machine learning model.

[0087] The statistical information acquisition unit 416 acquires statistical information of the inference results of the machine learning model executed by the inference unit 415. The statistical information acquisition unit 416 may store the number of times the machine learning model has not been detected as statistical information in the RAM 115. The statistical information acquisition unit 416 acquires the statistical information from the RAM 115 and outputs it to the statistical information transmission unit 418. The statistical information acquisition unit 416 may also output model information along with the statistical information to the statistical information transmission unit 418.

[0088] The statistical information transmission unit 418 transmits the received statistical information, along with the model information, to the statistical information receiving unit 409 of the server-side information processing device 10 via the network. The statistical information transmission unit 418 may also transmit the statistical information as part of the model information.

[0089] The statistical information receiving unit 409 is an example of an acquisition means, and receives and acquires statistical information and model information. The statistical information receiving unit 409 outputs the received statistical information and model information to the statistical information processing unit 407.

[0090] The statistical information processing unit 407 determines candidate machine learning models for modification based on the statistical information obtained by inference from the statistical information receiving unit 409, and outputs the model information to the database reference unit 402.

[0091] Figure 10 is a flowchart of the related model recommendation process for recommending related machine learning models in the second embodiment. In the related model recommendation process of the second embodiment, the information processing system selects machine learning models from statistical information representing the user's usage trends in the second embodiment and displays a list of related machine learning models. Figure 10 uses symbols corresponding to the flowchart of the first embodiment in Figure 5. For example, steps S3002 to S3006 in Figure 10 are the same processes as S1002 to S1006 in Figure 5, so the explanation is omitted or simplified.

[0092] In S3001, the statistical information acquisition unit 416 acquires statistical information of at least one of inference and usage, along with model information, for the purpose of modifying the user's machine learning model. The statistical information acquisition unit 416 may acquire the number of times the machine learning model has failed to detect an area as statistical information. The number of times the model has failed to detect an area is, for example, the number of times when the machine learning model performed inference once on a single input image and failed to obtain any detection areas. The acquired statistical information is compressed and encrypted by the statistical information acquisition unit 416 into a format suitable for transmission. The statistical information acquisition unit 416 then outputs the statistical information along with the model information to the statistical information transmission unit 418.

[0093] In S3002s and S3002r, the client-side statistical information transmission unit 418 transmits statistical information along with model information to the virtual server-side statistical information receiving unit 409.

[0094] In S3007, the statistical information processing unit 407 selects a machine learning model with high priority from the user's machine learning models based on statistical information. Here, we give an example in which the statistical information processing unit 407 selects a machine learning model with a high number of undetected objects, for example, based on the number of times detection has failed. However, the statistical information processing unit 407 may also select a machine learning model with high priority based on the number of inference operations (for example, a machine learning model with many inference operations) or a machine learning model with high priority based on the number of images taken by the imaging device (for example, a machine learning model with many images taken). The number of inference operations here refers to the number of times the machine learning model performs inference once for each input image in the object detection task, and the count is increased by 1. If there are more highly relevant machine learning models than a predetermined number, the statistical information processing unit 407 may preferentially select that number of high-priority machine learning models. The statistical information processing unit 407 outputs the selected high-priority machine learning models to the database reference unit 402.

[0095] After S3003 and S3004 are executed, the database reference unit 402 retrieves related model information from the database based on the model information of the high-priority machine learning model. Subsequently, the display control information generation unit 405 generates display control information corresponding to the high-priority machine learning model. The display control information generation unit 405 may also generate display control information corresponding to the highly related machine learning model along with the selected high-priority machine learning model based on the database.

[0096] In S3005s and S3005r, the display control information transmission unit 406 on the virtual server side transmits display control information to the display control information receiving unit 413 on the client side.

[0097] In S3006, the display unit 414 displays a display screen generated based on display control information corresponding to a high-priority machine learning model on the display device 114.

[0098] In this embodiment, an example was given in which the client-side information processing device 11 acquires statistical information, but the same process can be performed even when the virtual server-side information processing device 10 performs inference and acquires statistical information.

[0099] As described above, this embodiment can recommend high-priority machine learning models that have issues such as a high number of undetected results, and that are highly relevant to the machine learning model, by utilizing statistical information that represents the user's usage trends. Therefore, this embodiment can provide an information processing system that enables easy selection of a model suitable for the user's application.

[0100] (Modified version of the second embodiment) A modification of the second embodiment shows an example in which the machine learning model to recommend is selected using the usage period of the machine learning model as user statistics. This embodiment is based on the tendency that the longer the model usage period, the higher the accuracy and reliability of the machine learning model. Figures 11A to 11C illustrate the modification of the second embodiment.

[0101] The statistical information acquisition unit 416 acquires statistical information of the inference results of the model executed by the inference unit 415. The statistical information acquisition unit 416 may store the model usage period as statistical information in the RAM 115. The statistical information acquisition unit 416 acquires the statistical information from the RAM 115 and transmits it to the model information transmission unit 412.

[0102] Figure 11A is a flowchart of the related model determination process for determining related models in a modified example of the second embodiment. In the related model determination process, the database management unit 404 calculates the degree of relevance between models and determines the related models. Processes similar to those in the above-described embodiment are omitted or simplified in their explanation. For example, steps S4001 to S4006 in Figure 11A are the same as steps S2001 to S2006 in the first embodiment, so their explanation is omitted or simplified.

[0103] In S4007, the database management unit 404 calculates the average usage period of machine learning models across multiple client-side information processing devices 11 based on the usage period of the machine learning models transmitted from the client-side information processing devices 11, and updates the database (in this case, the average usage period database) for the usage period of each machine learning model. Figure 11B shows an example of the average usage period database 1101 of machine learning models in a modified version of the second embodiment. The average usage period database 1101 associates the model ID of the machine learning model with the average usage period (in days). Note that the unit of the average usage period is not limited to days and may be changed as appropriate. The machine learning model with model ID "0001" has the longest average usage period and is considered a highly reliable machine learning model that is frequently used by users.

[0104] As a result, the database management unit 404 terminates the processing of the related model determination process.

[0105] Figure 11C is a flowchart of the related model recommendation process for recommending related machine learning models in a modified version of the second embodiment. In the related model recommendation process, the information processing system selects machine learning models from statistical information representing the user's usage trends and displays a list of related machine learning models. Figure 11C is labeled with symbols corresponding to the flowchart of the first embodiment in Figure 10. Processes similar to those in the above-described embodiments are omitted or simplified in their explanation. For example, S4101 and S4102 are the same processes as S3001 and S3002, and S4103, S4105, and S4106 are the same processes as S3003, S3005, and S3006. Therefore, similar processes are omitted or simplified in their explanation.

[0106] In S4107, the database reference unit 402 refers to the database, which includes the average usage period, stored in the data holding unit 403, and selects a machine learning model with a long average usage period (or usage period). The database reference unit 402 outputs model information of the selected machine learning model to the display control information generation unit 405.

[0107] In S4108, the display control information generation unit 405 generates data acquisition and display control information necessary for display on the display unit 414, based on data such as model information received from the database reference unit 402. For example, the data to be acquired is data of machine learning models with a high degree of relevance and a long average usage period. The display control information generation unit 405 may generate display control information to display a predetermined number of machine learning models in order of decreasing relevance. If the number of machine learning models with a high degree of relevance exceeds a predetermined number, the display control information generation unit 405 may select that number of machine learning models with a long average usage period. The display control information generation unit 405 may, for example, generate display control information related to display settings such as the display position, window size, and display text of each model information and the model name associated with that model information. However, the types of data to be acquired and the display control information to be generated are not limited to these, and may be any information that can be managed in the database. The display control information generation unit 405 then outputs the acquired data and the generated display control information to the display control information transmission unit 406.

[0108] While this modification demonstrates the use of the average usage period, the median and mode of usage periods may also be used to obtain user usage trends. Furthermore, even with the same usage period, users with higher usage frequency provide more reliable information for the machine learning model, so weighting may be applied when calculating the average according to usage frequency.

[0109] Furthermore, while this modification uses a simple average usage period, it would also be possible to retain the machine learning model used immediately before changing the machine learning model, maintain a matrix of the average usage periods of the previously used machine learning model and the changed machine learning model, and recommend the model with the longest usage period after the change.

[0110] As explained above, this modified version can recommend publicly available machine learning models that are highly relevant to the model currently used by the user, and that have a long usage period, such as an average usage period. This modified version provides an information processing system that makes it easy for users to select more reliable machine learning models that they frequently use, through this recommendation method.

[0111] (Third embodiment) In the third embodiment, when a user downloads a machine learning model from the virtual server-side information processing device 10 to the client-side information processing device 11, the system recommends a machine learning model that is highly related to the downloaded machine learning model.

[0112] Figure 12 is a block diagram showing an example of the functional configuration of an information processing system according to the third embodiment. The virtual server-side information processing device 10 includes a model information receiving unit 401, a database reference unit 402, a data holding unit 403, a database management unit 404, a display control information generation unit 405, a display control information transmission unit 406, a model management unit 408, and a model transmission unit 420. The client-side information processing device 11 includes a model information acquisition unit 411, an inference unit 415, a model information transmission unit 412, a display control information receiving unit 413, a display unit 414, a model holding unit 417, and a model receiving unit 419.

[0113] The model management unit 408 manages machine learning models stored in the database of the information processing device 10 on the virtual server side. The model management unit 408 manages, for example, the registration of new machine learning models and the deletion of registered machine learning models.

[0114] The model transmission unit 420 receives a machine learning model from the model management unit 408. The model transmission unit 420 downloads the received machine learning model to the model receiving unit 419 of the client-side information processing device 11 and transmits it. The display control information transmission unit 406 and the model transmission unit 420 may have separate interfaces or the same interface.

[0115] The display control information generation unit 405 generates display control information to display the newly registered machine learning model as the associated machine learning model when the newly registered machine learning model is associated with the machine learning model indicated by the model information obtained from the client-side information processing device 11.

[0116] The model information receiving unit 401 may receive model information from the client-side information processing device 11 in response to the timing of the machine learning model download, or triggered by the download.

[0117] The model receiving unit 419 receives information from the server-side information processing device 10 via the network. For example, the model receiving unit 419 receives information about a machine learning model. The received display control information is output to the display unit 414. The display control information receiving unit 413 and the model receiving unit 419 may have separate interfaces or the same interface.

[0118] The model storage unit 417 stores the machine learning model downloaded from the information processing device 10 on the virtual server side.

[0119] Figure 13 is a flowchart of the related model recommendation process for recommending related machine learning models in the third embodiment. In the related model recommendation process of the third embodiment, when the machine learning model selected by the user is downloaded from the information processing device 10 on the virtual server side to the information processing device 10 on the client side, models highly related to the downloaded machine learning model are recommended.

[0120] In S5001, the display unit 414 and the like display a list of machine learning models related to the machine learning model selected by the user on the display device 114 as recommended machine learning models. The process in S5001 is the same as the process in steps S1001 to S1006 of the first embodiment.

[0121] In S5002, the model information acquisition unit 411 receives a selection of the machine learning model to download from the user. The user may, for example, select the machine learning model to download from the related machine learning models displayed by the display unit 414 using the input device 113. The model information acquisition unit 411 outputs the machine learning model received from the user to the model information transmission unit 412.

[0122] In S5003s, the model information transmission unit 412 transmits the model information of the machine learning model selected by the user in S5002 to the model information receiving unit 401 on the virtual server side. The model information transmission unit 412 may convert the acquired model information into a format suitable for transmission by compression, encryption, or other means before transmitting it.

[0123] In the S5003r, the model information receiving unit 401 receives model information of the machine learning model selected by the user and outputs it to the model management unit 408. If the received model information has been converted into data, the model information receiving unit 401 may perform data conversion such as data decoding and decompression on the model information to convert it back to the original format before outputting it to the model management unit 408.

[0124] In S5004, the model management unit 408 retrieves the machine learning model selected by the user via the database management unit 404 and outputs it to the model transmission unit 420.

[0125] In the S5005s, the model transmission unit 420 transmits the machine learning model selected by the user to the client-side model reception unit 419. The model transmission unit 420 may transmit the machine learning model after converting the data into a format suitable for transmission by compression, encryption, etc.

[0126] In the S5005r, the client-side model receiving unit 419 receives the machine learning model and outputs it to the model holding unit 417. If the received machine learning model has undergone data conversion, the model receiving unit 419 may perform data conversion such as data decoding and decompression on the machine learning model to convert it back to its original format before outputting it to the model holding unit 417.

[0127] In S5006, the model storage unit 417 stores the model information of the machine learning model selected and newly downloaded by the user. The model storage unit 417 may also store the machine learning model along with the model information.

[0128] Furthermore, the process in S5007 is executed. The process in S5007 is the same as the process in steps S1001 to S1006 of the first embodiment. Note that S5007 may be executed in parallel with other processes. For example, the process in S5007 may be executed in parallel from S5002 onwards. Here, the display control information transmission unit 406 may receive the acquisition data of the machine learning model related to the machine learning model selected by the user and the display control information from the display control information generation unit 405 and transmit it to the display control information receiving unit 413. Also, if a newly registered machine learning model is included in the related machine learning models, the display control information generation unit 405 may generate display control information that includes the display of the newly registered machine learning model. The display control information transmission unit 406 and the display control information receiving unit 413 may convert and transmit / receive the data. The display control information receiving unit 413 outputs the received display control information to the display unit 414. Based on the display control information, the display unit 414 causes the display device 114 to display a screen containing information such as the machine learning model selected by the user and related machine learning models.

[0129] As described above, in this embodiment, a highly relevant model is recommended at the time a machine learning model is downloaded. This recommendation method provides an information processing system that enables users to easily select a model suitable for their purpose when they want to search for other machine learning models.

[0130] (Fourth embodiment) In this embodiment, a UI (user interface) is provided for inputting information about failed shots. Information about the machine learning model used when the image with the failed information was taken is utilized to recommend a machine learning model that can resolve the failure.

[0131] Figure 14 is a block diagram showing an example of the functional configuration of the information processing system in the fourth embodiment. The fourth embodiment will be described with reference to Figure 14.

[0132] The virtual server-side information processing device 10 includes a failure information receiving unit 423, a database reference unit 402, a data holding unit 403, a database management unit 404, a display control information generation unit 405, and a display control information transmission unit 406.

[0133] The client-side information processing device 11 includes an inference unit 415, a failure information acquisition unit 421, a failure information transmission unit 422, a display control information receiving unit 413, a display unit 414, and an inference unit 415.

[0134] The inference unit 415 performs inference using a machine learning model.

[0135] The failure information acquisition unit 421 generates a display screen for inputting failure information from when the machine learning model performed inference, along with an image taken by the user with an imaging device (not shown). The failure information acquisition unit 421 refers to the input screen and acquires the failure information entered by the user and the model information of the machine learning model used during inference of the image. The failure information may be flag information of 0 or 1 indicating whether or not the inference failed, and the type and cause of the failure. Specifically, the failure information includes information about the type or cause of the failure, such as failure due to blurring or failure due to focusing on a subject other than the desired subject. If the user inputs that it failed, the failure information flag is set to 1; otherwise, the failure information flag is set to 0. Similarly, the flag indicating the type and cause of the failure is set to 1 if applicable, and to 0 if not applicable. Note that the initial value of all failure information may be 0. The failure information acquisition unit 421 outputs the acquired model information and information including the failure information to the failure information transmission unit 422.

[0136] The failure information transmission unit 422 receives the failure information and the model information of the machine learning model used for inference, and transmits it to the failure information receiving unit 423 of the server-side information processing device 10.

[0137] The failure information receiving unit 423 is an example of an acquisition means, and receives information including failure information and model information of machine learning models that failed to perform inference from the client-side information processing device 11 via the network. The failure information receiving unit 423 outputs the received failure information and model information to the database reference unit 402.

[0138] If the database reference unit 402 obtains failure information, it refers to a database that includes at least one of the detection rate and false positive rate, as described later. Based on the failure information and at least one of the detection rate and false positive rate, the database reference unit 402 obtains related model information for the machine learning model selected by the user.

[0139] The database reference unit 402 retrieves relevant model information from the database based on the failure information and model information.

[0140] Figures 15A and 15B are flowcharts of the related model recommendation process for recommending relevant machine learning models in the fourth embodiment. In the related model recommendation process of the fourth embodiment, failure information is entered when the user checks an image taken with an imaging device (not shown), and if a failure occurred, a model with a high degree of relevance is recommended. Figure 15A shows the client-side process. Figure 15B shows the virtual server-side process. The letters A, B, and C in the circles shown in Figures 15A and 15B indicate that they are connected to each other. Figure 16 shows an example of a screen for entering failure information in the fourth embodiment.

[0141] The explanation of processes similar to those in the embodiments described above will be omitted or simplified. For example, steps S6001 and S6002 are the same processes as steps S1001 and S1002 in the first embodiment, and steps S6004 to S6006 are the same processes as steps S1004 to S1006 in the first embodiment. Therefore, the explanation of these processes will be omitted or simplified.

[0142] In S6010, the failure information acquisition unit 421 displays a screen including a success button and a failure button on the display device 114 via the display unit 414, along with an image captured by an imaging device (not shown) and stored in the media drive 117. Figure 16 is an example of a display screen that the failure information acquisition unit 421 displays in order to acquire failure information.

[0143] In S6011, the failure information acquisition unit 421 determines whether or not the failure button has been pressed. Figure 16(a) is an example of a display screen that shows an out-of-focus image among the images displayed during image scrolling. The display screen in Figure 16(a) includes image 1601, success button 1602, and failure button 1603. If image 1601 in Figure 16(a) is out of focus on the person who is the subject to be photographed and is therefore a failure, the user selects and presses the failure button 1603. If the failure button 1603 is pressed, the failure information acquisition unit 421 holds the failure information indicating whether or not there was a failure as 1 and proceeds to S6012. If the failure button 1603 is not pressed, the failure information acquisition unit 421 proceeds to S6005r.

[0144] In S6012, the failure information acquisition unit 421 determines whether or not the blur button 1605 has been pressed. Figure 16(b) is an example of a display screen for inputting the cause of failure after the failure button has been pressed. The display screen in Figure 16(b) includes an image 1604, a blur button 1605, and a different subject button 1606. As shown in Figure 16(b), if the subject to be photographed is not in focus and the image has failed, the user selects and presses the blur button 1605. One possible cause of this failure is that the subject to be photographed has not been detected, i.e., the detection rate is low.

[0145] If the failure information acquisition unit 421 determines that the blur button 1605 has been pressed by the user, the process proceeds to S6019. In S6019, the failure information acquisition unit 421 updates the blur failure information to 1, outputs it to the failure information transmission unit 422, and proceeds to S6020. On the other hand, if the user has not pressed the blur button 1605, the failure information acquisition unit 421 proceeds to S6013.

[0146] In S6013, the failure information acquisition unit 421 determines whether or not a different subject button was pressed. Figure 16(c) is an example of a display screen that shows an image displayed during image scrolling where a different subject is in focus. The display screen in Figure 16(c) includes image 1607, success button 1608, and failure button 1609. Image 1607 in Figure 16(c) is in focus on a dog, which is a different subject, so it is a failure as the person, the subject to be photographed, is not in focus. One possible cause of this failure is a false detection of the subject, i.e., a high false detection rate. In this case, if the user presses the failure button 1609, the screen in Figure 16(d) is displayed. Figure 16(d) is an example of a display screen for inputting the cause of failure after pressing the failure button 1609. The display screen in Figure 16(d) includes image 1610, blur button 1611, and different subject button 1612. As shown in Figure 16(d), if a different subject than the intended subject is mistakenly detected and the focus is on the dog, resulting in a failed shot, the user selects and presses the "different subject" button 1612.

[0147] If the failure information acquisition unit 421 determines that a different subject button 1612 was pressed by the user, the process proceeds to S6019. In S6019, the failure information acquisition unit 421 updates the failure information due to a different subject to 1, outputs it to the failure information transmission unit 422, and proceeds to S6020. On the other hand, if the user did not press a different subject button 1612, the failure information acquisition unit 421 proceeds to S6020.

[0148] In S6020, the failure information transmission unit 422 transmits the failure information acquired from the failure information acquisition unit 421 to the failure information receiving unit 423 on the virtual server side. As described above, the failure information includes information such as whether or not there was a failure, failure due to blurring, and failure due to focusing on a different subject.

[0149] In S6021, the failure information receiving unit 423 on the virtual server side determines whether or not it has received failure information. If the failure information receiving unit 423 determines that it has received failure information, it outputs the failure information to the database and proceeds to S6022. On the other hand, if the failure information receiving unit 423 has not received failure information, it proceeds to S6004.

[0150] In S6022, the database reference unit 402 refers to the failure information obtained from the failure information receiving unit 423 to determine the cause of the failure. Specifically, the database reference unit 402 determines whether the cause of the failure is blur or focusing on a different subject. If the database reference unit 402 determines that the cause of the failure is blur, it proceeds to S6014. On the other hand, if the database reference unit 402 determines that the cause of the failure is not blur, that is, focusing on a different subject, it proceeds to S6015.

[0151] In S6014 and S6015, the database reference unit 402 retrieves relevant model information for a machine learning model from the model information and at least one of the detection rate and false positive rate registered in the database, based on at least one of the detection rate and false positive rate of the machine learning model selected by the user. For details on how to calculate the detection rate and the false positive rate described later, see “Powers, David MW (2011). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation”.

[0152] Figure 17 shows a database 1701 of the relevant machine learning model, detection rate, and false positive rate for the fourth embodiment. The detection rate may be, for example, a value relating to the ratio of the number of times the target object was detected to the number of times the machine learning model performed inference. The false positive rate may be, for example, a value relating to the ratio of the number of times a target other than the desired target was incorrectly detected to the number of times the machine learning model performed inference.

[0153] In S6014, the database lookup unit 402 may obtain related model information for machine learning models with a high detection rate if the cause of failure is blur. This is because if the cause of failure is blur, it is thought that the subject to be photographed was not detected, meaning that the detection rate of the machine learning model selected by the user is low. Here, the database lookup unit 402 recommends candidate model 2 with a high detection rate based on database 1701. After that, the database lookup unit 402 proceeds to S6004.

[0154] In S6015, the database lookup unit 402 retrieves relevant model information with a low false positive rate from the model information and the database if the cause of the failure is not blur, that is, if a different subject is detected. This is because detecting a different subject suggests that the machine learning model selected by the user has a high false positive rate, and therefore a machine learning model with a low false positive rate should be recommended to the user. Based on the database 1701, the database lookup unit 402 prioritizes recommending candidate model 3 with a low false positive rate. After that, the database lookup unit 402 proceeds to S6004.

[0155] In S6004, the display control information generation unit 405 generates display control information based on the database and outputs it to the display control information transmission unit 406. The display control information generation unit 405 may generate display control information based on the model information received in S6002r, as well as the related model information acquired in S6014 and S6015.

[0156] In S6005s, S6005r, and S6006, the display control information transmission unit 406 on the virtual server side transmits display control information to the display control information receiving unit 413 on the client side. The display unit 414 displays a display screen on the display device 114 based on the display control information generated by failure information, etc.

[0157] In this embodiment, a different machine learning model was recommended based on the failure information of a single image, but recommendations may also be made based on input information from multiple images. The system may also be configured to display a related model only when the input of failure information exceeds a predetermined number of images.

[0158] As described above, this embodiment provides a UI for inputting failure information during shooting, and recommends a machine learning model that can resolve the failure and has a high degree of relevance based on the failure information and information on the machine learning model used during shooting. Thus, this embodiment provides an information processing system that enables the easy identification and recommendation of a machine learning model that resolves the user's problem based on the user's failure information using this recommendation method.

[0159] This embodiment includes information about the type of failure, along with whether or not a failure occurred, in the failure information. This allows this embodiment to recommend a more appropriate machine learning model to the user based on the type of failure.

[0160] <Other Embodiments> For example, an information processing system may select a recommended machine learning model based on the number of detections in an object detection task. For instance, if the number of detections is small, the database lookup unit 402 may prioritize recommending a machine learning model with a high detection rate among the relevant machine learning models. Conversely, if the number of detections is large, the database lookup unit 402 may prioritize recommending a machine learning model with a low false positive rate among the relevant machine learning models.

[0161] Although examples of embodiments have been described in detail above, the present invention can take the form of, for example, an information processing system, an information processing device, an information processing method, a program, or a recording medium (storage medium). Specifically, it may be applied to a system consisting of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or to a device consisting of a single device.

[0162] The embodiments described above may be combined.

[0163] The order of each step in the flowchart of the above embodiment may be changed as appropriate.

[0164] In the above embodiment, the information processing device 10 on the virtual server side and the information processing device 11 on the client side were described as separate devices, but both information processing devices 10 and 11 may be integrated into one unit.

[0165] Furthermore, it goes without saying that the object of the present invention is achieved as follows: a recording medium (or storage medium) containing program code (computer program) of software that realizes the functions of the embodiments described above is supplied to a system or device. Such storage medium is, needless to say, a computer-readable storage medium. The computer (or CPU or MPU, etc.) of the system or device then reads and executes the program code stored on the recording medium. In this case, the program code read from the recording medium itself realizes the functions of the embodiments described above, and the recording medium containing that program code constitutes the present invention.

[0166] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0167] The disclosures herein include the following information processing devices, information processing methods, and programs. (Item 1) A means of obtaining model information, which is information about a machine learning model, A reference means that obtains related model information indicating machine learning models related to the acquired model information by referring to a database that shows the relationships between multiple machine learning models, A generation means for generating display control information for displaying a screen containing information on a recommended machine learning model based on the aforementioned related model information, An information processing device equipped with the following features. (Item 2) The aforementioned reference means refers to the database based on relevant information corresponding to the degree of relevance between machine learning models. The information processing device described in item 1. (Item 3) The aforementioned reference means refers to the database based on relevant information corresponding to the ancestral relationships of the machine learning model. An information processing device according to item 1 or item 2, characterized in that it is an information processing device according to item 1 or item 2. (Item 4) The reference means refers to the database based on the relevant information corresponding to at least one of the parent-child and sibling relationships of the machine learning model, as information about the ancestors. The information processing device described in item 3, characterized by the features described herein. (Item 5) The aforementioned reference means refers to the database based on the related information corresponding to the number of learning repetitions in the parent-child relationship. The information processing device described in item 4, characterized by the features described herein. (Item 6) The aforementioned reference means refers to the database based on the related information corresponding to the number of generations between machine learning models in the sibling relationship. An information processing device according to item 4 or item 5, characterized in that it is an information processing device. (Item 7) The reference means refers to the database based on relevant information depending on whether at least a portion of the training datasets used to train the machine learning model are the same. An information processing device according to any one of items 1 to 6, characterized by the features described in item 1 to 6. (Item 8) The aforementioned reference means refers to the database based on the relevant information corresponding to the ratio used in the same training dataset. The information processing device described in item 7, characterized by the features described herein. (Item 9) The aforementioned reference means refers to the database which contains relevant information corresponding to the information set by the creator of the machine learning model. An information processing device according to any one of items 1 to 8, characterized by the above. (Item 10) The reference means refers to the database based on the related information, which corresponds to at least one of the information set by the creator as to whether or not there is a relationship between machine learning models, and the information as to whether or not the creator has met predetermined conditions. The information processing device according to item 9, characterized in that it is a processing device. (Item 11) The acquisition means acquires model information of the machine learning model selected by the user. An information processing device according to any one of items 1 to 10, characterized by the features described in item 1 to 10. (Item 12) The database includes a model management means for managing multiple machine learning models contained in the aforementioned database, The acquisition means acquires model information in accordance with the timing at which the machine learning model managed by the model management means is downloaded. An information processing device according to any one of items 1 to 11, characterized by the features described in item 1 to 11. (Item 13) The acquisition means acquires model information and failure information related to the failure of the machine learning model whose inference failed. The reference means acquires the related model information corresponding to the model information and the failure information. An information processing device according to any one of items 1 to 12, characterized in that it is the same as described in item 1 to 12. (Item 14) The database includes a model management means for managing multiple machine learning models contained in the aforementioned database, The generation means generates the display control information, including the display of the registered machine learning model, if the machine learning model newly registered in the database is a related machine learning model. An information processing device according to any one of items 1 to 13, characterized by the features described in item 1 to 13. (Item 15) It includes statistical information processing means for acquiring and processing statistical information obtained from the inference of machine learning models, The acquisition means acquires the statistical information along with the model information. An information processing device according to any one of items 1 to 14, characterized by the features described in item 1 to 14. (Item 16) The statistical information processing means selects a machine learning model with a high priority based on the number of inference operations. The aforementioned reference means references relevant information of the high-priority machine learning model. The information processing device described in item 15, characterized by the features described herein. (Item 17) The statistical information processing means selects a machine learning model with a high priority based on the number of undetected cases in the inference. The aforementioned reference means references relevant information of the high-priority machine learning model. An information processing device as described in item 15 or item 16, characterized by the above. (Item 18) A database management means for managing the related model information is provided based on the degree of relevance, which indicates the degree of relationship between machine learning models. An information processing device according to any one of items 1 to 17, characterized by being equipped with the following: (Item 19) The generation means determines the display order of the information of the related machine learning models based on the degree of relevance, which indicates the degree of relevance. An information processing device according to any one of items 1 to 18, characterized by the features described herein. (Item 20) The reference means obtains the relevant model information by referring to the database based on at least one of the detection rate, which is the ratio of the number of times the target was detected to the number of times the machine learning model performed inference, and the false positive rate, which is the ratio of the number of false positives. An information processing device according to any one of items 1 to 19, characterized in that it is the same as described in item 1 to 19. (Item 21) If the detection rate of the acquired machine learning model is low, the reference means acquires the related model information of the machine learning model with a high detection rate among the related machine learning models. The information processing device described in item 20, characterized by the features described herein. (Item 22) The reference means, if the false positive rate of the acquired machine learning model is high, acquires the related model information of a machine learning model with a low false positive rate among the related machine learning models. An information processing device according to item 20 or item 21, characterized in that it is an information processing device. (Item 23) If the reference means indicates that the inference failed, and the failure information indicates that the subject to be photographed has not been detected, it retrieves the relevant model information of the relevant machine learning model from the database based on the detection rate. An information processing device according to any one of items 20 to 22, characterized in that it is an information processing device. (Item 24) If the reference means obtains failure information indicating that the inference failed, which indicates that a different subject was detected, it obtains the relevant model information of the relevant machine learning model from the database based on the detection rate and the false positive rate. An information processing device according to any one of items 20 to 23, characterized by the features described herein. (Item 25) The reference means obtains the relevant model information of the relevant machine learning model by referring to the database which includes the usage period of the machine learning model, The generation means generates the display control information based on the usage period. An information processing device according to any one of items 1 to 24, characterized by the features described in item 1 to 24. (Item 26) We obtain model information, which is information about the machine learning model. By referring to a database that shows the relationships between multiple machine learning models, related model information is obtained that shows the machine learning models related to the acquired model information. Based on the aforementioned related model information, display control information is generated for displaying a screen containing information on the recommended machine learning model. An information processing method characterized by the following: (Item 27) A program to cause a computer to function as one of the information processing devices described in any one of items 1 through 25.

[0168] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0169] 10... Information processing unit, 401... Model information receiving unit, 402... Database reference unit, 403... Data holding unit, 404... Database management unit, 405... Display control information generation unit.

Claims

1. A means of obtaining model information, which is information about a machine learning model, A reference means that obtains related model information indicating machine learning models related to the acquired model information by referring to a database that shows the relationships between multiple machine learning models, A generation means for generating display control information for displaying a screen containing information on a recommended machine learning model based on the aforementioned related model information, An information processing device equipped with the following features.

2. The aforementioned reference means refers to the database based on relevant information corresponding to the degree of relevance between machine learning models. The information processing apparatus according to claim 1.

3. The aforementioned reference means refers to the database based on relevant information corresponding to the ancestral relationships of the machine learning model. The information processing apparatus according to feature 1.

4. The reference means refers to the database based on the relevant information corresponding to at least one of the parent-child and sibling relationships of the machine learning model, as information about the ancestors. The information processing apparatus according to claim 3.

5. The aforementioned reference means refers to the database based on the related information corresponding to the number of learning repetitions in the parent-child relationship. The information processing apparatus according to feature 4.

6. The aforementioned reference means refers to the database based on the related information corresponding to the number of generations between machine learning models in the sibling relationship. The information processing apparatus according to feature 4.

7. The reference means refers to the database based on relevant information depending on whether at least a portion of the training datasets used to train the machine learning model are the same. The information processing apparatus according to feature 1.

8. The aforementioned reference means refers to the database based on the relevant information corresponding to the ratio used in the same training dataset. The information processing apparatus according to feature 7.

9. The aforementioned reference means refers to the database which contains relevant information corresponding to the information set by the creator of the machine learning model. The information processing apparatus according to feature 1.

10. The reference means refers to the database based on the related information, which corresponds to at least one of the information set by the creator as to whether or not there is a relationship between machine learning models, and the information as to whether or not the creator has met predetermined conditions. The information processing apparatus according to feature 9.

11. The acquisition means acquires model information of the machine learning model selected by the user. The information processing apparatus according to feature 1.

12. The database includes a model management means for managing multiple machine learning models contained in the aforementioned database, The acquisition means acquires model information in accordance with the timing at which the machine learning model managed by the model management means is downloaded. The information processing apparatus according to feature 1.

13. The acquisition means acquires model information and failure information related to the failure of the machine learning model whose inference failed. The reference means acquires the related model information corresponding to the model information and the failure information. The information processing apparatus according to feature 1.

14. The database includes a model management means for managing multiple machine learning models contained in the aforementioned database, The generation means generates the display control information, including the display of the registered machine learning model, if the machine learning model newly registered in the database is a related machine learning model. The information processing apparatus according to feature 1.

15. It includes statistical information processing means for acquiring and processing statistical information obtained from the inference of machine learning models, The acquisition means acquires the statistical information along with the model information. The information processing apparatus according to feature 1.

16. The statistical information processing means selects a machine learning model with a high priority based on the number of inference operations. The aforementioned reference means references relevant information of the high-priority machine learning model. The information processing apparatus according to feature 15.

17. The statistical information processing means selects a machine learning model with a high priority based on the number of undetected cases in the inference. The aforementioned reference means references relevant information of the high-priority machine learning model. The information processing apparatus according to feature 15.

18. A database management means for managing the related model information is provided based on the degree of relevance, which indicates the degree of relationship between machine learning models. The information processing apparatus according to claim 1, characterized by being equipped with

19. The generation means determines the display order of the information of the related machine learning models based on the degree of relevance, which indicates the degree of relevance. The information processing apparatus according to feature 1.

20. The reference means obtains the relevant model information by referring to the database based on at least one of the detection rate, which is the ratio of the number of times the target was detected to the number of times the machine learning model performed inference, and the false positive rate, which is the ratio of the number of false positives. The information processing apparatus according to feature 1.

21. If the detection rate of the acquired machine learning model is low, the reference means acquires the related model information of the machine learning model with a high detection rate among the related machine learning models. The information processing apparatus according to claim 20.

22. The reference means, if the false positive rate of the acquired machine learning model is high, acquires the related model information of a machine learning model with a low false positive rate among the related machine learning models. The information processing apparatus according to claim 20.

23. If the reference means indicates that the inference failed, and the failure information indicates that the subject to be photographed has not been detected, it retrieves the relevant model information of the relevant machine learning model from the database based on the detection rate. The information processing apparatus according to claim 20.

24. If the reference means obtains failure information indicating that the inference failed, which indicates that a different subject was detected, it obtains the relevant model information of the relevant machine learning model from the database based on the detection rate and the false positive rate. The information processing apparatus according to claim 20.

25. The reference means obtains the relevant model information of the relevant machine learning model by referring to the database which includes the usage period of the machine learning model, The generation means generates the display control information based on the usage period. The information processing apparatus according to feature 1.

26. We obtain model information, which is information about the machine learning model. By referring to a database that shows the relationships between multiple machine learning models, related model information is obtained that shows the machine learning models related to the acquired model information. Based on the aforementioned related model information, display control information is generated for displaying a screen containing information on the recommended machine learning model. An information processing method characterized by the following:

27. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 25.

Images