Information processing apparatus and method for controlling the same

Abstract

To acquire a learning model and learning data suitable for a user's purpose.SOLUTION: An information processing apparatus comprises: model management means that manages information on a second learning model used for generation of a first learning model and information on a first dataset used for generation of the first learning model, in association with the first learning model; data management means that manages information on a second dataset used for generation of the first dataset in association with the first dataset; detection means that detects that a third learning model generated through re-learning of the second learning model is added to targets to be managed by the model management means, or that a third dataset generated by a data operation for the second dataset is added to targets to be managed by the data management means; setting means that sets a notification condition for notifying that the third learning model or the third dataset is added; and notification means that performs notification when the notification condition set by the setting means is satisfied.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a management technique for a learning model and learning data. [Background technology]

[0002] In recent years, systems using machine learning technology have been put into practical use in various fields. Traditionally, the learning models and learning data used in such systems were often created by companies or universities. However, in recent years, it has become possible for general users to create learning models and learning data tailored to their own purposes. As a result, a huge number of learning models and learning data exist.

[0003] Additionally, it is anticipated that there will be Internet-based services that allow general users to publish and obtain training models and training data they have created. Users of the service (for example, users who want to create customized models) will need to select training models and training data that are suitable for their own purposes from the publicly available training models and training data. However, if even general users are able to create and publish training models, a huge number of training models and training data will exist, and they will be updated daily. This will make it difficult for users of the service to grasp the status of training models and training data.

[0004] Patent Document 1 discloses a method for searching and locating target training data based on tags attached to the training data. Patent Document 2 discloses a method for outputting new inference results and notifying the user of the results when a saved training model is updated. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-178892 [Patent Document 2] Japanese Patent Publication No. 2022-61191 Summary of the Invention [Problem to be solved by the invention]

[0006] However, it is difficult to grasp the update status of the learning data with the technology described in Patent Document 1. Furthermore, with the technology described in Patent Document 2, it becomes necessary to store the learning model and the inference results in a database every time the learning model is updated, which increases operational costs.

[0007] The present invention has been made in consideration of such problems, and aims to provide a technology that makes it easy to obtain learning models and learning data that are suitable for the user's purposes. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, an information processing device according to the present invention has the following arrangement. a model management means for managing information on a second learning model used to create a first learning model and information on a first dataset used to create the first learning model in association with the first learning model; a data management means for managing information on a second data set used in creating the first data set in association with the first data set; A detection means for detecting that a third learning model created by relearning the second learning model has been added to the objects to be managed by the model management means, or that a third dataset created by data manipulation of the second dataset has been added to the objects to be managed by the data management means; A setting means for setting notification conditions for notifying that the third learning model or the third dataset has been added; a notification means for issuing a notification when the notification condition set by the setting means is satisfied; Equipped with. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technology that makes it easy to obtain learning models and learning data that are suitable for the user's purposes. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating a hardware configuration of a server device. [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of a user device. [Figure 3] FIG. 1 is a diagram illustrating the overall configuration of a system. [Figure 4] FIG. 2 is a diagram illustrating a functional configuration of a server device. [Figure 5] 10A and 10B are diagrams illustrating examples of data management information and model management information. [Figure 6] FIG. 10 is a diagram illustrating an example of traceability information. [Figure 7] FIG. 10 is a diagram illustrating an example of notification information. [Figure 8] 10 is a flowchart of model data registration and notification settings. [Figure 9] 10 is a flowchart of a notification process. [Figure 10] FIG. 10 is a diagram illustrating a functional configuration of a server device (second embodiment). [Figure 11] 10 is a flowchart of model data update and evaluation result notification. [Figure 12] FIG. 10 is a diagram illustrating an example of traceability information (second embodiment). [Figure 13] FIG. 10 is a diagram illustrating an example of a notification setting GUI. [Figure 14] FIG. 10 is a diagram illustrating a functional configuration of a server device (third embodiment). [Figure 15] 10 is a flowchart of model data update and re-learning result notification. [Figure 16] FIG. 10 is a diagram illustrating an example of traceability information (third embodiment). DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] (First embodiment) As a first embodiment of an information processing device according to the present invention, a server device that manages learning models and learning data will be described below as an example.

[0013] <Summary> The server device of this embodiment manages models (trained models) and datasets together with traceability information (past update history of the models and datasets). When an update is made to a managed model or dataset, the server device notifies the target user of the update based on pre-registered notification settings.

[0014] <System and device hardware configuration> FIG. 3 is a diagram showing the overall configuration of the system. The server device 10 is an information processing device that operates as a server device that manages learning models and learning data. For example, it is deployed as a virtual server on a cloud server. The user devices 10 to 13 are information processing devices that operate as user devices that acquire and use learning models and learning data from the server device. As shown in FIG. 3, the virtual server is configured to be able to communicate with multiple user devices and to provide learning models and learning data to the user devices in response to requests from the user devices. The user devices are terminal devices that users view and operate on their screens, such as PCs (personal computers) and tablet terminals.

[0015] FIG. 1 is a diagram showing the hardware configuration of a server device. H101 is a CPU that controls various devices connected to a system bus H105. H102 is a ROM that stores a basic input / output system (BIOS) program and a boot program. H103 is a RAM that is used as the main memory device for the CPU H101. H104 is an interface (I / F) that communicates data with external devices. For example, it may be a communication interface such as Ethernet (registered trademark), or a general-purpose interface such as USB or serial communication. It may also be an interface for wired connection or a wireless connection.

[0016] Figure 2 shows the hardware configuration of the user device. H111 is a CPU that controls various devices connected to the system bus H119. H112 is a ROM that stores the BIOS program and boot program. H113 is an input device that performs processing related to the input of various information. Examples include a touch panel, keyboard, mouse, and robot controller. H114 is a display device that performs processing related to the display of various information. For example, it displays the results of processing performed by the user device itself or the results of processing performed by a server device and sent to the user device. The display device can be of any type, such as an LCD display device, projector, or LED indicator.

[0017] H115 is RAM, which is used as the main memory of the CPU H111. H116 is a hard disk, which is used to store application programs, data, libraries, etc. H117 is a media drive, which allows data to be written to and read from removable storage media, thereby enabling data to be transferred to external devices (digital still cameras, PCs, tablet devices). H118 is an interface (I / F), which communicates data with external devices. For example, it may be a communication interface such as Ethernet (registered trademark), or a general-purpose interface such as USB or serial communication. It may also be an interface for wired or wireless connection.

[0018] <Tasks processed by machine learning> As an example of a task handled in the first embodiment, an object detection task in which an image is input will be used for explanation. The object detection task is a task in which, when image data is input, a specific object in the image is detected and a bounding box (BB) surrounding the specific object is inferred. However, the type of task is not limited to object detection. For example, it can be applied to various tasks, such as a task of estimating and dividing an area, or a classification task of classifying subjects (people, cars, etc.).

[0019] <Server device functional configuration> 4 is a diagram showing the functional configuration of the server device 10. The server device 10 is made up of a control unit 100 and a storage unit 200. Each component will be described in detail below.

[0020] The control unit 100 includes a model management unit 101, a data management unit 102, a traceability information management unit 103, a registration unit 104, a notification setting unit 105, and a notification unit 106. Each of these functional units can be realized, for example, by a CPU executing various programs. However, some or all of these functional units may also be realized by hardware such as an application specific integrated circuit (ASIC).

[0021] The model management unit 101 generates management information for a model (trained model) and manages the management information by associating it with the model. Information such as that shown in FIG. 5(b) is generated and assigned to the managed model.

[0022] "Model ID" is the model's unique identification information (ID). "Creator User ID" is the ID of the user who trained the model. "Category" is information about the object that the model targets, and information necessary for users to search for models. "Task" is the type of processing that the model targets, such as object detection or image generation.

[0023] "Initial model" describes the ID of the model that served as the initial parameter when training the model. For example, as shown in FIG. 5(b), the initial model for model 262 is the model having model ID "md_0001", and the model having model ID "md_0001" is model 260. In other words, this indicates that model 262 is a model obtained as a result of (additional) training using model 260 as the initial model. "Training dataset" is the ID of the dataset used to train the model. "Evaluation data" is the ID of the data used when evaluating the model.

[0024] The data management unit 102 generates management information for a dataset used for learning and evaluation, and manages the management information by associating it with the dataset. Information such as that shown in Fig. 5(a) is generated and assigned to the managed dataset.

[0025] "Dataset ID" (hereafter referred to as "Data ID") is an ID unique to a dataset. "Number of images" is the number of images held by the dataset. "Number of GTs" is the number of GTs (Ground Truths) assigned to the dataset. "Category" is information about the objects targeted by the dataset and information necessary for users to search for data. "Task" is the type of processing targeted by the dataset, such as object detection or image generation.

[0026] "Subset" describes the ID of the dataset that was the basis for creating the dataset. For example, as shown in FIG. 5(a), the subset of dataset 213 is the dataset with data ID "ds_0002", and the model with data ID "ds_0002" is dataset 211. In other words, this indicates that dataset 213 is a dataset created based on dataset 211. "Creating user ID" is the ID of the user who created the dataset.

[0027] The traceability information management unit 103 manages traceability information of models and data sets. Here, traceability information is information indicating the process by which each model or data set was created in the past (i.e., the past history of the model or data set).

[0028] 6 is a diagram showing an example of traceability information. For example, it indicates that a model 601 having a model ID "md_0001" was created by learning using a dataset having a data ID "ds_0001." It indicates that a model 602 having a model ID "md_0002" was created by learning using a dataset having a data ID "ds_0001" and a dataset having a data ID "ds_0003," with the model having the model ID "md_0001" as the initial model.

[0029] This traceability information is created based on the model management information and data management information described above. As described above, the traceability information indicates the past history of each model or data set. Therefore, by referencing the traceability information, it is possible to extract information about the models and data sets used to create each model or data set.

[0030] The registration unit 104 registers, updates, and deletes models and datasets. When the registration unit 104 registers, updates, or deletes a model or dataset, the traceability information management unit 103 updates the traceability information. That is, the traceability information management unit 103 updates the traceability information based on the model management information and data management information related to the model or dataset that has been registered, updated, or deleted.

[0031] The notification setting unit 105 sets notification information requested by the user.

[0032] Figure 7 is a diagram showing an example of notification information. "Notification ID" is an ID unique to the notification. "User ID" describes the ID of the user who set the notification information / the user to whom the notification information is to be sent. "Related model data" describes the model ID of the model related to the notification information or the data ID of the data set.

[0033] "Notification status" describes the situation in which notification information is sent (e.g., a change in data). For example, notification information 310 having notification ID "rep_0001" is notification information set by a user with user ID "user_1001," and indicates that notification will be sent when an additionally trained model is registered for model ID "md_0001." "Notification options" sets options included in the notification (e.g., detailed condition settings for the notification status).

[0034] The notification unit 106 generates notification content based on the traceability information and the notification information set by the notification setting unit 105, and notifies the user of the content. The procedure for generating notification content will be described in detail later with reference to FIG.

[0035] <Device Operation> FIG. 8 is a flowchart for registering models and data and setting notifications. Here, it is assumed that a user collects dog images and creates and registers a dog detection model. However, server device 10 does not necessarily have to perform all of the steps described in this flowchart. Furthermore, the images are not limited to dogs; they may also be people or cars that are the detection target, and the detection model is not limited to dog detection. The flowchart in FIG. 8 starts when a user (user 1) operates user device 10 to register (send) the dog detection model and dog image data they created to server device 10.

[0036] In S1001, the model management unit 101 generates model management information for the dog detection model sent by the user. Here, model management information for model 260 shown in FIG. 5(b) is created as the model management information for the dog detection model. The model management information is assigned a model ID "md_0001", and the creating user ID "user_1001", category "dog", and task "object detection" are registered. No initial model is registered, and the data ID "ds_0001" of the dataset registered together with the model is entered in the training dataset. Also, here, the data ID "ds_1001" is entered as the dataset used as evaluation data.

[0037] In S1002, the data management unit 102 generates data management information for the dataset of dog images sent by the user. Here, data management information for dataset 210 shown in FIG. 5(a) is created as the data management information for the dog image data. The data management information is assigned a data ID "ds_0001", and the number of images "1000", the number of GTs "1200", the category "dog", and the task "object detection" are registered. Here, it is assumed that the data included in the registered dataset was created / collected by user 1 himself. Therefore, since there is no data linked to the dataset, "none" or a blank is registered in the subset. Then, the user ID of user 1, "user_1001", is entered in the creating user ID.

[0038] In S1003, the traceability information management unit 103 generates traceability information. Here, the traceability information shown in model 601 and data 603 (corresponding to the dog detection model and dog image data) in Fig. 6 is generated. The generated traceability information indicates that model 601 with model ID "md_0001" was created using the data set with data ID "ds_0001" as training data.

[0039] In S1004, the registration unit 104 registers the dog detection model and dog image data sent by the user.

[0040] In S1005, notification setting unit 105 sets the notification information requested by the user. Here, notification information 310 in Fig. 7 is set as the notification setting set by user 1. In other words, assume a situation in which user 1 "wants to be notified when another user uses the dog detection model that user 1 has registered as an initial model and registers a new detection model."

[0041] Here, the notification ID "rep_0001" is assigned to the notification setting 310, the user ID is "user_1001," and the related model data is "md_0001." In addition, the notification status is specified as "additional learning."

[0042] The above processing procedure is used to register the learning model and learning data and to set notifications. Here, an example is shown in which the model and data are registered and the notification settings are set together, but it is also possible to register only the model, only the data, or only the notification settings.

[0043] FIG. 9 is a flowchart of the notification process. Specifically, this is a processing procedure for notifying a user when an update is made to a model or data stored in server device 10. Here, assume a situation in which user 2 creates an updated dog detection model and dog data using the dog detection model and dog data described in FIG. 8. More specifically, user 2 creates a data set (Chihuahua image data) including an image of his or her pet "Chihuahua," and performs additional learning (relearning) on ​​the dog detection model to create a "Chihuahua detection model." The flowchart in FIG. 9 starts when a user (user 2) operates user device 12 to register (send) the created Chihuahua detection model and Chihuahua image data to server device 10.

[0044] In S2001, the model management unit 101 generates model management information for the Chihuahua detection model transmitted by the user, and the data management unit 102 generates data management information for the Chihuahua image data transmitted by the user.

[0045] Here, model management information for model 262 shown in Fig. 5(b) is created as the model management information for the Chihuahua detection model. The model management information is assigned a model ID of "md_0003", the model ID of "md_0001" is entered as the initial model, and the data IDs of "ds_0001" and "ds_0003" are entered as the training dataset.

[0046] Furthermore, data management information for the dataset 212 shown in FIG. 5(a) is created as data management information for the Chihuahua image data. The data management information is assigned a data ID "ds_0003", and the number of images "300", the number of GTs "300", the category "dog", and the task "object detection" are registered. Here, it is assumed that the data included in the registered dataset was created / collected by user 2 himself. Therefore, since there is no data linked to the dataset, "none" or a blank is registered in the subset. Furthermore, the user ID of user 2, "user_1002", is entered in the creating user ID.

[0047] In S2002, the traceability information management unit 103 generates traceability information from the model management information and data management information generated in S2001. Here, the traceability information shown in the model 602 and data 604 (corresponding to the Chihuahua detection model and Chihuahua image data) in FIG. 6 is generated. The generated traceability information indicates that the model 602 with the model ID "md_0003" was created using the data sets with the data IDs "ds_0001" and "ds_0003" as training data. In other words, it is detected that the model 602 created by re-training the model 601 has been added to the management targets.

[0048] In S2003, the notification unit 106 generates a notification based on the traceability information generated in S2002 and the notification information (FIG. 7) set by the notification setting unit 105. The notification information 310 is set to notify user 1 with user ID "user_1001" when the model with model ID "md_0001" is additionally trained. The notification is generated because the above-mentioned Chihuahua detection model (model ID "md_0003") was created using the model with model ID "md_0001" as the initial model.

[0049] In S2004, the notification unit 106 notifies the user of the notification generated in S2003. In an example of the generated notification, the user 1 (e.g., the user device 11) is notified that a model with model ID "md_0003", which is a re-learning model using the model with model ID "md_0001" as the initial model, has been registered.

[0050] Through the above process, User 1 can easily know that the dog detection model (model ID "md_0001") that he created has been retrained by User 2, and a Chihuahua detection model (model ID "md_0003") has been created.

[0051] In the example of Figure 9 above, a notification is provided when the model is updated (retrained), but a notification can also be provided when the data is updated. For example, assume that User 2 wants to be notified when the dataset with the data ID "ds_0001" used to create the Chihuahua detection model is changed. Here, it is assumed that the user ID, model ID, related model data, notification status, and notification options shown in the notification information 312 of Figure 7 have been set. Here, the notification option is set to "update 1,000 or more images (of data)." Note that a setting to notify when a single image is changed is also possible, but it is recommended to set the notification frequency above a certain level to avoid excessive notification frequency.

[0052] Assume that user 1 has modified (cleansed) the GT of a dataset with data ID "ds_0001." When registering the cleansed data, user 1 enters a subset "ds_0001 (cleansed)" as in dataset 213 in FIG. 5(a) and registers that the dataset is based on the dataset with data ID "ds_0001." As a result, the notification setting unit 105 generates a notification based on the notification information 312 in FIG. 7 and the data management information of dataset 213 in FIG. 5(a). Specifically, user 2 is notified that a change has been made to the dataset with data ID "ds_0001."

[0053] Through the above process, User 2 can easily know that the dataset used to create the Chihuahua detection model has been changed. In addition, because User 2 knows that cleansing has been performed on the dataset, User 2 has the opportunity to retrain the model and improve its accuracy.

[0054] Other examples of notification situations include an increase in the number of images included in a dataset or the addition of a new task. For example, in the case of dog image data, a notification may be received when new dog image data is registered. Furthermore, notification content may include information about the inclusion relationship of data, such as "A dataset has been created that specifically selects images of kingfishers from bird image data."

[0055] As described above, according to the first embodiment, when an update is made to a model or dataset that the server device manages, the server device notifies the target user of the update based on the notification settings registered in advance. This allows the user to easily understand the learning model and learning data suitable for use and their update status. The user can properly understand that learning data that has the potential to improve the learning model has been created.

[0056] (Second embodiment) In the second embodiment, a notification operation will be described when a data set created by user 1 is edited by user 3 and the edited data set is registered by user 3. In particular, a form in which the edited data set is evaluated and notified to user 1 will be described.

[0057] In the second embodiment, it is assumed that the user 1 has registered and saved the bird detection model (model ID "md_0002") and bird image data (data ID "ds_0002") in the server device 10 in advance.

[0058] <Device configuration> 10 is a diagram showing the functional configuration of a server device in the second embodiment. The functional configuration is almost the same as that of the first embodiment (FIG. 4), except that it further includes an inference and evaluation unit 107. Note that the user device 13 is assumed to be a terminal operated by a user 3.

[0059] The inference and evaluation unit 107 is a functional unit that performs inference and evaluation based on the learning model and learning data specified by the notification setting unit 105. The inference and evaluation results are sent to the notification unit .

[0060] <Notification Settings Graphical User Interface (GUI)> 13(a) is a diagram showing an example of a notification setting GUI for learning data. Specifically, this is a GUI displayed on the display unit of user device 11, which allows user 1 to set notification conditions for bird image data (data ID "ds_0002").

[0061] Here, we assume that User 1 wants to consider whether or not updated training data based on bird image data that he or she created is useful for training his or her own model. Therefore, he or she sets up the system so that he or she will be notified when training data based on data with data ID "ds_0002" is registered.

[0062] Setting item 1301 is an item for setting the notification situation, and three items are shown here. The first is a setting for notifying when a subset of the target dataset is created. The second is a setting for notifying when a dataset is created in which changes or additions have been made to the GT of the target dataset. The third is a setting for notifying when a learning model using the target dataset is registered. In Figure 13(a), a setting is set for notifying when a subset of the target dataset is created (the first checkbox is selected and blacked out).

[0063] Setting item 1302 is used to set options to be included in the notification. Here, it is configured so that it can be set whether or not the evaluation results are included in the notification content. In FIG. 13(a), the setting to include the evaluation results in the notification content is made. Note that when performing an evaluation, it is necessary to set the target learning model and the evaluation data to be used for the evaluation, so it is configured so that the learning model and evaluation data can also be specified. In FIG. 13(a), the learning model specified is the bird detection model (model ID "md_0002"), which is the learning model used when learning the data with data ID "ds_0002". And, as the evaluation data, "changed data" created based on the data with data ID "ds_0002" is specified.

[0064] The notification setting GUI shown in Figure 13(a) is just an example, and the items are not limited to check boxes; other methods such as pull-down menus may be used. Furthermore, more setting items may be added (such as when data is divided or integrated). Alternatively, more general setting items may be configured.

[0065] <Device Operation> 11 is a flowchart of the model / data update and evaluation result notification. Specifically, this is a processing procedure for notifying User 1 when an update is made to a model or data created by User 1 and stored in the server device 10.

[0066] Here, it is assumed that user 3 wants to create a kingfisher detection model based on the bird detection model and bird learning data created by user 1. Therefore, it is assumed that user 3 uses user device 13 to extract only kingfisher images from bird image data (data ID "ds_0002") and create kingfisher image data. It is further assumed that user 3 performs additional learning using the kingfisher image data with the bird detection model (model ID "md_0002") as the initial model to create a kingfisher detection model. The flowchart in FIG. 11 starts when the user (user 3) operates user device 13 to register (send) the kingfisher detection model and kingfisher image data created to server device 10. Note that the detection model and learning data are not limited to birds but may also be people, cars, etc., and the learning model is not limited to a detection model.

[0067] In S3001, the model management unit 101 generates model management information for the kingfisher detection model sent from the user, and the data management unit 102 generates data management information for the kingfisher image data sent from the user.

[0068] Here, model management information for model 263 shown in Fig. 5(b) is created as the model management information for the kingfisher detection model. The model management information is assigned the model ID "md_0004", the model ID "md_0002" is entered as the initial model, and the data IDs "ds_0002" and "ds_0005" are entered as the training dataset.

[0069] Furthermore, data management information for the data set 214 shown in FIG. 5(a) is created as data management information for the kingfisher image data. The data management information is assigned a data ID of "ds_0005," and the number of images is "300," the number of GTs is "560," the category is "bird," and the task is "object detection." In this case, the image data included in the registered kingfisher image data is selected from the bird image data (data ID "ds_0002"), so "ds_0002" is entered in the subset. And the user ID of user 3, "user_1003," is entered in the creating user ID.

[0070] In S3002, the traceability information management unit 103 generates traceability information from the model management information and data management information generated in S3001. Here, the traceability information shown in the model 612 and data 613 (corresponding to the kingfisher detection model and kingfisher image data) in Fig. 12 is generated.

[0071] FIG. 12 is a diagram showing an example of traceability information in the second embodiment. Note that model 610 (model ID "md_0002") and data 611 (data ID "ds_0002") correspond to the bird detection model and bird image data created and registered by user 1. The generated traceability information indicates that model 612 with model ID "md_0004" was created using model 610 with model ID "md_0002" as the initial model. It also indicates that it was created using datasets with data IDs "ds_0002" and "ds_0005" as training data. It also indicates that the dataset with data ID "ds_0005" was created based on the dataset with data ID "ds_0002." In other words, it is detected that a dataset created by a data operation on the dataset with data ID "ds_0002" has been added to the management targets.

[0072] In S3003, the notification unit 106 determines whether evaluation processing by the inference / evaluation unit 107 is necessary, based on the traceability information generated in S3002 and the notification information (FIG. 7) set by the notification setting unit 105. The notification setting set by the user 1 described with reference to FIG. 13(a) corresponds to the notification setting 311 in FIG. 7. Here, since the data with data ID "ds_0005" was created based on the data with data ID "ds_0002", it is determined that the data matches the notification status indicated in the notification setting 311. Furthermore, since "evaluation" is set in the notification options indicated in the notification setting 311, it is determined that evaluation is necessary, and the process proceeds to S3004. Note that if it is determined that evaluation is not necessary, for example, if "evaluation" is not set in the notification options, the process proceeds to S3005.

[0073] In S3004, the inference and evaluation unit 107 executes inference and evaluation using the model with model ID "md_0002" and the data with data ID "ds_0005." The model and data used here are specified in the setting item 1302 in FIG. 13(a). The inference and evaluation result is sent to the notification unit 106.

[0074] In S3005, the notification unit 106 generates a notification based on the notification information (FIG. 7) and including the evaluation result in S3004. In S3006, the notification unit 106 notifies the user 1 of the notification generated in S3005.

[0075] Through the above process, User 1 can easily know that kingfisher image data based on the bird detection model (model ID "md_0002") that he or she created has been created and registered. Furthermore, based on the evaluation results included in the notification, User 1 can obtain the inference and evaluation results based on his or her own bird detection model (model ID "md_0002") and the created kingfisher image data. For example, User 1 can easily know the accuracy of his or her bird detection model for the kingfisher image data (updated training data). This makes it possible to consider whether or not using updated training data is useful for training his or her own model.

[0076] As described above, according to the second embodiment, when an update is made to a model or dataset that the server device manages, the server device notifies the target user of the update along with the evaluation result based on the notification settings registered in advance. This allows the user to easily understand whether the derivatives of the learning model or learning data that he or she created are useful to him or her.

[0077] (Third embodiment) In the third embodiment, a notification operation will be described when an initial model used by user 2 when additionally learning a learning model is updated. In particular, a form will be described in which the updated initial model is re-learned and notified to user 2.

[0078] In the third embodiment, it is assumed that user 1 has previously registered and saved a dog detection model (model ID "md_0001") and dog image data (data ID "ds_0001") in server device 10. User 2 also performs re-learning using the dog detection model created by user 1 as the initial model, registers the re-learned model, and sets up notifications.

[0079] <Device configuration> 14 is a diagram showing the functional configuration of a server device in the third embodiment. The functional configuration is almost the same as that of the first embodiment (FIG. 4), except that it further includes an inference and evaluation unit 107 and a learning unit 108. The inference and evaluation unit 107 is the same as in the second embodiment.

[0080] The learning unit 108 is a functional unit that performs re-learning based on the learning model and learning data specified by the notification setting unit 105. The learning result is sent to the notification unit 106.

[0081] <Notification Settings Graphical User Interface (GUI)> 13(b) is a diagram showing an example of a notification setting GUI for a learning model. Specifically, this is a GUI displayed on the display unit of user device 12, which allows user 2 to set notification conditions for a dog detection model (model ID "md_0001"). Here, it is assumed that user 2 wants to re-learn the dog detection model (model ID "md_0001") used for re-learning (additional learning) again when the model is updated.

[0082] Setting item 1303 is an item for setting the notification status for the learning model, and two items are shown here. As mentioned above, the target here is the dog detection model (model ID "md_0001"). The first setting is a setting that notifies when the target learning model is updated. The second setting is a setting that notifies when the target learning model is used as the initial model and a new model is registered. In Figure 13(b), a setting is made to notify when the target learning model is updated (the first checkbox is selected and blacked out).

[0083] Setting item 1304 is used to set options to be included in notifications. In this case, it is set whether or not to include learning results in the notification content. In Figure 13(b), the setting to include learning results in the notification content is selected. When performing learning, it is necessary to set the target learning model and the learning data to be used for learning. In Figure 13(b), the initial model specifies the "updated model" that is the notification target. Two types of learning data are specified, and their proportions are also specified. The proportions may be specified numerically, or in an intuitively understandable way such as a diagram, graph, or table, as in graph 1314.

[0084] A setting item 1305 sets whether or not the evaluation result is to be included in the notification content. This is the same as in the second embodiment, so a description thereof will be omitted.

[0085] The notification setting GUI in Figure 13(b) is just one example, and the items are not limited to check boxes; other methods such as pull-down menus may be used. Furthermore, the learning data ratio and other items may be configured to present selection options and accept user specification.

[0086] <Device Operation> 15 is a flowchart of the process of updating model data and notifying the results of re-learning. The flowchart in FIG. 15 starts when user 1 re-learns the dog detection model that he or she created, creates a new dog detection model, and registers (transmits) it to server device 10.

[0087] In S4001, the model management unit 101 generates model management information for the new dog detection model sent by the user, and the data management unit 102 generates data management information for the dog image data sent by the user.

[0088] Here, model management information for model 264 shown in Fig. 5(b) is created as the model management information for the dog detection model. The model management information is assigned a model ID of "md_0005", and the model ID of "md_0001" is entered as the initial model.

[0089] Furthermore, data management information for the dog image data is created for data set 213 shown in Figure 5(a). The data management information is assigned a data ID "ds_0004" and registered. Here, it is written that the registered dog image data (data ID "ds_0004") has been created based on data set "ds_0001".

[0090] In S4002, the traceability information management unit 103 generates traceability information from the model management information and data management information generated in S4001. Here, the traceability information shown in model 621 and data 623 (corresponding to the new dog detection model and dog image data) in Fig. 16 is generated.

[0091] FIG. 16 is a diagram showing an example of traceability information in the third embodiment. Note that model 601 (model ID "md_0001") and data 603 (data ID "ds_0001") correspond to a dog detection model and dog image data previously created and registered by user 1. The generated traceability information indicates that model 621 with model ID "md_0005" was created using model 601 with model ID "md_0001" as the initial model. It also indicates that it was created using the dataset with data ID "ds_0004" as training data. It also indicates that the dataset with data ID "ds_0004" was created based on the dataset with data ID "ds_0001."

[0092] In S4003, the notification unit 106 determines whether learning processing by the learning unit 108 is necessary, based on the traceability information generated in S4002 and the notification information (FIG. 7) set by the notification setting unit 105. The notification setting set by user 2 described with reference to FIG. 13(b) corresponds to the notification setting 313 in FIG. 7. Here, since a model with model ID "md_0005" has been created based on a model with model ID "md_0001", it is determined that this matches the notification status indicated in the notification setting 313. Furthermore, since "learning" is set in the notification options indicated in the notification setting 313, it is determined that learning is necessary, and the process proceeds to S4004. Note that if it is determined that learning is not necessary, for example, if "learning" is not set in the notification options, the process proceeds to S4005.

[0093] In S4004, the learning unit 108 executes the learning process a specified number of times using the model with model ID "md_0005" and the data with data IDs "ds_0001" (80%) and "ds_0003" (20%). The model, data, ratio, and number of learning iterations used here are specified in the setting item 1304 in FIG. 13(b). The learning results are sent to the notification unit 106.

[0094] In S4005, the notification unit 106 determines whether or not evaluation processing by the inference / evaluation unit 107 is necessary, based on the notification information (FIG. 7) set by the notification setting unit 105. The notification setting set by user 2 described with reference to FIG. 13(b) corresponds to the notification setting 313 in FIG. 7. Since "evaluation" is set in the notification options indicated in the notification setting 313, it is determined that evaluation is necessary, and the process proceeds to S4006. Note that if it is determined that evaluation is not necessary, for example, if "evaluation" is not set in the notification options, the process proceeds to S4007.

[0095] In S4006, the inference and evaluation unit 107 executes inference and evaluation using the "trained model" obtained by the training in S4004 and the data with the data ID "ds_1003." The model and data used here are specified in the setting item 1305 in FIG. 13(b). The inference and evaluation result is sent to the notification unit 106.

[0096] In S4007, the notification unit 106 generates a notification based on the notification information (FIG. 7) that includes the learning result from S4004 and the evaluation result from S4006. In S4008, the notification unit 106 notifies the user 2 of the notification generated in S4007.

[0097] Through the above process, User 2 can easily know that the learning model used as the initial model has been updated. Furthermore, User 2 can easily know what characteristics the updated model has based on the learning results and evaluation results included in the notification.

[0098] As described above, according to the third embodiment, when an update is made to a model or dataset that the server device manages, the server device notifies the target user of the update together with the evaluation result based on the notification setting registered in advance. This allows the user to easily understand what characteristics the learning model that the user previously created has acquired as a result of the update.

[0099] The disclosure of this specification includes the following information processing device, control method, and program. (Item 1) a model management means for managing information on a second learning model used to create a first learning model and information on a first dataset used to create the first learning model in association with the first learning model; a data management means for managing information on a second data set used in creating the first data set in association with the first data set; A detection means for detecting that a third learning model created by relearning the second learning model has been added to the objects to be managed by the model management means, or that a third dataset created by data manipulation of the second dataset has been added to the objects to be managed by the data management means; A setting means for setting notification conditions for notifying that the third learning model or the third dataset has been added; a notification means for issuing a notification when the notification condition set by the setting means is satisfied; An information processing device comprising: (Item 2) The first user who created the first learning model is different from the second user who created the second learning model, The notification means notifies the second user. 2. The information processing device according to item 1, (Item 3) The third dataset is a dataset created by performing at least one of the following data operations on the second dataset: adding data, deleting data, correcting data, adding a ground truth (GT), deleting a ground truth (GT), and correcting a ground truth (GT). 3. The information processing device according to item 1 or 2. (Item 4) The setting means includes a condition regarding the data operation as a first notification condition for notifying that the third data set has been added. 4. The information processing device according to any one of items 1 to 3. (Item 5) Further, the method includes: evaluating the learning model based on the designated learning model and the data set; The setting means can further set whether or not to perform evaluation by the evaluation means, When the setting means sets the evaluation to be performed, the notification means includes the evaluation result by the evaluation means in the notification. 5. The information processing device according to claim 1, wherein the information processing device is a computer. (Item 6) The setting means can further specify a learning model and a dataset to be used in the evaluation. 6. The information processing device according to claim 5, (Item 7) Further comprising a learning means for learning the learning model based on the designated learning model and the data set, The setting means can further set whether or not learning is to be performed by the learning means, When the setting means sets the learning to be performed, the notification means includes the learning result by the learning means in the notification. 7. The information processing device according to claim 5, wherein: (Item 8) The setting means can further specify a learning model and a data set to be used in the learning. 8. The information processing device according to claim 7, (Item 9) The setting means accepts settings relating to the notification conditions via a graphical user interface (GUI) displayed on a display unit of a terminal operated by a user. 9. The information processing device according to claim 1, wherein the information processing device is a computer. (Item 10) A control method for an information processing device, comprising: The information processing device includes: a model management means for managing information on a second learning model used to create a first learning model and information on a first dataset used to create the first learning model in association with the first learning model; a data management means for managing information on a second data set used in creating the first data set in association with the first data set; The control method comprises: a detection step of detecting that a third learning model created by relearning the second learning model has been added to the objects to be managed by the model management means, or that a third dataset created by data manipulation of the second dataset has been added to the objects to be managed by the data management means; a setting step of setting notification conditions for notifying that the third learning model or the third dataset has been added; a notification step of issuing a notification when the notification condition set in the setting step is satisfied; A control method comprising: (Item 11) Item 11. A program for causing a computer to execute the control method according to Item 10.

[0100] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0101] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0102] 10 Server device; 11-13 User device; 100 Control unit; 200 Storage unit; 101 Model management unit; 102 Data management unit; 103 Traceability information management unit; 104 Registration unit; 105 Notification setting unit; 106 Notification unit

Claims

1. a model management means for managing information on a second learning model used to create a first learning model and information on a first dataset used to create the first learning model in association with the first learning model; a data management means for managing information on a second data set used in creating the first data set in association with the first data set; A detection means for detecting that a third learning model created by relearning the second learning model has been added to the objects to be managed by the model management means, or that a third dataset created by data manipulation of the second dataset has been added to the objects to be managed by the data management means; a setting means for setting notification conditions for notifying that the third learning model or the third dataset has been added; a notification means for issuing a notification when the notification condition set by the setting means is satisfied; An information processing device comprising:

2. A first user who created the first learning model is different from a second user who created the second learning model, The notification means notifies the second user.

2. The information processing apparatus according to claim 1, wherein:

3. The third data set is a data set created by performing at least one of the following data operations on the second data set: adding data, deleting data, modifying data, adding a ground truth (GT), deleting a GT, and modifying a GT.

2. The information processing apparatus according to claim 1, wherein:

4. The setting means includes a condition regarding the data operation as a first notification condition for notifying that the third data set has been added.

2. The information processing apparatus according to claim 1, wherein:

5. Further, the method includes: evaluating the learning model based on the designated learning model and the data set; The setting means can further set whether or not to perform evaluation by the evaluation means, When the setting means sets the evaluation to be performed, the notification means includes the evaluation result by the evaluation means in the notification.

2. The information processing apparatus according to claim 1, wherein:

6. The setting means can further specify a learning model and a dataset to be used in the evaluation.

6. The information processing apparatus according to claim 5,

7. Further comprising a learning means for learning the learning model based on the designated learning model and the data set, The setting means can further set whether or not learning is to be performed by the learning means, When the setting means sets the learning to be performed, the notification means includes the learning result by the learning means in the notification.

6. The information processing apparatus according to claim 5,

8. The setting means can further specify a learning model and a data set to be used in the learning.

8. The information processing apparatus according to claim 7,

9. The setting means accepts settings relating to the notification conditions via a graphical user interface (GUI) displayed on a display unit of a terminal operated by a user.

2. The information processing apparatus according to claim 1, wherein:

10. A control method for an information processing device, comprising: The information processing device includes: a model management means for managing information on a second learning model used to create a first learning model and information on a first dataset used to create the first learning model in association with the first learning model; a data management means for managing information on a second data set used in creating the first data set in association with the first data set; The control method comprises: a detection step of detecting that a third learning model created by relearning the second learning model has been added to the objects to be managed by the model management means, or that a third dataset created by data manipulation of the second dataset has been added to the objects to be managed by the data management means; a setting step of setting notification conditions for notifying that the third learning model or the third dataset has been added; a notification step of issuing a notification when the notification condition set in the setting step is satisfied; A control method comprising:

11. A program for causing a computer to execute the control method according to claim 10.

Images