Information processing device, system, and information processing method

Abstract

To provide a technique for enabling high-quality supervised learning.SOLUTION: An information processing device is configured to acquire a dataset shared among users and instruct supervised learning based on the acquired dataset. Attribute information of the dataset includes the number of sets of data included in the dataset and an evaluation index representing evaluation of the dataset by users.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to learning techniques.

Background Art

[0002] In recent years, the utilization of AI (Artificial Intelligence) has been promoted in various fields. Among them, there is supervised learning in which machine learning is performed based on teacher data including correct data to generate an inference model.

[0003] In supervised learning, in order to obtain a machine learning model with high generalization performance, teacher data consisting of various inputs determined by the task to be solved and annotation data with high-quality GT (Ground Truth) assigned thereto is required. Generally, high-quality teacher data uses a publicly available dataset created for the purpose of a competition or the like, or if there is no teacher data suitable for the purpose, it is necessary to create a dataset by oneself. Thus, the creation of teacher data in supervised learning requires specialized knowledge and is time-consuming.

[0004] Patent Document 1 discloses a method of presenting teacher data that is compatible with the sensor device used by the purchaser of the teacher data in a marketplace where teacher data used for machine learning can be bought and sold.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, there was a problem that teacher data that is compatible with the sensor device does not necessarily have high annotation quality. The present invention provides a technique for realizing high-quality supervised learning.

Means for Solving the Problem

[0007] One aspect of the present invention includes an acquisition means for acquiring a dataset shared among users, and an instruction means for instructing supervised learning based on the dataset acquired by the acquisition means, wherein the attribute information of the dataset includes the number of data included in the dataset and an evaluation index representing the user's evaluation of the dataset.

Effect of the Invention

[0008] According to the present invention, it is possible to provide a technique for realizing high-quality supervised learning.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [First embodiment] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the system according to this embodiment includes a web server 103, a storage server 104, a learning server 105, and a client terminal 102.

[0012] The web server 103, storage server 104, and learning server 105 are connected to the Internet 100, and the client terminal 102 is connected to a local network 101. The local network 101 is also connected to the Internet 100 via devices not shown.

[0013] The client terminal 102 is a terminal operated by a user for supervised learning. For the sake of explanation, the number of client terminals 102 is one in FIG. 1, but in reality, a plurality of client terminals 102 are connected to the local network 101 or the Internet 100. The client terminal 102 may be, for example, a personal computer (PC), a tablet terminal, a smartphone, or other device.

[0014] The WEB server 103 functions as a portal site for a machine learning system that performs processes such as creation of annotations, upload of datasets, management of supervised learning, provision of an SNS (Social Networking Service) function, and the like.

[0015] The storage server 104 stores the datasets created for supervised learning. Attribute information of the dataset is attached to the dataset. The content of the dataset varies depending on the task to be inferred. For example, a dataset for an object detection task includes one or more images and the coordinate positions of the objects to be detected in the images.

[0016] The learning server 105 performs execution of supervised learning, creation of a report on the test results of the learned model (inference model) generated by the supervised learning, storage of learning parameters for the supervised learning, deployment of the inference model, and the like.

[0017] Next, a hardware configuration example of a computer device applicable to the WEB server 103, the storage server 104, the learning server 105, and the client terminal 102 will be described with reference to the block diagram of FIG. 2. Hereinafter, for simplicity of explanation, it will be described that the WEB server 103, the storage server 104, the learning server 105, and the client terminal 102 all have the hardware configuration shown in FIG. 2, but it is not limited thereto.

[0018] The CPU 200 is a central processing unit (Central Processing Unit), and executes various processes using computer programs and data stored in the RAM (Random Access Memory) 220. Thereby, the CPU 200 controls the operation of the entire computer device and executes or controls various processes described as the processes performed by the device to which this computer device is applied.

[0019] The ROM (Read-Only-Memory) 210 stores the setting data of the computer device, computer programs and data related to the startup of the computer device, computer programs and data related to the basic operations of the computer device, and so on.

[0020] The RAM 220 has an area for storing computer programs and data loaded from the ROM 210 or the HDD 230, and an area for storing computer programs and data received from an external device via the communication unit 260. Further, the RAM 220 has a work area used when the CPU 200 executes various processes. In this way, the RAM 220 can appropriately provide various areas.

[0021] The HDD 230 stores an OS (Operating System), computer programs and data for causing the CPU 200 to execute or control various processes described as processes performed by the device to which this computer device is applied, and so on. Note that an external storage device may be used instead of or in addition to the HDD 230. The external storage device can be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. As such a medium, for example, a flexible disk (FD), CD-ROM, DVD, USB memory, MO, flash memory, etc. are known. Also, the external storage device may be a server device or the like connected via a network.

[0022] The input unit 240 is a user interface such as a keyboard, mouse, touch panel screen, etc., and various instructions and information can be input to the computer device by the user's operation.

[0023] The display unit 250 has a liquid crystal screen or a touch panel screen, and can display the processing result by the CPU 200 as an image, characters, etc. Note that the display unit 250 may be a projection device such as a projector that projects an image or characters.

[0024] When this computer device is applied to the WEB server 103 and the learning server 105, the input unit 240 and the display unit 250 may be omitted. The communication unit 260 performs data communication with external devices via the Internet 100 or the local network 101.

[0025] The CPU 200, ROM 210, RAM 220, HDD 230, input unit 240, display unit 250, and communication unit 260 are all connected to a system bus 270. Note that the hardware configuration shown in Fig. 2 is merely an example and can be changed / modified as appropriate.

[0026] The block diagram of FIG. 3 shows an example of the functional configuration of each of the web server 103, the storage server 104, the learning server 105, and the client terminal 102. The following describes a case where the functional units shown in FIG. 3, except for the data storage units 302, 308, and 310, are implemented as software (computer programs). In this case, the data storage units 302, 308, and 310 are implemented as memory devices such as the ROM 210, the RAM 220, and the HDD 230. In the following, the functional units, except for the data storage units 302, 308, and 310, are sometimes described as the main processing units. However, in reality, the functions of the functional units are realized by the CPU 200 executing the computer programs corresponding to the functional units. Note that one or more of the functional units shown in FIG. 3, except for the data storage units 302, 308, and 310, may be implemented as hardware.

[0027] The process performed to learn and test an inference model through supervised learning in the system according to this embodiment will be described with reference to the flowchart in Fig. 4. As an example, the following describes a case where supervised learning is performed on an object detection task to detect a black toy poodle from an image.

[0028] 4 starts, the web browser 301 of the client terminal 102 displays the web screen (browser screen of the dataset SNS) illustrated in Fig. 5(a) on the display screen of the display unit 250. For example, the web browser 301 accesses the web server 103 to request the web screen, receives the web screen transmitted from the web server 103 in response to the request, and displays it on the display unit 250. Hereinafter, unless otherwise specified, it is assumed that the web browser 301 performs all display control of the web screen and processing of the web screen in response to user operations.

[0029] The search window 501 is a text box for inputting a query to search for an object that is the target of the object detection task, and the user can input a query into the search window 501 by operating the input unit 240 of the client terminal 102. In FIG. 5(a), "black toy poodle" is input into the search window 501 as the query.

[0030] The search button 502 is a button for issuing an instruction to execute a search. When the user operates the input unit 240 to instruct the search button 502, the web browser 301 transmits to the web server 103 an instruction to search for a dataset based on the query entered in the search window 501. In this embodiment, since "black toy poodle" is specified as the query, the web browser 301 transmits to the web server 103 an instruction to search for datasets to which labels including "black toy poodle" have been assigned.

[0031] When the web browser 301 receives a dataset group that is the result of a search in response to a search instruction from the web server 103, for each dataset in the dataset group, it generates a thumbnail 510 representing the dataset and displays it in the area 509. The dataset received from the web server 103 includes an image containing a black toy poodle and the coordinate position of "black toy poodle" in the image. The web browser 301 generates a thumbnail 510 by arranging each "image containing a black toy poodle" included in the dataset. At this time, the web browser 301 superimposes a "bounding box as GT for object detection" on the coordinate position of "black toy poodle" in the "image containing a black toy poodle".

[0032] The learning button 503 is a button for instructing the execution of supervised learning, and becomes valid when one or more datasets to be used for learning (learning datasets) and datasets to be used for testing (test datasets) are each selected. When the user operates the input unit 240 to instruct the learning button 503, the web browser 301 transmits a learning instruction including the identification information of the learning dataset and the identification information of the test dataset to the web server 103. The identification information of the learning dataset includes information for specifying each dataset added to the learning dataset. The identification information of the test dataset includes information for specifying the test dataset.

[0033] Radio button 504 is a radio button for specifying the "number of favorites registered by the user" as a condition for specifying the order of datasets to be displayed in area 509. Datasets with a large number of favorites are datasets that are popular with many users who are performing learning. Therefore, these datasets are likely to have high annotation quality, and preferentially searching and adopting such datasets makes it easier to improve the accuracy of the inference model. When the user operates input unit 240 to select radio button 504, web browser 301 sorts and arranges each of the generated thumbnails 510 by the "number of favorites registered by the user" and displays them in area 509.

[0034] Radio button 505 is a radio button for specifying the "number of users using the dataset" as a condition for specifying the order in which datasets are displayed in area 509. When the user operates input unit 240 to select radio button 505, web browser 301 sorts and arranges each of the generated thumbnails 510 by the "number of users using the dataset" and displays them in area 509.

[0035] Radio button 506 is a radio button for specifying the "number of projects using" as a condition for specifying the order of datasets to be displayed in area 509. When the user operates input unit 240 to select radio button 506, web browser 301 sorts and arranges each of the generated thumbnails 510 in area 509 by the "number of projects that use the dataset corresponding to that thumbnail 510."

[0036] The radio button 507 is a radio button for specifying "the number of deployed projects" as a condition for defining the order of the data sets to be displayed in the area 509. When the user operates the input unit 240 to indicate the radio button 507, the web browser 301 sorts and arranges the generated respective thumbnails 510 in the area 509 in the order of "the number of deployed projects among the projects using the data sets corresponding to the thumbnails 510".

[0037] In this way, the radio buttons 504 to 507 can indirectly estimate the annotation quality from the popularity of the data sets. Note that the web browser 301 controls so that only one of the radio buttons 504 to 507 can be selected. Also, the sorting conditions of the thumbnails 510 are not limited to the above conditions, and the method of specifying the sorting conditions is not limited to the above method. An annotation that increases at least any one of the accuracy rate (how correct the estimation is), recall rate (how many of the positive examples can be estimated as positive examples among the correct answers), and precision rate (how many of the estimated positive examples are correct answers) of the learned learning model can be said to have high quality. Annotations that appropriately represent an object (tag), annotations that appropriately surround the area of the object, correct annotations for objects that are difficult to distinguish (for example, long-haired Norwegian Forest Cats and Maine Coons), etc. However, it is not easy to distinguish whether an annotation has high quality or to collect annotations with high quality. Therefore, using the evaluation index described later, obtain annotations (teacher data that is considered to have high quality) that are considered to have high quality and enable learning.

[0038] The indicator 508 is an indicator that shows how much the data set selected in response to the user operation from the area 509 has been selected in learning and testing, respectively. The indicator 508 represents the ratio of positive and negative examples in the learning data set, the ratio of the test data set to the learning data set, etc. in different colors.

[0039] The scroll bar 511 is used to scroll the contents of the area 509, and also to visually represent the number of search hits in the dataset using the thickness of the scroll bar 511.

[0040] In step S401, when the user operates the input unit 240 to select the search button 502, the web browser 301 transmits to the web server 103 a search instruction for datasets (positive case datasets) that have been assigned labels containing the query "black toy poodle" entered in the search window 501 as a search word for the positive case datasets. The web browser 301 then receives from the web server 103 a group of positive case datasets that are the results of the search in response to the transmitted search instruction. The web browser 301 then generates thumbnails 510 corresponding to each positive case dataset in the received group of positive case datasets and displays them in the area 509.

[0041] In step S402, the web browser 301 adds to the training dataset the positive case dataset corresponding to the thumbnail 510 selected by the user through the input unit 240 from the group of thumbnails 510 displayed in the area 509.

[0042] Here, when radio button 504 is selected and the user operates the mouse as input unit 240 to place the mouse cursor over thumbnail 510, web browser 301 displays the number of images contained in the data set corresponding to thumbnail 510 and the number of favorites for that data set, as shown in Figure 5(b).

[0043] Furthermore, when radio button 505 is selected and the user operates the mouse as input unit 240 to place the mouse cursor over thumbnail 510, web browser 301 displays the number of images included in the data set corresponding to thumbnail 510 and the number of users using that data set.

[0044] Furthermore, when radio button 506 is selected and the user operates the mouse as input unit 240 to place the mouse cursor over thumbnail 510, web browser 301 displays the number of images contained in the data set corresponding to thumbnail 510 and the number of projects using the image.

[0045] Furthermore, when radio button 507 is selected and the user operates the mouse as input unit 240 to place the mouse cursor over thumbnail 510, web browser 301 displays the number of images contained in the data set corresponding to thumbnail 510 and the number of deployed projects.

[0046] In this embodiment, the attribute information of a dataset includes the number of images included in the dataset and the rating indexes of the dataset by various users (in the example of FIG. 5(b) , the number of favorites, the number of users, the number of projects used, and the number of projects). When a user operates the mouse serving as the input unit 240 to hover the mouse cursor over a thumbnail 510, the web browser 301 can display the "number of images included in the dataset" and the "rating indexes of the dataset by various users (the rating index corresponding to the selected radio button among the number of favorites, the number of users, the number of projects used, and the number of projects)" included in the attribute information of the dataset corresponding to the thumbnail 510. Therefore, the user can use such display as a reference to consider which dataset to add to the training dataset.

[0047] Note that the operation of hovering the mouse cursor over the thumbnail 510 is merely an example of an operation by the user to select the data set corresponding to the thumbnail 510 for reference, and the data set attribute information may be displayed in response to other user operations. Furthermore, the data set attribute information may be displayed at all times. Furthermore, the method of displaying the data set attribute information is not limited to a specific display method.

[0048] In step S403, the web browser 301 determines whether the user has operated the input unit 240 to input an instruction to end the process of adding the positive case data set to the training data set.

[0049] If the result of this determination is that the user has operated the input unit 240 to input an instruction to end the process of adding a positive example dataset to the training dataset, the process proceeds to step S404. On the other hand, if the user has not operated the input unit 240 to input an instruction to end the process of adding a positive example dataset to the training dataset, the process proceeds to step S402.

[0050] The user operates the input unit 240 to input a query in the search window 501 to search for a dataset (negative example dataset) for performing negative learning to prevent the detection of objects that the user wishes to exclude from detection targets or objects that are easily erroneously detected. In this embodiment, in order to learn a brown toy poodle as a negative example, the user operates the input unit 240 to input "brown toy poodle" in the search window 501 as a query for searching for the negative example dataset.

[0051] In step S404, when the user operates the input unit 240 to select the search button 502, the web browser 301 transmits to the web server 103 a search instruction for datasets (negative example datasets) that have been assigned labels including the query "brown toy poodle" entered in the search window 501 as a search word for negative example datasets. The web browser 301 then receives from the web server 103 a group of negative example datasets that are the results of the search in response to the transmitted search instruction. The web browser 301 then generates thumbnails (negative example thumbnails) corresponding to each negative example dataset in the received group of negative example datasets, and displays them in area 509.

[0052] In step S405, the web browser 301 adds the negative case dataset corresponding to the negative case thumbnail selected by the user operating the input unit 240 among the negative case thumbnail group displayed in the area 509 to the learning dataset.

[0053] An example of the display of the WEB screen after the completion of the process in step S405 is shown in FIG. 5(c). As shown in FIG. 5(c), in the WEB screen of FIG. 5(a), in the area where “Train” was displayed in the indicator 508, the number of images (the number on the left) included in the positive case dataset added to the learning dataset and the number of images (the number on the right) included in the negative case dataset added to the learning dataset are displayed in different display colors. Also, the ratio of the horizontal width of the display area of “the number of images included in the positive case dataset added to the learning dataset” to the horizontal width of the display area of “the number of images included in the negative case dataset added to the learning dataset” is, for example, the ratio of “the number of images included in the positive case dataset added to the learning dataset” to “the number of images included in the negative case dataset added to the learning dataset”.

[0054] In step S406, the web browser 301 determines whether or not an instruction to end the process of adding the negative case dataset to the learning dataset has been input by the user operating the input unit 240.

[0055] As a result of this determination, if the user has input an instruction to end the process of adding the negative case dataset to the learning dataset by operating the input unit 240, the process proceeds to step S407. On the other hand, if the user has not input an instruction to end the process of adding the negative case dataset to the learning dataset by operating the input unit 240, the process proceeds to step S405.

[0056] In step S407, when the user operates the input unit 240 to instruct the search button 502, the web browser 301 sends a search instruction for a data set (test data set) labeled with a query "Toy Poodle Black" entered in the search window 501 as a search word for the test data set to the WEB server 103. Then, the web browser 301 receives a group of test data sets that are the results of the search corresponding to the sent search instruction from the WEB server 103. Then, the web browser 301 generates thumbnails (test thumbnails) corresponding to the respective test data sets in the received group of test data sets and displays them in the area 509.

[0057] In step S408, the web browser 301 acquires, as a selected test data set, the test data set corresponding to the test thumbnail selected by the user operating the input unit 240 among the group of test thumbnails displayed in the area 509.

[0058] An example of the display of the WEB screen after the completion of the process in step S408 is shown in FIG. 5(d). As shown in FIG. 5(d), in the WEB screen of FIG. 5(a), in the area where "Test" is displayed in the indicator 508, a horizontally striped area corresponding to the number of images included in the selected test data set is colored and displayed, and the number ( "100") is also displayed. The ratio of the horizontal width of the display area of "the number of images included in the positive example data set / negative example data set added to the learning data set" to the horizontal width of the display area of "the number of images included in the selected test data set" is, for example, the ratio of "the number of images included in the positive example data set / negative example data set added to the learning data set" to "the number of images included in the selected test data set".

[0059] Then, when the addition of the dataset to the learning dataset and the acquisition of the selected test dataset are completed, as shown in FIG. 5(d), the learning button 503, which was in an invalid state (a state where the user cannot make a selection using the input unit 240) in FIG. 5(a), becomes an effective state (a state where the user can make a selection using the input unit 240).

[0060] In step S409, when the user operates the input unit 240 to instruct the learning button 503, the web browser 301 transmits a learning instruction including the identification information of the learning dataset and the identification information of the test dataset to the WEB server 103.

[0061] Here, the operation and configuration examples of each of the WEB server 103, the storage server 104, and the learning server 105 will be described. First, the WEB server 103 will be described. In the data storage unit 302, data managed on the WEB server 103 is stored (saved). The data stored in the data storage unit 302 includes, for example, registration information such as the user name and password of the user who uses this system, and SNS-related information such as follows and followers.

[0062] The SNS control unit 303 provides SNS functions such as following between users, direct messages, and favorite registration to datasets, and operates the data stored in the data storage unit 302 according to user interactions. The calculation unit 304 quantifies the popularity and quality of the dataset as an index for making it easier to find a desired dataset based on various calculation methods.

[0063] The learning control unit 305 is responsible for the management and control of supervised learning to achieve a desired task, and transmits a learning job, which is a job for causing the learning server 105 to execute learning and testing based on a learning instruction transmitted from the web browser 301, to the learning server 105. The learning job includes the identification information of the learning dataset used for learning and the identification information of the test dataset used for testing.

[0064] The management department 306 manages data sets, such as uploading the "data set to be uploaded" sent from the client terminal 102 to the storage server 104 or deleting the data set already uploaded to the storage server 104.

[0065] The data reception department 307 receives various types of information (for example, user interaction, data set to be uploaded to the storage server 104) sent from the web browser 301.

[0066] Next, the storage server 104 will be described. The data storage department 308 stores (saves) the data set uploaded by the management department 306. The data providing department 309 transmits, as a learning data set, the data set corresponding to the "identification information of the learning data set" included in the learning job received from the learning control department 305 among the data sets stored in the data storage department 308 to the learning server 105. Also, the data providing department 309 transmits, as a test data set, the data set corresponding to the "identification information of the test data set" included in the learning job received from the learning control department 305 among the data sets stored in the data storage department 308 to the learning server 105.

[0067] When the management department 306 receives a search instruction for a data set (positive example data set / negative example data set / test data set) from the web browser 301 via the data reception department 307, it performs a search according to the search instruction from the data sets stored in the data storage department 308. Then the data reception department 307 transmits the result of the search to the web browser 301.

[0068] In this way, the data storage unit 308 can register data sets uploaded from the client terminals 102 of various users, and can select a positive case data set / negative case data set / test data set from the registered data set group. That is, in the system according to the present embodiment, the data sets are shared among users, and a positive case data set / negative case data set / test data set can be selected from such data sets.

[0069] Next, the learning server 105 will be described. In the data storage unit 310, a definition file which is a file of information defining an inference model, data sets used for progress management of a learning project and learning / testing, hyperparameters set during learning, an inference model, etc. are stored for each project. That is, every time the learning server 105 receives a learning job, it stores the learning data set used for learning and the test data set used for testing in the data storage unit 310.

[0070] The generation unit 311 performs operations such as adjusting the individual appearance frequencies of the data sets received from the data providing unit 309 and preprocessing for input to the inference model. The generation unit 313 performs preprocessing for input to the inference model, but does not perform adjustment of the individual appearance frequencies of the data sets received from the data providing unit 309.

[0071] The learning unit 312 performs learning of an inference model (a selected model selected as a model suitable for the task) using the learning data set transmitted from the data providing unit 309 based on the learning job from the learning control unit 305.

[0072] For each test data set transmitted from the data providing unit 309, the test unit 314 inputs the test data set into the inference model to perform the arithmetic processing (inference processing) of the inference model, obtains "performance information indicating the performance of the inference model obtained in this project", such as the recall rate and precision rate, based on the specified test conditions, and notifies the user of the performance information. For example, the test unit 314 may display the performance information on the display unit 250 of the learning server 105 using images or characters, or may transmit the performance information to the web browser 301 and cause the web browser 301 to display the performance information. The deployment unit 315 places the "completed inference model" generated through the above learning and testing in an operation state that can be used in response to an HTTP request.

[0073] Therefore, in step S409, after the web browser 301 transmits a learning instruction to the WEB server 103, the inference model learning process is performed by the operations of the above WEB server 103, storage server 104, and learning server 105.

[0074] Then, in step S410, the above processing by the test unit 314 is performed to obtain performance information for each test data set. Then, the test unit 314 generates notification information such as performance information indicating the highest performance among the obtained performance information (highest performance information) and the data set corresponding to the highest performance information, and transmits the notification information to the web browser 301. The web browser 301 displays the notification information using images or characters. Note that the information included in the notification information is not limited to the above example.

[0075] In step S411, the web browser 301 determines whether the learning button 503 has been instructed again. For example, the user views the screen of the web browser 301 based on the notification information, determines whether re-learning is necessary, and if it is determined that re-learning is necessary, instructs the learning button 503.

[0076] When the learning button 503 is instructed again, the process proceeds to step S409. If the learning button 503 is not instructed again, the process according to the flowchart of FIG. 4 ends.

[0077] <Modification Example 1> The operation of the system when the radio button 506 is instructed will be described by taking the table in FIG. 6 as an example. The table in FIG. 6 is held and managed in the storage server 104.

[0078] FIG. 6(a) shows an example of a dataset management table for managing the dataset uploaded to the storage server 104. When a dataset is uploaded to the storage server 104, the storage server 104 issues a Dataset ID unique to the dataset. Then, the storage server 104 registers in the dataset management table by associating the Dataset ID issued for the dataset, the User ID (identification information) unique to the creator of the dataset, the number of images (Images) included in the dataset, the number of GTs in the image, and the number of favorite registrations (Fav) for the dataset.

[0079] FIG. 6(b) shows an example of a project management table. The storage server 104 issues a unique Project ID for each learning project of the user. Then, the storage server 104 registers in the project management table by associating the Project ID issued for the project, the User ID unique to the creator of the project, the learning status (Trained) of the inference model corresponding to the project, and the deployment status (Deployed) of the inference model. Trained = 0 indicates unlearned, and Trained = 1 indicates learned. Also, Deployed = 0 indicates undeployed, and Deployed = 1 indicates deployed.

[0080] Figure 6(c) shows an example of a reference dataset management table. The reference dataset management table is a table that manages datasets used for learning or testing for each project. For ease of viewing, Figure 6(c) shows a table that combines management tables for each project.

[0081] The Train / Test column indicates whether the dataset is used for training or testing: "Train" indicates that the dataset is used for training, and "Test" indicates that the dataset is used for testing.

[0082] The Pos / Neg column indicates whether the dataset is a positive-case dataset or a negative-case dataset. "Pos" indicates that the dataset is a positive-case dataset, and "Neg" indicates that the dataset is a negative-case dataset.

[0083] The number of projects using a dataset can be counted for each Dataset ID by filtering the reference dataset management table by Dataset ID. For example, in the example in Figure 6(c), dataset_0000 corresponds to three Project IDs (pi_0000, pi_0001, pi_0002), so the number of projects using the dataset for dataset_0000 is 3. The number of projects using a dataset can be extracted using the following SQL statement:

[0084] SELECT DatasetID,COUNT(*) FROM Reference Data Set Management Table GROUP BY DatasetID To get the popularity of the dataset used in the learning phase, simply count the results filtered by Dataset ID, Train, and Pos. This can be retrieved using the following SQL statement:

[0085] SELECT DatasetID,COUNT(*) FROM Reference Data Set Management Table WHERE Train / Test=Train AND Pos / Neg=Pos GROUP BY DatasetID If the radio button 507 to sort the dataset search results by the number of deployed projects is checked, you can further filter using the Deployed column in the project management table. This allows you to limit the results to learning projects that are actually being used in production environments, strongly suggesting the usefulness of the datasets that appear at the top of the search results. Also, if this model can be applied to edge devices, you can use the number of installations on edge devices as a substitute for the number of deployments.

[0086] In this way, by allowing users to search for the dataset they want to use and displaying search results that take popularity into account, the system supports effective and efficient dataset selection in the pre-learning stage, enabling effective learning to obtain an inference model that suits the purpose.

[0087] In the first embodiment, a search is performed from the datasets stored in the data storage unit 308 in response to the search instruction. However, for example, when selecting a positive example dataset, a search may be performed from a dataset corresponding to "Pos" in the Pos / Neg column or a dataset corresponding to "Train" in the Train / Test column in response to the search instruction. This is also true when searching for a negative example dataset. Furthermore, when selecting a test dataset, a search may be performed from a dataset corresponding to "Test" in the Train / Test column in response to the search instruction.

[0088] [Second embodiment] In each of the following embodiments including this embodiment, differences from the first embodiment will be described. Unless otherwise specified below, it is assumed that the hardware configuration and the like are the same as those in the first embodiment. In the system according to this embodiment, the processing performed to learn and test an inference model by supervised learning will be described according to the flowchart of FIG. 7. In the flowchart of FIG. 7, the same step numbers are assigned to the processing steps similar to those shown in FIG. 4, and the description of the processing steps will be omitted.

[0089] At the start of the processing according to the flowchart of FIG. 7, the web browser 301 of the client terminal 102 causes the display unit 250 to display a WEB screen (browser screen of the dataset SNS) exemplified in FIG. 8(a). For example, the web browser 301 accesses the WEB server 103 to request the WEB screen, receives the WEB screen received from the WEB server 103 in response to the request, and causes the display unit 250 to display it.

[0090] In step S701, the web browser 301 causes a menu related to the selected dataset selected by the user operating the input unit 240 from among the dataset group displayed in the area 509 to be displayed. For example, when the user operates the mouse as the input unit 240 and overlays the mouse cursor on the dataset (selected dataset), as shown in FIG. 8(a), the web browser 301 displays a menu 801 for selecting whether to add the selected dataset as a positive example dataset to the learning dataset or add the selected dataset as a negative example dataset to the learning dataset.

[0091] In step S702, when the user operates the mouse as the input unit 240 and selects "Add to Learning Dataset" and "Positive Example" in the menu 801, the web browser 301 adds the selected dataset to the learning dataset as a positive example dataset. On the other hand, when the user operates the mouse as the input unit 240 and selects "Add to Learning Dataset" and "Negative Example", the web browser 301 adds the selected dataset to the learning dataset as a negative example dataset.

[0092] In step S703, when the state of selecting "Add to Learning Dataset" and "Positive Example" in the menu 801 in step S702 continues for a specified time or more, the web browser 301 displays a menu 810 for selecting weights as shown in Fig. 8(b). The weights mentioned here are for adjusting the appearance probability between learning datasets. The larger the value, the higher the probability of appearing in the learning dataset, so the contribution rate to learning can be controlled for each dataset. The user can operate the input unit 240 to select a desired weight (the weight of the positive example dataset in step S703) from the menu 810.

[0093] In step S704, as shown in Fig. 8(c), the web browser 301 displays the menu 801 related to the selected dataset selected by the user operating the input unit 240 from among the dataset group displayed in the area 509.

[0094] In step S705, when the user operates the mouse as the input unit 240 and selects "Add to Learning Dataset" and "Negative Example" in the menu 801, the web browser 301 adds the selected dataset to the learning dataset as a negative example dataset.

[0095] In step S706, if the "Add to learning dataset" and "Negative examples" selections in menu 801 in step S705 continue for a specified time or longer, web browser 301 displays menu 810 for selecting weights, as shown in Fig. 8(d). The user can operate input unit 240 to select a desired weight from menu 810 (the weight of the negative example dataset in step S706).

[0096] Then, in this embodiment, in step S409, when the user operates the input unit 240 to select the learning button 503, the web browser 301 transmits a learning instruction including the identification information of the learning dataset, the identification information of the test dataset, and the dataset weight to the web server 103. The web server 103 transmits the "dataset weight" included in the learning instruction received from the web browser 301 to the storage server 104.

[0097] First, the storage server 104 calculates and registers the effective read probability of the data set for each project in advance. The effective read probability of the data set ds_j used in the project pj_i can be calculated according to the following formula (1).

[0098]

number

[0099] The denominator is a normalization term, and the summation represented by ds_k∈Dataset applies to the entire dataset used for training by project pj_i. For example, in the reference dataset management table of FIG. 9, the effective read probability of dataset ds_0000 of project pj_0001 is calculated as 1×100 / (1×100+9×5)≒0.69. The EffReadRate column in FIG. 9 registers the effective read probability calculated for each dataset for each project. Since the effective read probability is not calculated for datasets used in testing, it is set to Nan. Thus, in FIG. 9, the effective read probability is calculated by performing a weighted average of the positive and negative example datasets used for training. However, the effective read probability may be calculated only for the positive example dataset to narrow down the training dataset to the target task. The reference dataset management table of FIG. 9 is stored and managed by the storage server 104.

[0100] The storage server 104 then sums up the effective read probabilities calculated for each dataset for each project across all projects that use the dataset as training data, and calculates the sum as the evaluation index for the dataset. For example, the calculation unit 304 calculates the evaluation index for the dataset according to the following formula (2):

[0101]

number

[0102] The storage server 104 may then update the evaluation index included in the attribute information of the dataset to the evaluation index determined for that dataset, in which case thumbnails of the dataset sorted based on the evaluation index can be displayed in area 509.

[0103] Furthermore, the evaluation index of a dataset can be considered the expected value of the images included in the dataset being used for learning during one iteration when each project is trained equally. The learning server 105 performs supervised learning so that the higher the evaluation index of the dataset, the higher the probability that images included in the dataset will be used for supervised learning. This supports effective and efficient dataset selection in the pre-learning stage, enabling effective learning to obtain an inference model that meets the purpose.

[0104] [Third embodiment] In this embodiment, the storage server 104 calculates an evaluation index for a data set based on attributes associated with the identification information of a user associated with the data set. In this embodiment, the table in Fig. 10 will be used as an example. The table in Fig. 10 is held / managed by the storage server 104.

[0105] The dataset management table in Figure 10(a) is similar to the dataset management table in Figure 6(a). In the user account management table in Figure 10(b), followers represents the number of accounts followed by the user, and follows represents the number of accounts the user is following. Furthermore, dataset represents the number of datasets that the user has uploaded to the storage server 104. In such a case, the storage server 104 calculates the evaluation index of the dataset based on the following formula (3).

[0106]

number

[0107] For example, when calculating the evaluation index of the dataset with Dataset ID = ds_0000, since the number of follows of user_0000 corresponding to ds_0000 is "20" from the user account management table, the "number of user followers" is "20". Also, the GTs corresponding to user_0000 corresponding to ds_0000 are "100", "147", and "23" from the dataset management table. Therefore, the accumulation server 104 calculates 100 + 147 + 23 with the scaling term as "1", and obtains "270", which is the result of this calculation (total number), as the "total number of GTs given to the user". Also, the number of dataset GTs, which are the GTs corresponding to ds_0000, is "100". Therefore, the accumulation server 104 calculates 20 x 270 x 100 from the above formula (3), and obtains "540,000", which is the result, as the "evaluation index". Note that the scaling term can be scaled with an arbitrary monotonically increasing function according to the expansion of the service.

[0108] This evaluation index regards users with a large number of followers and a high record of GT grants as annotators with high experience values, and the higher the number of GTs given by such users to a dataset, the higher the index.

[0109] And the accumulation server 104 may update the evaluation index included in the attribute information of the dataset to the evaluation index obtained for the dataset. In this case, the thumbnail of the dataset sorted based on this evaluation index can be displayed in area 509.

[0110] Also, the learning server 105 performs supervised learning so that the images included in the dataset are adopted for supervised learning with a higher probability as the evaluation index of the dataset is higher. As a result, it is possible to selectively use datasets containing a large number of high-quality GTs by annotators with high experience values for learning, and the possibility of obtaining a highly accurate inference model becomes high.

[0111] In this embodiment, the evaluation index is calculated on the assumption that the dataset creator and the annotator are the same, but by linking a user to each GT, it is possible to weight the annotator for each GT. Even if the dataset is made up of multiple annotators, the evaluation index can be used as an index to measure the annotation quality of the dataset.

[0112] 1 used in the above embodiments uses three server devices: the web server 103, the storage server 104, and the learning server 105. However, the functions of two or more of these server devices may be implemented in a single server device. Also, at least one of the server devices: the web server 103, the storage server 104, and the learning server 105 may be implemented in two or more server devices, each with its own function.

[0113] Furthermore, in each of the above embodiments, a case has been described in which the dataset includes one or more images and the coordinate position in the image of the object to be detected. However, the configuration of the dataset is not limited to this configuration, and a dataset including an image and the position of a GT in the image may also be used. Furthermore, a dataset including data other than images (e.g., audio data) and the position of a GT in the image (e.g., the position / section of a GT in the audio data) may also be used. In such a case, the attribute information of the dataset will include the number of data included in the dataset and an evaluation index representing the user's evaluation of the dataset.

[0114] The numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in the above-mentioned embodiments and variant examples are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0115] Further, some or all of the embodiments and modifications described above may be used in appropriate combination. Also, some or all of the embodiments and modifications described above may be selectively used.

[0116] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in the computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0117] The invention described in this specification includes the following information processing apparatus, system, information processing method, and computer program. (Item 1) An acquisition means for acquiring a data set shared among users, An instruction means for instructing supervised learning based on the data set acquired by the acquisition means and comprising wherein the attribute information of the data set includes the number of data included in the data set and an evaluation index representing a user's evaluation of the data set. An information processing apparatus characterized by the above. (Item 2) The acquisition means acquires a positive example data set retrieved as a positive example data set from a group of data sets shared among users and a negative example data set retrieved as a negative example data set from the group of data sets, the information processing apparatus according to Item 1. (Item 3) The instruction means displays the positive example data set on a display screen, adds a first data set selected according to a user operation from the positive example data set to a learning data set, displays the negative example data set on the display screen, and adds a second data set selected according to a user operation from the negative example data set to the learning data set. The information processing apparatus according to item 2, characterized by instructing execution of supervised learning based on the learning dataset. (Item 4) The information processing apparatus according to item 3, characterized in that the instruction means causes the attribute information of the dataset to be displayed on the display screen in response to a user operation on the dataset displayed on the display screen. (Item 5) The information processing apparatus according to item 1, characterized in that the dataset includes an image and GT in the image. (Item 6) An information processing apparatus according to item 1, and a server device, a system comprising: wherein the server device a first processing means for obtaining a read probability of the dataset based on the number of data included in the dataset used in the project and the weight input in response to a user operation in the information processing apparatus for the dataset; a second processing means for obtaining an evaluation index representing a user's evaluation of the dataset based on the sum of the read probabilities obtained for each project for the dataset; a third processing means for executing supervised learning based on the instruction by the instruction means and the evaluation index; and characterized by comprising. (Item 7) An information processing apparatus according to item 1, and a server device, a system comprising: wherein the server device a first processing means for obtaining an evaluation index representing a user's evaluation of the dataset based on an attribute associated with identification information of a user associated with the dataset; a second processing means for executing supervised learning based on the instruction by the instruction means and the evaluation index; and characterized by comprising. (Item 8) The system according to item 7, wherein the first processing means obtains the evaluation index based on the number of GTs included in the dataset, the total number of GTs included in each dataset associated with the identification information, and the number of accounts followed by the user. (Item 9) An information processing method performed by an information processing apparatus, an acquisition step in which an acquisition means of the information processing apparatus acquires a dataset shared among users; an instruction step in which an instruction means of the information processing apparatus instructs supervised learning based on the dataset acquired in the acquisition step; and comprising: wherein the attribute information of the dataset includes the number of data included in the dataset and an evaluation index representing a user's evaluation of the dataset. An information processing method characterized by the above. (Item 10) A computer program for causing a computer to function as each means of the information processing apparatus according to any one of items 1 to 5.

[0118] The invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the claims are attached to disclose the scope of the invention.

Description of Reference Numerals

[0119] 100: Internet 101: Local network 102: Client terminal 103: Web server 104: Storage server 105: Learning server

Claims

1. An acquisition means for acquiring a dataset shared among users, An instruction means for instructing supervised learning based on the dataset acquired by the acquisition means and comprising The attribute information of the dataset includes the number of data included in the dataset and an evaluation index representing the user's evaluation of the dataset. An information processing apparatus characterized by the above.

2. The acquisition means acquires a positive example dataset retrieved as a positive example dataset from a group of datasets shared among users and a negative example dataset retrieved as a negative example dataset from the group of datasets. The information processing apparatus according to claim 1, characterized by this.

3. The instruction means displays the positive example dataset on a display screen, adds a first dataset selected according to a user operation from the positive example dataset to a learning dataset, displays the negative example dataset on the display screen, adds a second dataset selected according to a user operation from the negative example dataset to the learning dataset, and instructs execution of supervised learning based on the learning dataset. The information processing apparatus according to claim 2, characterized by this.

4. The instruction means displays the attribute information of the dataset on the display screen according to a user operation on the dataset displayed on the display screen. The information processing apparatus according to claim 3, characterized by this.

5. The dataset includes an image and GT in the image. The information processing apparatus according to claim 1, characterized by this.

6. A system having the information processing apparatus according to claim 1 and a server device, wherein the server device has a first processing means for obtaining a read probability of the dataset based on the number of data included in the dataset used in a project and the weight input according to a user operation in the information processing apparatus for the dataset, a second processing means for obtaining an evaluation index representing the user's evaluation of the dataset based on the sum of the read probabilities obtained for each project for the dataset, and a third processing means for executing supervised learning based on the instruction by the instruction means and the evaluation index. A system characterized by the above.

7. A system having the information processing apparatus according to claim 1 and a server device, wherein the server device First processing means for obtaining an evaluation index representing a user's evaluation of the dataset based on attributes associated with the identification information of the user associated with the dataset; Second processing means for performing supervised learning based on the instruction by the instruction means and the evaluation index; A system characterized by comprising the above. **Claim 8** The system according to claim 7, wherein the first processing means obtains the evaluation index based on the number of GTs included in the dataset, the total number of GTs included in each dataset associated with the identification information, and the number of accounts followed by the user. **Claim 9** An information processing method performed by an information processing apparatus, comprising: An acquisition step in which an acquisition means of the information processing apparatus acquires a dataset shared among users; An instruction step in which an instruction means of the information processing apparatus instructs supervised learning based on the dataset acquired in the acquisition step; The information processing method is characterized by comprising: The attribute information of the dataset includes the number of data included in the dataset and an evaluation index representing a user's evaluation of the dataset. **Claim 10** A computer program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 5. ​

Images