Label assigning device, label assigning method, and program

Abstract

To efficiently apply a label to data.SOLUTION: A label assignment device includes a learning unit that learns a feature extractor on the basis of target data to which a label is not assigned, a feature extraction unit that extracts feature data from the target data using the feature extractor, a clustering unit that classifies the feature data into a plurality of clusters, and a label assignment unit that assigns a label corresponding to a cluster to the target data corresponding to the feature data belonging to the cluster.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a labeling device, a labeling method, and a program. [Background technology]

[0002] Supervised learning, a type of machine learning technique, requires a large amount of training data with correct labels. Techniques aimed at efficiently creating training data have been proposed (see, for example, Patent Documents 1 to 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-9314 [Patent Document 2] Japanese Patent Application Publication No. 2023-180330 [Patent Document 3] Patent Publication No. 2021-128573 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional techniques leave room for improvement in the efficiency of creating training data. For example, the conventional techniques involve a heavy workload for assigning correct labels to training data.

[0005] One aspect of the present disclosure provides a technique for efficiently labeling data. [Means for solving the problem]

[0006] A labeling device according to one aspect of the present disclosure includes a learning unit that trains a feature extractor based on unlabeled target data, a feature extraction unit that extracts feature data from the target data using the feature extractor, a clustering unit that classifies the feature data into a plurality of clusters, and a labeling unit that assigns a label corresponding to the cluster to target data corresponding to feature data belonging to the cluster. [Effects of the Invention]

[0007] According to one aspect of the present disclosure, labels can be efficiently assigned to data. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of the overall configuration of a machine learning system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer. [Figure 3] 1 is a block diagram showing an example of a functional configuration of a labeling device according to a first embodiment. FIG. [Figure 4] FIG. 2 is a diagram illustrating an example of a labeling assistance tool according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a labeling assistance tool according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a folder configuration according to the first embodiment. [Figure 7] 4 is a flowchart showing an example of a labeling method according to the first embodiment. [Figure 8] FIG. 10 is a block diagram showing an example of the functional configuration of a labeling device according to a second embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a sub-cluster according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a labeling assistance tool according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a folder configuration according to the second embodiment. [Figure 12]10 is a flowchart showing an example of a labeling method according to the second embodiment. [Figure 13] FIG. 10 is a block diagram showing an example of the functional configuration of a labeling device according to a third embodiment. [Figure 14] 11 is a flowchart showing an example of a labeling method according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] [First embodiment] The first embodiment of the present disclosure is an example of an information processing system that generates a machine learning model. Hereinafter, the information processing system according to this embodiment will be referred to as a "machine learning system." The machine learning system has a function of supporting the creation of training data used for training the machine learning model. Specifically, the machine learning system includes a labeling device that assigns labels to unlabeled data.

[0011] In recent years, advances have been made in the automation and labor-saving of visual inspections and monitoring processes using machine learning technology, also known as artificial intelligence (AI). However, training a machine learning model requires the preparation of a large amount of training data, which places a heavy burden on the system. In particular, the task of assigning correct labels to each piece of training data is a heavy burden.

[0012] Techniques for efficiently creating training data have been proposed. For example, Patent Document 1 discloses a graphical user interface (GUI) for a tool that facilitates labeling work. Patent Document 2 discloses a technique for dividing an image into similar regions by clustering the color information of the pixels, and assigning a class to each region. Patent Document 3 discloses a technique for reducing the load of labeling work by using a machine learning model such as a convolutional neural network (CNN) to extract features of data to be labeled and clustering the data by appearance features based on the extracted features.

[0013] However, conventional techniques have room for further improvement in the efficiency of creating training data. For example, Patent Document 1 requires assigning labels to each piece of data containing a variety of appearance features, which does little to reduce the burden of labeling work. Patent Document 2 performs clustering based on pixel color information, which prevents labeling based on shape and limits the tasks to which it can be applied. Patent Document 3 classifies data by appearance feature before labeling, which can reduce the burden of labeling work to a certain extent. However, in Patent Document 3, to improve the accuracy of clustering, it is necessary to train a machine learning model in advance using the data to be labeled, which requires assigning labels to a portion of the data (e.g., approximately 100 to 1,000 images per class).

[0014] The present embodiment aims to efficiently assign labels to data by training a feature extractor based on unlabeled data, classifying feature data extracted from the data using the feature extractor into multiple clusters, and assigning labels corresponding to the clusters to data corresponding to feature data belonging to the clusters.

[0015] In one aspect, this embodiment eliminates the need to create training data for training a feature extractor, thereby enabling efficient labeling of data. In another aspect, this embodiment enables efficient creation of training data, thereby enabling efficient generation of machine learning models.

[0016] <Overall structure> The overall configuration of the machine learning system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the overall configuration of the machine learning system.

[0017] 1, the machine learning system 1000 includes a data generating device 10, a labeling device 20, and a model learning device 30. The data generating device 10, the labeling device 20, and the model learning device 30 are connected to each other so as to be able to communicate data with each other via a communication network N. The communication network N may be, for example, a network such as a LAN (Local Area Network), a VPN (Virtual Private Network), or the Internet.

[0018] The data generating device 10 is an example of a device that generates data to be labeled. For example, the data generating device 10 may be a visual inspection device or a monitoring device. For example, the visual inspection device may be a device that detects defects in an inspection object based on an image of the inspection object. For example, the monitoring device may be a device that detects abnormalities that occur in the monitoring object based on an image of the monitoring object.

[0019] Hereinafter, data to be labeled will be referred to as "target data." In this embodiment, the target data is image data. The image data may be, for example, a still image or a video. Note that the target data is assumed to have no labels attached to it.

[0020] The labeling device 20 is an example of an information processing device such as a personal computer, a workstation, or a server that assigns labels to target data. The labeling device 20 may assign labels to target data generated by the data generating device 10. The labeling device 20 may output the target data to which the labels have been assigned.

[0021] The model learning device 30 is an example of an information processing device such as a personal computer, workstation, or server that generates a machine learning model. The model learning device 30 may generate a machine learning model using the target data labeled by the labeling device 20 as training data. In this embodiment, the machine learning model may be a classification model that performs a classification task of classifying image data. The classification model may be, for example, a support vector machine or a deep neural network.

[0022] Note that the overall configuration of the machine learning system 1000 shown in FIG. 1 is one example, and various system configuration examples are possible depending on the application and purpose. For example, the machine learning system 1000 may include multiple units of one or more of the data generating device 10, the labeling device 20, and the model learning device 30. For example, the data generating device 10, the labeling device 20, and the model learning device 30 may be implemented by multiple computers, or may be implemented as a cloud computing service. The classification of devices such as the data generating device 10, the labeling device 20, and the model learning device 30 shown in FIG. 1 is one example.

[0023] <Hardware configuration> The hardware configuration of each device included in the machine learning system 1000 will be described with reference to Fig. 2. The data generating device 10, labeling device 20, and model learning device 30 included in the machine learning system 1000 are realized by, for example, a computer. Fig. 2 is a block diagram showing an example of the hardware configuration of a computer.

[0024] 2, the computer 500 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HDD (Hard Disk Drive) 504, an input device 505, a display device 506, a communication I / F (Interface) 507, and an external I / F 508. The CPU 501, the ROM 502, and the RAM 503 form a so-called computer. The hardware components of the computer 500 are connected to each other via a bus line 509. The input device 505 and the display device 506 may be connected to the external I / F 508 for use.

[0025] The CPU 501 is a computing device that reads programs and data from a storage device such as the ROM 502 or the HDD 504 onto the RAM 503 and executes the processes, thereby realizing the overall control and functions of the computer 500. The computer 500 may have a GPU (Graphics Processing Unit) in addition to or instead of the CPU 501.

[0026] The ROM 502 is an example of a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 502 functions as a main storage device that stores various programs, data, etc. required for the CPU 501 to execute various programs installed in the HDD 504. Specifically, the ROM 502 stores boot programs such as a Basic Input Output System (BIOS) and an Extensible Firmware Interface (EFI) that are executed when the computer 500 starts up, as well as data such as OS (Operating System) settings and network settings.

[0027] The RAM 503 is an example of a volatile semiconductor memory (storage device) in which programs and data are erased when the power is turned off. The RAM 503 is, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 503 provides a working area in which various programs installed in the HDD 504 are expanded when executed by the CPU 501.

[0028] The HDD 504 is an example of a non-volatile storage device that stores programs and data. The programs and data stored in the HDD 504 include an OS, which is basic software that controls the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use a storage device that uses flash memory as a storage medium (e.g., an SSD (Solid State Drive)) instead of the HDD 504.

[0029] The input device 505 includes a touch panel, operation keys and buttons, a keyboard and mouse, a microphone for inputting sound data such as voice, and the like, which are used by the user to input various signals.

[0030] The display device 506 is composed of a display such as a liquid crystal display or organic EL (Electro-Luminescence) display for displaying a screen, a speaker for outputting sound data such as voice, and the like.

[0031] The communication I / F 507 is an interface that connects to a communication network and enables the computer 500 to perform data communication.

[0032] The external I / F 508 is an interface with external devices, such as a drive device 510.

[0033] The drive device 510 is a device for loading a recording medium 511. The recording medium 511 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk. The recording medium 511 may also include semiconductor memories that record information electrically, such as ROMs and flash memories. This allows the computer 500 to read from and / or write to the recording medium 511 via the external I / F 508.

[0034] The various programs to be installed in the HDD 504 are installed, for example, by setting the distributed recording medium 511 in a drive device 510 connected to the external I / F 508 and reading out the various programs recorded on the recording medium 511 by the drive device 510. Alternatively, the various programs to be installed in the HDD 504 may be installed by being downloaded via the communication I / F 507 from a network different from the communication network.

[0035] <Functional configuration> The functional configuration of the machine learning system 1000 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of the labeling device according to the first embodiment.

[0036] 3, the labeling device 20 includes a target data storage unit 101, an extractor storage unit 102, a temporary label storage unit 103, a correct label storage unit 104, an acquisition unit 110, an unlabeled learning unit 120, a feature extraction unit 130, a dimension reduction unit 140, a clustering unit 150, a label assignment unit 160, a label correction unit 170, and a display unit 180. By executing a pre-installed label assignment program, the labeling device 20 functions as the target data storage unit 101, the extractor storage unit 102, the temporary label storage unit 103, the correct label storage unit 104, the acquisition unit 110, the unlabeled learning unit 120, the feature extraction unit 130, the dimension reduction unit 140, the clustering unit 150, the label assignment unit 160, the label correction unit 170, and the display unit 180. The unlabeled learning unit 120 is an example of a learning unit.

[0037] For example, the target data storage unit 101, the extractor storage unit 102, the temporary label storage unit 103, and the correct label storage unit 104 are realized by the HDD 504 shown in Fig. 2. For example, the acquisition unit 110, the unlabeled learning unit 120, the feature extraction unit 130, the dimension reduction unit 140, the clustering unit 150, the label assignment unit 160, the label correction unit 170, and the display unit 180 are realized by processing that is executed by the CPU 501 in accordance with a program loaded from the HDD 504 onto the RAM 503 shown in Fig. 2.

[0038] One or more pieces of unlabeled target data are stored in the target data storage unit 101. The target data stored in the target data storage unit 101 is generated by the data generating device 10 and acquired by the acquiring unit 110.

[0039] The extractor storage unit 102 stores a feature extractor that extracts feature data from target data. The feature extractor may be a deep neural network based on deep learning. For example, the feature extractor may be configured with a convolutional neural network or a Vision Transformer. The feature extractor stored in the extractor storage unit 102 is generated by the unlabeled learning unit 120.

[0040] The temporary label storage unit 103 stores target data to which a temporary label has been assigned. The temporary label is a label that is temporarily assigned to target data. The temporary label is assigned to the target data by the label assignment unit 160.

[0041] A folder corresponding to each temporary label may be generated in the temporary label storage unit 103. In this case, the target data is stored in each folder, and a temporary label is assigned to the target data. The temporary label storage unit 103 may store target data whose file name includes a temporary label. In this case, the temporary label included in the file name is assigned to the target data.

[0042] The target data to which a correct label has been assigned is stored in the correct label storage unit 104. The correct label is a label that the model learning device 30 uses as training data for the classification model. The correct label is assigned to the target data by the label correction unit 170.

[0043] The correct label storage unit 104 may generate folders corresponding to the respective correct labels. In this case, the target data is stored in each folder, and a correct label is assigned to the target data. The correct label storage unit 104 may store target data whose file name includes the correct label. In this case, the correct label included in the file name is assigned to the target data.

[0044] The acquiring unit 110 acquires one or more pieces of target data. The acquiring unit 110 may acquire the target data from the data generating device 10. The acquiring unit 110 may receive the target data transmitted by the data generating device 10. The acquiring unit 110 may read the target data stored in a storage device of the data generating device 10.

[0045] The acquiring unit 110 stores the acquired target data in the target data storage unit 101. The acquiring unit 110 may store the target data in the target data storage unit 101 by changing the file name, folder name, folder structure, or the like of the target data.

[0046] The unlabeled learning unit 120 trains a feature extractor based on unlabeled target data. The unlabeled learning unit 120 may train the feature extractor based on target data read from the target data storage unit 101. As an example, the unlabeled learning unit 120 may train the feature extractor by self-supervised learning. The unlabeled learning unit 120 may train the feature extractor by any method as long as it can learn features from unlabeled data. The feature extractor learns appearance features indicated in image data, thereby becoming able to extract feature data indicating appearance features from unlabeled image data. The unlabeled learning unit 120 stores the trained feature extractor in the extractor storage unit 102.

[0047] The feature extraction unit 130 extracts feature data from the target data. The feature extraction unit 130 may extract feature data from each piece of target data read out from the target data storage unit 101. The feature extraction unit 130 may extract feature data using a feature extractor read out from the extractor storage unit 102.

[0048] In this embodiment, the feature extraction unit 130 extracts n-dimensional feature data from L pieces of target data. Therefore, the number of dimensions obtained by integrating all the feature data is (L, n). Note that the number of dimensions n of the feature data depends on the model structure of the feature extractor.

[0049] The dimension reduction unit 140 reduces the dimensions of the feature data extracted by the feature extraction unit 130. Examples of the dimension reduction method include PCA (Principal Component Analysis), t-SNE (t-Distributed Stochastic Neighbor Embedding), and UMAP (Uniform Manifold Approximation and Projection).

[0050] The dimension reduction unit 140 converts n-dimensional feature data into lower-dimensional (e.g., two-dimensional or three-dimensional) feature data. In this embodiment, the dimension reduction unit 140 reduces the dimension of n-dimensional feature data into two-dimensional feature data. Therefore, the number of dimensions obtained by integrating all the feature data after dimension reduction is (L, 2).

[0051] The clustering unit 150 classifies the feature data whose dimensions have been reduced by the dimension reduction unit 140 into a plurality of clusters. Examples of the clustering method include k-means clustering, Gaussian Mixture Model, and Agglomerative Clustering. The clustering unit 150 may treat one set generated by the clustering as one cluster. The clustering unit 150 may divide one set into a plurality of clusters. The clustering unit 150 may combine a plurality of sets into a single cluster.

[0052] The labeling unit 160 assigns temporary labels to the target data based on the clustering result by the clustering unit 150. For each cluster included in the clustering result, the labeling unit 160 may assign a temporary label corresponding to the cluster to the target data corresponding to feature data belonging to the cluster. Note that the target data corresponding to feature data may be the target data input to the feature extractor to extract the feature data.

[0053] The labeling unit 160 stores the target data to which the temporary label has been assigned in the temporary label storage unit 103. The labeling unit 160 may assign a temporary label to the target data by storing the target data in a folder corresponding to the temporary label. The labeling unit 160 may assign a temporary label to the target data by adding the temporary label to the file name of the target data.

[0054] The label correction unit 170 corrects the temporary label assigned to the target data by the label assignment unit 160. The label correction unit 170 may display the target data and the temporary label assigned to the target data, and correct the temporary label assigned to the target data in response to a user operation.

[0055] The label correction unit 170 may correct the temporary label assigned to the target data by moving the target data from a folder corresponding to the pre-correction temporary label to a folder corresponding to the corrected temporary label in the temporary label storage unit 103. The label correction unit 170 may correct the temporary label assigned to the target data by changing the temporary label included in the file name of the target data stored in the temporary label storage unit 103.

[0056] When the label correction unit 170 completes the correction of the temporary label assigned to the target data, it assigns a correct label to the target data based on the corrected temporary label. The label correction unit 170 may assign the temporary label assigned by the label assignment unit 160 to the target data as a correct label without correcting it.

[0057] The label correction unit 170 stores the target data to which the correct label has been assigned in the correct label storage unit 104. The label correction unit 170 may assign the correct label to the target data by storing the target data in a folder corresponding to the correct label. The label correction unit 170 may assign the correct label to the target data by adding the correct label to the file name of the target data.

[0058] The display unit 180 displays a labeling support tool that serves as a user interface for the labeling device 20. The labeling support tool is a graphical user interface that has a screen that requests the labeling device 20 to perform labeling-related processing and displays the processing results by the labeling device 20.

[0059] The display unit 180 may display the labeling support tool on the display device 506 of the labeling device 20. The display unit 180 may transmit screen data for displaying the labeling support tool to a terminal device operated by a user. The terminal device may display the labeling support tool on the display device 506 based on the screen data received from the labeling device 20. The terminal device may be, for example, an information processing terminal such as a personal computer, a tablet terminal, or a smartphone.

[0060] The functional configuration of the labeling device 20 shown in Fig. 3 is an example, and it goes without saying that there are various examples of functional configurations depending on the application and purpose. The division of the storage unit, such as the target data storage unit 101, the extractor storage unit 102, the temporary label storage unit 103, and the correct label storage unit 104 shown in Fig. 3, is an example. The division of the processing unit, such as the acquisition unit 110, the unlabeled learning unit 120, the feature extraction unit 130, the dimensionality reduction unit 140, the clustering unit 150, the labeling unit 160, the label correction unit 170, and the display unit 180 shown in Fig. 3, is an example.

[0061] For example, at least two of the target data storage unit 101, the extractor storage unit 102, the temporary label storage unit 103, and the correct label storage unit 104 may be integrated into one storage unit. Also, for example, at least one of the target data storage unit 101, the extractor storage unit 102, the temporary label storage unit 103, and the correct label storage unit 104 may be divided into multiple storage units.

[0062] For example, at least two of the acquisition unit 110, the unlabeled learning unit 120, the feature extraction unit 130, the dimension reduction unit 140, the clustering unit 150, the label assignment unit 160, the label correction unit 170, and the display unit 180 may be integrated into one processing unit. Also, for example, at least one of the acquisition unit 110, the unlabeled learning unit 120, the feature extraction unit 130, the dimension reduction unit 140, the clustering unit 150, the label assignment unit 160, the label correction unit 170, and the display unit 180 may be divided into multiple processing units.

[0063] <User Interface> A labeling support tool, which is an example of a user interface of the labeling device 20, will be described with reference to FIGS.

[0064] Fig. 4 is a diagram showing an example of a labeling support tool according to the first embodiment. Fig. 4 shows an example of a main screen 600 of the labeling support tool. As shown in Fig. 4, the main screen 600 has an input folder setting unit 601, an output folder setting unit 602, an extractor learning operation unit 603, a feature extraction operation unit 604, a dimension reduction operation unit 605, a clustering operation unit 606, a cluster correction operation unit 607, a temporary label assignment operation unit 608, a temporary label correction operation unit 609, a dimension reduction result display unit 610, and a clustering result display unit 611.

[0065] The input folder setting unit 601 accepts the setting of a folder in which unlabeled target data is stored. In the input folder setting unit 601, the folder path can be directly entered in the input field, or the folder can be selected from a folder selection dialog box launched by pressing the "Specify Folder" button.

[0066] The output folder setting unit 602 accepts the setting of the folder in which the labeled target data will be stored. In the output folder setting unit 602, the folder path can be directly entered in the input field, or a folder can be selected from a folder selection dialog box launched by clicking the "Specify Folder" button.

[0067] The extractor learning operation unit 603 accepts operations related to the learning of the feature extractor. The extractor learning operation unit 603 may display a setting screen for setting parameters related to the learning of the feature extractor by pressing a "parameter setting" button. The extractor learning operation unit 603 may start learning of the feature extractor by pressing a "learning execution" button. The parameters related to the learning of the feature extractor may include, for example, the type of feature extractor, learning parameters, etc.

[0068] The feature extraction operation unit 604 accepts operations related to the extraction of feature data. The feature extraction operation unit 604 may start the extraction of feature data by pressing the "execute feature extraction" button.

[0069] The dimension reduction operation unit 605 accepts operations related to the dimension reduction of feature data. The dimension reduction operation unit 605 may display a setting screen for setting parameters related to dimension reduction by pressing a "Parameter Setting" button. The dimension reduction operation unit 605 may start the dimension reduction of feature data by pressing a "Execute Dimension Reduction" button. The parameters related to dimension reduction may include, for example, the type of dimension reduction algorithm, parameters at the time of execution, etc. After the dimension reduction is completed, the dimension reduction operation unit 605 may display the dimension reduction results in the dimension reduction result display unit 610 by pressing a "Display Results" button.

[0070] The clustering operation unit 606 accepts operations related to clustering. The clustering operation unit 606 may display a setting screen for setting parameters related to clustering by pressing a "Parameter setting" button. The clustering operation unit 606 may start clustering by pressing a "Execute clustering" button. The parameters related to clustering may include, for example, the type of clustering algorithm, parameters at runtime, etc. The parameters at runtime differ depending on the type of clustering algorithm, but may include, for example, the number of clusters k in the case of k-means. After clustering is completed, the clustering operation unit 606 may display the clustering results in the clustering result display unit 611 by pressing a "Display results" button.

[0071] The cluster correction operation unit 607 accepts operations related to correction of the clustering results. The cluster correction operation unit 607 may divide, merge, correct, etc. clusters using a "Cluster Correction" button. The cluster correction operation unit 607 may also reflect the correction results in the clustering results using an "Update Results" button. Note that if correction of the clustering results is not required, operation of the cluster correction operation unit 607 may be skipped.

[0072] The temporary label assignment operation unit 608 accepts an operation to assign a temporary label to target data. The temporary label assignment operation unit 608 may assign a temporary label to target data belonging to each cluster based on the clustering result by pressing an "output result" button. When the cluster correction operation unit 607 corrects the clustering result, the temporary label assignment operation unit 608 may assign a temporary label to target data belonging to each cluster based on the corrected clustering result. For example, the temporary label assignment operation unit 608 may store each target data in a folder corresponding to the temporary label.

[0073] The temporary label correction operation unit 609 accepts an operation to correct the temporary label assigned to the target data. The temporary label correction operation unit 609 may launch a label correction screen (see FIG. 5) by pressing a "correct temporary label" button. The label correction screen is a screen for checking the temporary label assigned to the target data and correcting the temporary label as necessary. After the label correction screen is closed, the temporary label correction operation unit 609 may correct the temporary label assigned to the target data based on the correction result by pressing an "update result" button. For example, the temporary label correction operation unit 609 may move the target data whose temporary label has been corrected from the folder corresponding to the temporary label before the correction to the folder corresponding to the temporary label after the correction.

[0074] The dimension reduction result display unit 610 displays the dimension reduction result. The dimension reduction result display unit 610 may display the distribution of the dimension-reduced feature data in a two-dimensional scatter plot.

[0075] The clustering result display unit 611 displays the clustering results. The clustering result display unit 611 may display the distribution of the dimension-reduced feature data in a two-dimensional scatter plot in a different manner for each cluster. For example, the clustering result display unit 611 may plot the feature data for each cluster in a different color or shape, or a combination thereof.

[0076] Fig. 5 is a diagram showing an example of the labeling support tool according to the first embodiment. Fig. 5 shows an example of a label correction screen 700 of the labeling support tool. As shown in Fig. 5, the label correction screen 700 has a temporary label folder setting section 701, a temporary label selection section 702, a number display section 703, a scroll button 704, an automatic update button 705, and a data display section 706.

[0077] The temporary label folder setting section 701 accepts the setting of a folder for storing target data to which a temporary label has been assigned. In the temporary label folder setting section 701, a folder path may be directly entered in an input field, or a folder may be selected in a folder selection dialog launched by pressing the "Specify Folder" button. The folder set in the output folder setting section 602 of the main screen 600 may be automatically set in the temporary label folder setting section 701.

[0078] The temporary label selection unit 702 accepts the selection of a temporary label to display the target data. The temporary label selection unit 702 may display a list of temporary labels assigned to the target data in a selectable manner. As an example, the temporary label selection unit 702 may display the list of temporary labels in a pull-down menu. The temporary label selection unit 702 may also allow a temporary label to be directly input. When a temporary label is selected in the temporary label selection unit 702, the target data to which the selected temporary label has been assigned may be displayed in the data display unit 706.

[0079] The temporary label selection unit 702 may accept settings related to the display of target data using a "Settings" button. The display settings may include the number of target data items to display at one time, adding and deleting temporary labels that can be set, the amount of scrolling using the scroll button 704, etc.

[0080] The number display unit 703 displays information regarding the number of target data displayed in the data display unit 706. The number display unit 703 displays the number of target data to be displayed (in other words, the number of target data to which the temporary label selected by the temporary label selection unit 702 has been assigned) and the order of the target data currently being displayed among the target data to be displayed. Fig. 5 shows an example of the number display unit 703 when the first piece of target data out of 150 pieces of target data is displayed.

[0081] The scroll button 704 is a button for manually scrolling the target data displayed in the data display section 706. When the left-facing button of the scroll button 704 is pressed, the target data displayed in the data display section 706 may be scrolled forward. When the right-facing button of the scroll button 704 is pressed, the target data displayed in the data display section 706 may be scrolled backward. When the target data displayed in the data display section 706 is scrolled, the order of the target data in the number display section 703 may be updated.

[0082] The automatic update button 705 is a button for automatically scrolling the target data displayed in the data display section 706. When the automatic update button 705 is pressed, the target data displayed in the data display section 706 is scrolled at a predetermined timing without the need to press the scroll button 704.

[0083] The data display unit 706 displays the target data classified into the temporary label selected by the temporary label selection unit 702 in association with the temporary label assigned to the target data. As an example, the data display unit 706 may display the temporary label by superimposing it on a part of the target data. Furthermore, the temporary label may be displayed in a different manner for each type of temporary label. For example, the temporary label may be displayed using a different symbol, icon, character color, or background color, or a combination of these, for each type of temporary label.

[0084] The data display unit 706 may display multiple target data at once. The target data displayed on the data display unit 706 may accept corrections to the temporary labels. For example, when a temporary label displayed in association with the target data is selected, the data display unit 706 may display a list of selectable temporary labels, and when a different temporary label is selected from the list, accepts corrections to the selected temporary label. Figure 5 shows an example in which images 1 to 10 are displayed with the temporary label "Cluster 1" assigned, and the temporary label of image 8 has been corrected to "Cluster 4."

[0085] Closing the label correction screen 700 returns to the main screen 600. The label correction screen 700 may display all of the target data and close automatically once the temporary label has been corrected as necessary. When returning to the main screen 600 and pressing the "Update Results" button in the temporary label correction operation section 609, the correction results of the temporary label may be reflected in the target data.

[0086] Fig. 6 is a diagram showing an example of a folder configuration according to the first embodiment. As shown in Fig. 6, a target data folder 801, temporary label folders 811 to 814, and correct label folders 821 to 824 may be created in the storage device of the labeling device 20. The target data folder 801 may be stored in the target data storage unit 101. The temporary label folders 811 to 814 may be stored in the temporary label storage unit 103. The correct label folders 821 to 824 may be stored in the correct label storage unit 104. Note that Fig. 6 shows an example in which four temporary labels and four correct labels are set, but the number or names of the labels may be set arbitrarily.

[0087] All target data to which no label has been assigned is stored in the target data folder 801. The label assignment unit 160 assigns a temporary label to the target data by storing the target data in one of the temporary label folders 811 to 814.

[0088] If the temporary label assigned to the target data is correct, the label correction unit 170 moves the target data as is to the correct label folders 821-824 corresponding to the temporary label folders 811-814 (solid arrows). As a result, the target data is assigned a correct label that is the same as the temporary label. On the other hand, if the temporary label assigned to the target data is incorrect, the label correction unit 170 moves the target data from the temporary label folders 811-814 to the correct correct label folders 821-824 (dashed arrows). As a result, the target data is assigned a correct label that is different from the temporary label assigned to the target data.

[0089] 6 shows only an example of correcting temporary label 1 to correct labels 1 to 4 for the sake of visibility, but temporary labels 2 to 4 may also be corrected to correct labels 1 to 4, respectively. Also, while an example of correcting a temporary label by moving it from a temporary label folder to a correct label folder has been described here, a temporary label may also be corrected by moving the target data between temporary label folders. In this case, the configuration may be such that when the temporary label is finally determined, the target data is moved from the temporary label folder to the corresponding correct label folder.

[0090] <Processing Procedure> The processing procedure of the labeling method executed by the labeling device 20 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the labeling method according to the first embodiment.

[0091] In step S101, the acquisition unit 110 of the labeling device 20 acquires one or more target data from the data generating device 10. The acquisition unit 110 stores the acquired target data in the target data storage unit 101. A user of the labeling device 20 performs an operation to set an input folder and an output folder in the labeling support tool.

[0092] In step S102, a user of the labeling device 20 performs an operation related to learning of a feature extractor in the labeling support tool. The unlabeled learning unit 120 of the labeling device 20 reads target data from the target data storage unit 101 in response to the user's operation. The unlabeled learning unit 120 learns a feature extractor by self-supervised learning based on the read target data. The unlabeled learning unit 120 stores the learned feature extractor in the extractor storage unit 102.

[0093] In step S103, a user of the labeling device 20 performs an operation related to feature data extraction in the labeling support tool. The feature extraction unit 130 of the labeling device 20 reads a feature extractor from the extractor storage unit 102 in response to the user's operation. The feature extraction unit 130 also reads target data from the target data storage unit 101. The feature extraction unit 130 uses the read feature extractor to extract feature data from each of the read target data. The feature extraction unit 130 sends the extracted feature data to the dimension reduction unit 140. The dimension reduction unit 140 receives the feature data from the feature extraction unit 130.

[0094] In step S104, the user of the labeling device 20 performs an operation related to dimensional reduction of feature data in the labeling support tool. The dimension reduction unit 140 of the labeling device 20 reduces the dimensions of the feature data received from the feature extraction unit 130 in response to the user's operation. The dimension reduction unit 140 sends the dimension-reduced feature data to the clustering unit 150. The clustering unit 150 receives the dimension-reduced feature data from the dimension reduction unit 140. The dimension reduction unit 140 also sends the dimension-reduced feature data to the display unit 180. The display unit 180 displays the distribution of the feature data on the dimension reduction result display unit 610 of the labeling support tool.

[0095] In step S105, the user of the labeling device 20 determines whether the feature data has been appropriately extracted by the feature extractor. For example, the user refers to the distribution of the feature data displayed in the dimension reduction result display unit 610 of the labeling support tool and determines whether an appropriate number of clusters have been formed. If the feature data has not been appropriately extracted (NO), the labeling device 20 returns the process to step S102. In this case, the labeling device 20 may return the process to step S101 or step S104. On the other hand, if the feature data has been appropriately extracted (YES), the labeling device 20 proceeds to step S106.

[0096] When the process returns to step S102, the user of the labeling device 20 changes parameters related to the learning of the feature extractor in the labeling support tool and performs an operation to start learning of the feature extractor. When the process returns to step S101, the user of the labeling device 20 adds new target data to the input folder set in the labeling support tool and performs an operation to start learning of the feature extractor. When the process returns to step S104, the user of the labeling device 20 changes parameters related to dimensionality reduction in the labeling support tool and performs an operation to start dimensionality reduction of the feature data. In this way, the labeling device 20 repeatedly performs the processes from step S101 to step S105 until it is determined in step S105 that feature data has been appropriately extracted.

[0097] In step S106, the user of the labeling device 20 performs an operation related to clustering in the labeling support tool. The clustering unit 150 of the labeling device 20 classifies the feature data received from the dimension reduction unit 140 into multiple clusters in response to the user's operation. The clustering unit 150 sends the clustering results to the labeling unit 160. The labeling unit 160 receives the clustering results from the clustering unit 150. The clustering unit 150 also sends the clustering results to the display unit 180. The display unit 180 displays the clustering results on the clustering result display unit 611 of the labeling support tool.

[0098] In step S107, a user of the labeling device 20 performs an operation in the labeling support tool to assign temporary labels to target data. In response to the user's operation, the labeling unit 160 of the labeling device 20 assigns temporary labels corresponding to each cluster to target data corresponding to feature data belonging to that cluster, based on the clustering result received from the clustering unit 150. The labeling unit 160 stores the target data to which the temporary labels have been assigned in the temporary label storage unit 103.

[0099] In step S108, the user of the labeling device 20 performs an operation in the labeling assistance tool to correct the temporary label assigned to the target data. In response to the user's operation, the label correcting unit 170 of the labeling device 20 reads the target data to which the temporary label has been assigned from the temporary label storage unit 103. The label correcting unit 170 displays the target data and the temporary label assigned to the target data, and corrects the temporary label assigned to the target data in response to the user's operation. Note that if the temporary label assigned to the target data is appropriate, step S108 may be skipped.

[0100] In step S109, the label correction unit 170 of the label assignment device 20 assigns a correct label to the target data based on the temporary label corrected in step S108 or the temporary label assigned in step S107. The label correction unit 170 stores the target data to which the correct label has been assigned in the correct label storage unit 104.

[0101] <Effects of the first embodiment> The labeling device 20 according to this embodiment trains a feature extractor based on unlabeled target data, classifies feature data extracted from the target data using the feature extractor into multiple clusters, and assigns a label corresponding to the cluster to target data corresponding to feature data belonging to a cluster.

[0102] In one aspect, this embodiment eliminates the need to create training data for training a feature extractor, thereby enabling efficient labeling of data. In another aspect, this embodiment enables efficient creation of training data, thereby enabling efficient generation of machine learning models.

[0103] The labeling device 20 may correct the labels assigned to the target data. The labeling device 20 may display the target data classified into each cluster, and correct the labels assigned to the target data in response to a user operation. In one aspect, according to this embodiment, it is only necessary to assign labels to only data to which an incorrect label has been assigned, thereby enabling efficient assignment of labels to data.

[0104] The labeling device 20 may reduce the dimensions of the feature data. The labeling device 20 may classify the dimension-reduced feature data into a plurality of clusters. In one aspect, according to the present embodiment, the amount of calculation required for clustering is reduced, and therefore labels can be efficiently assigned to data.

[0105] The labeling device 20 may display the distribution of the feature data. The labeling device 20 may repeatedly train the feature extractor in response to a user operation. In one aspect, according to this embodiment, the feature extractor can be repeatedly trained until it can appropriately extract feature data.

[0106] As described above, according to this embodiment, a feature extractor is trained using unlabeled target data, and then feature data from the target data is extracted, enabling clustering of data with similar features. Furthermore, by training the feature extractor using a machine learning technique using unlabeled data, such as self-supervised learning, the task of labeling training data for training the feature extractor is eliminated, reducing the workload of creating training data. Furthermore, by assigning temporary labels based on the clustering results and then correcting the labels, the workload of labeling can be significantly reduced compared to labeling data with a variety of mixed features.

[0107] [Second embodiment] In the first embodiment, a temporary label is assigned to each cluster classified by clustering. In the second embodiment, target data classified into the same cluster is further divided into multiple clusters, and temporary labels corresponding to the divided clusters are assigned.

[0108] The machine learning system 1000 according to this embodiment will be described below, focusing on the differences from the first embodiment.

[0109] <Functional configuration> The functional configuration of the machine learning system 1000 will be described with reference to Fig. 8. Fig. 8 is a block diagram showing an example of the functional configuration of a label assignment device according to the second embodiment.

[0110] 8, the labeling device 20 includes a target data storage unit 101, an extractor storage unit 102, a temporary label storage unit 103, a correct label storage unit 104, an acquisition unit 110, an unlabeled learning unit 120, a feature extraction unit 130, a dimension reduction unit 140, a clustering unit 150, a cluster division unit 155, a labeling unit 160, a label correction unit 170, and a display unit 180. That is, the labeling device 20 according to this embodiment differs from the labeling device 20 according to the first embodiment (see FIG. 3) in that it further includes a cluster division unit 155.

[0111] The cluster dividing unit 155 divides into multiple clusters one or more clusters into which the feature data has been classified by the clustering unit 150. Hereinafter, each cluster divided by the cluster dividing unit 155 is also referred to as a "sub-cluster." The cluster dividing unit 155 may divide the feature data included in a cluster into multiple sub-clusters based on the distance from the center of the cluster.

[0112] For example, the cluster dividing unit 155 may automatically set the center of gravity of feature data included in a cluster as the center of the cluster. The cluster dividing unit 155 may also set an arbitrary position selected by the user as the center of the cluster. The cluster dividing unit 155 may display the automatically set center of the cluster and correct the center of the cluster in response to a user operation.

[0113] The cluster dividing unit 155 may exclude outliers when calculating the center of gravity of feature data included in a cluster. For example, the cluster dividing unit 155 may calculate the center of gravity of all feature data included in a cluster, exclude feature data whose distance from the center of gravity is equal to or greater than a predetermined threshold, and then recalculate the center of gravity of the remaining feature data.

[0114] For example, the cluster dividing unit 155 may divide a cluster into a plurality of sub-clusters by comparing the distance from the center of the cluster with a predetermined threshold. Specifically, the cluster dividing unit 155 may divide the cluster into a first sub-cluster consisting of feature data whose distance from the center of the cluster is equal to or greater than the predetermined threshold, and a second sub-cluster consisting of feature data whose distance from the center of the cluster exceeds the predetermined threshold.

[0115] For example, the cluster dividing unit 155 may divide a cluster into a plurality of sub-clusters based on the order in which the clusters are sorted in ascending order of distance from the center of the cluster. Specifically, the cluster dividing unit 155 may divide the cluster into a first sub-cluster consisting of a predetermined number of feature data and a second sub-cluster consisting of other feature data, starting from the cluster having the shortest distance from the center of the cluster.

[0116] The reliability of a cluster formed by clustering decreases as the distance from the center of the cluster increases due to the inclusion of data indicating other features. Therefore, feature data closer to the center of the cluster is more reliable, and feature data farther from the center is less reliable. By dividing a cluster into multiple sub-clusters based on the distance from the center of the cluster and assigning each sub-cluster a different label, it is possible to distinguish between feature data with high reliability and feature data with low reliability.

[0117] 9A and 9B are diagrams illustrating an example of sub-clusters according to the second embodiment. Fig. 9A shows an example of a first division method based on the distance from the center of the cluster. In Fig. 9A, a cluster is divided into sub-clusters at a distance R A The feature data within the range is treated as a highly reliable sub-cluster (tentative label 1), and the distance R A The feature data outside the range of distance R is classified as a sub-cluster with low reliability (tentative label 2). A may be set by a user operation for each cluster, or may be set automatically by, for example, statistical analysis of variance within the cluster.

[0118] An example of the second division method based on the order of proximity to the center of the cluster is shown in Figure 9(B). In Figure 9(B), the distance R from the center of the cluster (star) is i A predetermined number (e.g., 100) of feature data items in ascending order of length are designated as a subcluster with high reliability (tentative label 1), and the remaining feature data items are designated as a subcluster with low reliability (tentative label 2). Here, a configuration in which the data are divided based on the number of data items to be included in the subcluster is shown, but the number of data items to be included in the subcluster may be dynamically determined based on the proportion of data items to be included in the subcluster, etc. For example, the 25% closest to the center of the cluster may be designated as a subcluster with high reliability, and the remaining 75% may be designated as a subcluster with low reliability.

[0119] The cluster dividing unit 155 may divide the clusters by combining the methods shown in Fig. 9. The cluster dividing unit 155 may also divide the clusters into three or more sub-clusters (for example, with high, medium, and low reliability). The cluster dividing unit 155 may divide the clusters based on a measure other than reliability or multiple measures including reliability.

[0120] The labeling unit 160 according to this embodiment assigns temporary labels to target data based on the clustering result obtained by dividing the data into sub-clusters by the cluster dividing unit 155. The labeling unit 160 may assign temporary labels corresponding to the sub-clusters to target data corresponding to feature data belonging to each of the sub-clusters divided by the cluster dividing unit 155.

[0121] <User Interface> The labeling assistance tool according to this embodiment will be described with reference to FIGS.

[0122] FIG. 10 is a diagram showing an example of a labeling assistance tool according to the second embodiment. FIG. 10 shows an example of a main screen 600 according to this embodiment. As shown in FIG. 10, the main screen 600 has an input folder setting unit 601, an output folder setting unit 602, an extractor learning operation unit 603, a feature extraction operation unit 604, a dimension reduction operation unit 605, a clustering operation unit 606, a cluster correction operation unit 607, a temporary label assignment operation unit 608, a temporary label correction operation unit 609, a dimension reduction result display unit 610, a clustering result display unit 611, and a cluster division operation unit 612. That is, the main screen 600 according to this embodiment differs from the main screen 600 according to the first embodiment (see FIG. 4) in that it has a cluster division operation unit 612.

[0123] The cluster division operation unit 612 accepts operations related to cluster division. The cluster division operation unit 612 may display a setting screen for setting parameters related to cluster division using a "parameter setting" button. The cluster division operation unit 612 may automatically set the center of the cluster using an "automatic center setting" button. The cluster division operation unit 612 may accept the setting of the center of the cluster using a "manual center setting" button.

[0124] The parameters related to the division of clusters may include, for example, a cluster division method, a threshold value for division, etc. The cluster division method may include, for example, a first division method based on the distance from the center of the cluster (see FIG. 9(A)), or a second division method based on the order of proximity to the center of the cluster (see FIG. 9(B)). In the first division method, the threshold may be the distance for dividing the sub-clusters. In the second division method, the threshold may be the number or proportion of feature data to be included in the sub-cluster. After the division of the clusters is completed, the cluster division operation unit 612 may display the clustering results divided into sub-clusters on the clustering result display unit 611.

[0125] Fig. 11 is a diagram showing an example of a folder configuration according to the second embodiment. As shown in Fig. 11, a target data folder 801, temporary label folders 811 to 816, and correct label folders 821 to 826 may be created in the storage device of the labeling device 20. The target data folder 801 may be stored in the target data storage unit 101. The temporary label folders 811 to 816 may be stored in the temporary label storage unit 103. The correct label folders 821 to 826 may be stored in the correct label storage unit 104. Note that while Fig. 11 shows an example in which six temporary labels and six correct labels are set, the number or names of the labels may be set arbitrarily.

[0126] Temporary label folder 811 stores target data classified into sub-cluster 1-1, which has a high reliability, out of the target data classified into cluster 1. Temporary label folder 812 stores target data classified into sub-cluster 1-2, which has a low reliability, out of the target data classified into cluster 1. Similarly, temporary label folders 813 and 815 store target data classified into sub-clusters 2-1 and 3-1, which have a high reliability, out of the target data classified into clusters 2 and 3, and temporary label folders 814 and 816 store target data classified into sub-clusters 2-2 and 3-2, which have a low reliability, out of the feature data classified into clusters 2 and 3.

[0127] As in the first embodiment, the target data stored in the temporary label folders 811 to 816 is moved and stored in the correct label folders 821 to 826. That is, if the temporary labels assigned to the target data are correct, the target data stored in the temporary label folders 811 to 816 is moved directly to the correct label folders 821 to 826 (solid arrows). If the temporary labels assigned to the target data are incorrect, the target data stored in the temporary label folders 811 to 816 is moved to the correct correct label folders 821 to 826 (dashed arrows).

[0128] <Processing Procedure> The processing procedure of the labeling method executed by the labeling device 20 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the labeling method according to the second embodiment.

[0129] The processes from step S201 to step S206 are the same as the processes from step S101 to step S106 in the first embodiment (see FIG. 7). That is, the labeling device 20 learns a feature extractor that can appropriately extract feature data based on target data to which no labels are assigned, and classifies the feature data extracted by the feature extractor into multiple clusters.

[0130] In step S207, the user of the labeling device 20 performs an operation related to dividing the clusters in the labeling support tool. The cluster dividing unit 155 of the labeling device 20 divides one or more clusters included in the clustering result received from the clustering unit 150 into multiple sub-clusters in response to the user's operation. The cluster dividing unit 155 sends the clustering result divided into multiple sub-clusters to the labeling unit 160. The labeling unit 160 receives the clustering result divided into multiple sub-clusters from the cluster dividing unit 155. The cluster dividing unit 155 also sends the clustering result divided into multiple sub-clusters to the display unit 180. The display unit 180 displays the clustering result divided into multiple sub-clusters on the clustering result display unit 611 of the labeling support tool.

[0131] In step S208, the user of the labeling device 20 performs an operation in the labeling support tool to assign temporary labels to target data. In response to the user's operation, the labeling unit 160 of the labeling device 20 assigns temporary labels corresponding to the sub-clusters to target data corresponding to feature data belonging to the sub-clusters, based on the clustering results received from the cluster dividing unit 155. The labeling unit 160 stores the target data to which the temporary labels have been assigned in the temporary label storage unit 103.

[0132] The processing from step S209 to step S210 is the same as the processing from step S108 to step S109 in the first embodiment (see FIG. 7). That is, the labeling device 20 appropriately corrects the temporary labels assigned to the target data, and assigns correct labels to the target data based on the temporary labels.

[0133] <Effects of the second embodiment> The labeling device 20 according to this embodiment divides a cluster into multiple clusters and assigns labels corresponding to the divided clusters. The labeling device 20 may divide the clusters based on the distance from the center of the cluster. In one aspect, according to this embodiment, the distance from the center of the cluster indicates the reliability of the clustering result, so that highly reliable data can be extracted.

[0134] As described above, even among target data classified into the same cluster, there is a mixture of highly reliable and low reliable data. By distinguishing between highly reliable and low reliable data and assigning temporary labels based on the distance from the cluster center, it is possible to further reduce the amount of data that needs to be checked in detail. This further reduces the labeling load.

[0135] [Third embodiment] In the first embodiment, a feature extractor is trained by self-supervised learning based on unlabeled target data. In the third embodiment, a feature extractor trained on unlabeled target data is used to assign temporary labels to the target data, and the feature extractor is then trained again by supervised learning based on the target data with the temporary labels.

[0136] The machine learning system 1000 according to this embodiment will be described below, focusing on the differences from the first embodiment.

[0137] <Functional configuration> The functional configuration of the machine learning system 1000 will be described with reference to Fig. 13. Fig. 13 is a block diagram showing an example of the functional configuration of a labeling device according to the third embodiment.

[0138] 13, the labeling device 20 includes a target data storage unit 101, an extractor storage unit 102, a temporary label storage unit 103, a correct label storage unit 104, an acquisition unit 110, an unlabeled learning unit 120, a labeled learning unit 125, a feature extraction unit 130, a dimension reduction unit 140, a clustering unit 150, a cluster division unit 155, a labeling unit 160, a label correction unit 170, and a display unit 180. That is, the labeling device 20 according to this embodiment differs from the labeling device 20 according to the first embodiment (see FIG. 3) in that it further includes a labeled learning unit 125. The labeled learning unit 125 is an example of a second learning unit.

[0139] The labeled learning unit 125 trains the feature extractor based on target data to which temporary labels have been assigned. The labeled learning unit 125 may train the feature extractor based on target data read out from the temporary label storage unit 103. As an example, the labeled learning unit 125 may train the feature extractor by supervised learning. The labeled learning unit 125 may train the feature extractor by any method as long as it is a method that can learn features from labeled data.

[0140] The labeled learning unit 125 may train a feature extractor based on highly reliable target data. The labeled learning unit 125 may extract highly reliable target data based on feature data extracted from target data using the feature extractor trained by the unlabeled learning unit 120. The labeled learning unit 125 may extract highly reliable target data based on the clustering results of the feature data. The labeled learning unit 125 may extract highly reliable target data for each cluster included in the clustering results based on the distance from the center of the cluster.

[0141] The labeled learning unit 125 updates the feature extractor stored in the extractor storage unit 102 with a feature extractor that has been trained by supervised learning. The labeled learning unit 125 may overwrite the feature extractor stored in the extractor storage unit 102 with the feature extractor that has been trained by supervised learning.

[0142] The labeled learning unit 125 may associate a trained feature extractor with a generation time and store it as a new trained feature extractor in the extractor storage unit 102. In this case, multiple feature extractors are stored in the extractor storage unit 102. The feature extraction unit 130 may read a desired feature extractor from the extractor storage unit 102 based on the generation time associated with the feature extractor.

[0143] The configuration of the machine learning system 1000 according to the third embodiment may be combined with the machine learning system 1000 according to the second embodiment. For example, the labeling device 20 according to this embodiment may further include a cluster dividing unit 155. Similarly, the labeling device 20 according to the second embodiment may further include a labeled learning unit 125.

[0144] <Processing Procedure> The processing procedure of the labeling method executed by the labeling device 20 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the labeling method according to the third embodiment.

[0145] The processes from step S301 to step S306 are the same as the processes from step S101 to step S106 in the first embodiment (see FIG. 7). That is, the labeling device 20 learns a feature extractor that can appropriately extract feature data based on target data to which no labels are assigned, and classifies the feature data extracted by the feature extractor into multiple clusters.

[0146] In step S307, the labeled learning unit 125 of the labeling device 20 receives the clustering result from the clustering unit 150. For each cluster included in the clustering result, the labeled learning unit 125 extracts highly reliable target data based on the distance from the center of the cluster. The labeled learning unit 125 assigns a temporary label corresponding to the cluster to the extracted target data. This generates training data in which the temporary label is assigned to the target data.

[0147] In step S308, the labeled learning unit 125 of the labeling device 20 learns a feature extractor by supervised learning based on the training data generated in step S307. The labeled learning unit 125 updates the feature extractor stored in the extractor storage unit 102 with the trained feature extractor.

[0148] In step S309, the user of the labeling device 20 performs an operation related to feature data extraction in the labeling support tool. The feature extraction unit 130 of the labeling device 20 reads a feature extractor from the extractor storage unit 102 in response to the user's operation. The feature extraction unit 130 also reads target data from the target data storage unit 101. The feature extraction unit 130 uses the read feature extractor to extract feature data from each of the read target data. The feature extraction unit 130 sends the extracted feature data to the dimension reduction unit 140. The dimension reduction unit 140 receives the feature data from the feature extraction unit 130.

[0149] In step S310, the user of the labeling device 20 performs an operation related to dimensionality reduction of feature data in the labeling support tool. The dimension reduction unit 140 of the labeling device 20 reduces the dimensions of the feature data received from the feature extraction unit 130 in response to the user's operation. The dimension reduction unit 140 sends the dimension-reduced feature data to the clustering unit 150. The clustering unit 150 receives the dimension-reduced feature data from the dimension reduction unit 140.

[0150] In step S311, a user of the labeling device 20 performs an operation related to clustering in the labeling support tool. The clustering unit 150 of the labeling device 20 classifies the feature data received from the dimension reduction unit 140 into multiple clusters in response to the user's operation. The clustering unit 150 sends the clustering result to the labeling unit 160. The labeling unit 160 receives the clustering result from the clustering unit 150.

[0151] In step S312, the labeling device 20 determines whether a termination condition for terminating the learning of the feature extractor has been met. For example, the labeling device 20 may determine that the termination condition has been met if the number of times the feature extractor has learned is equal to or greater than a predetermined threshold. Alternatively, the labeling device 20 may calculate the variance of the distance from the center of each cluster shown in the clustering result, and determine that the termination condition has been met if the amount of change in the variance is equal to or less than a predetermined threshold.

[0152] If the termination condition is not met (NO), the labeling device 20 returns the process to step S307. On the other hand, if the termination condition is met (YES), the labeling device 20 proceeds to step S313.

[0153] The processing from step S313 to step S315 is the same as the processing from step S107 to step S109 in the first embodiment (see FIG. 7). That is, the labeling device 20 assigns temporary labels to the target data based on the clustering result, appropriately corrects the temporary labels assigned to the target data, and assigns correct labels to the target data based on the temporary labels.

[0154] <Effects of the third embodiment> The labeling device 20 according to this embodiment trains a feature extractor based on labeled target data. The labeling device 20 may train a feature extractor based on target data selected based on the distance from the center of a cluster. In one aspect, this embodiment allows a highly accurate feature extractor to be trained based on training data to which highly reliable labels have been assigned.

[0155] As described above, supervised learning using target data with desired temporary labels as training data improves the accuracy of the feature extractor, and clustering accuracy can be improved compared to when self-supervised learning is used. Furthermore, performing supervised learning using highly reliable data makes it possible to train the feature extractor more accurately.

[0156] [supplement] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0157] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0158] 1000: Machine Learning Systems 10: Data generation device 20: Labeling device 30: Model learning device 101: Target data storage unit 102:Extractor storage unit 103: Temporary label storage unit 104: Correct label memory 110: Acquisition Department 120: Unlabeled learning unit 125: Labeled learning section 130: Feature extraction unit 140: Dimension reduction part 150: Clustering Department 155: Cluster division part 160: Label assignment unit 170: Label correction section 180: Display section

Claims

1. a training unit configured to train a feature extractor based on unlabeled target data; a feature extraction unit configured to extract feature data from the target data using the feature extractor; a clustering unit configured to classify the feature data into a plurality of clusters; a label assignment unit configured to assign the label corresponding to the cluster to the target data corresponding to the feature data belonging to the cluster; A labeling device comprising:

2. a label correction unit configured to correct the label assigned to the target data; 2. The labeling device of claim 1.

3. a display unit configured to display the target data classified into each cluster for each cluster, the label correction unit is configured to correct the label assigned to the target data in response to a user operation.

3. The labeling device according to claim 2.

4. a dimension reduction unit configured to reduce the dimension of the feature data; the clustering unit is configured to classify the dimension-reduced feature data into the plurality of clusters.

2. The labeling device of claim 1.

5. a display unit configured to display the distribution of the feature data, the learning unit is configured to repeatedly learn the feature extractor in response to a user operation.

5. The labeling device according to claim 4.

6. a cluster dividing unit configured to divide the cluster into a plurality of clusters; the labeling unit is configured to assign the labels corresponding to the divided clusters. A labeling device according to any one of claims 1 to 5.

7. the cluster dividing unit is configured to divide the clusters based on a distance from a center of the cluster.

7. The labeling device of claim 6.

8. a second training unit configured to train the feature extractor based on the labeled target data; A labeling device according to any one of claims 1 to 5.

9. the second learning unit is configured to learn the feature extractor based on the target data selected based on a distance from the center of the cluster.

9. The labeling device of claim 8.

10. The computer training a feature extractor based on unlabeled target data; extracting feature data from the target data using the feature extractor; classifying the feature data into a plurality of clusters; assigning the label corresponding to the cluster to the target data corresponding to the feature data belonging to the cluster; A labeling method that performs

11. On the computer, training a feature extractor based on unlabeled target data; extracting feature data from the target data using the feature extractor; classifying the feature data into a plurality of clusters; assigning the label corresponding to the cluster to the target data corresponding to the feature data belonging to the cluster; A program to execute.

Images