Labeling support system, labeling support method, and labeling support program

Abstract

A first classification means 181 generates a first plurality of clusters by classifying a first group of data, which is a group of data to be labeled, by unsupervised learning. A second classification means 182 generates a second plurality of clusters by classifying a second group of data, which is a group of data including at least a part of the group of data to be labeled. An output means 183 outputs data that is included in the second plurality of clusters and is classified into a different cluster for the first plurality of clusters.

Description

[Technical Field]

[0001] The present invention relates to a labeling support system, a labeling support method, and a labeling support program that support labeling of unlabeled data. [Background technology]

[0002] In the IoT (Internet of Things) society, it is now possible to collect data from a variety of devices. However, it is extremely difficult to simply search for a specific video among a large amount of data. Therefore, there is a need for a system to search the collected data.

[0003] One method for searching data is to label the data. However, since manually labeling large amounts of data requires a huge amount of time and cost, various methods for classifying data have been proposed.

[0004] For example, Patent Document 1 describes a sensor data classification device that classifies sensor data obtained by a large number of sensors according to their characteristics. The device described in Patent Document 1 associates sets of sensor data divided into preset time intervals with sensor identifiers and divided interval identifiers, and calculates multiple types of characteristic parameters from the data included in the divided data sets. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2016-99888 Summary of the Invention [Problem to be solved by the invention]

[0006] For example, automatic rule-based labeling could be considered, but maintaining the rules in response to changes in the environment is cumbersome, and adding rules is not easy.

[0007] In the device described in Patent Document 1, the calculation method of feature parameters for classification and the division intervals are determined in advance. However, even if data is classified based on values ​​calculated based on some criteria, there is still a problem in that it is costly to perform meaningful labeling work on unlabeled data.

[0008] Therefore, an object of the present invention is to provide a labeling support system, a labeling support method, and a labeling support program that can support the labeling work for clusters into which unlabeled data has been classified. [Means for solving the problem]

[0009] The labeling support system according to the present invention is characterized by comprising a first classification means for generating a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning; a second classification means for generating a second plurality of clusters by classifying a second data group, which is a data group including at least a portion of the data to be labeled; and an output means for outputting data included in the second plurality of clusters that was classified into a different cluster in the first plurality of clusters.

[0010] The labeling support method according to the present invention is characterized in that a computer generates a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning; the computer generates a second plurality of clusters by classifying a second data group, which is a data group including at least some of the data to be labeled; and the computer outputs data included in the second plurality of clusters that was classified into a different cluster in the first plurality of clusters.

[0011] The labeling support program according to the present invention is characterized in that it causes a computer to execute a first classification process that generates a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning; a second classification process that generates a second plurality of clusters by classifying a second data group, which is a data group containing at least some of the data to be labeled; and an output process that outputs data included in the second plurality of clusters that was classified into a different cluster in the first plurality of clusters. [Effects of the Invention]

[0012] According to the present invention, it is possible to assist in labeling clusters into which unlabeled data has been classified. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of the configuration of an embodiment of a labeling support system according to the present invention. [Figure 2] FIG. 1 is an explanatory diagram showing an example of data used in the labeling support system. [Figure 3] FIG. 10 is an explanatory diagram illustrating an example of feature amounts. [Figure 4] FIG. 10 is an explanatory diagram showing an example of graph visualization of dimension-reduced data. [Figure 5] FIG. 10 is an explanatory diagram showing another example of graph visualization of dimension-reduced data. [Figure 6] FIG. 10 is an explanatory diagram showing an example of a process for labeling data within a cluster. [Figure 7] FIG. 10 is an explanatory diagram illustrating an example of a process for selecting some clusters. [Figure 8] FIG. 10 is an explanatory diagram showing an example of processing for excluding part of data. [Figure 9] FIG. 10 is an explanatory diagram showing an example of overlay display of results before and after refinement. [Figure 10] FIG. 10 is an explanatory diagram showing an example of displaying results before and after refinement in parallel windows. [Figure 11] FIG. 10 is an explanatory diagram showing an example of displaying results before and after refinement in parallel windows. [Figure 12] FIG. 10 is an explanatory diagram showing an example of data that shows different results before and after refinement being displayed in a list in a separate window. [Figure 13] FIG. 10 is an explanatory diagram showing an example in which the refinement results of multiple times are overlaid. [Figure 14] FIG. 10 is an explanatory diagram showing an example of data that has been refined multiple times and has different results being displayed in a list in a separate window. [Figure 15] FIG. 10 is an explanatory diagram showing an example of displaying statistical information for each cluster. [Figure 16] FIG. 10 is an explanatory diagram showing another example of displaying statistical information of each cluster. [Figure 17] 10 is a flowchart showing an example of the operation of the labeling support system. [Figure 18] 1 is a block diagram showing an overview of a labeling support system according to the present invention. [Figure 19] FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, moving images (video data) will be used as an example of unlabeled data. However, unlabeled data is not limited to moving images, and may be, for example, still images, music data, text data, etc. Furthermore, unlabeled data (data to be labeled) may be referred to as unclassified data hereinafter.

[0015] 1 is a block diagram showing an example of the configuration of an embodiment of a labeling support system according to the present invention. The labeling support system 1 of this embodiment includes a data acquisition unit 10, a related information acquisition unit 20, an object identification unit 30, a data processing unit 40, a text information input unit 50, a feature extraction unit 60, a feature storage unit 70, a visualization processing unit 80, an input / output device 90, and a data refinement unit 100.

[0016] The data acquiring unit 10 acquires data to be labeled (i.e., unclassified data). For example, when a moving vehicle is captured by a camera (not shown), the data acquiring unit 10 may acquire a video of the vehicle captured by the camera as data to be labeled. Note that the data acquired by the data acquiring unit 10 is not limited to data acquired in real time. For example, the data acquiring unit 10 may acquire the data to be labeled from a storage server (not shown) in which the data to be labeled is stored.

[0017] The related information acquisition unit 20 acquires information related to the data to be labeled (hereinafter referred to as related information). In this embodiment, the related information is information indicating the circumstances under which the data to be labeled was generated, such as information indicating the place (place where the image was taken) and time when the data was generated, and data acquired by a sensor (hereinafter referred to as sensor data).

[0018] For example, if the data to be labeled is video data captured by an in-vehicle camera (drive recorder), relevant information may include GPS (Global Positioning System) information indicating the vehicle position, information acquired based on a CAN (Controller Area Network), etc. Examples of sensor data acquired in this case include speed, acceleration, and position (latitude, longitude, altitude, etc.).

[0019] Furthermore, when video showing the operating status of a thermal power plant is used as the data to be labeled, examples of sensor data include fuel flow rate, pressure, temperature, rotation speed, power generation, etc. In addition, when video showing the status of a farm is used as the data to be labeled, examples of sensor data include time, temperature, humidity, pH, soil moisture content, solar radiation, wind direction and speed, water level, etc.

[0020] The object identification unit 30 identifies objects included in the acquired data and generates information specifying the identified objects (hereinafter referred to as an object list). For example, if the object to be identified is a vehicle, the object identification unit 30 may identify the vehicle from the data acquired by the data acquisition unit 10 and generate information specifying the vehicle (for example, coordinates indicating its position in the image) as an object list. Note that methods for identifying objects from images or videos are widely known, and detailed explanations will be omitted here.

[0021] The data processing unit 40 processes the data (more specifically, the object list) into a format that can be used when the feature extraction unit 60, which will be described later, performs processing. Specifically, the data processing unit 40 processes the data so as to improve the accuracy of feature extraction and clustering. The data processing unit 40 performs, for example, data thinning, missing value interpolation, outlier removal, and deletion of unnecessary data items. Furthermore, for example, if the data to be labeled is video data, the data processing unit 40 may convert the video data into numerical time-series data.

[0022] The text information input unit 50 accepts input of text data including information to be added to each piece of data to be labeled (hereinafter referred to as additional information). Additional information is information indicating the content of the data to be labeled that can be obtained other than related information. Examples of categories indicating additional information include weather, plant types, and traffic participants. Examples of weather category values ​​include sunny, cloudy, rainy, and snowy. Examples of plant type category values ​​include rice, wheat, and barley. Examples of traffic participants include automobiles, bicycles, and pedestrians.

[0023] Note that input of text data is optional. That is, additional information does not need to be input for the data to be labeled. However, it is preferable to input additional information because the more additional information there is for the data to be labeled, the more accurate the classification can be. In the following description, data to be labeled that is associated with additional information will also be simply referred to as data to be labeled.

[0024] FIG. 2 is an explanatory diagram showing an example of data used in the labeling support system 1 of this embodiment. The example shown in FIG. 2 shows that the data acquisition unit 10 acquires video 11 as data to be labeled, and the related information acquisition unit 20 acquires related information 21 related to the location where the video 11 was taken, etc. The example shown in FIG. 2 also shows that the data processing unit 40 processes the video 11 and related information 21 (more specifically, the object list generated by the object identification unit 30) to generate numerical time-series data 41. Furthermore, the example shown in FIG. 2 shows that the text information input unit 50 has received input of text data 51 including information related to weather, scene, time period, and object as additional information.

[0025] The feature extraction unit 60 extracts features from each piece of data to be labeled. In this embodiment, the feature extraction unit 60 first generates multiple clusters by automatically classifying each piece of data to be labeled, including additional information, through unsupervised learning. Any method for generating clusters through unsupervised learning may be used, and examples include the k-means method and a Gaussian mixture model.

[0026] Hereinafter, the process in which the feature extraction unit 60 generates multiple clusters by classifying a group of data to be labeled using unsupervised learning will be referred to as a first classification process. The multiple clusters generated by the first classification process will be referred to as a first multiple clusters, and the group of data classified into the first multiple clusters will be referred to as a first data group. Because the feature extraction unit 60 performs the process of classifying the data to be labeled using unsupervised learning, the feature extraction unit 60 can also be referred to as a classification means.

[0027] Then, the feature extraction unit 60 extracts a feature amount of each data included in the generated cluster. For example, the feature extraction unit 60 may extract additional information included in text data as a feature amount. Alternatively, the feature extraction unit 60 may extract a feature amount indicated by the numerical time series data. Specifically, the feature extraction unit 60 may extract a feature amount based on a sensor value included in the data to be labeled (more specifically, the numerical time series data).

[0028] The method for extracting features from the numerical time series data is arbitrary. For example, for each cluster generated by the k-means method, the feature extraction unit 60 may extract a feature called the distance from the center of gravity of the numerical time series data included in the cluster to each data (cluster distance feature).

[0029] In addition, in this embodiment, the case has been described in which the object identification unit 30 identifies an object from the information acquired by the data acquisition unit 10 and the related information acquisition unit 20, and the data processing unit 40 processes the data of the identification result into a format used by the feature extraction unit 60. However, the data acquisition unit 10 may directly acquire data in a format used by the feature extraction unit 60 and input the acquired data to the feature extraction unit 60. In this case, the labeling support system 1 may not include the related information acquisition unit 20, the object identification unit 30, and the data processing unit 40.

[0030] The feature storage unit 70 stores the feature amount of each data item extracted by the feature extraction unit 60. The feature storage unit 70 may also store information about labels added by the data refinement unit 100, which will be described later. The feature storage unit 70 may store the feature amount for each data item in any manner.

[0031] Fig. 3 is an explanatory diagram showing an example of feature amounts stored in the feature storage unit 70. In the example shown in Fig. 3, the vertical direction represents one feature point, and the horizontal direction represents feature amounts (category values) of each category (e.g., weather, traffic participants, types of plants, etc.). The feature storage unit 70 is realized by, for example, a magnetic disk or the like.

[0032] The visualization processing unit 80 performs processing to visualize information that contributes to the labeling work for the generated clusters. The visualization processing unit 80 of this embodiment visualizes the dimension-reduced data to be labeled by drawing a graph on the input / output device 90 so that a person can observe how the data to be labeled is clustered.

[0033] The visualization processing unit 80 may reduce the dimension of the data to be labeled into two or three dimensions using, for example, UMAP (Uniform Manifold Approximation and Projection) or the like, and visualize the dimension-reduced data as a graph such as a distribution map. In this case, the visualization processing unit 80 may display data classified into the same cluster in a different manner from other clusters (for example, by using a different color or symbol).

[0034] Fig. 4 is an explanatory diagram showing an example of graph visualization of dimension-reduced data. The graph shown in Fig. 4 shows an example of data that has been reduced to two dimensions using UMAP, with different display styles (diagonal lines, black boxes, etc.) for each cluster to which it belongs.

[0035] Fig. 5 is an explanatory diagram showing another example of graph visualization of dimension-reduced data. The graph shown in Fig. 5 is a graph in which plotted symbols are changed for each type of video data. As shown in Fig. 5, the visualization processing unit 80 may also display the range of data included in a cluster by enclosing it with a dotted line so that the range can be identified.

[0036] Furthermore, when drawing a graph, the visualization processing unit 80 may display all data, or may determine to display only data that satisfies a specific condition, or not to display any data at all. For example, the visualization processing unit 80 may determine whether to display clusters that satisfy a specific condition (e.g., clusters with a larger number of data points than a predetermined number) or unclassified data (i.e., unlabeled data).

[0037] Furthermore, the visualization processing unit 80 of this embodiment outputs data that has come to belong to a different cluster as a result of a re-learning process, which will be described later. Note that a method for outputting data will be described later.

[0038] The input / output device 90 displays the output results from the visualization processing unit 80. The input / output device 90 also receives input from the user regarding the displayed results and executes processing according to the input. In this embodiment, processing by the data refinement unit 100 (described later) is performed based on the input of a cluster specified by the user in response to the output of the input / output device 90.

[0039] The input / output device 90 may be realized by a tablet terminal, etc. Alternatively, the input / output device 90 may be realized by a device having a display device and a pointing device, etc.

[0040] The data refinement unit 100 performs various processes on the data group to be labeled based on the clusters generated by the feature extraction unit 60. Specifically, the data refinement unit 100 generates a second data group from the data group to be labeled in accordance with the generated first plurality of clusters. In this embodiment, the data refinement unit 100 performs the following three types of processes.

[0041] First, the first processing will be described. The first processing is processing for labeling data within a cluster. In the first processing, the data refinement unit 100 generates a second data group by labeling data classified into one of the first plurality of clusters, out of the data group to be labeled, for each cluster. The data refinement unit 100 can label any cluster. The data refinement unit 100 may label all clusters, or may label a cluster specified by the user via the input / output device 90.

[0042] Furthermore, as long as the same label is assigned to data within a cluster, the content of the label is arbitrary. The data refinement unit 100 may assign any temporary label to the data within the target cluster, or may assign a label with content specified by the user. The data refinement unit 100 may then associate the data (more specifically, the feature of the data) with the assigned label and store them in the feature storage unit 70.

[0043] Fig. 6 is an explanatory diagram showing an example of a process for labeling data within a cluster. The example shown in Fig. 6 shows that the data refinement unit 100 has added temporary labels "A," "B," and "C" to the clusters shown in Fig. 5. If the user specifies a target cluster among the clusters shown in Fig. 5, the data refinement unit 100 need only add temporary labels to the specified cluster.

[0044] Thereafter, the feature extraction unit 60 generates multiple clusters again by learning using labeled data (supervised learning). Note that the feature extraction unit 60 may also perform learning (unsupervised learning) by adding unlabeled data. Hereinafter, the process by which the feature extraction unit 60 generates multiple clusters by classifying a data group including at least some of the data to be labeled is referred to as a second classification process. Furthermore, the multiple clusters generated by the second classification process are referred to as a second multiple clusters, and the data group classified into the second multiple clusters is referred to as a second data group.

[0045] In this way, the second classification process generates and refines multiple clusters again using at least some of the data to be labeled used in the first classification process, so the second classification process can be called a re-learning process or refinement. This allows for semi-automated labeling through unsupervised learning and can also contribute to the discovery of new labels.

[0046] The feature extraction unit 60 may extract a feature amount of each piece of data included in the clusters (second plurality of clusters) generated by the second classification process, and store the extracted feature amount in the feature storage unit .

[0047] After the second classification process, the visualization processing unit 80 outputs data included in the second plurality of clusters that was classified into a different cluster in the first plurality of clusters. This corresponds to the process of visualizing data that has come to belong to a different cluster as a result of re-learning. The specific visualization process will be described later.

[0048] Next, the second processing will be described. The second processing is a processing in which at least some of the clusters are selected and learning is performed again (unsupervised learning). The data refining unit 100 generates, as a second data group, a data group that is classified into a cluster selected from the first plurality of clusters from among the data group to be labeled.

[0049] First, the data refinement unit 100 selects at least some clusters from the first plurality of clusters. The data refinement unit 100 may select clusters specified by a user via the input / output device 90, or may automatically select clusters that satisfy certain conditions. The conditions here are arbitrary, and may include, for example, a cluster with a predetermined number of data or more, or a proportion of classified data greater than a predetermined threshold. The data group within the selected cluster corresponds to the second data group described above.

[0050] 7 is an explanatory diagram showing an example of a process for selecting some clusters. In the example shown in FIG. 7, two clusters are selected out of the three clusters generated. Note that the data refining unit 100 may assign arbitrary cluster identification information to the data in each cluster so that the clusters classified in the first classification process can be identified in the second process as well.

[0051] Thereafter, the feature extraction unit 60 generates multiple clusters again by learning (unsupervised learning) on ​​the data in the selected cluster (i.e., performs a re-learning process). This process corresponds to the second classification process described above, and the generated multiple clusters correspond to the second multiple clusters. Note that the feature extraction unit 60 may also perform learning by separately adding new data. This allows for deeper digging into the data in the cluster, which is expected to result in more detailed data classification.

[0052] Then, after the second classification process, the visualization processing unit 80 outputs data included in the second plurality of clusters that was classified into a different cluster in the first plurality of clusters, similar to the first process described above. Note that, since there is a possibility that the selected cluster will be subdivided, the visualization processing unit 80 may output data in a cluster whose cluster identification information is in the minority (other than the largest proportion) as data that was classified into a different cluster in the first plurality of clusters.

[0053] Next, the third processing will be described. The third processing is a processing in which at least a part of the data that has not been classified into a cluster, such as an outlier, is excluded, and learning (unsupervised learning or supervised learning) is performed again. The data refinement unit 100 generates a data group as a second data group by excluding one or more pieces of data that have not been classified into any of the first plurality of clusters from the data group to be labeled.

[0054] FIG. 8 is an explanatory diagram showing an example of processing to exclude a portion of data. In the example shown in FIG. 8, data within the range enclosed by a solid circle is excluded as an outlier. For example, if the data to be labeled is video data, this corresponds to processing to exclude noise scenes. Thereafter, at least one of the first processing and the second processing described above, or both, is performed. This is expected to improve classification accuracy.

[0055] The above describes three types of processing performed by the data refinement unit 100. However, the processing performed by the data refinement unit 100 is not limited to the above three types of processing. The data refinement unit 100 may also perform data maintenance processing. Furthermore, after each of the first processing, second processing, and third processing, the same processing may be performed again, or a different processing may be performed.

[0056] An example of the data maintenance process is the process of maintaining data to be used for learning by the feature extraction unit 60. The data refinement unit 100 may output a file including a data group with labels added or a data group with outliers removed.

[0057] For example, suppose that a group of data to be labeled has been labeled in the first process described above. In this case, the data refinement unit 100 may perform processes such as creating a label file containing the specified labels, copying only the labeled data to a folder for next learning, and sorting (moving and copying) the original data into folders for each label based on the labels.

[0058] Also, for example, if clusters are selected in the second process described above, the data refining unit 100 may perform processes such as creating a data list file that lists only the data belonging to the selected clusters, or copying only the data belonging to the selected clusters to a folder for next learning.

[0059] Also, for example, if a process of excluding outliers is performed in the third process described above, the data refining unit 100 may perform processes such as creating a data list file that lists only data other than the specified data (outliers) and copying the data other than the specified data (outliers) to a folder for next learning.

[0060] Below, we will specifically explain how the visualization processing unit 80 visualizes data that has come to belong to a different cluster as a result of relearning. First, the visualization processing unit 80 reduces the dimension of the data group to be labeled, and plots a graph in a manner that allows the dimension-reduced data included in the first plurality of clusters and the dimension-reduced data included in the second plurality of clusters to be distinguished by cluster. Then, the visualization processing unit 80 displays, among the dimension-reduced data included in the second plurality of clusters, data that was classified into a different cluster from the first plurality of clusters in a manner different from the other data.

[0061] Examples of different modes include changing the shade of the color, changing the color itself, changing the outline of the outer frame, and blinking the display.

[0062] FIG. 9 is an explanatory diagram showing an example of overlay display of results before and after refinement. The example shown in FIG. 9 shows that the visualization processing unit 80 displays the distribution of data for each refinement in an overlapping manner, and also displays data other than the layer of interest (i.e., refinement) in a manner different from the data of the layer of interest. Specifically, in the example shown in FIG. 9, the results of the first refinement and the results of the second refinement are displayed in an overlapping manner. In this case, when focusing on the results of the first refinement, data d1 included in the target cluster only in the second refinement is displayed in a manner different from the other data. Similarly, when focusing on the results of the second refinement, data d2 included in the target cluster only in the first refinement is displayed in a manner different from the other data.

[0063] 10 and 11 are explanatory diagrams showing examples in which the results before and after refinement are displayed in parallel windows. As shown in Fig. 10, the visualization processing unit 80 may display the results before and after refinement in separate windows. In this case, as shown in Fig. 11, the visualization processing unit 80 may display the state of data that has changed before and after refinement in a different state from other data.

[0064] Furthermore, the visualization processing unit 80 may display a list of data for which the results before and after refinement differ (i.e., data classified into different clusters). Fig. 12 is an explanatory diagram showing an example in which data d3 for which the results before and after refinement differ is displayed in a list in a separate window. In the example shown in Fig. 12, the results are shown by displaying a list of coordinates at which data for which the results before and after refinement differ are displayed.

[0065] 9 to 12 illustrate an example in which two refinement results are compared. However, the comparison targets are not limited to two results, and may be three or more results. FIG. 13 is an explanatory diagram showing an example in which multiple refinement results are overlaid. FIG. 14 is an explanatory diagram showing an example in which data that has resulted in different results after multiple refinements is listed in a separate window. The example shown in FIG. 13 is an example in which there are four refinement results, compared to the example shown in FIG. 9. Similarly, the example shown in FIG. 14 is an example in which there are four refinement results, compared to the example shown in FIG. 12.

[0066] Furthermore, the visualization processing unit 80 may display statistical information of clusters for each classification process (i.e., refinement) of the data group, separately from or together with the above-described graph. Note that the creation of the statistical information may be performed by the visualization processing unit 80 or the feature extraction unit 60.

[0067] Fig. 15 is an explanatory diagram showing an example of displaying statistical information for each cluster. In the example shown in Fig. 15, the number of data in a cluster, the center of gravity of the data, and the variance (x direction and y direction) are displayed as statistical information for the cluster. As shown in Fig. 15, the visualization processing unit 80 may switch between displaying the statistical information for each refinement, or may display the information side by side.

[0068] FIG. 16 is an explanatory diagram showing another example of displaying statistical information for each cluster. As shown in FIG. 16, the visualization processing unit 80 may display statistical information for the cluster (e.g., false positive rate) in graph and table format. The example shown in FIG. 16 shows the degree of agreement between the label and the assigned cluster when supervised learning is performed. Note that in the example shown in FIG. 16, unsupervised learning is assumed for the first time, and no evaluation results are available.

[0069] The data acquisition unit 10, related information acquisition unit 20, object identification unit 30, data processing unit 40, text information input unit 50, feature extraction unit 60, visualization processing unit 80, and data refinement unit 100 are realized by a computer processor (e.g., a CPU (Central Processing Unit)) that operates according to a program (labeling support program).

[0070] For example, the program may be stored in a storage unit (not shown) of the labeling support system 1, and the processor may read the program and operate in accordance with the program as the data acquisition unit 10, the related information acquisition unit 20, the object identification unit 30, the data processing unit 40, the text information input unit 50, the feature extraction unit 60, the visualization processing unit 80, and the data refinement unit 100. Furthermore, the functions of the labeling support system 1 may be provided in a SaaS (Software as a Service) format.

[0071] The data acquisition unit 10, related information acquisition unit 20, object identification unit 30, data processing unit 40, text information input unit 50, feature extraction unit 60, visualization processing unit 80, and data refinement unit 100 may each be realized by dedicated hardware. Furthermore, some or all of the components of each device may be realized by general-purpose or dedicated circuits, processors, etc., or a combination thereof. These may be configured by a single chip, or by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and a program.

[0072] Furthermore, when some or all of the components of the labeling support system 1 are realized by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, etc., in a form in which each device is connected via a communication network.

[0073] Next, an operation of the labeling support system 1 of this embodiment will be described. Fig. 17 is a flowchart showing an operation example of the labeling support system 1. The operation example shown in Fig. 17 is an operation example in which the data acquisition unit 10 directly acquires data in a format used by the feature extraction unit 60 and inputs the acquired data to the feature extraction unit 60.

[0074] The feature extraction unit 60 generates a first plurality of clusters from a data group to be labeled (a first data group) (step S11). Thereafter, the feature extraction unit 60 generates a second plurality of clusters from a data group (a second data group) including at least a portion of the data to be labeled (step S12). Then, the visualization processing unit 80 outputs data included in the second plurality of clusters that was classified into a different cluster in the first plurality of clusters (step S13).

[0075] As described above, in this embodiment, the feature extraction unit 60 generates a first plurality of clusters by classifying a first data group through unsupervised learning. The feature extraction unit 60 also generates a second plurality of clusters by classifying a second data group. The visualization processing unit 80 then outputs data included in the second plurality of clusters that was classified into a different cluster from the first plurality of clusters. This can assist in the labeling process for the clusters into which unlabeled data has been classified.

[0076] In this embodiment, the data refining unit 100 generates a second data group from the data group to be labeled in accordance with the generated first plurality of clusters, thereby improving the accuracy of re-learning using the generated second data group.

[0077] Next, an overview of the present invention will be described. Fig. 18 is a block diagram showing an overview of a labeling support system according to the present invention. A labeling support system 180 (e.g., labeling support system 1) according to the present invention includes first classification means 181 (e.g., feature extraction unit 60) that generates a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning, second classification means 182 (e.g., feature extraction unit 60) that generates a second plurality of clusters by classifying (i.e., relearning) a second data group, which is a data group including at least a portion of the data to be labeled, and output means 183 (e.g., visualization processing unit 80) that outputs data included in the second plurality of clusters that was classified into a different cluster in the first plurality of clusters.

[0078] Such a configuration can assist in labeling the unlabeled data into classified clusters.

[0079] The labeling support system 180 may also include a data refinement means (e.g., a data refinement unit 100) that generates a second data group from the data group to be labeled in accordance with the first plurality of clusters generated.

[0080] Specifically, the data refinement means may generate a second data group by labeling data classified into one of a first plurality of clusters from the data group to be labeled (for example, the first processing by the data refinement unit 100).

[0081] In addition, the data refinement means may generate a second data group from the data group to be labeled, the second data group being a data group classified into a cluster selected from the first plurality of clusters (for example, a second process by the data refinement unit 100).

[0082] In addition, the data refinement means may generate a second data group by excluding one or more pieces of data from the data group to be labeled that are not classified into any of the first plurality of clusters (for example, the third processing by the data refinement unit 100).

[0083] The output means may also reduce the dimension of the data group to be labeled, plot the dimension-reduced data contained in the first plurality of clusters and the dimension-reduced data contained in the second plurality of clusters in a manner that allows each cluster to be identified, and display, among the dimension-reduced data contained in the second plurality of clusters, data that was classified into a different cluster in the first plurality of clusters in a manner different from the other data.

[0084] The output means may also display statistical information of the cluster for each classification process of the data group.

[0085] 19 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 1000 includes a processor 1001, a main memory device 1002, an auxiliary memory device 1003, and an interface 1004.

[0086] The above-described labeling support system 180 is implemented in a computer 1000. The operations of the above-described processing units are stored in the form of a program (labeling support program) in an auxiliary storage device 1003. The processor 1001 reads the program from the auxiliary storage device 1003, loads it into the main storage device 1002, and executes the above-described processing in accordance with the program.

[0087] In at least one embodiment, the auxiliary storage device 1003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM (Compact Disc Read-only memory), a DVD-ROM (Read-only memory), and a semiconductor memory connected via the interface 1004. In addition, when this program is distributed to the computer 1000 via a communication line, the computer 1000 that receives the program may load the program into the main storage device 1002 and execute the above processing.

[0088] The program may also be a program for realizing part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-described functions in combination with another program already stored in the auxiliary storage device 1003.

[0089] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0090] (Supplementary Note 1) A first classification means for generating a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning; a second classification means for classifying a second data group, which is a data group including at least a part of the data to be labeled, to generate a second plurality of clusters; and an output means for outputting data included in the second plurality of clusters that has been classified into a different cluster in the first plurality of clusters. A labeling support system characterized by:

[0091] (Appendix 2) A data refinement means for generating a second data group from the data group to be labeled in accordance with the first plurality of clusters generated. Attachment 1: A labeling assistance system.

[0092] (Appendix 3) The data refinement means generates a second data group by labeling data classified into one of the first plurality of clusters from the data group to be labeled. 1. A labeling assistance system according to claim 1 or 2.

[0093] (Appendix 4) The data refinement means generates, as a second data group, a data group classified into a cluster selected from the first plurality of clusters from among the data group to be labeled. 1. A labeling assistance system according to claim 1 or 2.

[0094] (Appendix 5) The data refinement means generates a second data group by excluding one or more pieces of data that are not classified into any of the first plurality of clusters from the data group to be labeled. 10. A labeling assistance system according to any one of appendices 1 to 4.

[0095] (Supplementary Note 6) The output means reduces the dimensions of the data group to be labeled, plots the dimension-reduced data included in the first plurality of clusters and the dimension-reduced data included in the second plurality of clusters in a manner that allows each cluster to be identified, and displays, among the dimension-reduced data included in the second plurality of clusters, data that has been classified into a different cluster from the first plurality of clusters in a manner different from the other data. 6. A labeling assistance system according to any one of appendices 1 to 5.

[0096] (Appendix 7) The output means displays statistical information of clusters for each classification process of data groups. 7. A labeling assistance system according to any one of appendices 1 to 6.

[0097] (Appendix 8) A computer generates a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning; the computer generates a second plurality of clusters by classifying a second data group that is a data group including at least a portion of the data to be labeled; The computer outputs data included in the second plurality of clusters that has been classified into a different cluster in the first plurality of clusters. A labeling support method comprising:

[0098] (Appendix 9) A second data group is generated from the data group to be labeled according to the generated first plurality of clusters. The labeling assistance method described in Appendix 8.

[0099] (Appendix 10) To the computer, a first classification process for generating a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning; a second classification process for generating a second plurality of clusters by classifying a second data group that is a data group including at least a portion of the data to be labeled; and an output process for outputting data included in the second plurality of clusters that has been classified into a different cluster in the first plurality of clusters; A program storage medium that stores a labeling assistance program for executing the above.

[0100] (Appendix 11) To the computer, Execute a data refinement process to generate a second data group from the data group to be labeled according to the generated first plurality of clusters. 11. A program storage medium according to claim 10, storing a labeling assistance program for:

[0101] (Appendix 12) To the computer, a first classification process for generating a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning; a second classification process for generating a second plurality of clusters by classifying a second data group that is a data group including at least a portion of the data to be labeled; and an output process for outputting data included in the second plurality of clusters that has been classified into a different cluster in the first plurality of clusters; A labeling support program to help implement this.

[0102] (Appendix 13) To the computer, Execute a data refinement process to generate a second data group from the data group to be labeled according to the generated first plurality of clusters. Labeling assistance program as described in Appendix 12.

[0103] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0104] 1. Labeling support system 10 Data Acquisition Section 20 Related Information Acquisition Department 30 Object Recognition Unit 40 Data Processing Department 50 Text information input section 60 Feature Extraction Unit 70 Feature Memory Unit 80 Visualization processing unit 90 Input / Output Devices 100 Data Refinement Department

Claims

1. a first classification means for generating a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning; a second classification means for classifying a second data group, which is a data group including at least a part of the data to be labeled, to generate a second plurality of clusters; and an output means for outputting data included in the second plurality of clusters that has been classified into a different cluster in the first plurality of clusters. A labeling support system characterized by:

2. A data refinement means for generating a second data group from the data group to be labeled in accordance with the generated first plurality of clusters is provided. The labeling support system according to claim 1 .

3. The data refinement means generates a second data group by labeling data classified into one of the first plurality of clusters from the data group to be labeled. The labeling support system according to claim 2.

4. The data refinement means generates, as a second data group, a data group classified into a cluster selected from the first plurality of clusters from among the data group to be labeled.

3. The labeling support system according to claim 1 or 2.

5. The data refinement means generates a second data group by excluding one or more pieces of data that are not classified into any of the first plurality of clusters from the data group to be labeled. The labeling support system according to any one of claims 1 to 4.

6. The output means reduces the dimensions of the data group to be labeled, plots the dimension-reduced data included in the first plurality of clusters and the dimension-reduced data included in the second plurality of clusters in a manner that allows each cluster to be identified, and displays, among the dimension-reduced data included in the second plurality of clusters, data that has been classified into a different cluster from the first plurality of clusters in a manner different from the other data. The labeling support system according to any one of claims 1 to 5.

7. The output means displays statistical information of the clusters for each classification process of the data group. The labeling support system according to any one of claims 1 to 6.

8. a computer classifying a first data group, which is a data group to be labeled, by unsupervised learning to generate a first plurality of clusters; the computer generates a second plurality of clusters by classifying a second data group that is a data group including at least a portion of the data to be labeled; The computer outputs data included in the second plurality of clusters that has been classified into a different cluster in the first plurality of clusters. A labeling support method comprising:

9. A second data group is generated from the data group to be labeled according to the generated first plurality of clusters. The labeling support method according to claim 8.

10. On the computer, a first classification process for generating a first plurality of clusters by classifying a first data group, which is a data group to be labeled, through unsupervised learning; a second classification process for generating a second plurality of clusters by classifying a second data group that is a data group including at least a portion of the data to be labeled; and an output process for outputting data included in the second plurality of clusters that has been classified into a different cluster in the first plurality of clusters; A labeling support program to help implement this.

Images