Teaching support device, teaching support method, and teaching support program

Abstract

To provide a teaching work support device, a teaching work support method, and a teaching work support program that can support label confirmation work.SOLUTION: A teaching work support device includes: a label acquisition part; a data acquisition part; a feature amount extraction part; a label assignment part; a coincidence calculation part; and an output control part. The label acquisition part acquires a first label which has been assigned to data. The data acquisition part acquires data. The feature amount extraction part extracts a feature amount of data. The label assignment part groups data on the basis of similarity or distance of a feature amount, and assigns a second label to the grouped data. The coincidence calculation part calculates a degree of coincidence of the first label and the second label. The output control part outputs information related to a combination of the first label and the second label of low coincidence.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This embodiment relates to a teaching support device, a teaching support method, and a teaching support program. [Background technology]

[0002] In the manufacturing sector, there is a growing trend to improve productivity by using machine learning techniques to classify product images to monitor the occurrence of product defects and flaws. One machine learning technique for classifying product images is supervised learning, which generates a classification model by training a model using deep learning or other methods on product images that have been pre-labeled manually. In supervised learning, accurately labeling a large number of images is necessary to generate a highly accurate classification model. Therefore, the labor cost of labeling in supervised learning tends to be high. Technologies that support labeling in such supervised learning are also known. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 6307822 [Overview of the project] [Problems that the invention aims to solve]

[0004] When labels are applied manually, there can be inconsistencies in the judgment of the person applying the label. Due to these inconsistencies, it is possible that similar images may be labeled as defective in one instance and as not defective in another, resulting in contradictory labels. In this case, it is conceivable to appoint a label manager to ensure label quality. While the establishment of a label manager would maintain label quality, checking the labels applied to a large number of images is a difficult task for the manager.

[0005] An embodiment provides an instruction work support device, an instruction work support method, and an instruction work support program that can assist in the label confirmation work.

Means for Solving the Problems

[0006] The instruction work support device of the embodiment includes a label acquisition unit, a data acquisition unit, a feature amount extraction unit, a label assignment unit, a degree of coincidence calculation unit, and an output control unit. The label acquisition unit acquires a first label assigned to the data. The data acquisition unit acquires the data. The feature amount extraction unit extracts the feature amount of the data. The label assignment unit groups the data based on the similarity or distance of the feature amounts, and assigns a second label to the grouped data. The degree of coincidence calculation unit calculates the degree of coincidence between the first label and the second label. The output control unit outputs information regarding the combination of the first label and the second label with a low degree of coincidence. The output control unit displays plot points representing the data using features on the display screen of the display device, and highlights the plot points representing the data corresponding to combinations of first and second labels with low degree of agreement.

Brief Description of the Drawings

[0007] [Figure 1] FIG. 1 is a block diagram showing the overall configuration of an instruction work support device according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an example of an integrated label as an integration result of an AI label and a manual label. [Figure 3] FIG. 3 is a diagram showing an example of the degree of coincidence. [Figure 4] FIG. 4 is a diagram showing an example of the hardware configuration of the instruction work support device. [Figure 5] FIG. 5 is a flowchart showing the operation of the instruction work support device. [Figure 6A] FIG. 6A is a diagram showing an example of a display screen of plot points with an AI label displayed based on two-dimensional plot feature amounts as an output screen for information regarding label contradiction. <明 [Figure 6B] FIG. 6B is a diagram showing an example of a display screen of plot points with a manual label displayed based on two-dimensional plot feature amounts as an output screen for information regarding label contradiction. [Figure 7A] Figure 7A shows an example of the display screen for plot points with AI labels, shown in two dimensions, in Modification Example 1. [Figure 7B] Figure 7B shows an example of a display screen for plot points with hand labels, shown in two dimensions, in Modification Example 1. [Figure 8] Figure 8 is a block diagram showing the overall configuration of the teaching support device according to Modification Example 2. [Figure 9A] Figure 9A shows an example of a display screen for manually labeled plot points in Modification Example 2. [Figure 9B] Figure 9B shows an example of a temporary AI label. [Figure 9C] Figure 9C shows an example of the degree of agreement between temporary AI labels and human-generated labels. [Figure 9D] Figure 9D shows an example of highlighting plot points corresponding to images with a low degree of match to the temporary AI label. [Figure 10A] Figure 10A shows an example of a display screen for manually labeled plot points in Modification Example 3. [Figure 10B] Figure 10B shows an example of a listed image. [Figure 10C] Figure 10C shows an example of highlighting plot points in images where the same type of label discrepancy may occur as with images corresponding to plot points selected by the user. [Figure 11] Figure 11 shows an example of a combined manual label. [Figure 12] Figure 12 shows an example of the degree of agreement. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. Figure 1 is a block diagram showing the overall configuration of a teaching support device according to one embodiment. The teaching support device of the embodiment assists in the teaching work of assigning labels to training data used in machine learning. In particular, the teaching support device of the embodiment assists the user in confirming the labels assigned by the teaching work. The user is, for example, a label manager. On the other hand, the user is not necessarily limited to a label manager, but may be an operator who assigns labels. Also, in the following, the training data is assumed to be images. However, the training data may be any data used in machine learning. For example, the training data may be audio, documents, etc.

[0009] The teaching support device 1 according to this embodiment includes a label acquisition unit 101, a data acquisition unit 102, a feature extraction unit 103, a label assignment unit 104, a degree of agreement calculation unit 105, and an output control unit 106.

[0010] The label acquisition unit 101 acquires the labels assigned to the images. The label acquisition unit 101 may acquire the labels assigned to each of multiple images at once. Furthermore, the labels assigned by the label acquisition unit 101 may be labels assigned by any method. For example, the labels may be manually assigned labels, or labels may be assigned using a classification model learned by a machine learning method. The classification model is a machine learning model that classifies images according to their features and assigns labels according to the classified images. The classification model may be trained using supervised learning or unsupervised learning. In the following explanation, the labels acquired by the label acquisition unit 101 are assumed to be manually assigned labels by an operator. Also, for the purposes of the following explanation, the manually assigned label to the i-th image by operator j is y ij Let it be represented as follows: Here, j is an integer from 1 to M. Also, i is an integer from 1 to N. In other words, in this embodiment, manual labels can be assigned to N images by M workers. Also, manual label y ij The value represents the type of label assigned to the image.

[0011] The data acquisition unit 102 acquires the actual data of the i-th image to which the label acquired by the label acquisition unit 101 is attached. For example, the image is, for example, an appearance image of a product obtained by a product inspection device. The resolution of the image may be any resolution such as, for example, 128 pixels × 128 pixels. The data acquisition unit 102 may acquire an image data set constituted by arranging the data of a plurality of images.

[0012] The feature extraction unit 103 extracts a feature amount of a lower dimension than the feature amount of the original image from the actual data of the image acquired by the data acquisition unit 102. Extraction of the low-dimensional feature amount can be performed by a label-free representation learning method using a deep neural network as proposed in, for example, "Clustering Friendly representation learning via instance discrimination and feature decorrelation, Yaling Tao, Kentaro Takagi, Kouta Nakata. arXiv:2106.00131 (ICLR2021)". For the following explanation, the feature amount extracted from the actual data of the i-th image is X i =(x i1 , x i2 , …, x ik ) is represented as. Here, k is the number of dimensions of the feature amount extracted by the feature extraction unit 103. For example, when the feature extraction unit 103 is configured to extract a 64-dimensional feature amount from the actual data of the image, k = 64.

[0013] The labeling unit 104 groups the images using the feature amount X i extracted by the feature extraction unit 103. Then, a label is attached to the images in each group. The labeling unit 104 determines the similarity between the feature amounts X i of each image or the feature amounts X iImages are grouped using the distance between them. Various unsupervised learning methods, such as clustering, can be used for grouping. For example, the k-means method can be used for clustering. For the purposes of the following explanation, the labels assigned to images by the labeling unit 104 are called AI labels. Also, the AI ​​label assigned to the i-th image is y iA Let's assume it is represented as follows: AI label y iA The value of can be, for example, the cluster number obtained by the k-means algorithm. Furthermore, the number of groups L' formed by the k-means algorithm may be equal to the number of types of labels L assigned as manual labels. As will be explained later, L' may also be greater than L.

[0014] The matching degree calculation unit 105 generates an integrated label by combining the AI ​​label and the manual label, and extracts information about label inconsistencies by calculating the matching degree between the AI ​​label and the manual label from the integrated label.

[0015] Figure 2 shows an example of a combined label resulting from the integration of AI labels and manual labels. In Figure 2, the integration is performed by associating the AI ​​label and manual label assigned to the i-th image. Here, in Figure 2, the AI ​​label is one of the labels 1, 2, or 3 corresponding to the cluster number. On the other hand, the manual label is one of the labels A, B, or C. As shown in Figure 2, the AI ​​label and manual label do not need to be assigned in the same format.

[0016] Also, AI label y iA and a manual label y assigned by worker j ij The degree of agreement can be calculated, for example, according to equation (1).

number

[0017] Figure 3 shows an example of the degree of agreement. In Figure 3, the combinations of AI labels and human labels are sorted in descending order of degree of agreement. Note that when the number of types of human labels is L and the number of groups of AI labels is L', there can be L' × L possible combinations of AI labels and human labels. However, the degree of agreement for combinations that do not exist as a result of the assignment does not need to be calculated. On the other hand, the degree of agreement for combinations that do not exist as a result of the assignment may be calculated as, for example, 0.

[0018] The output control unit 106 outputs information about label inconsistencies that should be presented to the user based on the degree of agreement. The information about label inconsistencies concerns combinations of AI labels and manual labels with a low degree of agreement. A combination of AI labels and manual labels with a low degree of agreement means that the labeling unit 104 and the operator have made different evaluations. In this case, the operator may have assigned different labels to similar images. In this embodiment, a state in which the labeling unit 104 and the operator have made different evaluations is considered a state in which there is a possibility of label inconsistency. For example, if the numbers of images with a low degree of agreement are displayed in a list format, the user can find candidate images for which label inconsistencies should be checked. Here, the output of information about label inconsistencies can be done, for example, by displaying the information about label inconsistencies on a display device. In addition, the output of information about label inconsistencies can be done by any method, such as printing the information about label inconsistencies on paper from a printing device.

[0019] Figure 4 shows an example of the hardware configuration of the teaching support device 1. The teaching support device 1 is a computer and has, for example, a processor 201, memory 202, input device 203, display device 204, communication device 205, and storage 206 as hardware. The processor 201, memory 202, input device 203, display device 204, communication device 205, and storage 206 are connected to a bus 207.

[0020] The processor 201 is a processor that controls the overall operation of the teaching support device 1. The processor 201 operates as a label acquisition unit 101, a data acquisition unit 102, a feature extraction unit 103, a label assignment unit 104, a matching degree calculation unit 105, and an output control unit 106 by executing a program stored in the storage 206, for example. The processor 201 is, for example, a CPU. The processor 201 may also be an MPU, GPU, ASIC, FPGA, etc. The processor 201 may be a single CPU, etc., or multiple CPUs, etc.

[0021] Memory 202 includes ROM and RAM. ROM is non-volatile memory. ROM stores the startup program for the teaching support device 1, etc. RAM is volatile memory. RAM is used, for example, as working memory during processing in processor 201.

[0022] The input device 203 is an input device such as a touch panel, keyboard, or mouse. When the input device 203 is operated, a signal corresponding to the operation is input to the processor 201 via the bus 207. The processor 201 performs various processes in response to this signal.

[0023] The display device 204 is a display device such as a liquid crystal display or an organic EL display, as well as various information output devices such as a printer.

[0024] The communication device 205 is a communication device for the teaching support device 1 to communicate with external equipment. The communication device 205 may be a communication device for wired communication or a communication device for wireless communication.

[0025] Storage 206 is a storage device such as a hard disk drive or a solid-state drive. Storage 206 stores various programs executed by the processor 201, such as the teaching support program 2061.

[0026] Storage 206 also stores the integrated label 2062 and the matching score 2063. The integrated label 2062 is data generated as a result of integrating the AI ​​label and the human label. The matching score 2063 is the degree of matching between the AI ​​label and the human label calculated from the integrated label 2062.

[0027] Bus 207 is a data transfer path for exchanging data between the processor 201, memory 202, input device 203, display device 204, communication device 205, and storage 206.

[0028] Next, the operation of the teaching support device 1 will be explained. Figure 5 is a flowchart showing the operation of the teaching support device 1. The process in Figure 5 is performed, for example, each time manual label information is input. In the following explanation, the manual label is assumed to be one of labels A, B, or C. The AI ​​label is assumed to be one of labels 1, 2, or 3 corresponding to the cluster number. Furthermore, in the following explanation, the manual label is assumed to have been assigned by only one worker.

[0029] In step S1, the label acquisition unit 101 acquires a manually written label. The data acquisition unit 102 acquires the actual data of the image to which the manually written label acquired by the label acquisition unit 101 is attached. The manually written label and the actual image data are input, for example, by the operation of the user's input device 203. Alternatively, if the manually written label and the actual image data are recorded as a single file, the manually written label and the actual image data may be acquired by inputting that file. The acquired manually written label and the actual image data are stored, for example, in memory 202.

[0030] In step S2, the feature extraction unit 103 extracts feature X from the actual image data. i The key features are extracted. Feature extraction can be performed, for example, by using the deep neural network method described above.

[0031] In step S3, the labeling unit 104 extracts feature X from the actual data of the i-th image. i Based on this, an AI label is assigned to the i-th image. The labeling unit 104 assigns an AI label to the i-th image's feature quantity X i The labeling unit 104 calculates the similarity or distance between the features of the i-th image and other images belonging to each group. The labeling unit 104 then groups the i-th image based on the similarity or distance of the features. The labeling unit 104 then assigns an AI label to the i-th image depending on which group it was grouped into.

[0032] In step S4, the degree of agreement calculation unit 105 generates a combined label by integrating the AI ​​label and the manual label. The degree of agreement calculation unit 105 then calculates the degree of agreement from the combined label based on formula (1). For example, when the manual label is one of labels A, B, or C, and the AI ​​label is one of labels 1, 2, or 3, the degree of agreement calculation unit 105 calculates the degrees of agreement r(A,1), r(A,2), r(A,3), r(B,1), r(B,2), r(B,3), r(C,1), r(C,2), and r(C,3), respectively. As mentioned above, the degree of agreement for combinations that do not exist does not need to be calculated.

[0033] In step S5, the output control unit 106 outputs information regarding label inconsistencies. After that, the process shown in Figure 5 is completed.

[0034] For example, the output control unit 106 may display labeled plot points representing each image on the display device 204 using feature quantities. In this case, the output control unit 106 converts the feature quantities of each image into lower-dimensional plot features so that the feature quantities of each image can be displayed as plot points. For example, the k-dimensional feature quantity X extracted from the i-th image. i This is a 2D plot feature X' i =(x' i1 ,x' i2 It can be converted to ). Conversion to 2D features can be achieved using dimensionality reduction methods that utilize unsupervised learning clustering, such as t-SNE (t-Distributed Stochastic Neighbor Embedding) and UMAP (Uniform Manifold Approximation and Projection). Note that methods such as t-SNE and UMAP can also reduce to a number of dimensions other than 2. The number of dimensions of the plotted features should be such that the image can be displayed as plot points. Therefore, the number of dimensions of the plotted features may be, for example, 3.

[0035] Figure 6A shows an example of a display screen showing AI-labeled plot points based on 2D plot features, which serves as an output screen for information regarding label inconsistencies. Each plot point PA corresponds to one image. The horizontal display position of each plot point PA is, for example, determined by the plot feature x'. i1 It supports this, and the vertical display position is, for example, the plotted feature x' i2 It corresponds to the following. And, one of the numbers 1, 2, or 3 representing the AI ​​label is displayed at each plot point PA. The user can check the distribution of AI labels assigned by the labeling unit 104 by looking at the screen shown in Figure 6A. Here, if any of the plot points PA shown in Figure 6A are selected, the corresponding image may be displayed.

[0036] Figure 6B shows an example of a display screen showing manually labeled plotted points based on two-dimensional plotted features, which serves as an output screen for information regarding label inconsistencies. The user can switch from the display screen in Figure 6A to the display screen in Figure 6B, for example, by operating the user's input device 203.

[0037] Similar to plotted points with AI labels, each plotted point P11 corresponds to a single image. The horizontal display position of each plotted point P11 is, for example, the plotted feature x'. i1 It supports this, and the vertical display position is, for example, the plotted feature x' i2 This corresponds to the same image. Therefore, plot point PA and plot point P11 for the same image are displayed in the same position on the display screen of the display device 204. Each plot point P11 displays one of the letters A, B, or C, which represent a human label. The user can check the distribution of human labels assigned by worker 1 by looking at the display screen shown in Figure 6B. Here, if any of the plot points P11 shown in Figure 6B are selected, the corresponding image may be displayed.

[0038] Furthermore, in the display shown in Figure 6B, plot point P12, which has a low degree of agreement with the AI-labeled plot points, for example, an agreement of 0.5 or less, is highlighted more than the other plot points. For example, in Figure 6B, plot point P12 is displayed with a different type of line than the other plot points. By looking at the display screen shown in Figure 6B, the user can easily identify images where label inconsistencies may be occurring. Here, highlighting is not necessarily limited to using a different type of line. For example, highlighting may be done by using a different color, a different shape, or a different size from the other plot points.

[0039] Here, the display screens in Figure 6A and Figure 6B are not necessarily configured to be switchable. For example, the display screens in Figure 6A and Figure 6B may be displayed side by side on a single screen.

[0040] Furthermore, the output of information regarding label inconsistencies is not limited to a two-dimensional display of plotted points. Information regarding label inconsistencies may also be output by displaying a list of combinations with low degree of agreement on the display device 204. Additionally, the output may not be a display method such as printing.

[0041] As described above, according to this embodiment, the degree of agreement between the first label, which is manually assigned by the label acquisition unit 101, and the second label, which is assigned by the label assignment unit 104 based on the image to which the first label is assigned, is calculated by the degree of agreement calculation unit 105. Based on the degree of agreement, information regarding label inconsistencies is output. This allows the user to notice label inconsistencies without having to review the actual image. In this way, the workload for the user in checking labels can be reduced in this embodiment.

[0042] Furthermore, the labeling unit 104 assigns a second label to the image using an unsupervised learning method such as clustering. In other words, there is no human intervention in the labeling process by the labeling unit 104. Therefore, even when labeling images during the launch of a new product or inspection process, there is little variation in the labeling decisions. Moreover, labels are assigned without requiring knowledge of the new product or inspection process, as is the case when labels are assigned manually, thus ensuring the quality of the assigned labels.

[0043] The following describes some modifications of the embodiment. [Modification Example 1: Automatic setting of the number of groups] In the embodiment described above, the number of image data groups L' grouped by the labeling unit 104 is assumed to be the same as the number of types of manual labels L. In reality, the number of image data groups L' only needs to be greater than or equal to the number of types of manual labels L. That is, the number of groups L' may be set to L' = L + K using a predetermined K (where K is an integer of 1 or more).

[0044] Figures 7A and 7B illustrate the effect of setting the number of groups to be greater than L. Figure 7A shows an example of the display screen for plot points with AI labels displayed in two dimensions in Modification Example 1. On the other hand, Figure 7B shows an example of the display screen for plot points with manual labels displayed in two dimensions in Modification Example 1.

[0045] Here, in Modification 1, L is 2. In this case, the image is assigned either manual label A or B. On the other hand, in Modification 1, K is 1, meaning L' is 3. Therefore, the image is assigned either AI label 1, 2, or 3. In other words, the number of types of AI labels is one more than the number of types of manual labels. In this case, similar images may be assigned different AI labels. For example, an image assigned manual label B in Figure 7B may be assigned AI label 1 or 2 as shown in Figure 7A. This means that the image assigned manual label B has been further divided into two groups based on its features.

[0046] Even if the number of AI label types is one more than the number of manual label types, the degree of agreement r(A,1), r(A,2), r(A,3), r(B,1), r(B,2), and r(B,3) can still be calculated according to equation (1). Then, plot points with low degree of agreement are highlighted as shown in Figure 7B. Here, even if the images to which manual label B is assigned are further divided into two groups, the degree of agreement within each group can still be calculated correctly. Therefore, images where label inconsistencies occur can be identified, just as when the number of manual label types and the number of AI label types are the same. Specifically, in the example in Figure 7B, the degree of agreement r(A,1), r(A,2), and r(B,3) is considered low. The location of the plot points with low degree of agreement is the same even if we assume that AI label 1 and AI label 2 are the same label.

[0047] Thus, in Modification 1, the number of AI label types can be greater than the number of human label types. A larger number of AI label types, i.e., a larger number of groups, means that features with similar characteristics are grouped together. This is expected to improve the accuracy of AI label assignment. In practice, it is desirable to set L', i.e., K, which is suitable for highlighting label inconsistencies, taking into account processing time and other factors.

[0048] Furthermore, it is undesirable for the number of AI label types to be less than the number of human label types. This is because groups that should be separated may not be separated, resulting in inappropriate AI labeling.

[0049] [Variation 2: Checking for discrepancies in neighboring data labels] In the embodiment described above, AI labels are assigned to all acquired images. Alternatively, AI labels may be assigned only to images similar to the image that the user has focused on.

[0050] Figure 8 is a block diagram showing the overall configuration of the teaching support device according to Modification 2. The teaching support device 1 according to Modification 2 includes a label acquisition unit 101, a data acquisition unit 102, a feature extraction unit 103, a label assignment unit 104, a degree of agreement calculation unit 105, and an output control unit 106, in addition to an operation unit 107. The operation unit 107 accepts the user's operation to select a plot point on a display screen of manually labeled plot points. The operation unit 107 then inputs information that identifies the selected plot point, such as the coordinates of the plot point, to the label assignment unit 104. The operation unit 107 may be configured, for example, by an input device 203.

[0051] In the modified example 2, the labeling unit 104, when a plot point is selected by the operation unit 107, uses feature quantities or plot features to assign temporary AI labels to the images corresponding to the selected plot point and the P plot points in its vicinity.

[0052] The matching degree calculation unit 105 in the modified example 2 calculates the degree of matching between the image to which a temporary AI label has been assigned and the manually written label.

[0053] In the modified example 2, the output control unit 106 highlights plot points corresponding to images with low matching scores among the images to which temporary AI labels have been assigned.

[0054] The operation of Modification 2 will now be explained in detail. Figure 9A shows an example of the display screen for manually labeled plot points in Modification 2. The display screen for manually labeled plot points can be displayed in the same manner as in the previously described embodiment and Modification 1. Here, let's assume that the user has selected plot point P21. In response, the labeling unit 104 searches for images that have features similar to or close in distance to the image features of plot point P21. For example, when P is 10, the labeling unit 104 searches for 10 neighboring points of plot point P21, indicated by point P22 in Figure 9A. Here, P may be a fixed value or may be set by the user, for example.

[0055] After searching for images, the labeling unit 104 assigns temporary AI labels to the searched images. For example, the labeling unit 104 assigns temporary AI label 4 as shown in Figure 9B.

[0056] The matching degree calculation unit 105 calculates the matching degree of the manual label to the temporary AI label. The matching degree between the manual label shown in Figure 9A and the temporary AI label shown in Figure 9B is shown in Figure 9C.

[0057] As shown in Figure 9D, the output control unit 106 highlights plot points P23 corresponding to images with a low degree of match to the temporary AI label.

[0058] In the modified example 2 described above, the user can select an image of interest and, without any cumbersome work, discover images that are similar to that image but may have label inconsistencies.

[0059] In the modified example 2, AI labels may or may not be assigned to images corresponding to plot points other than the plot point selected by the user and its neighbors. If AI labels are assigned, they may be assigned using the same method as in the embodiment described above.

[0060] [Variation 3: Checking for inconsistencies in labels of the same type] In Modification 2, when a plot point with a manual label is selected, a temporary AI label is assigned to the selected plot point and its neighboring plot points. The degree of agreement between the temporary AI label and the manual label is calculated, and plot points with manual labels that have a low degree of agreement with the temporary AI label are highlighted. Alternatively, a plot point with a manual label selected by the user may be considered a plot point where a label discrepancy occurs, and plot points with similar label discrepancies may be highlighted. The block diagram showing the overall configuration of the teaching support device according to Modification 3 may be the same as in Modification 2.

[0061] In the modified example 3, when a plot point is selected by the operation unit 107, the matching degree calculation unit 105 lists images that have the same combination of AI label and human label as the image corresponding to the selected plot point.

[0062] The output control unit 106 outputs a list of images listed by the matching degree calculation unit 105. Alternatively, the output control unit 106 highlights the manually labeled plot points corresponding to the images listed by the matching degree calculation unit 105.

[0063] The operation of Modification 3 will now be explained in detail. Figure 10A shows an example of the display screen for manually labeled plot points in Modification 3. The display screen for manually labeled plot points can be displayed in the same manner as in the previously described embodiment and Modification 1. Here, it is assumed that the user has selected plot point P31. Plot point P31 is a plot point that the user has determined to have a label inconsistency.

[0064] Upon selecting plot point P31, the matching degree calculation unit 105 lists images that have the same combination of AI label and human label as the image of plot point P31. For example, if plot point P31 corresponds to image number 1, the matching degree calculation unit 105 calculates the AI ​​label y associated with image number 1. 1A and manual label y 1j List images with the same combination as the given combination. Figure 10B shows an example of the listed images. For example, the AI ​​label y for number 1. 1A = 1, and manual label y 1j When =A, the degree of agreement calculation unit 105 calculates the AI ​​label y iA = 1, and manual label y ij List the images that are equal to A.

[0065] Based on the results of the matching degree calculation unit 105, the output control unit 106 highlights plot points corresponding to images that may have the same type of label mismatch as the image corresponding to the plot point selected by the user. For example, as shown in Figure 10C, the output control unit 106 highlights plot point P32, which, like plot point P31, has been assigned manual label A and also has been assigned AI label 1. The output control unit 106 may also output the list of images shown in Figure 10B.

[0066] In the modified example 3 described above, the user can find images with similar label inconsistencies to the image they themselves discovered, without any cumbersome work.

[0067] [Variation 4: When manual labels are applied by multiple workers] In the embodiments and modifications described above, it is explained that the manual labels are applied by one worker. In contrast, if the manual labels are applied by two or more workers, the same processing as in the embodiments and modifications described above is performed, for example, by integrating the manual labels applied by multiple people.

[0068] In the modified example 4, the degree of agreement calculation unit 105 creates a combined manual label by combining the manual labels of multiple workers. For example, suppose two workers have assigned manual labels to the i-th image. In this case, the manual label assigned by worker j1 is y ij1 =l j1 , the manual label assigned by worker j2 is y ij2 =l j2 Therefore, the combined manual label y ij1_j2 is, i j1_j2 =l j1 _l j2 Figure 11 shows an example of a combined manual label. In Figure 11, manual labels for two workers, worker 1 and worker 2, are obtained. As shown in Figure 11, for example, if worker 1 assigns manual label A and worker 2 assigns manual label B, the combined manual label y i1_2 It is A_B.

[0069] Furthermore, in the modified example 4, the degree of agreement calculation unit 105 calculates the degree of agreement between the AI ​​label and the combined manual label. The degree of agreement may be calculated according to formula (1). Figure 12 shows an example of the degree of agreement. Similar to the embodiment described above, the degree of agreement calculation unit 105 sorts the combinations of AI labels and combined manual labels in descending order of degree of agreement.

[0070] Here, the agreement rate calculation unit 105 in Modification 4 may further calculate the agreement rate. The agreement rate is the ratio of the total number of workers to the number of workers who assigned the most frequently assigned manual label among the corresponding combined manual labels. For example, regarding the agreement rate for combined manual label A_B, the total number of workers is 2. Also, the most frequently assigned manual labels are both manual label A and manual label B. And the number of workers who assigned manual label A and manual label B is 1 each. Therefore, the agreement rate for combined manual label A_B is 1 / 2.

[0071] The output control unit 106 outputs the list shown in Figure 12, or highlights plot points where the degree of agreement between the AI ​​label and the combined human label is low. This allows the user to easily identify images with potentially inconsistent labels from human labels assigned by multiple workers without cumbersome work. Furthermore, by outputting images with low agreement rates and low degree of agreement at the top of the list, the user can identify images that have been assigned different human labels by different workers.

[0072] Here, Modification 4 shows an example where two workers apply manual labels. Modification 4 can also be applied when three or more workers apply manual labels.

[0073] The instructions shown in the processing procedure described in the above-described embodiment can be executed based on a software program. A general-purpose computer system can store this program in advance and, by reading this program, can obtain the same effects as those of the teaching support device described above. The instructions described in the above-described embodiment are recorded as a program that can be executed by a computer on a magnetic disk (flexible disk, hard disk, etc.), optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, Blu-ray® Disc, etc.), semiconductor memory, or similar recording medium. Any storage format is acceptable as long as it is a recording medium that can be read by a computer or embedded system. The computer can read the program from this recording medium and, based on this program, have the CPU execute the instructions described in the program, thereby achieving the same operation as the teaching support device of the above-described embodiment. Of course, when the computer acquires or reads the program, it may do so via a network. Furthermore, an operating system (OS) running on a computer, a database management software, a network, or other middleware (MW) operating on a computer, based on instructions from a program installed on a computer or embedded system from a recording medium, may execute some of the processes necessary to realize this embodiment. Furthermore, the recording medium in this embodiment is not limited to a medium independent of the computer or embedded system, but also includes recording media that store or temporarily store programs downloaded via LAN, the Internet, etc. Furthermore, the recording medium is not limited to one; even when the processing in this embodiment is performed from multiple media, these are also included as recording media in this embodiment, and the configuration of the media may be any configuration.

[0074] In this embodiment, the computer or embedded system is used to execute each process in this embodiment based on a program stored on a recording medium, and may be configured as any of the following: a single device such as a personal computer or microcomputer, or a system in which multiple devices are connected via a network. Furthermore, the term "computer" in this embodiment is not limited to personal computers, but also includes arithmetic processing units, microcontrollers, and the like included in information processing equipment, and refers collectively to any equipment or device capable of realizing the functions of this embodiment through a program.

[0075] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0076] 1 Teaching support device, 101 Label acquisition unit, 102 Data acquisition unit, 103 Feature extraction unit, 104 Label assignment unit, 105 Matching degree calculation unit, 106 Output control unit, 107 Operation unit, 201 Processor, 202 Memory, 203 Input device, 204 Display device, 205 Communication device, 206 Storage, 207 Bus, 2061 Teaching support program, 2062 Integrated label, 2063 Matching degree.

Claims

1. A label acquisition unit that acquires a first label assigned to the data, A data acquisition unit that acquires the aforementioned data, A feature extraction unit for extracting features from the aforementioned data, A labeling unit that groups the data based on the similarity or distance of the aforementioned features and assigns a second label to the grouped data, A degree of agreement calculation unit that calculates the degree of agreement between the first label and the second label, An output control unit that outputs information regarding combinations of the first label and the second label with a low degree of agreement, It is equipped with, The output control unit, Using the aforementioned features, plot points representing the data are displayed on the display screen of the display device. The plot points representing the data corresponding to the combination of the first label and the second label with a low degree of agreement are highlighted. A teaching support device.

2. The teaching support device according to claim 1, wherein the number of groups to be grouped is set based on the number of types of the first label.

3. The system further comprises an operation unit that accepts the operation of selecting the plot points displayed on the aforementioned display screen, The labeling unit assigns a third label to the first data corresponding to the selected plot point and to the second data corresponding to plot points in the vicinity of the selected plot point. The matching degree calculation unit calculates the matching degree between the first label and the third label, The output control unit outputs information regarding combinations of the first label and the third label with a low degree of matching. The teaching support device according to claim 1.

4. The system further comprises an operation unit that accepts the operation of selecting the plot points displayed on the aforementioned display screen, The matching degree calculation unit lists the second data for which the combination of the first label and the second label is the same as the first data corresponding to the selected plot point. The output control unit highlights the plot points corresponding to the listed second data. The teaching support device according to claim 1.

5. The first label mentioned above is a label that was applied manually. A teaching support device according to any one of claims 1 to 4.

6. The first label mentioned above is a label that has been assigned by multiple people, The aforementioned matching degree calculation unit, A combined label is generated by combining the first labels that were assigned by multiple people. The degree of agreement between the combined label and the second label is calculated. The output control unit outputs information regarding combinations of the combined label with a low degree of match and the second label. The teaching support device according to claim 5.

7. To obtain the first label assigned to the data, To obtain the aforementioned data, Extracting the features of the aforementioned data, The data is grouped based on the similarity or distance of the aforementioned features, and a second label is assigned to the grouped data. The degree of agreement between the first label and the second label is calculated, To output information regarding combinations of the first label and the second label with a low degree of agreement, It is equipped with, Outputting information regarding the combination of the first label and the second label with a low degree of agreement is: The plot points representing the data using the aforementioned features are displayed on the display screen of the display device, To highlight the plot points representing the data corresponding to the combination of the first label and the second label with a low degree of agreement, including, Methods for supporting teaching tasks.

8. To obtain the first label assigned to the data, To obtain the aforementioned data, Extracting the features of the aforementioned data, The data is grouped based on the similarity or distance of the aforementioned features, and a second label is assigned to the grouped data. The degree of agreement between the first label and the second label is calculated, To output information regarding combinations of the first label and the second label with a low degree of agreement, A teaching support program for causing a computer to execute, Outputting information regarding the combination of the first label and the second label with a low degree of agreement is: The plot points representing the data using the aforementioned features are displayed on the display screen of the display device, To highlight the plot points representing the data corresponding to the combination of the first label and the second label with a low degree of agreement, including, A teaching support program.

Images