Content classification method

Abstract

To provide a novel content classification method.SOLUTION: A content classification method using machine learning for a learning model and a classifier fabrication method are provided. In Step 1, a data set containing a plurality of contents is acquired. Learning labels are attached to m contents, and the learning labels are not attached to remaining contents. In Step 2, a first learning model is created by machine learning using the m contents. In Step 3, judgment labels are attached to the plurality of contents using the first learning model and are displayed on a GUI. In Step 4, new learning labels are attached to k contents in the plurality of contents. In Step 5, a second learning model is created by the machine learning using the k contents. In Step 6, judgment labels are attached to the plurality of contents using the second learning model and are displayed on the GUI.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the invention relates to a computer device, a graphical user interface, and a method for classifying content and creating a classifier.

[0002] One aspect of the present invention relates to a computer device. One aspect of the present invention relates to a method for classifying digitized content (text data, image data, audio data, or video data) using a computer device. In particular, one aspect of the present invention relates to a computer device that efficiently classifies a collection of content using machine learning. One aspect of the present invention relates to a content classification method and a classifier creation method using a graphical user interface managed by a program by a computer device. [Background technology]

[0003] Users want to easily extract information related to topics specified by the users from a collection of content. In recent years, new classification systems using learning models trained by machine learning have been proposed. For example, Patent Document 1 discloses a machine learning approach for determining documents that are highly relevant to topics specified by users. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-104630 A Summary of the Invention [Problem to be solved by the invention]

[0005] There are cases where a collection of documents (such as patents or papers) needs to be classified according to a certain purpose. The classification is done using various items such as summaries, keywords, drawings, and various notes that accompany the documents. The accuracy and efficiency of classification depends on the content of the documents in question, but is also likely to vary depending on the experience and skill of the worker. In addition, classification of large volumes of documents requires relying on human labor, which poses a challenge for efficiency.

[0006] When creating a classifier using machine learning, there are issues such as the need to prepare a large amount of training data, which can place an excessive burden on the user, and the fact that variation in the number of classified contents in the training data can affect the accuracy of the classifier.

[0007] In view of the above problems, an object of one embodiment of the present invention is to provide a method for efficiently classifying information. Another object of one embodiment of the present invention is to provide a graphical user interface for efficiently classifying information. Another object of one embodiment of the present invention is to provide a program for efficiently classifying information.

[0008] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract problems other than these from the description of the specification, drawings, claims, etc.

[0009] The problems of one embodiment of the present invention are not limited to the problems listed above. The problems listed above do not preclude the existence of other problems. The other problems are problems not mentioned in this section, which will be described below. Problems not mentioned in this section can be derived by a person skilled in the art from the description of the specification or drawings, etc., and can be appropriately extracted from these descriptions. One embodiment of the present invention solves at least one of the problems listed above and / or other problems. [Means for solving the problem]

[0010] The proposed system uses machine learning, and performs machine learning based on a relatively small amount of training data from the initial set of documents to obtain a learning model, and then classifies the remaining documents using the obtained learning model. The accuracy of the classification results obtained here depends on the quality of the learning model, but on the premise that a satisfactory accuracy cannot be obtained with a single machine learning run, classification, verification, and machine learning can be repeated on the same graphical interface to create a high-quality learning model.

[0011] An operator evaluates whether the classification performed by the machine was appropriate, and the evaluated data is added as training data. Machine learning is then performed again based on the initial training data and the added training data to obtain a second learning model. The accuracy of the second learning model has improved due to the increased training data.

[0012] By repeating these steps, a learning model with sufficient accuracy can be obtained. The present invention also provides a graphical user interface for efficiently performing the steps of obtaining this learning model.

[0013] The classification may be two types or three or more types depending on the purpose of the operator.

[0014] This allows the machine to learn from the training data that the worker has given to some of the data for classification, and enables the worker to classify all the documents in a shorter time than if the worker were to judge all the documents manually or visually. As for the mechanism of machine learning, a known technology may be used, for example, Naive Bayes or Random Forest.

[0015] In the first step, a file containing text data is prepared. Note that a file containing text data may be referred to as content. Content includes not only text data, but also image data, audio data, and video data. Note that an example of content is text contained in a patent document.

[0016] The second step is to read the file and display the text data on the screen.

[0017] In the third step, classifications for a portion of the total data are input to the learning labels as training data. Note that in the second step, a file may be prepared in which a portion of the documents has already been classified as training data (learning labels), and these may be imported as learning labels when the file is read.

[0018] In the fourth step, the learning start button is pressed to obtain a learning model, and the classification is assigned to the judgment label using the learning model. At this time, a label corresponding to the classification is assigned as the judgment label. The label given by the calculation of the learning model is regarded as the judgment label.

[0019] In the fifth step, an operator evaluates the assigned judgment labels and inputs the evaluation results into the learning labels as additional training data.

[0020] In the sixth step, the start learning button is pressed to obtain a learning model, and a classification is assigned to the judgment label using the learning model.

[0021] In the seventh step, steps five and six are repeated until sufficient classification accuracy is achieved.

[0022] One aspect of the present invention is a method for classifying content and a method for creating a classifier, comprising the steps of: acquiring a dataset having a plurality of contents including m pieces of content (m represents a natural number) to which learning labels have been assigned, and n pieces of content (n represents a natural number) to which the learning labels have not been assigned; creating a first learning model by machine learning using the m pieces of content; assigning judgment labels to the plurality of contents using the first learning model and presenting them in a graphical user interface; assigning new learning labels to q pieces of content (q represents a natural number) out of the n pieces of content; creating a second learning model by machine learning using the (q+m) pieces of content to which the learning labels have been assigned; the classification method comprising the steps of assigning new judgment labels to the plurality of contents using the second learning model and presenting them in the graphical user interface.

[0023] One aspect of the present invention is a method for classifying content and a method for creating a classifier, comprising the steps of: acquiring a data set consisting of a plurality of contents; a user assigns learning labels to m pieces of content among the plurality of contents, and the remaining pieces of content are not assigned learning labels; a step of creating a first learning model by machine learning using the m pieces of content to which the learning labels have been assigned; a step of assigning judgment labels to the plurality of contents using the first learning model and presenting the judgment labels in a graphical user interface; a step of assigning new learning labels to k pieces of content among the plurality of contents; a step of creating a second learning model by machine learning using the k pieces of content to which the learning labels have been assigned; a classification method comprising the steps of assigning new judgment labels to the plurality of contents using the second learning model and presenting the plurality of contents in a graphical user interface, where k represents a natural number greater than m. The k pieces of content may completely include the m pieces of content, or may only partially include the m pieces of content.

[0024] One aspect of the present invention is a method for classifying content and a method for creating a classifier, comprising the steps of: acquiring a data set consisting of a plurality of contents; m pieces of content among the plurality of contents are given a learning label; and the remaining pieces of content are not given a label; calculating a first score for estimating a judgment label of the plurality of contents using the m pieces of content to which the learning label is given; presenting a list in a graphical user interface in which the plurality of contents are given a label determined based on the first score; next, assigning a new learning label to k pieces of content among the plurality of contents included in the list; calculating a second score for estimating a judgment label of the plurality of contents using the k pieces of content to which the learning label is given; and presenting a list in a graphical user interface in which the plurality of contents are given a new judgment label determined based on the second score. Here, k represents a natural number greater than m. The set of k pieces may completely include the set of m pieces, or may only partially include the set of m pieces.

[0025] In the above configuration, the classification method includes a step of specifying a specific numerical range of the first score and assigning a learning label to the corresponding content.

[0026] In each of the above configurations, the plurality of contents are classified as text data.

[0027] In each of the above configurations, the classification method further includes a step of performing clustering by unsupervised learning on a data set including a plurality of contents.

[0028] In each of the above configurations, the plurality of contents is a classification method including text contained in the patent document. Effect of the Invention

[0029] One aspect of the present invention can provide a method for efficiently classifying information. Alternatively, one aspect of the present invention can provide a user interface for efficiently classifying information. Alternatively, one aspect of the present invention can provide a program for efficiently classifying information.

[0030] In addition, one aspect of the present invention can provide a user with an interactive interface for creating a classifier using machine learning, thereby reducing the burden on the user, such as preparing training data and evaluating learning results.

[0031] The effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. The other effects are effects not mentioned in this section, which will be described below. Effects not mentioned in this section can be derived by a person skilled in the art from the description of the specification or drawings, and can be appropriately extracted from these descriptions. One embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 is a diagram for explaining the classification method. [Diagram 2] FIG. 2 is a diagram for explaining the classification method. [Diagram 3] FIG. 3 is a flow chart illustrating the classification method. [Figure 4] FIG. 4 is a diagram illustrating the user interface. [Diagram 5] FIG. 5 is a flow chart illustrating the classification method. [Figure 6] 6A and 6B are diagrams illustrating a user interface. [Figure 7] FIG. 7 is a diagram illustrating the user interface. [Figure 8]FIG. 8 is a block diagram illustrating a computer device that runs a program. [Figure 9] FIG. 9 is a diagram illustrating a user interface. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] (Embodiment) In this embodiment, a content classification method will be described with reference to FIGS.

[0034] The content classification method or classifier creation method described in this embodiment is controlled by a program running on a computer device. The program is stored in a memory or storage device of the computer device. Alternatively, the program is stored in a computer device connected via a network (such as a local area network (LAN), a wide area network (WAN), or the Internet) or in a server computer device having a database.

[0035] The display device of the computer device can display data provided to the program by the user and the results of calculations performed on the provided data by the arithmetic device of the computer device. The configuration of the computer device will be described in detail with reference to FIG. 8.

[0036] By following a list display format, data displayed on a display device connected to a computer device can be easily recognized by the user, thereby improving operability. As an example, a display format that allows a user to easily interact with a program held by a computer device via a display device will be described as a graphical user interface (hereinafter, abbreviated as GUI).

[0037] A user can use the program's content classification method or classifier creation method via the GUI. The GUI can simplify the user's content classification operation. The GUI also makes it easier for the user to visually determine the content classification results. The GUI also allows the user to easily operate the program. The content in question refers to text data, image data, audio data, video data, etc.

[0038] Next, a method for classifying content using the GUI and a method for creating a classifier will be described according to the operation procedure of the GUI. First, a user has a step of acquiring a data set consisting of multiple contents via the GUI. The multiple contents are files stored in a memory or storage device of a computer device, or files stored in a computer, server, etc. connected to a network.

[0039] For example, it is preferable that a plurality of contents are stored in a list in the file, or the contents may be stored in different files.

[0040] As an example, a case where the content stored in the file is text data will be described. Note that in this specification, a case where the text data is text included in a patent document will be described. It is preferable that the file stores a list of multiple contents. Alternatively, the multiple contents may be stored in different files. For example, the multiple files may store different types or amounts of text data. Each text data can be read from the multiple files via a GUI.

[0041] The program has a step of displaying the read text data on the GUI. The text data is preferably in a list format. The GUI displays the text data in accordance with the display format of the GUI. The listed text data is preferably managed in units called records. For example, each record is composed of a label ID linked to a unique number indicating the order of the record, and content (text data).

[0042] For example, when the content is text data, it is difficult to handle the text data as data for a program to perform machine learning. Therefore, it is necessary to convert the target text data into a format that the program can handle. There is a method of analyzing and vectorizing text data for machine learning by a computer device. One example of a vectorization method is a method called Bag of Words (BoW). BoW can vectorize the frequency of occurrence of feature words contained in text data from text data. As an example, a feature word is a character string that appears repeatedly, or a character string that includes multiple adjectives or predicates. The vectorized text data can be easily handled by a computer device as input data for machine learning. In addition, distributed representations, such as Word2Vec, may be used to vectorize text data. Distributed representations are also called embedded representations.

[0043] In the above-mentioned content classification method or classifier creation method, the content can be classified by unsupervised machine learning, i.e., clustering. For example, K-means or DBSCAN (density-based spatial clustering of applications with noise) functions as a classifier. In addition, when the object of clustering is a document, a topic model may be used.

[0044] Next, the user has a step of reading n records into the GUI from a file stored in a memory or storage of the computer device. The GUI displays the n records in the GUI according to a display format provided by the GUI. The user has a step of assigning learning labels to m records selected by the user from the n records displayed in the GUI. Note that no learning labels are assigned to the remaining records. m and n are natural numbers.

[0045] Next, the program has a step of creating a learning model by machine learning using the m pieces of text data to which the learning labels have been added. For the learning model, a machine learning algorithm such as a decision tree, Naive Bayes, KNN (k Nearest Neighbor), SVM (Support Vector Machines), Perceptron, logistic regression, or neural network can be used.

[0046] Furthermore, there may be a step of switching the learning model according to the number of learning labels. When the number of learning labels is small, a decision tree, naive Bayes, or logistic regression may be used, and when the number of learning labels is a certain amount or more, an SVM, random forest, or neural network may be used. The learning model used in this embodiment uses a random forest, which is one of the decision tree algorithms.

[0047] In this embodiment, a supervised learning model is used in which the program updates the learning model through multiple machine learning rounds. Therefore, the classification accuracy improves every time the learning model is updated. Therefore, the learning model generated by the program in the first machine learning round is called the first learning model, and the learning model generated by the program in the second machine learning round is called the second learning model. Note that the learning model generated by the program in the pth machine learning round is called the pth learning model. p is a natural number.

[0048] Next, the program classifies the text data using the generated learning model, and the program has a step of assigning a classification result or a score to each record. In detail, first, the program generates a first learning model using the m pieces of text data to which the learning labels have been assigned. Next, the program classifies n pieces of text data using the first learning model. The program assigns a judgment label and a first score to the n records as a classification result. The first score is a result of a calculation performed by the program to estimate the judgment label using the first learning model. The first score can preferably have a numerical range of 0 to 1.

[0049] The first score may be rephrased as a probability for the judgment label generated by the first learning model. For example, a case will be described where the learning label is given as two types of data, "Yes" and "No." When the judgment label given by the program is "Yes," the program presents the probability that the judgment label is "Yes" as the first score. If the learning label is three types of data, it is preferable that the judgment label is three types of data. In other words, it is preferable that the types of judgment labels are equal to the types of learning labels.

[0050] Next, the program has a step of displaying the judgment label and the first score determined based on the first score on the GUI. In detail, the program additionally presents (displays) the judgment label and the first score classified by the first learning model to a list of n records displayed on the GUI. Note that the judgment label and the first score are also given to the record to which the learning label is assigned.

[0051] Next, the user judges each judgment label and the first score. The user has a step of assigning additional learning labels to k records from among a plurality of records to which learning labels have not been assigned. Here, k is a natural number greater than m. Note that the k records can include some or all of the m records. When the learning labels are assigned to the k records, it is preferable that the m learning labels are not updated. By not updating the m learning labels, the information of the first learning model is inherited. However, if it can be determined that any of the learning labels assigned to the m records is a learning label different from the user's judgment, it is preferable to assign a new learning label to the learning label. Note that, for records to which no learning labels have been assigned, it is preferable to present to the user those with the lowest first score labels with priority. For example, in the case of two-class classification, it is preferable to encourage the user to assign labels from records with a first score in the vicinity of 0.5 (for example, 0.5 or more and less than 0.65). Here, the first score means a probability.

[0052] In addition, as a result of the user judging the judgment label and the first score of the first learning model, the learning label assigned to the record selected by the user from among the m records may be updated. A new learning label may be assigned to the first learning model to create a second learning model, which will be described next.

[0053] Next, the program has a step of creating a second learning model by machine learning using the k pieces of text data to which the learning labels have been assigned. The user judges the first judgment label assigned to the first learning model, and the program generates a second learning model based on the k learning labels added by the user. The program reclassifies the n pieces of text data using the second learning model. The classification results are assigned to the n records by the program as judgment labels and a second score. The second score is the result of a calculation performed by the program to estimate the second judgment label using the second learning model.

[0054] Next, the program has a step of displaying a second judgment label and a second score determined based on the second score on the GUI. In detail, the program updates the presentation (display) of the second judgment label and the second score classified by the second learning model in the list of n records displayed on the GUI. The second judgment label and the second score are also given to the record to which the new learning label is assigned.

[0055] As described above, the user can provide learning labels to records displayed in the GUI, and the program can generate and update a learning model using the provided learning labels. For example, a learning label is provided as teacher information for a portion of a large amount of content (text data, image data, audio data, or video data). The program generates a learning model using the learning labels. The program uses the learning model to present a judgment label for content to which a learning label has not been assigned. The user updates the learning model by providing further learning labels to the judgment labels displayed by the program in the GUI.

[0056] The user can further provide a learning label to the judgment label presented by the first learning model via the GUI. In other words, the program can update the first learning model generated by machine learning to a second learning model that adds the user's judgment conditions to the first learning model. Thus, the program can provide a classification method that allows the user to classify necessary content from a large amount of content with fewer processing steps.

[0057] Next, a content classification method will be described with reference to the drawings. Fig. 1 is a diagram for explaining a content classification method according to one embodiment of the present invention. Each step explained in Fig. 1 indicates the processing contents of a program or a GUI.

[0058] In step S11, the program displays the initial screen of the GUI, which will be described in detail with reference to FIG.

[0059] The user operates the program to load a dataset via the GUI. A dataset is a file stored in the memory or storage of a computer device, or a file stored in a computer or server connected to a network. A file stores a list of multiple contents.

[0060] In step S12, the program displays the content read from the file. The GUI displays the content according to the display format of the GUI. It is preferable that the listed content is managed in units called records. For example, each record is composed of a label ID linked to a number, the content, and the like.

[0061] In step S13, the user assigns learning labels. For example, learning labels are assigned to m records selected by the user from n records displayed in the GUI. Note that no labels are assigned to the remaining records. Note that m or n is a natural number.

[0062] In step S14, the program performs machine learning using the m pieces of content to which the learning labels have been assigned, and creates a first learning model. The program also has a step of classifying the content using the first learning model generated by the program, and assigning the classification results to each record. Note that the first learning model can use machine learning algorithms such as decision trees, naive Bayes, KNN, SVM, perceptron, logistic regression, and neural networks.

[0063] In detail, the program generates a first learning model, and then classifies n pieces of content using the first learning model. The program assigns a judgment label and a first score to the n records as a classification result. The first score is a result of a calculation performed by the program to estimate a judgment label using the first learning model.

[0064] In step S15, the program displays the classification result of the first learning model on the GUI. The classification result of the first learning model is given as a judgment label and a first score determined based on the first score. In detail, the program additionally presents (displays) the judgment label and the first score classified by the first learning model in a list of n records displayed on the GUI. Note that the judgment label and the first score are also given to the records to which the learning label is assigned.

[0065] Next, the user judges the displayed judgment label. It is not necessary to judge all judgment labels, and any record may be selected from records to which no learning label has been assigned and judged. The result of the judgment here may be used as the learning label in the next step. The user has a step of assigning additional learning labels to k records from among a plurality of records to which no learning label has been assigned. Here, k is greater than m and is a natural number. Note that the k records may include m records. When assigning learning labels to the k records, it is preferable that the m learning labels are not updated. By not updating the m learning labels, the information of the first learning model is inherited. However, if it can be determined that any of the learning labels assigned to the m records is a learning label different from the user's judgment, it is preferable to assign a new learning label to the learning label.

[0066] Return to step S14. The program has a step of creating a second learning model by machine learning using the k pieces of content to which the learning labels have been assigned. The user judges the judgment label assigned to the first learning model, and the program generates a second learning model based on the k pieces of learning labels added by the user. The program reclassifies the n pieces of content using the second learning model. The classification results are assigned to the n records by the program as judgment labels and a second score. The second score is the result of a calculation performed by the program to estimate the judgment label using the second learning model.

[0067] In step S15, the program displays the classification results of the second learning model on the GUI. The classification results of the second learning model are given as a judgment label and a second score determined based on the second score. In detail, the program updates the presentation (display) of the judgment label and the second score classified by the second learning model in a list of n records displayed on the GUI. The judgment label and the second score are also given to the records to which the learning label is assigned.

[0068] By repeating steps S14 and S15, the classification accuracy of the learning model improves each time it is updated. Therefore, the learning model generated by the program in the first machine learning is called the first learning model, and the learning model generated by the program in the second machine learning is called the second learning model. Note that the learning model generated by the program in the pth machine learning is called the pth learning model.

[0069] Next, a content classification method different from that in FIG. 1 will be described with reference to FIG. 2. FIG. 2 differs from FIG. 1 in that content with learning labels is read. Therefore, in FIG. 2, it is not necessary to assign learning labels in step S13. The other steps are the same processes as in FIG. 1, so the description of FIG. 1 can be referred to.

[0070] Fig. 3 is a flowchart for explaining in detail the explanation of Fig. 1. Each step explained in Fig. 3 indicates the processing contents of the user, the program, or the GUI of the program. Also, when the explanation overlaps with Fig. 1, the detailed explanation may be omitted.

[0071] Step S21 is a step for loading content. The user performs an operation via the GUI to load a data set, which is a collection of content, into the program. The program displays the loaded data set according to the display format of the GUI. The content loaded in step S21 corresponds to a mother set for classification.

[0072] In step S22, the user assigns learning labels. For example, learning labels are assigned to m records selected by the user from the n records (population) displayed on the GUI. Note that learning labels are not assigned to the remaining records. Note that m or n is a natural number.

[0073] In step S23, the program creates a first learning model through machine learning using the m pieces of content to which the learning labels have been assigned.

[0074] In step S24, the program classifies the content using the first learning model generated by the program, and the program assigns the classification results to each record. The program displays the classification results using the first learning model on the GUI. The classification results of the first learning model are given as a judgment label and a first score determined based on the first score.

[0075] In step S25, the user may evaluate the classification results of each record displayed on the GUI to determine whether the classification results are appropriate.

[0076] In step S26, the user determines whether additional learning is required for the result of classification using the first learning model. If the user determines that additional learning is required, the process proceeds to step S27. If the user determines that additional learning is not required, the classification of the content ends.

[0077] In step S27, the user assigns additional learning labels. For example, from among the n records (population) displayed on the GUI, new learning labels are assigned to k records by additionally assigning learning labels to records other than the m records to which no learning labels have been assigned. Note that no labels are assigned to the remaining records. Note that k is a natural number.

[0078] In step S23, the program creates a second learning model through machine learning using the k pieces of content to which the learning labels have been assigned.

[0079] In step S24, the content is classified using the second learning model generated by the program, and the program assigns the classification result to each record. The program displays the classification result using the second learning model on the GUI. The classification result of the second learning model is given as a judgment label and a second score determined based on the second score.

[0080] In step S25, the user determines the classification result of each record displayed on the GUI.

[0081] In step S26, the user determines whether additional learning is required for the result of classification using the second learning model. If the user determines that additional learning is required, the process proceeds to step S27. If the user determines that additional learning is not required, the classification of the content ends.

[0082] The flowchart in Figure 3 shows an example in which the user is responsible for judging the classification results. However, some of the criteria for judging the classification results may be given to the program. The program that receives the judgment criteria from the user can continue learning until the classification results converge.

[0083] For example, when a learning model provides a judgment label and a score for a record, if the score is greater than the judgment criterion, the learning label "Yes" can be automatically assigned. As a different example, if the score is less than the judgment criterion, the learning label "No" can be automatically assigned. If the probability of the learning model classifying the content is high or low, the learning model can automatically assign a re-training label. Therefore, since part of the learning of the learning model can be automated, it is suitable for classifying a large amount of content.

[0084] It is desirable to encourage users to label records that have not been given a learning label and have a score close to 0.5 (for example, 0.5 or more and less than 0.65). The score can be said to be a probability.

[0085] FIG. 4 is a diagram for explaining the GUI 30 of the above-mentioned program. The GUI is composed of a plurality of columns and a plurality of records. Column 31 displays the record number. Column 32 displays the label ID associated with the record number. Column 33 displays the learning label "U-Label" associated with the record number. Column 34 displays the judgment label "A-Label" associated with the record number. Column 35 displays the score "Score" associated with the record number. Column 36 displays the contents "Contents" associated with the record number. The user can provide learning labels as teacher data to columns 33a to 33n. It is preferable to provide the learning label "U-Label" to the record to be learned.

[0086] Fig. 5 is a flowchart different from Fig. 3. Note that when the explanation overlaps with Fig. 3, the detailed explanation may be omitted.

[0087] In step S41, text data is loaded. The user operates the program via the GUI to load a data set, which is a collection of text data. The program displays the loaded text according to the display format of the GUI. The text data loaded in step S41 corresponds to a parent set for classification.

[0088] In step S42, the user assigns a learning label. For example, the user assigns a learning label, either "Yes" or "No", to m records selected by the user from among n records (population) displayed on the GUI. Note that no learning label is assigned to the remaining records. Note that m or n is a natural number.

[0089] In step S43, the program creates a first learning model through machine learning using the m pieces of content to which the learning labels have been assigned.

[0090] In step S44, the text data is classified using the first learning model generated by the program, and the program assigns the classification results to each record. The program displays the results of classification using the first learning model on the GUI. The results of classification by the first learning model are given as a judgment label and a first score determined based on the first score. The first score is a value of 0 or more and 1 or less for n records. Also, the judgment label is assigned to the n records as either "Yes" or "No".

[0091] In step S45, the user determines the classification result of each record displayed on the GUI.

[0092] In step S46, the user determines whether additional learning is required for the result of classification using the first learning model. If the user determines that additional learning is required, the process proceeds to step S47. If the user determines that additional learning is not required, the classification of the content ends.

[0093] In step S47, the user assigns additional learning labels. For example, from among the n records (population) displayed on the GUI, the user assigns additional learning labels to records other than the m records to which no learning labels have been assigned, thereby assigning new learning labels to k records. Here, k is a natural number.

[0094] In step S43, the program creates a second learning model through machine learning using the k pieces of content to which the learning labels have been assigned.

[0095] In step S44, the content is classified using the second learning model generated by the program, and the program assigns the classification result to each record. The program displays the classification result using the second learning model on the GUI. The classification result of the second learning model is given as a judgment label and a second score determined based on the second score.

[0096] In step S45, the user determines the classification result of each record displayed on the GUI.

[0097] In step S46, the user judges whether or not additional learning is required for the result of classification using the second learning model. If the user judges that additional learning is required, the process proceeds to step S47. If the user judges that additional learning is not required, the classification of the content ends.

[0098] The flowchart in Figure 5 shows an example in which the user is responsible for judging the classification results. However, the user may also provide the program with criteria for judging the classification results. The program that has been given the criteria can continue learning until the classification results converge.

[0099] For example, when a learning model provides a judgment label and a score for a record, if the score is greater than the judgment criterion, the learning label "Yes" can be automatically assigned. As a different example, if the score is less than the judgment criterion, the learning label "No" can be automatically assigned. If the probability of the learning model classifying the content is high or low, the learning model can automatically assign a re-training label. Therefore, since part of the learning of the learning model can be automated, it is suitable for classifying large amounts of text data.

[0100] It is desirable to encourage the user to assign labels to records with no learning labels, starting from those with a score close to 0.5 (for example, between 0.5 and 0.65).

[0101] 6A and 6B are diagrams for explaining GUI 30. FIG. 6A corresponds to step S44 in the flowchart of FIG. 5. In step S44, the classification results obtained by the first learning model are displayed on GUI 30. The content shows an example in which n records are loaded. Each record has a number No., a label ID, and text data. In FIG. 6A, the text data is displayed in column 36. However, for ease of explanation, the text data is replaced with symbols and displayed.

[0102] The program shows an example in which learning labels ("Yes" or "No") are assigned to records 1 to 3 in columns 33a to 33c as "Group A" to generate a first learning model. Note that although this example shows a binary classification, it may also be a classification with three or more values.

[0103] FIG. 6A shows an example in which the program generates a first learning model and assigns a classification result to each record. The classification result displays a judgment label ("Yes" or "No") determined based on the first score and a first score. The first score is assigned a value of 0 or more and 1 or less for n records.

[0104] For example, let us consider the case where the record number is 1. The judgment label "A-Label" displays "Yes", and the score "Score" displays "0.874". The user gives the learning label "U-Label" to the results classified by the learning model. When the judgment label "A-Label" displays "Yes", the score "Score" indicates the probability of "Yes", and when the judgment label "A-Label" displays "No", the score "Score" indicates the probability of "No".

[0105] As one example, if the user judges that the result displayed in the judgment label "A-Label" and score "Score" is correct, the user assigns "Yes" to the learning label "U-Label". As a different example, if the user judges that the result displayed in the judgment label "A-Label" and score "Score" is incorrect, the user assigns "No" to the learning label "U-Label". The accuracy of the classifier can be improved by the user assigning judgment conditions for records whose score "Score" as a result of classification by the learning model is close to a score of 0.5 (for example, greater than or equal to 0.5 and less than 0.65).

[0106] FIG. 6B shows an example in which the program assigns learning labels ("Yes" or "No") in columns 33d to 33f to records 4 to 6 as "Group B" to generate a second learning model.

[0107] The program shows an example of generating a second learning model and assigning a classification result to each record. The classification result displays a judgment label ("Yes" or "No") and a second score determined based on the second score. In the first learning model shown in FIG. 6A, the judgment label of record 7 was "Yes" and the first score was "0.538", but in the second learning model shown in FIG. 6B, the judgment label of record 7 has changed to "No" and the second score to "0.625". As shown in FIG. 6, by adding a learning label to the first learning model, it is possible to improve the probability of classification of records to which a learning label has not been assigned. Note that the judgment label and score, which are the classification results, are only examples, and the contents of the judgment label and score are not limited. They change depending on the content to be loaded and the learning label to be assigned.

[0108] FIG. 7 is a diagram for explaining GUI 30. Differences between FIG. 7 and FIG. 6A will be described. FIG. 7 shows an example in which n records are loaded as content. Each record has a number No., a label ID, and image data. That is, FIG. 7 shows an example in which image data is classified as content in column 36. Therefore, in one aspect of the present embodiment, a classification method for classifying image data can be provided. Note that audio data, video data, and the like can be handled as different content.

[0109] FIG. 8 is a diagram illustrating an electronic device 10, a database 21, a remote computer 22, and a remote computer 23 to which the electronic device 10 is connected via a network. The electronic device 10 has a calculation device 11, a memory 12, an input / output interface 13, a communication device 14, and a storage 15. The electronic device 10 is electrically connected to a display device 16a and a keyboard 16b via the input / output interface 13. The electronic device 10 is also electrically connected to a network interface 17 via the communication device 14, and the network interface 17 is electrically connected to the database 21, the remote computer 22, and the remote computer 23 via a network. The network may be a LAN, a WAN, or the Internet.

[0110] A computer device according to one aspect of the present embodiment corresponds to an electronic device. A program possessed by the computer device is stored in a memory 12 or a storage 15. The program generates a learning model using a calculation device 11. The program can be displayed on a display device via an input / output interface 13. A user can provide input data such as learning labels to the program from a keyboard in response to a GUI displayed on the display device.

[0111] The program can also be used in the remote computer 22 or the remote computer 23 via a network. Alternatively, the program stored in the memory or storage of the database 21, the remote computer 22, or the remote computer 23 can be used to operate the electronic device 10. The remote computer 22 may be a portable information terminal, or a portable terminal such as a tablet computer or a notebook computer. In the case of a portable information terminal or a portable terminal, communication can be performed using wireless communication.

[0112] FIG. 9 shows an example in which a GUI 30A for classifying content is displayed on a display device 16a connected to the electronic device 10 described with reference to FIG.

[0113] The GUI 30A is made up of a plurality of columns and a plurality of records. The record is made up of columns 51 to 56. The GUI 30A further includes a button 57a, a button 58, and a display area 57b.

[0114] Column 51 displays the record number. Column 52 displays the Index associated with the record number. Column 53 displays the user specified label "User Specified Label" associated with the record number. Column 54 displays the automatic classification label "Automatic classification Label" associated with the record number. Column 55 displays the classification probability "probability" associated with the record number. Column 56 displays the claim "CLAIM" associated with the record number. The user specified label corresponds to the learning label, the automatic classification label corresponds to the judgment label, and the probability corresponds to the score.

[0115] For example, by pressing button 57a, a file stored in memory 12 or storage 15 of electronic device 10 can be selected. The storage location of the selected file is displayed in display area 57b. When the file is selected, information stored in the file is displayed in GUI 30A. In FIG. 9, record information stored in the file is displayed in columns 51, 52, and 56. Note that in FIG. 9, patent information is stored as content. Patent numbers are displayed in column 52, and claim information, which is text data, is displayed in column 56.

[0116] Next, the user can assign user-specified labels as training data to the column 53a including a plurality of records. Note that it is preferable to assign the user-specified labels to the records that the user wishes to train the learning model.

[0117] Next, by pressing button 58, the program updates the learning model using the conditions given to GUI 30A. The learning model can be named and stored in memory 12 or storage 15. Thus, the learning model stored in memory 12 or storage 15 can be used to learn new data and update the learning model. Alternatively, the learning model stored in memory 12 or storage 15 can be used to classify content.

[0118] The program uses the updated learning model to classify the content displayed in GUI 30A. As an example, the program assigns an automatic classification label and a probability, which are classification results, to each record. For example, the automatic classification label is displayed in column 54a included in each record. The classification probability can be displayed in column 55a included in each record. Detailed text data of the claim can be displayed in column 56a included in each record.

[0119] The user looks at the automatic classification labels and probabilities that are the classification results assigned to the records, and if sufficient classification accuracy is obtained, ends the update of the learning model. If the user looks at the automatic classification labels and probabilities that are the classification results assigned to the records and determines that the classification accuracy is not sufficient, the user can update the learning model by assigning user-specified labels to records that do not have user-specified labels assigned to them and pressing button 58.

[0120] Although not shown, the order of display can be changed based on the numerical values ​​and label information contained in each column, or a filter function can be used to prevent display. This allows the user to efficiently evaluate the classification accuracy. For example, by displaying only records whose scores (probabilities) are below a certain level, the learning labels for those records can be changed or new ones added.

[0121] The content classification method described with reference to Figs. 1 to 9 can provide a method for classifying information with a high probability. For example, a GUI is suitable for classifying information with a high probability. A program can update a learning model by providing new training data (learning labels) to the learning model. A program can classify information with a high probability by updating the learning model.

[0122] In addition, the results of unsupervised clustering performed on the entire records before the first or second or subsequent learning labels are assigned may be displayed on the GUI, and the user may adjust the records to which learning labels are assigned according to the clustering results. Such a clustering method may be a known method such as k-means clustering or DBSCAN.

[0123] Furthermore, the created learning model can be stored in the electronic device itself or in an external memory and can be called up and used when classifying a new data set. Furthermore, the learning model can be updated according to the method described above while adding new training data.

[0124] As described above, the structure and method described in this embodiment mode can be used in appropriate combination with structures and methods other than those described in this embodiment mode. [Explanation of symbols]

[0125] 10: Electronic device, 11: Computing device, 12: Memory, 13: Input / output interface, 14: Communication device, 15: Storage, 16a: Display device, 16b: Keyboard, 17: Network interface, 21: Database, 22, 23: Remote computer, 30, 30A: GUI

Claims

1. A method for generating a dataset of a plurality of patent document contents, the method comprising: obtaining a dataset of a plurality of patent document contents, m of the plurality of patent document contents being assigned learning labels and the remaining contents being unlabeled; A first score is calculated to estimate labels of the contents of the plurality of patent documents using the m contents to which the learning labels have been assigned; A list in which judgment labels determined based on the first scores are assigned to the contents of the plurality of patent documents is displayed in a graphical user interface; A learning label is assigned to k pieces (k is a natural number greater than m) of contents of the plurality of patent documents included in the list; A second score is calculated to estimate a judgment label of the content of the plurality of patent documents using the k pieces of content to which the learning labels are assigned; A method of classifying content, wherein the list in which the judgment labels determined based on the second scores are assigned to the contents of the plurality of patent documents is displayed within the graphical user interface.

2. In claim 1, A content classification method, wherein the learning labels have three or more values.

Images