Content classification methods

Abstract

Provide a method for classifying new content. 【Solution means】A method for classifying content and a method for creating a classifier using machine learning in a learning model. Step 1 is to obtain a dataset consisting of a plurality of contents. Learning labels are assigned to m contents, and learning labels are not assigned to the remaining contents. Step 2 is to create a first learning model by machine learning using the m contents. Step 3 is to assign judgment labels to a plurality of contents using the first learning model and display them on the GUI. Step 4 is to assign new learning labels to k contents among the plurality of contents. Step 5 is to create a second learning model by machine learning using the k contents. Step 6 is to assign judgment labels to a plurality of contents using the second learning model and display them on the GUI. ​

Description

Technical Field

[0001] One aspect of the present invention relates to a computer device, a graphical user interface, and a method for classifying content and a method for creating a classifier.

[0002] Note that one aspect of the present invention relates to a computer device. One aspect of the present invention relates to a computer device and a method for classifying electronic content (text data, image data, audio data, or video data) using the computer device. In particular, one aspect of the present invention relates to a computer device that efficiently classifies a collection of content using machine learning. Note that one aspect of the present invention relates to a method for classifying content and a method for creating a classifier using a graphical user interface managed by a computer device through a program.

Background Art

[0003] A user wants to easily extract information related to topics specified by the user from a collection of content. In recent years, new classification systems using learning models learned by machine learning have been proposed. For example, in Patent Document 1, a machine learning approach for determining a document highly relevant to topics specified by a user is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Sometimes, a collection of documents (for example, patents or academic papers) is classified according to its purpose. The documents are classified using various items such as summaries, keywords, diagrams, and various notes that accompany them. The accuracy and efficiency of the classification depend on the content of the documents being classified, as well as the experience and skill of the worker. Differences are easily apparent, and classifying large volumes of documents requires manual effort, posing challenges to efficiency. there were.

[0006] Creating a classifier using machine learning requires preparing a large amount of training data. This may impose an excessive burden on the user, or the number of classified content items in the training data may be excessive. One challenge is that variability affects the accuracy of the classifier.

[0007] In view of the above problems, one aspect of the present invention aims to provide a method for efficiently classifying information. This is one aspect of the present invention. Alternatively, one aspect of the present invention is a graphical user that efficiently classifies information. One of the objectives is to provide an interface. Alternatively, one aspect of the present invention is to efficiently One of the objectives is to provide a program that classifies information.

[0008] Furthermore, the description of these problems does not preclude the existence of other problems. One approach does not require that all of these issues be resolved. The title will become clear from the description in the specification, drawings, claims, etc. It is possible to extract other issues from the descriptions in the drawings, claims, etc.

[0009] The problems addressed by one embodiment of the present invention are not limited to those listed above. This does not preclude the existence of other issues. These other issues are described in the following section. This is a problem not mentioned. Problems not mentioned in this section can be derived from the descriptions in the specification or drawings by those skilled in the art, and can be appropriately extracted from these descriptions. One aspect of the present invention solves at least one of the above-listed problems and / or other problems.

Means for Solving the Problem

[0010] The proposed system utilizes machine learning. Machine learning is performed based on a relatively small number of training data in the initial set of documents to obtain a learning model, and classification is performed on the remaining documents using the obtained learning model. Here, the accuracy of the classification results obtained depends on the quality of the learning model. Assuming that satisfactory accuracy cannot be obtained with a single machine learning, on the same graphical interface, classification, verification, and machine learning are repeated to enable the creation of a high-quality learning model.

[0011] An operator evaluates whether the classification performed by the machine is appropriate, and the evaluated data is added as training data. Based on the initial training data and the added training data, machine learning is performed again to obtain a second learning model. The second learning model has improved accuracy due to the increase in training data.

[0012]

[0013] By repeating these operations, a learning model with sufficient accuracy can be obtained. In addition, the present invention provides a graphical user interface for efficiently obtaining this learning model.

[0014]

[0015] The classification can be either two types or three or more types depending on the purpose of the operator. Thus, the machine learns the teacher data given to some of the data for the operator to classify , and the operator can classify the entire document in a shorter time than manually or visually judging all the documents. Regarding the mechanism of machine learning, known techniques may be used, for example Naive Bayes or Random Forest can be used.

[0015] In the first step, prepare a file containing text data. Note that the file containing text data may be referred to as content. Also, the content includes not only text data, but also image data, audio data, or video data. As an example of the content, the text included in the patent document can be used.

[0016] In the second step, read the file and display the text data on the screen.

[0017] In the third step, input the classification for a part of all the data into the learning label as teacher data. Note that in the second step, prepare the file in a state where the classification has been given as teacher data ( learning label) for some of the documents in advance, and when reading, it may be taken in as the learning label.

[0018] In the fourth step, press the learning start button to obtain the learning model, and use the learning model to assign the classification to the judgment label. At this time, as the judgment label, assign the label corresponding to the classification. Use the label given by the calculation of the learning model as the judgment label.

[0019] In the fifth step, the worker evaluates the assigned judgment label and adds the evaluation results. This data is entered into the training labels as training data.

[0020] In the sixth step, you press the start learning button, the learning model is acquired, and the learning model is used. The classification is then assigned to the judgment label.

[0021] In the seventh step, steps 5 and 6 are repeated until sufficient classification accuracy is achieved. To retaliate.

[0022] One aspect of the present invention is a method for classifying content and a method for creating a classifier, and a learning label m pieces of content (where m is a natural number) that have been assigned a label, and the learning label that has been assigned to them. n pieces of content (where n is a natural number) that are not present, and a data containing multiple pieces of content that include The process includes a step of acquiring a dataset. Using m content items, the first step is performed by machine learning. The process includes the step of creating a learning model for the first learning model. The step of assigning a judgment label to the items and presenting them within the graphical user interface It possesses. Of the n content items, q (where q is a natural number) of content items contain new learning. The process includes a step of assigning learning labels. The (q+m) learning labels are then assigned to (q+m) items. The process includes a step of creating a second learning model using machine learning with the content. Using the learning model 2, new judgment labels are assigned to multiple pieces of content and the graphics This is a classification method that has steps presented within the user interface.

[0023] One aspect of the present invention is a method for classifying content and a method for creating a classifier, wherein multiple components The process includes a step of obtaining a dataset consisting of elements. The user can select multiple elements. Of the contents, m items are assigned learning labels, and the remaining contents are assigned learning labels. The learning label is not assigned. Using the m pieces of content to which the learning label has been assigned. The process includes the step of creating a first learning model using machine learning. By assigning judgment labels to multiple pieces of content, it can be displayed within a graphical user interface. The process involves the following steps: assigning new learning labels to k of the multiple contents. The process includes the step of assigning a label. Using k pieces of content to which the learning label has been assigned The process includes a step of creating a second learning model using machine learning. By assigning new judgment labels to multiple content items, a graphical user interface is created. This is a classification method that has steps to be presented internally, where k represents a natural number greater than m. Furthermore, the k sets may completely contain the m sets, or they may contain only a part of them. It's okay to be there.

[0024] One aspect of the present invention is a method for classifying content and a method for creating a classifier, wherein multiple components The process involves obtaining a dataset consisting of multiple contents. The selected content will be labeled for learning, while the remaining content will not. The determination of multiple content items is performed using the m content items to which the learning label has been assigned. The process includes a step of calculating a first score for estimating the label. A list of multiple content items with predetermined labels assigned to them is displayed in a graphical user interface. It has a step to present within the face. Next, of the multiple contents included in the list The process includes a step of assigning new learning labels to k pieces of content. A second method for estimating the determination labels of multiple content items using the given k content items. The process includes a step of calculating the score. A new judgment is determined based on the second score. A list with bells attached to multiple content items is presented within the graphical user interface. This is a classification method that has the steps shown, where k represents a natural number greater than m. The k sets may contain all of the m sets, or only a part of them. stomach.

[0025] In the above configuration, a specific numerical range is specified from the first score, and the corresponding content This classification method includes a step of assigning training labels to the data.

[0026] In each of the above configurations, the classification method is that multiple content items are text data.

[0027] In each of the above configurations, unsupervised learning is applied to a dataset containing multiple content items. This classification method further includes the step of performing clustering.

[0028] In each of the above configurations, the multiple contents include a classification method that includes text contained in the patent document. That is the case. [Effects of the Invention]

[0029] One aspect of the present invention can provide a method for efficiently classifying information. Alternatively, this One aspect of the invention can provide a user interface for efficiently classifying information. Alternatively, one aspect of the present invention can provide a program for efficiently classifying information. Cut.

[0030] Furthermore, one aspect of the present invention relates to an interactive interface for creating a classifier using machine learning. - It can provide users with a face, and can handle tasks such as preparing training data and evaluating learning results. This can reduce the burden on users.

[0031] The effects of one embodiment of the present invention are not limited to those listed above. This does not preclude the existence of other effects. These other effects are described in the following section. This is an effect not mentioned in the specification. Effects not mentioned in this section can be understood by those skilled in the art from the specification. These can be derived from drawings and other descriptions, and can be appropriately extracted from these descriptions. Furthermore, one aspect of the present invention includes, at least, the effects listed above and / or other effects. It has at least one effect. Therefore, one aspect of the present invention may, in some cases, It may not always have the effects listed above. [Brief explanation of the drawing]

[0032] [Figure 1] Figure 1 is a diagram illustrating the classification method. [Figure 2] Figure 2 is a diagram illustrating the classification method. [Figure 3] Figure 3 is a flowchart explaining the classification method. [Figure 4] Figure 4 is a diagram illustrating the user interface. [Figure 5] Figure 5 is a flowchart illustrating the classification method. [Figure 6] Figures 6A and 6B illustrate the user interface. [Figure 7] Figure 7 is a diagram illustrating the user interface. [Figure 8] Figure 8 is a block diagram illustrating the computer device that runs the program. [Figure 9]Figure 9 is a diagram illustrating the user interface. [Modes for carrying out the invention]

[0033] (Embodiment) In this embodiment, the content classification method will be explained using Figures 1 to 9.

[0034] The content classification method or classifier creation method described in this embodiment is computer It is controlled by a program running on the computer device. The program is controlled by the computer device. It is stored in the memory or storage that it possesses. Or, on the network (LAN) Local Area Network), WAN (Wide Area Network) k) Computer devices or databases connected via the internet, etc. It is stored in a server computer device that has a memory slot.

[0035] Furthermore, the display device of the computer device will show the data that the user provides to the program, Display the result of calculation performed by the arithmetic unit of the computer device using the given data. This is possible. Further details regarding the computer system configuration are provided in Figure 8. ru.

[0036] The data displayed on the display device connected to the computer device is in a listed format. By following the formula, it becomes easier for users to recognize and improves usability. For example... This allows the user to easily interact with the programs on the computer device via the display device. The display format used to achieve this is called a graphical user interface (hereafter, This will be explained using the term GUI (which can be written as GUI).

[0037] The user can use the GUI to configure the program's content classification methods or classifiers. A method can be used. The GUI simplifies the content classification operations performed by the user. This is possible. Furthermore, users can visually view the content classification results via a GUI. This makes it easier to make decisions. Furthermore, using a GUI allows users to operate the program easily. This is possible. Note that the content in question includes text data, image data, and audio data. This indicates data, such as video data.

[0038] Next, we will explain how to classify content using a GUI and how to create a classifier, using the GUI. The procedure will be explained below. First, the user accesses data consisting of multiple contents via the GUI. The process includes a step to acquire a set of content. Multiple content refers to the content that a computer device possesses. Files stored in memory or storage, or connected to a network. These are files stored on computers, servers, etc.

[0039] For example, the file in question may contain multiple contents listed and stored together. This is preferable. Alternatively, the content may be stored in separate files.

[0040] As an example, let's consider the case where the content stored in the file is text data. To clarify. In this specification, if the text data is text included in a patent document... I will explain this. The file in question contains multiple contents listed and stored. It is preferable that the content is stored in different files. That's also good. For example, multiple files could each contain different types or amounts of text data. Data may be saved. Text can be selected from multiple files via the GUI. It can read data.

[0041] The program has a step of displaying the read text data on a GUI. The data is preferably in a listed format. The GUI has The text data will be displayed according to the display format. Note that the listed text data It is preferable that data be managed in units called records. For example, each record is , a label ID associated with a unique number representing the order of the record, and content ( It consists of text data, etc.

[0042] For example, if the content is text data, the program will be able to process the text data mechanically. It is difficult to treat the target text data as learning data. It needs to be in a format that Grams can handle, so that the computer device can perform machine learning. One method is to analyze text data and convert it into vectors. As an example of a vectorization method... There is a method called Bag of Words (BoW). BoW is a text data From this, the frequency of occurrence of characteristic words contained in the text data can be vectorized. For example, characteristic words are strings of characters that appear repeatedly, or multiple adjectives or predicates that modify them. It is a string. Vectorized text data is used by computer devices for machine learning. It can be easily handled as input data. Furthermore, to vectorize text data... Distributed representations, such as Word2Vec, may also be used. Distributed representations are embedded representations and They are also called that.

[0043] Furthermore, in the above-mentioned method for classifying content or creating a classifier, the content in question is... In contrast, it can perform classification using unsupervised machine learning, i.e., clustering. For example, K-means and DBSCAN (density-based spatial Clustering of applications with noise is It functions as a classifier. Furthermore, if the target of clustering is documents, it uses a topic model. You may use it.

[0044] Next, the user accesses the GUI from the memory or storage of the computer device. The process involves a step of reading n records from an existing file. The GUI is GU The GUI displays n records according to the display format provided by I. From the displayed n records, the user selects m records and assigns them training labels. The process includes a step of assigning a label. Note that the remaining records are not assigned learning labels. m and n are natural numbers.

[0045] Next, the program uses the m text data that have been assigned training labels to perform the operation. The process includes a step of creating a learning model through machine learning. The learning model may include decision trees, Naïve trees, etc. Bubayz, KNN (K Nearest Neighbor), SVM (Support Vector Machines, perceptrons, logistic regression, neural Machine learning algorithms such as multi-networks can be used.

[0046] Furthermore, there may be a step to switch the training model depending on the number of training labels. When the number of labels to be used is small, use decision trees, naive Bayes, logistic regression, or training labels. If the number is above a certain amount, you can use SVM, Random Forest, or Neural Network. It is also possible to do so. Note that the learning model used in this embodiment is one of the decision tree algorithms. This uses a random forest architecture.

[0047] In this embodiment, the program updates the learning model through multiple machine learning iterations. A learning model is used. Therefore, the accuracy of the classification improves each time the learning model is updated. Therefore, the learning model generated by the program in the first machine learning run is referred to as the first learning model. The learning model generated by the program during the second machine learning run is called the second learning model. The learning model generated by the program in the p-th machine learning session is called the p-th learning model. p is a natural number.

[0048] Next, the program classifies the text data using the generated learning model, and the program The process includes a step of assigning a classification result or score to each record. To explain, first, the program uses m text data to which training labels have been assigned. The first learning model is generated using [this method]. Next, the program uses the first learning model to [this method]. Classify n text data. The program will determine the classification result in n records. A label and a first score are assigned. The first score is the score at which the program performs in the first academic year. This is the result of the calculation performed to estimate the judgment label using the training model. The first score is The possible numerical range is preferably between 0 and 1.

[0049] Note that the first score can be rephrased as the probability for the judgment label generated by the first learning model. It is also possible to do this. For example, training labels may be given as two types of data, "Yes" and "No". Let me explain the case. If the judgment label assigned by the program is "Yes", the program The first score presents the probability that the judgment label is "Yes". If the labels for use are three types of data, then the judgment labels must be three types of data. This is preferable. In other words, it is preferable that the type of judgment label is the same as the type of training label.

[0050] Next, the program determines the judgment label and the first score based on the first score The program has a step of displaying A on the GUI. In more detail, the program has a first learning The classification labels and first scores determined by the model are displayed in the GUI for n records. The list will be added (displayed) to the list of items. Note that the judgment label and the first score are used for learning purposes. It is also given to records that have been assigned a bell.

[0051] Next, the user determines each judgment label and the first score. The user learns For k records from among multiple records that do not have a learning label assigned, additional learning The process includes a step of assigning a practice label, where k is a natural number greater than m. k records can contain some or all of m records. When assigning training labels to records, it is preferable that the m training labels remain unchanged. It is because the m training labels are not updated, and the information from the first training model is carried over. However, if any of the training labels assigned to the m records differ from the user's judgment, If it can be determined that a label is for learning purposes, a new learning label will be assigned to that label. It is preferable to do so. Records that have not been assigned a learning label will be the first score It is preferable to present the lowest bell value to the user first. For example, two-class classification. In this case, the first score is selected from records in the vicinity of 0.5 (for example, 0.5 or greater and less than 0.65). It is desirable to encourage labeling by the user. Here, the first score represents the probability.

[0052] Furthermore, if the user determines the judgment label and first score of the first learning model, m Even if you update the learning labels assigned to the records selected by the user from among the individual records Good. Assign a new training label to the first learning model, and then the second learning model described below. It may be created.

[0053] Next, the program uses the k text data that have been assigned training labels to perform the operation. The process includes a step of creating a second learning model through machine learning. The user then creates the first learning model. The first judgment label assigned to it is determined, and further, the k training labels added by the user are used to make the decision. The program then generates a second learning model. The program then uses the second learning model. Therefore, the n text data items are classified again. The classification result is n items by the program. The record is assigned a judgment label and a second score. The second score is... The computations the program performs to estimate the second decision label using the second learning model This is the result.

[0054] Next, the program determines a second judgment label based on the second score and the second The program has a step of displaying the score on the GUI. More specifically, the program has a second step The second classification label and second score, determined by the learning model, are displayed in the GUI. Update the presentation (display) of the list of n records. Note that the second judgment label and the second The score is also given to records that have been assigned new training labels.

[0055] As mentioned above, the user assigns learning labels to the records displayed in the GUI. The program can generate and update a learning model based on the given learning labels. For example, a large amount of content (text data, image data, audio data, or video data) The program assigns training labels as training information to a portion of the data. The program generates a learning model using this learning model. A judgment label is displayed for content that was not assigned a label. The user is informed that the program is GU By assigning further training labels to the judgment labels displayed in I, the learning model is updated. ru.

[0056] The user, via the GUI, can further train the model based on the judgment labels presented by the first learning model. A label can be assigned. In other words, the program generates the first one through machine learning. The learning model is updated to a second learning model, which adds the user's decision conditions to the first learning model. Therefore, the program allows users to access a large amount of content with fewer processing steps. This provides a classification method that allows for the classification of necessary content.

[0057] Next, we will explain the content classification method using diagrams. Figure 1 shows one example of this implementation. This diagram illustrates the classification method for various types of content. Note that each of the types of content described in Figure 1 is... A step indicates the processing steps of a program or GUI.

[0058] In step S11, the program displays the initial GUI screen. The screen will be explained in detail in Figure 4.

[0059] The user loads the dataset into the program via a GUI. A dataset is a collection of data stored in the memory or storage of a computer device. Files stored on, or on computers and servers connected to a network. It is a file that is stored. The file contains multiple contents listed and stored. It is being done.

[0060] In step S12, the program displays the content it read from the file. The GUI displays the content according to the display format of the GUI. The content is preferably managed in units called records. Each record is structured by a label ID associated with a number, and content, etc. It will be accomplished.

[0061] In step S13, the user assigns learning labels. For example, displayed in the GUI. From n records, assign training labels to m records selected by the user. The remaining records will not be assigned labels. Note that m or n is a natural number. be.

[0062] In step S14, the program assigns learning labels to the m pieces of content. We will use this to perform machine learning and create the first learning model. Steps to classify content using a model and assign the classification results to each record. It has the following features. The first learning model includes decision trees, naive Bayes, KNN, SVM, and - Machine learning algorithms such as septrons, logistic regression, and neural networks You can use 'zum'.

[0063] To explain in detail, after the program generates the first learning model, the program then... Classify n pieces of content using a learning model. The program will classify n pieces of content as a result of the classification. Assign a judgment label and a first score to each record. Note that the first score is... This is the result of the computation that the program performs to estimate the decision label using the first learning model. .

[0064] In step S15, the program displays the results of the first learning model's classification in the GUI. The results classified by the first learning model are assigned a judgment label based on the first score. and is given as the first score. In more detail, the program is given the first learning model n records display the classification labels and first scores determined by Dell in the GUI. The items will be added to the list (displayed). Note that the judgment label and the first score are learning labels. It is also given to records that have been assigned the "L" trait.

[0065] Next, the user evaluates the displayed judgment label. It is necessary to evaluate all judgment labels. Instead, select any record from among those that do not have a learning label and make a judgment. You may do so. You may also use the results of this decision as training labels for the next step. User This involves selecting k records from among multiple records that do not have training labels assigned to them, and then adding them to the list of records. The process includes a step of assigning learning labels to the values, where k is greater than m and is a natural number. Yes. Note that k records may contain m records. When assigning training labels, it is preferable that the m training labels are not updated. The learning labels are not updated, so the information from the first learning model is carried over. However, If any of the training labels assigned to m records differ from the user's judgment, If it can be determined that it is a bell, it is preferable to assign a new learning label to that learning label. It seems so.

[0066] Return to step S14. The program then processes the k contents that have been assigned learning labels. The process includes a step of creating a second learning model using machine learning with the first The system evaluates the judgment labels assigned to the learning model, and further evaluates the k training labels added by the user. The program generates a second learning model. Dell reclassifies n items of content. The classification results are then processed by the program. Each record is assigned a judgment label and a second score. This is the result of the calculations the program performs to estimate the decision label using a second learning model. That is the case.

[0067] In step S15, the program displays the results of the second learning model's classification in the GUI. The results classified by the second learning model are assigned a judgment label based on the second score. and is given as the second score. In more detail, the program uses the second learning model n records display the classification label and second score determined by Dell in the GUI. Update the list of items to be presented (displayed). Note that the judgment label and the second score are used for learning. It is also given to records that have been assigned a bell.

[0068] By repeating steps S14 and S15, the learning model is updated. Furthermore, the accuracy of the classification improves. Therefore, the learning model that the program generates in the first machine learning run We call the first learning model the first learning model, and the learning model generated by the program in the second machine learning iteration is called the second. This is called the second learning model. Note that the learning model generated by the program in the p-th machine learning session is the second learning model. This is called the learning model of p.

[0069] Next, Figure 2 explains a different content classification method than that shown in Figure 1. Figure 2 is... Figure 1 differs from Figure 1 in that it loads content with learning labels. Therefore, Figure 2 Therefore, the assignment of learning labels in step S13 is not required. The other steps are the same as in Figure 1. For processing purposes, please refer to the explanation in Figure 1.

[0070] Figure 3 is a flowchart that explains the explanation in Figure 1 in detail. Each step is for the user, the program, or the GUI that the program has. This document describes the processing details. Furthermore, if the explanation overlaps with Figure 1, detailed explanations are omitted. It may be abbreviated.

[0071] Step S21 is the step to load content. The user loads content via the GUI. This involves loading a dataset, which is a collection of content, into the program. Ram displays the loaded dataset according to the GUI display format. Step S21 The content loaded corresponds to the population set used for classification.

[0072] In step S22, the user assigns learning labels. For example, displayed in the GUI. From n records (in the population), the user selects m records and assigns them training labels. A label will be assigned. Note that the remaining records will not be assigned learning labels. Also, m, n is a natural number.

[0073] In step S23, the program processes the m content items to which the learning labels have been assigned. We will use this to create a first learning model through machine learning.

[0074] In step S24, the program uses the first learning model generated to divide the content. The program then assigns a classification result to each record. The results of classification using the trained model are displayed in the GUI. The results of classification by the first trained model are The judgment label and the first score are determined based on the first score.

[0075] In step S25, the user evaluates the classification result of each record displayed in the GUI. You may evaluate whether the classification result is appropriate.

[0076] In step S26, the user performs additional learning on the results of the classification using the first learning model. Determine if further learning is needed. If the user determines that additional learning is needed, proceed to step S27. Proceed to the next step. Note that if the user determines that no further learning is required, the content classification will end. I'm done.

[0077] In step S27, the user adds additional learning labels. For example, they can be displayed in the GUI. From the n records (population) that were selected, training labels are assigned to all records except the m records in question. By adding learning labels to records that have not yet been assigned, k records can be assigned to k records. A new learning label will be assigned. Note that the remaining records will not be labeled. k is a natural number.

[0078] In step S23, the program assigns learning labels to the k content items. We will use this to create a second learning model through machine learning.

[0079] In step S24, the program uses the second learning model generated to divide the content. The program then assigns a classification result to each record. The results of the classification using the learning model are displayed in the GUI. The result is given as a judgment label and second score determined based on the second score. ru.

[0080] In step S25, the user determines the classification result of each record displayed in the GUI. To determine.

[0081] In step S26, the user performs additional learning on the results of classification using the second learning model. Determine if further learning is needed. If the user determines that additional learning is needed, proceed to step S27. Proceed to the next step. Note that if the user determines that no further learning is required, the content classification will end. I'm done.

[0082] The flowchart in Figure 3 shows an example where the user is responsible for determining the classification result. However, The program may be given some of the criteria for determining the classification result. The program can continue learning until the classification results converge.

[0083] For example, if a learning model assigns a judgment label and a score to a record, If the core exceeds the specified criteria, the learning label "Yes" will be automatically assigned. Yes, it is possible. Also, as a different example, if the score is lower than the relevant criterion, it will automatically use learning data. A label "No" can be assigned. The probability of content classification by the learning model is high. If the value is high or low, the learning model can automatically assign labels for retraining. Therefore, because part of the learning process of the learning model can be automated, a large amount of content It is suitable for classification.

[0084] Note that among records that have not been assigned training labels, those with a score near 0.5 (for example, 0.5) It is desirable to encourage users to label records with a value between 0.65 and above. The score can be rephrased as probability.

[0085] Figure 4 is a diagram illustrating the GUI30 of the program described above. Multiple GUIs are available. It consists of a column and multiple records. Column 31 displays the record number No. Column 32 displays the label ID associated with the record number. Mu33 displays a learning label "U-Label" linked to the record number. Column 34 is the judgment label “A-Label” associated with the record number No. This is displayed. Column 35 is the score "Score" associated with the record number No. This is displayed. 36 is the content associated with the record number No. "s" is displayed. The user then applies training data to columns 33a through 33n. A learning label can be provided. Note that the learning label "U-Label" is... It is preferable to provide this to the record you want to train.

[0086] Figure 5 is a different flowchart from Figure 3. Note that if the explanation overlaps with Figure 3, Detailed explanations may be omitted.

[0087] Step S41 loads text data. The user interacts with the program. This operation involves loading a dataset, which is a collection of text data, via a GUI. The program displays the loaded text according to the GUI display format. Step S41 The text data loaded corresponds to the population set used for classification.

[0088] In step S42, the user assigns learning labels. For example, displayed in the GUI. From n records (in the population), the user selects m records and assigns them training labels. Assign either "Yes" or "No" to each record. The remaining records will have learning labels. The attribute is not applied. Note that m or n is a natural number.

[0089] In step S43, the program processes the m content items to which the learning labels have been assigned. We will use this to create a first learning model through machine learning.

[0090] In step S44, the program uses the first learning model generated to process text data. The program classifies the data and assigns the classification result to each record. The results of the classification using the first learning model are displayed in the GUI. The results are given as a judgment label and first score, determined based on the first score. The first score is assigned a value between 0 and 1 for each of the n records. Additionally, the judgment label is assigned as either "Yes" or "No" to each of the n records. It will be done.

[0091] In step S45, the user determines the classification result of each record displayed in the GUI. To determine.

[0092] In step S46, the user performs additional learning on the results of the classification using the first learning model. Determine if further learning is needed. If the user determines that additional learning is needed, proceed to step S47. Proceed to the next step. Note that if the user determines that no further learning is required, the content classification will end. I'm done.

[0093] In step S47, the user adds additional learning labels. For example, they can be displayed in the GUI. From the n records (population) that were selected, training labels are assigned to all records except the m records in question. By adding learning labels to records that have not yet been assigned, k records can be assigned to k records. A new learning label is assigned. Note that k is a natural number.

[0094] In step S43, the program assigns learning labels to the k content items. We will use this to create a second learning model through machine learning.

[0095] In step S44, the content is divided using the second learning model generated by the program. The program then assigns a classification result to each record. The results of classification using the learning model are displayed in the GUI. Results of classification by the second learning model. This is the judgment label determined based on the second score and is given as the second score. .

[0096] In step S45, the user determines the classification result of each record displayed in the GUI. To determine.

[0097] In step S46, the user performs additional learning on the results of classification using the second learning model. Determine if further learning is needed. If the user determines that additional learning is needed, proceed to step S47. Proceed to the next step. Note that if the user determines that no further learning is required, the content classification will end. I'm done.

[0098] The flowchart in Figure 5 shows an example where the user is responsible for determining the classification result. However, The criteria for determining the classification result may be given to the program. The system can continue learning until the classification results converge.

[0099] For example, if a learning model assigns a judgment label and a score to a record, If the core exceeds the specified criteria, the learning label "Yes" will be automatically assigned. Yes, it is possible. Also, as a different example, if the score is lower than the relevant criterion, it will automatically use learning data. A label "No" can be assigned. The probability of content classification by the learning model is high. If the value is high or low, the learning model can automatically assign labels for retraining. Therefore, because part of the learning of the learning model can be automated, large amounts of text data It is suitable for classifying things.

[0100] Note that among records that have not been assigned training labels, those with a score near 0.5 (for example, 0.5) It is desirable to encourage users to label records with a value between 0.65 and 0.65.

[0101] Figures 6A and 6B illustrate GUI30. Figure 6A shows the flowchart of Figure 5. This corresponds to step S44 in the first learning model. In step S44, the first learning model The classified results are displayed in GUI30. The content is loaded with n records. This shows an example. Each record contains a number, a label ID, and text data. It has. In Figure 6A, text data is displayed in column 36. However, the explanation is simplified. To make it easier to understand, the text data is replaced with symbols for display.

[0102] The program records as “Group A” in order to generate the first learning model. Records 1 through 3 contain learning labels ("Yes" or "No") in columns 33a through 3. This shows an example given in Mu33c. Note that this example is a binary classification, but it can also be a classification of three or more values. They could be considered similar.

[0103] Figure 6A shows the program generating the first learning model and classifying each record. An example of assigning results is shown. The classification result is determined based on the first score. The bell ("Yes" or "No") and the first score are displayed. The core is assigned values ​​between 0 and 1 for n records.

[0104] For example, let's consider the case where the record number No. is 1. The judgment label is "A-Label". "Yes" is displayed in the field, and "0.874" is displayed in the score field. Yes. Users can assign a training label "U-Label" to the results classified by the learning model. Give. The score "Sco" when the judgment label "A-Label" displays "Yes". "re" indicates the probability of "Yes," and the judgment label "A-Label" is labeled "No." The score "Score" shown indicates the probability of "No".

[0105] As an example, the user displays the judgment label "A-Label" and the score "Score". If the result is judged to be correct, the learning label "U-Label" will be set to "Yes". As a different example, the user gives a judgment label "A-Label" and a score "Sco". If the result displayed as "re" is deemed incorrect, the learning label "U-La" will be used. Give "No" to "bel". The score "Score" of the classification result from the learning model is The user makes a judgment on records with a score near 0.5 (for example, between 0.5 and 0.65). The accuracy of the classifier can be improved by providing conditions.

[0106] Figure 6B shows how the program generates a second learning model, “Group B” and Then, in records 4 through 6, the learning label ("Yes" or "No") is entered in column 33. Examples are shown in columns d through 33f.

[0107] The program generates a second learning model and assigns a classification result to each record. An example of agreement is shown. The classification result is determined by the judgment label based on the second score (“ The first learning is shown in Figure 6A. In the model, the judgment label for record 7 was "Yes," and the first score was "0.538." However, in the second learning model shown in Figure 6B, the judgment label "No" for record 7, the second score A has changed to "0.625". As shown in Figure 6, the first learning model has a learning label. Adding a label improves the classification probability of records that were not assigned a training label. It is possible to do so. Note that the classification result, the judgment label and score, are merely examples and may not reflect the actual classification. The content of the labels and scores is not limited. Load content and assign learning labels. It changes depending on the circumstances.

[0108] Figure 7 is a diagram illustrating GUI30. The differences between Figure 7 and Figure 6A will be explained below. Figure 7 shows an example where n records are loaded as content. Each record contains a number, a label ID, and image data. In other words, in Figure 7, Column 36 shows an example of classifying image data as content. Therefore, in this implementation In one form of this configuration, a classification method for classifying image data can be provided. The content can include audio data, video data, and other similar data.

[0109] Figure 8 shows the electronic device 10 and the database to which the electronic device 10 is connected via a network. This is a diagram illustrating remote computers 21, 22, and 23. The device 10 includes a computing unit 11, memory 12, input / output interface 13, and communication device 1 4, and storage 15. The electronic device 10 has input / output interface 13 It is then electrically connected to the display device 16a and the keyboard 16b. 10 is electrically connected to the network interface 17 via the communication device 14. The network interface 17 then transmits data via the network. The base 21, remote computer 22, and remote computer 23 are electrically connected. The network can utilize a LAN, WAN, or the Internet. can.

[0110] A computer device in one aspect of this embodiment corresponds to an electronic device. The program installed in the device is stored in memory 12 or storage 15. Gram generates a learning model using the arithmetic unit 11. The program uses an input / output interface. -It can be displayed on the display device via face 13. The user can view what is displayed on the display device. Input data such as learning labels is provided to the program via the keyboard to the displayed GUI. It is possible.

[0111] Furthermore, this program can be accessed via the network from a remote computer 22, or a remote computer 22. It can also be used with the remote computer 23. Alternatively, it can be used with database 21, remote computer 23. Memory or storage of the remote computer 22 or the remote computer 23 The program stored in can be used to operate on the electronic device 10. Computer 22 is a personal information terminal, or a tablet computer, or a notebook computer. Mobile devices such as data terminals are also acceptable. In the case of mobile information terminals or other mobile devices, wireless communication is required. It can be used for communication.

[0112] Figure 9 shows the content being distributed to the display device 16a connected to the electronic device 10 described in Figure 8. This is an example of displaying GUI30A for similar purposes.

[0113] GUI30A consists of multiple columns and multiple records. A record is a column. It consists of columns 51 to 56, etc. Furthermore, GUI30A has buttons 57a, buttons 5 8. It has a display area 57b.

[0114] Column 51 displays the record number No. Column 52 displays the record number N The index associated with o is displayed. Column 53 is associated with the record number No. The user-specified label is displayed. Column 54 contains the automatic classification label “Automat” which is linked to the record number No. The "ic classification Label" is displayed. Column 55 contains: The probability of classification associated with the code number ("No.") is displayed. Column 56 displays the claim “CLAIM” associated with the record number No. The user-specified label corresponds to the training label, and the automatically classified label corresponds to the judgment label. Probability corresponds to the score.

[0115] For example, by pressing button 57a, the memory 12 or storage 1 of the electronic device 10 can be accessed. You will be able to select a file saved in 5. The save location of the file is displayed in display area 57b. If the file is selected, the file The information stored in the file is displayed in GUI30A. In Figure 9, the information stored in the file is The record information is displayed in columns 51, 52, and 56. In section 9, patent information is stored as content. Column 52 contains the patent number, Column 56 displays claim information, which is text data.

[0116] Next, the user uses the user as training data for column 53a, which contains multiple records. You can specify a label. Note that the user-specified label is used to train the model. It is preferable to provide it to the record.

[0117] Next, by pressing button 58, the program learns using the conditions given to GUI30A. Update the training model. Note that the training model will be named and stored in memory 12 or storage. It can be stored in 15. Therefore, the data stored in memory 12 or storage 15 The trained model can be used to train on new data, and the trained model can be updated. Alternatively, using the learning model stored in memory 12 or storage 15, the content It is possible to classify them as follows.

[0118] The program uses the updated learning model to process the content displayed in GUI30A. The program performs classification of the items. For example, the program automatically generates classification labels and confirmation labels as classification results. Assign a rate to each record. For example, the auto-classification label is assigned to each record. The contents are displayed in column 54a. The classification probability is displayed in column 55a, which contains each record. It can be displayed. The detailed text data of the claim is included in each record. It can be displayed in column 56a.

[0119] When a user sees the auto-classification label and probability, which are the classification results assigned to a record, If sufficient classification accuracy is achieved, the update of the learning model is terminated. The user assigns to the record. By examining the automatically generated classification labels and probabilities, we determine that the classification accuracy is insufficient. If this occurs, user-specified labels will be added to records that do not have user-specified labels. By agreeing to this and pressing button 58, the learning model can be updated.

[0120] Although not shown in the diagram, the display order is determined based on the numerical and label information contained in each column. It has a function that allows you to change it, or to hide it using a filter function. It is acceptable to allow this. This allows users to efficiently evaluate classification accuracy. For example, By displaying only records where the score (probability) is below a certain level, the record... You may change or add new learning labels to the code.

[0121] The content classification method explained using Figures 1 to 9 is a method for classifying information with a high probability. It can provide a method. For example, a GUI is suitable for classifying information with a high probability. The program works by providing the learning model with new training data (training labels). The learning model can be updated. The program's probability increases as the learning model is updated. It can classify a large amount of information.

[0122] Furthermore, before assigning learning labels to the entire record for the first or second time or later, unsupervised learning is performed on the entire record. The results of clustering performed by Xi can be displayed on a GUI, and users who see it can The records that provide training labels may be adjusted according to the clustering. Known rasterization methods include k-means clustering and DBSCAN. You may use this method.

[0123] Furthermore, the created learning model can be saved to the electronic device itself or to external memory. You can call it up and use it when classifying a new dataset, or you can use it with new training data. You can update the learning model by adding data and following the method described above.

[0124] The configurations and methods described in this embodiment can be appropriately combined with configurations and methods described in other embodiments. They can be used together. [Explanation of symbols]

[0125] 10: Electronic equipment, 11: Processing unit, 12: Memory, 13: Input / output interface, 14 :Communication device, 15:Storage, 16a:Display device, 16b:Keyboard, 17:Net Network interface, 21: Database, 22, 23: Remote computer ,30,30A:GUI

Claims

[Claim 1] A step of obtaining a dataset having (m+n) content items, consisting of m content items (where m is a natural number) that are assigned a first training label, and n content items (where n is a natural number) that are not assigned the first training label. The process involves creating a first learning model using machine learning with the aforementioned m pieces of content, A step of assigning a first judgment label and a first score to the m content items and the n content items using the first learning model, A step of creating a second learning model by machine learning using (q + m) content pieces, which are the n content pieces that have been assigned the first judgment label, q content pieces (where q is a natural number) that have been assigned the second learning label, and the m content pieces. The process includes the step of assigning a second judgment label and a second score to the m content items and the n content items using the second learning model, If it is determined that no further learning is necessary, the process terminates the classification of the m content items and the n content items. A method for classifying content, comprising the steps of: determining whether the second score of the m content items and the n content items is greater than or less than the judgment criterion when it is determined that additional learning is necessary, and assigning a label for retraining.

Images