Content Classification System

Abstract

To provide a classification model for classifying a content.SOLUTION: A learning content and a content are included. The learning content is given a first feature quantity and a learning label while the content is given a second feature quantity. A classification method of a content includes the steps of: generating a plurality of first classification models using plural learning contents by machine learning; generating a second classification model using the plurality of first classification models; and giving determination information to the plural contents using the second classification model to display the determination information on GUI. The determination information includes a classification label or a score. The GUI can specify a specific numerical range from the score, lists up the corresponding content to be displayed. The feature quantity given to the content is a management parameter (meta information).SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] One aspect of the present invention relates to a content classification method using a computer device, a content classification system, a classification model generation method, and a graphical user interface.

[0002] One aspect of the present invention relates to a computer device. One aspect of the present invention relates to a method for classifying digitized content (text data, image data, audio data, or video data) using a computer device. In particular, one aspect of the present invention relates to a content classification system that efficiently classifies a collection of content using machine learning. One aspect of the present invention relates to a content classification method, a content classification system, and a classification model generation method that use a graphical user interface managed by a computer device using a program. [Background technology]

[0003] Users want to easily classify and extract information related to topics they specify from a collection of content. However, when classifying content that meets the desired criteria from a large amount of content, the results of content classification vary depending on the knowledge and experience of each individual.

[0004] In recent years, there have been proposals to provide a computer with training data based on the results of content classifications made by individuals based on their knowledge and experience, and to use machine learning to learn how to classify content. For example, Patent Literature 1 discloses a machine learning approach for determining documents that are highly relevant to topics specified by a user. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-104630 Summary of the Invention [Problem to be solved by the invention]

[0006] A collection of content may be classified according to a purpose. In one aspect of the present invention, the content is a patent. Each patent is assigned a unique patent number. Therefore, hereinafter, content may be referred to as a patent number. Note that the content classification method used in one aspect of the present invention focuses on multiple management parameters assigned to patent numbers. However, the content is not limited to patent documents. The content can include information such as text data, image data, audio data, or video data.

[0007] Content is managed using various meta-information. Meta-information does not refer to the content itself, but rather to the attributes to which the content belongs or data describing related information. For example, a patent number is associated with the claims, abstract, drawings, and specification as content. Furthermore, patent numbers are given meta-information (evaluation information, number of days since issue, family information, etc.) and are managed using this meta-information. Patent numbers are classified according to importance using this meta-information. The accuracy and efficiency of classification depend on the content of the target document, but are easily affected by the user's experience and expertise. In addition, the need to classify a large number of documents poses challenges for efficiency.

[0008] Generating classification models using machine learning requires the preparation of a large amount of training data, which places an excessive burden on users. Another issue is that the variation in the number of classified contents included in the training data affects the accuracy of the classification model.

[0009] In view of the above problems, an object of one embodiment of the present invention is to provide a method for efficiently generating a classification model and classifying information using the same. Another object of one embodiment of the present invention is to provide a graphical user interface for interactively generating a classification model. Another object of one embodiment of the present invention is to provide a program for classifying information with a high probability.

[0010] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract other problems from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0011] A program is stored in a storage device of a computer device. The program can display various information on a display device of the computer device via a graphical user interface (hereinafter referred to as GUI). Note that a user can operate the program, add information, respond to a database, and give machine learning instructions to the computer device via the GUI. The program can also display the results of machine learning calculations, downloaded learning content or uncategorized content from a database, and the like on the display device via the GUI. Note that hereinafter, when simply referring to content, this includes learning content, uncategorized content, and categorized content.

[0012] The proposed content classification system uses machine learning to generate a content classification model, and then classifies unclassified content using the generated content classification model. For example, content with multiple pieces of meta information is used as training content. Training content is further assigned training labels to generate a feature vector from the training content. When generating a feature vector, the meta information or training labels can be treated as features of the training content.

[0013] The training content is treated as training data. A classification model can be obtained by performing machine learning based on the training content. The classification model obtained here classifies content that has multiple pieces of meta information. The classification types may be two or three or more, depending on the user's purpose. By using the classification model, the user can classify all documents in a shorter time than if they were to manually or visually judge all documents.

[0014] The training content can be downloaded from training content stored in a database. Alternatively, training content stored in a storage device of a computer device can be used. The training content may include and be managed as training labels. Furthermore, a classification model stored in a database can be downloaded. Alternatively, a classification model stored in a storage device of a computer device can be used.

[0015] One aspect of the present invention is a content classification method including: a training content and a content, the training content being assigned a first feature and a training label, and the content being assigned a second feature, the method including: generating a plurality of first classification models by machine learning using the plurality of training contents; generating a second classification model using the plurality of first classification models; and assigning judgment information to the plurality of contents using the second classification model and displaying the judgment information in a graphical user interface.

[0016] One aspect of the present invention is a content classification method including: a training content and a content, the training content being assigned a first feature and a training label, and the content being assigned a second feature, the method including: generating a plurality of first classification models by machine learning using the plurality of training contents; calculating an average value from the outputs of the plurality of first classification models; generating a second classification model using the plurality of average values; and assigning judgment information to the plurality of contents using the second classification model and displaying the judgment information in a graphical user interface.

[0017] One aspect of the present invention is a content classification method including: a learning content and a content, the learning content being assigned a first feature and a learning label, and the content being assigned a second feature, the method including: generating a plurality of first classification models by machine learning using the plurality of learning contents; evaluating each of the plurality of first classification models according to a first evaluation criterion; evaluating each of the plurality of first classification models according to a second evaluation criterion; generating a second classification model from the evaluation results according to the plurality of first evaluation criterion and the second evaluation criterion; and assigning judgment information to the plurality of contents using the second classification model and displaying the judgment information in a graphical user interface.

[0018] In the above configuration, it is preferable that the content classification method be such that the first evaluation criterion is precision and the second evaluation criterion is sensitivity.

[0019] In each of the above configurations, the content classification method preferably includes a step of generating a first classification model using any training content.

[0020] In each of the above configurations, it is preferable that the content classification method further includes a step of providing classification information to the learning content, selecting content having the same judgment information as the classification information from multiple contents to which classification labels have been assigned using the output of the second classification model, and displaying the content within the graphical user interface.

[0021] In each of the above configurations, the feature amount given to the learning content or content is preferably a content classification method that is a management parameter.

[0022] In each of the above configurations, the determination information preferably includes a content classification method including a classification label or a score.

[0023] In the above configuration, the graphical user interface preferably has a content classification method including a step of specifying a specific numerical range of the scores and displaying the corresponding content as a list. [Effects of the Invention]

[0024] One aspect of the present invention can provide a method for accurately classifying information, a user interface for accurately classifying information, or a program for accurately classifying information.

[0025] In addition, one aspect of the present invention can provide users with an interactive interface for generating classification models using machine learning, thereby reducing the burden on users of preparing training data and evaluating learning results.

[0026] The effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. The other effects are described below and are not mentioned in this section. Effects not mentioned in this section can be derived by a person skilled in the art from the description in the specification or drawings, and can be extracted as appropriate from these descriptions. One embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a flowchart illustrating the classification method. [Figure 2] FIG. 2 is a flowchart illustrating the classification method. [Figure 3] FIG. 3 is a diagram illustrating the connection between classification system 100 and a network. [Figure 4] FIG. 4 is a block diagram illustrating the classification system. [Figure 5] 5A and 5B are diagrams illustrating a graphical user interface. [Figure 6] FIG. 6 is a diagram illustrating a method for generating a classification model. [Figure 7] FIG. 7 is a diagram illustrating a method for generating a classification model. [Figure 8] FIG. 8 is a diagram illustrating a method for generating a classification model. [Figure 9] FIG. 9 is a diagram illustrating a graphical user interface. [Figure 10] FIG. 10 is a diagram illustrating a graphical user interface. DETAILED DESCRIPTION OF THE INVENTION

[0028] In this embodiment, a content classification method will be described with reference to FIGS.

[0029] The content classification method described in this embodiment is controlled by a program running on a computer device. The program is stored in the memory or storage of the computer device, or in a computer connected via a network (such as a local area network (LAN), a wide area network (WAN), or the Internet) or a server computer having a database.

[0030] The display device of the computer device can display data that the user provides to the program and the results of calculations performed on the data by the arithmetic device of the computer device. The configuration of the device will be described in detail with reference to Figure 4.

[0031] The data displayed on the display device can be easily recognized by the user and the operability can be improved by following a display format such as a list. Therefore, the interface that allows the user to easily interact with the programs of the computer device via the display device will be described as a GUI.

[0032] The user can use the content classification method of the program via the GUI. The user can easily classify the content using the GUI. Furthermore, the user can easily visually determine the content classification results via the GUI. Furthermore, the user can easily operate the program via the GUI. Note that the content refers to information such as text data, image data, audio data, or video data.

[0033] Next, a method for classifying content using a GUI will be described in accordance with the GUI operation procedure. First, the data processing unit will be described. The data processing unit has a data collection unit and a data generation unit. For example, the data collection unit acquires a file consisting of multiple pieces of content from a database via the GUI. The data generation unit can generate training content by having a user assign training labels to the content via the GUI. Alternatively, training content with training labels may be acquired from a database. Note that the multiple pieces of content are files stored in memory or storage of a computer device, or data stored in a database, computer, data server, or the like connected to a network.

[0034] Therefore, it is preferable that the database stores a list of multiple learning content items or multiple unclassified content items. The learning content items and unclassified content items are assigned multiple features and learning labels. The learning labels can be modified by the user via a GUI. When the learning labels are assigned to the learning content items, the learning content items with the learning labels can be stored in the database.

[0035] The training content may include validation content that is not labeled for training purposes and can be used to validate a classification model generated using the training content.

[0036] As an example, let us consider the case where the content is a patent number. A patent number is assigned multiple pieces of meta information as the patent number's feature quantities. For example, the meta information may include evaluation information, number of days elapsed, number of families, family status, application type, life, number of pending applications in the family, number of abandoned applications in the family, cost, number of inventors, field, or number of claims. In other words, the meta information is a content management parameter. Note that family includes patent families, patent families, etc.

[0037] Next, the learning processing unit will be described. The learning processing unit has a step of generating a classification model using training content. The learning processing unit has a classification model generation unit or a classification model evaluation unit.

[0038] The classification model generation unit can generate a classification model. The classification model generation unit includes a step of generating a plurality of first classification models by machine learning using a plurality of training contents, and a step of generating a second classification model using the plurality of first classification models. An output value of the first classification model or the second classification model can be displayed on a GUI. A user can assign (including correct) a training label of the first classification model to the output value. Alternatively, a user can add new training content to the output value.

[0039] The classification model evaluation unit evaluates the classification model generated by the classification model generation unit using the verification content. When the classification model is used to infer the verification content, the classification model outputs the inference result as judgment information. The GUI can display each evaluation content by adding the judgment information.

[0040] The user can evaluate the output result of the classification model evaluation unit, modify the learning labels as necessary, and update the classification model in the classification model generation unit. Alternatively, the user can update the classification model in the classification model generation unit by adding learning content.

[0041] Next, the judgment processing unit will be described. The judgment processing unit has a classification inference unit and a list generation unit. For example, the classification inference unit infers and classifies multiple unclassified content items using the first learning model and the second learning model generated by the classification model generation unit. The classification model assigns the inference results to each piece of content as judgment information.

[0042] The list generation unit can generate a list in a format desired by the user from the content given the determination information and display it on a GUI. For example, if each piece of content is managed by filing country, the filing country can be used as classification information. When the filing country is used as classification information, it is preferable that a different classification model be generated for each filing country. Note that classification information is not limited to the filing country. For example, one piece of meta information contained in the content can be used as classification information.

[0043] A case where meta-information is used as classification information will be described. For example, the status of a patent family may be used as classification information. Meta-information may include the patent number of a parent application, the patent number of a divisional application, etc. Different classification models can be generated based on whether a divisional application is possible from the patent number of a parent application, whether a divisional application is not possible from the patent number of a parent application, whether the patent number of a parent application is in patent pending status, whether the patent number of a parent application is in patent void status, whether a divisional application is possible from the patent number of a divisional application, whether a divisional application is not possible from the patent number of a divisional application, whether the patent number of a divisional application is in patent pending status, or whether the patent number of a divisional application is in patent void status, and different classification models can be used to make inferences.

[0044] That is, the determination processing unit can infer multiple unclassified pieces of content using the classification model. The method includes a step of assigning the inferred results to each piece of content as classification information and displaying it on a GUI. The classification information includes at least a classification label and a score (probability). The GUI also includes a step of specifying a specific numerical range of the scores and displaying the corresponding pieces of content as a list.

[0045] An example different from the classification model generation unit described above is shown. The classification model generation unit includes the steps of generating a plurality of first classification models by machine learning using a plurality of training contents, calculating an average value from the outputs of the plurality of first classification models, and generating a second classification model using the plurality of average values. The output values ​​of the first classification model or the second classification model can be displayed on a GUI. A user can modify the training labels of the first classification model for the output values. Alternatively, a user can add training content to the output values. The average value means that it is calculated using any one of an arithmetic mean calculation, a geometric mean calculation, and a harmonic mean calculation.

[0046] The second classification model is generated using the average values ​​of the first classification model, and the second classification model averages the outputs of the first classification model, thereby reducing the influence of noise components such as outliers in the training content.

[0047] Next, an example different from the classification model generation unit described above will be described. The classification model generation unit includes the steps of generating a plurality of first classification models by machine learning using a plurality of training contents, evaluating each of the plurality of first classification models according to a first evaluation criterion, evaluating each of the plurality of first classification models according to a second evaluation criterion, and generating a second classification model from the evaluation results according to the plurality of first evaluation criterion and the evaluation results according to the second evaluation criterion. The output values ​​of the first classification model or the second classification model can be displayed on a GUI. A user can modify the training labels of the first classification model for the output values. Alternatively, a user can add training content for the output values. The first evaluation criterion is the accuracy of the confusion matrix, and the second evaluation criterion is the sensitivity of the confusion matrix.

[0048] A second classification model is generated using the results of evaluating the outputs of multiple first classification models using a first evaluation criterion and a second evaluation criterion. Regarding the first evaluation criterion, the accuracy of the confusion matrix can be rephrased as the precision rate for the training labels. Regarding the second evaluation criterion, the sensitivity of the confusion matrix can be rephrased as the recall rate for the training labels. The generated classification model can include the precision rates and recall rates of the multiple first classification models. The second classification model generated using the multiple first classification models has improved classification accuracy.

[0049] In the above-described classification model generation unit, for example, a second classification model may be generated using m (m represents a natural number) generated first classification models.

[0050] If the classification model generation unit has k training contents (k is a natural number), the first classification model can generate any training contents up to k. If k training contents are selected, any training contents can include two different training models. The k training contents with different numbers can be sorted by number, and the first classification model can be generated using q training contents (q is a natural number) at a time.

[0051] The program can display content read from the database on a GUI. The content preferably has listed meta information. The GUI displays the content according to its display format. The listed meta information assigned to the content is preferably managed in units called records. For example, each record is composed of an ID (Identification) linked to a number, content (image data, audio data, or video data), meta information, etc.

[0052] In this specification, machine learning is performed with a focus on meta information, and a classification model is generated by the machine learning. The classification model analyzes the meta information and classifies content that has been converted into feature vectors.

[0053] Furthermore, the content classification method described above can perform classification by machine learning without using learning labels as training data. For example, algorithms such as K-means or DBSCAN (density-based spatial clustering of applications with noise) can be used as the classification model.

[0054] The program can also generate a classification model using machine learning with the training content to which the meta information and training labels have been added, using algorithms such as decision trees, Naive Bayes, kNearest Neighbor (KNN), Support Vector Machines (SVM), Perceptron, logistic regression, or neural networks.

[0055] Furthermore, the program can switch classification models depending on the number of training contents. For example, when the number of training contents is small, a decision tree, naive Bayes, or logistic regression may be used, while when the number of training contents is equal to or greater than a certain amount, an SVM, random forest, or neural network may be used. The classification model used in this embodiment uses random forest, which is a decision tree algorithm. Furthermore, random sampling or cross-variation may be used as a method for selecting meta information, a method for selecting training contents, or a method for selecting a first classification model. Alternatively, q items may be selected in the sorted order of the assigned numbers.

[0056] Next, a content classification method will be described using drawings. FIG. 1 is a flowchart illustrating a content classification method according to one embodiment of the present invention. The content classification method is controlled by a program running on a computer device. Therefore, the program can classify content by having a data processing unit, a learning processing unit, or a judgment processing unit. The program can classify content as desired by the user via a GUI. In other words, the content processed by each of the above-mentioned processing units corresponds to the steps of the program.

[0057] In step S11, the user can instruct the loading of a file containing content via the GUI. The file is stored in a database held by the data processing unit. The file may include learning content, unclassified content, etc.

[0058] Therefore, it is preferable that the database stores a list of multiple training contents or multiple unclassified contents. The user can assign or modify training labels to the training contents displayed in the GUI. Note that the file may also include verification content that has not been assigned a training label.

[0059] Step S12 is a learning processing unit that generates a classification model using the loaded file. The generated classification model can evaluate the verification content and display the evaluation results in the GUI. The user can then instruct the system to modify the learning labels, add learning content, etc. based on the evaluation results.

[0060] The user can predict changes over time in the meta information of the learning content and update the meta information. When the user updates the meta information, the classification model can include changes over time in the classification model. Therefore, the user can obtain changes in the classification of the content over time. The classification model can classify content into groups whose value is expected to increase or decrease.

[0061] Step S13 is a judgment processing unit. Uncategorized content is inferred using the classification model generated in step S12. The classification model can assign judgment information to the uncategorized content based on the inference results. The judgment processing unit can display the content to which the judgment information has been assigned in a GUI in a format desired by the user. The judgment information includes at least a classification label and a score. The GUI can also specify a specific numerical range of the score and display the corresponding content.

[0062] Next, the flowchart of Fig. 1 will be explained in more detail with reference to Fig. 2. First, the details of step S11 will be explained. The data processing section of step S11 has a data collection section of step S21 and a data generation section of step S22.

[0063] The data collection unit in step S21 will now be described. The data collection unit in step S21 can load files from a database. Meta information or content can be managed by different databases. Meta information may differ depending on the company, organization, or user that handles the content. Therefore, the data collection unit has a function of collecting meta information related to content from different databases. Each database can be installed in a different building, a different region, or a different country.

[0064] Next, the data generation unit in step S22 will be described. The data generation unit can manage content and meta information in units called records. For example, each record is composed of an ID associated with a number, content (image data, audio data, or video data), meta information, etc. In addition, the user can generate study content by assigning study labels to content displayed in the GUI.

[0065] Next, details of step S12 will be described. The learning processing unit in step S12 has a classification model generation unit in step S23, a classification model evaluation unit in step S24, and an output result determination process in step S25.

[0066] The classification model generation unit in step S23 will be described. The classification model generation unit can generate a classification model of content. The classification model generation unit can generate a plurality of first classification models by machine learning using a plurality of learning contents. The classification model generation unit can generate a second classification model using the plurality of first classification models. The GUI can display the output value of the first classification model or the second classification model.

[0067] After the output result determination process in step S25 (described later), the user can assign (including correct) a learning label for the first classification model to the output value. Alternatively, the user can add new learning content to the output value. The user can predict changes over time in the meta information of the learning content and update the meta information. The effect of updating the meta information by the user can be seen in the explanation of step S12.

[0068] Next, the classification model evaluation unit in step S24 will be described. The classification model evaluation unit can evaluate the classification model generated by the classification model generation unit by using the verification content. The classification model outputs the result of inferring the verification content as judgment information. The GUI can display each evaluation content by adding judgment information to it.

[0069] Next, the output result determination process of step S25 will be described. For example, the user can determine that the content classification model has been sufficiently trained by judging the output result of the classification model evaluation unit in step S24. The user issues an instruction to the GUI indicating that generation of the classification model is complete (OK). For example, the user can determine that training of the content classification model is insufficient (NG). The user returns to step S23 and updates the classification model by changing the training labels, adding training content, or updating meta information.

[0070] Next, details of step S13 will be described. The determination processing unit in step S13 has a classification inference unit in step S26 and a list creation unit in step S27.

[0071] The classification inference unit in step S26 will be described. The classification inference unit infers and classifies a plurality of unclassified contents using the first learning model and the second learning model generated by the classification model generation unit. The unclassified contents generated by the data generation unit in step S22 are provided to the classification inference unit. The classification model assigns the inference result to each content as judgment information.

[0072] The list creation unit in step S27 will now be described. The list creation unit can list the content to which the determination information has been given in a format desired by the user and display it on a GUI. Note that each piece of content may be given classification information different from the meta information. For example, if classification information is given to learning content, a different classification model can be generated for each piece of classification information. Alternatively, one piece of meta information possessed by the content can be used as the classification information.

[0073] The judgment information includes at least a classification label and a score. The GUI can also specify a specific numerical range of the score and display the corresponding content in a list on the GUI.

[0074] FIG. 3 is a diagram illustrating the connection between a classification system 100 having the above-described content classification method and a network.

[0075] The classification system 100 is connected to a communication network LAN1. A database DB1 or client computers CL1 to CLn (n is a natural number) are connected to the communication network LAN1. The communication network LAN1 can be connected to a communication network LAN2 via a network. The network can be the Internet, a communication network WAN, or satellite communication. The communication network LAN2 is connected to a database DB2 or client computers CL11 to CL1n.

[0076] The classification system 100 can generate content, classify content, generate models, and classify unclassified content using files containing content stored in database DB1, database DB2, client computers CL1 to CLn, or client computers CL11 to CL1n.

[0077] A user can also issue commands to the GUI from a program running on classification system 100. For example, a user can generate the above-described classification model using information from databases installed in different countries via the Internet to classify unclassified content. In other words, content or meta information may be stored in different databases or client computers.

[0078] The GUI can display the classification results classified by the classification system 100 stored in the storage device of the database DB1, the database DB2, the client computers CL1 to CLn, or the computer device of the client computers CL11 to CL1n.

[0079] FIG. 4 is a block diagram illustrating the classification system 100 described in FIG. 3. The classification system 100 has a GUI (Graphical User Interface) 110, a calculation unit 120, and a storage unit 130. The GUI 110 has an input unit 111 and an output unit 112. The input unit 111 has a function of selecting a content loading source and a function of inputting learning labels. The output unit 112 has a function of displaying a list of content loaded from a database or the like and a function of displaying determination information output by the classification model. Note that the user can modify meta information included in the displayed content via the GUI.

[0080] The calculation unit 120 has a data processing unit 121, a learning processing unit 122, and a judgment processing unit 123. The data processing unit 121 has a data collection unit and a data generation unit. The learning processing unit 122 has a classification model generation unit that creates a classification model, and a classification model evaluation unit that classifies the classification model. The output results of the classification model evaluation unit have the function of an evaluation result judgment process in which a user makes a judgment. The judgment processing unit 123 has a classification inference unit and an output list creation unit that lists the results classified by the classification inference unit. The calculation unit 120 performs arithmetic processing using a microprocessor based on a program stored in a memory unit of the computer device. However, the program can also be processed using a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit).

[0081] The storage unit 130 temporarily stores a list of content and meta information that have been generated by loading from a database or the like.

[0082] The storage unit 130 may be, for example, a DRAM (Dynamic Random Access Memory) having 1T (transistor) 1C (capacitor) type memory cells. The transistors used in the DRAM memory cells may be OS transistors. An OS transistor is a transistor having a metal oxide in its semiconductor layer. A memory device using OS transistors in its memory cells is called an "OS memory." Here, as an example of an OS memory, a RAM having 1T1C type memory cells is called a "Dynamic Oxide Semiconductor RAM (DOSRAM)."

[0083] The off-state current of an OS transistor is very small. Therefore, DOSRAM can reduce the frequency of refresh operations, thereby reducing the power required for refresh operations. The off-state current here refers to the current that flows between the source and drain when the transistor is in the off state. If the transistor is an n-channel type and the threshold voltage is, for example, about 0 V to 2 V, the current that flows between the source and drain when the voltage between the gate and source is negative can be called the off-state current.

[0084] FIG. 5A is a diagram illustrating the configuration of GUI 30. GUI 30 shows, as an example, a management screen that displays a list of p study contents. Study contents are managed on a record-by-record basis. The record includes a number (No) 31, content (ID) 32, meta information (Feature) 33 indicating features (meta information (F1) 33a through meta information (Fm) 33m), classification information (Case) 34 (classification information (C1) 34a through classification information (Cq) 34q), and study label (J-Label) 35. As an example, in FIG. 5A, study label 35 is assigned one of two values, "Yes" or "No," but study label 35 is not limited to two values ​​and may be three or more values.

[0085] 5B is a diagram illustrating the configuration of GUI 30A. GUI 30A shows a management screen on which n pieces of unclassified content are inferred by the evaluation inference unit and the inferred judgment information is displayed in a list. Like learning content, unclassified content has a number 31, content 32, meta information 33, and classification information 34. Furthermore, each record is assigned a classification label (A-Label) 36 and a score (Score) 37 as judgment information.

[0086] Note that GUI 30 and GUI 30A can be managed on the same display screen. Figures 9 and 10, which will be described later, show examples of GUI displays that can display study content and assessment information on the same management screen.

[0087] 6 is a diagram illustrating a method for generating a classification model using multiple features associated with the training content Sample through machine learning. Each feature indicates one piece of meta information and corresponds to a management parameter for managing content. In one embodiment, a method for generating a classification model is described using a calculation unit F, a calculation unit S, a calculation unit V, a first classification model, and a second classification model.

[0088] Each of the training content Sample(1) through Sample(k) is assigned j features (Feature) and a training label (Label). As an example, the calculation unit F1 can generate a feature vector Vlabel1(1) from the training content Sample(1) in a format that can be processed by a computer. The calculation unit Fk can generate a feature vector Vlabel1(k) from the training content Sample(k) in a format that can be processed by a computer. The feature vector Vlabel1(1) can be generated by the calculation unit F1 assigning different weighting coefficients to each feature. The feature vector Vlabel1(1) can also be generated using j or fewer randomly selected features (Feature).

[0089] Next, a plurality of first classification models are generated. The calculation units S1 to Sm correspond to different first classification models. As an example, the calculation unit S1 can generate the first classification model using the feature vectors Vlabel1(1) to Vlabel1(k). Note that the number of feature vectors Vlabel1 provided to the calculation unit S1 needs to be k or less. As a different example, the calculation unit Sm can generate the first classification model using different feature vectors Vlabel1(1) to Vlabel1(k). Thus, two different first classification models each include k or less feature vectors Vlabel1, and any one of the feature vectors Vlabel1 includes the same feature vector.

[0090] The k training content samples selected to generate the first classification model may be selected randomly or in a sorted order based on the numbers assigned to the training content. If the training content samples are selected randomly, the first classification model may include variations in the training content. If the training content samples are selected in a sorted order based on the numbers assigned to the training content, the first classification model may include trends along the assigned numbers based on a time series or a feature of meta information.

[0091] Therefore, the first classification model can generate the feature vector Vlabel2 using the feature vector Vlabel1 generated from the training content Sample(1) through Sample(k).

[0092] The second classification model is generated by the calculation unit V1. As an example, the calculation unit V1 has a step of generating the second classification model using m feature vectors Vlabel2. Note that the second classification model can generate classification models with different features by using the feature vectors Vlabel2(1) to Vlabel2(m).

[0093] Therefore, the second classification model can output an output value POUT using the feature vector Vlabel1 generated from the learning content Sample(1) to Sample(k). The GUI can display the output value POUT. The output value POUT includes a classification label and a score, which are judgment information. Therefore, the second classification model can classify content. Furthermore, the second classification model can assign judgment information to each piece of content.

[0094] To perform inference using the classification model described in Figure 6, a judgment result is obtained by providing unclassified content as the training content Sample of the classification model. However, unlike the training content, the unclassified content does not have a training label attached to it.

[0095] Figure 7 is a diagram for explaining a method for generating a classification model different from that in Figure 6. In Figure 7, differences from Figure 6 are explained, and in the configuration of the invention (or the configuration of the embodiment), the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and repeated explanations thereof will be omitted.

[0096] In FIG. 7, the average value Av of m feature vectors Vlabel2 is calculated to generate a feature vector Vlabel_a. A second classification model can be generated using p feature vectors Vlabel_a. The second classification model can generate a classification model with different features by calculating the average value Av of m feature vectors Vlabel2. The generated classification model can accurately classify content.

[0097] Figure 8 is a diagram for explaining a method for generating a classification model different from that shown in Figure 7. Figure 8 explains the differences from Figure 7, and in the configuration of the invention (or the configuration of the embodiment), the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and repeated explanations thereof will be omitted.

[0098] In FIG. 8, evaluation criteria for evaluating m feature vectors Vlabel2 are given to the evaluation and judgment unit JG. For example, the evaluation and judgment unit JG1 is given precision as a first evaluation criterion and can evaluate each feature vector Vlabel2(1). Next, the evaluation and judgment unit JG1 is given sensitivity as a second evaluation criterion and can evaluate each feature vector Vlabel2(1). The evaluation and judgment unit JG1 outputs the evaluation result Vlabel_b(1).

[0099] The second classification model is generated using the evaluation results Vlabel_b(1) to Vlabel_b(p). For example, multiple feature vectors Vlabel2 may be evaluated using different first and second evaluation criteria, or may be evaluated using the same evaluation criteria. Although not shown in FIG. 8, the average value of the evaluation results Vlabel_b using the first and second evaluation criteria can be calculated, as in FIG. 7.

[0100] The second classification model can generate a classification model with different features by using the evaluation results of the m feature vectors Vlabel2. The generated classification model can accurately classify the content.

[0101] 9 is a diagram illustrating the GUI 50. The GUI 50 has a display area for content (learning content, uncategorized content, and categorized content), an icon 58a for selecting the download source of a file containing the content, a text box 58b for displaying address information where the selected file will be saved, and an icon (Learning Start) 59 for executing machine learning.

[0102] The display area shows an example in which eight records are loaded. Each record has components including a number (No) 51, an ID (Index) 52, a feature (Feature) 53, classification information (Case) 54, a learning label (JL) 55, a classification label (AL) 56, and a score (Prob) 57. The feature 53 can display feature F(1) 53a through feature F(j) 53j as detailed information, where j is a natural number. The classification information 54 can display classification information C(1) 54a through classification information C(4) 54d as detailed information. The classification information can have types that can be expressed by natural numbers.

[0103] FIG. 9 shows an example in which the results of classifying learning content and unclassified content using a classification model are displayed on the GUI 50.

[0104] As an example, record numbers No. 1 to No. 3 correspond to study content. Study content is assigned a study label, and record numbers No. 1 to No. 3 are assigned classification information.

[0105] Record numbers 4 to 8 correspond to classified content. Classification labels 56 and scores 57 are assigned to the classified content. Note that FIG. 9 shows the results of classifying record numbers 4 to 7 using a classification model obtained by learning record numbers 1 and 3. As an example, the results of classifying record number 8 using a classification model obtained by learning record number 2 are shown. Due to space limitations, FIG. 9 only shows up to 8 records, but multiple types of records can be handled.

[0106] However, handling a large number of records poses display issues. Therefore, it is preferable to provide a sorting function for the classification label 56 or score 57. As an example of a sorting condition, the GUI can select and display the determination result for which the classification label 56 is "Yes." The GUI can also specify a numerical range for the score 57. When the above-mentioned sorting conditions are given to the GUI, the GUI can classify and display content that has similar characteristics to the learning content to which the training data is given.

[0107] For example, let's consider the case where the content is a patent number. Patent numbers are assigned multiple pieces of meta information. For patent numbers where the patent rights are maintained, the learning label is assigned "Yes." For patent numbers where the patent rights have been abandoned, the learning label is assigned "No." Next, machine learning is performed to generate a classification model.

[0108] The classification model described above can assign judgment information to unclassified content. The judgment information displays a classification label 56 and a score 57. As an example, a user can use the sort function to assign "No" to the classification label 56. Furthermore, the score 57 can be set to a value between "0.8" and "1.0." By providing the GUI with the sorting conditions described above, the GUI can select and display records that have similar characteristics to study content with abandoned patent numbers.

[0109] Fig. 10 is a diagram illustrating a GUI 50A that is different from Fig. 9. Fig. 10 is an example of an efficient GUI display when handling a large number of records. Note that Fig. 10 explains the differences from Fig. 9, and in the configuration of the invention (or the configuration of the embodiment), the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and repeated explanations thereof will be omitted.

[0110] Fig. 10 differs from Fig. 9 in that records relating to any selected classification information can be categorized and displayed. In Fig. 10, the display can be switched according to the type of classification information C(1) to C(4).

[0111] The user reviews the multiple features 53 assigned to the records and the classification model's judgment information, and if sufficient classification accuracy is obtained, the user ends the classification model update. If the user reviews the judgment information assigned to the records and determines that the classification accuracy is insufficient, the user can update the classification model by assigning a learning label to records that do not have a user-specified label and pressing icon 59. Note that the features 53 may be updated by predicting changes over time in the meta information of the learning content. When the user updates the features 53, the classification model can include changes over time in the classification model. Therefore, the user can obtain changes in the classification of content over time. The classification model can classify content groups that are expected to increase in value or content groups that are expected to decrease in value.

[0112] Although not shown, the display order of the numerical values ​​and label information included in the feature 53, the classification information 54a to 54d, the learning label 55, the classification label 56, or the score 57 can be changed, or selected numerical values ​​and label information can be sorted and displayed in a required order using a filter function. This allows the user to efficiently evaluate the judgment results of the classification model.

[0113] The content classification method described using FIGS. 1 to 10 can provide a method for classifying information with a high probability. For example, a GUI is suitable for classifying information with a high probability. A program can update a classification model by providing new training data (learning labels) to the classification model. By updating the classification model, the program can classify information with a high probability.

[0114] Furthermore, the generated classification model can be stored in the electronic device itself or in an external memory, and can be recalled and used when classifying new files. Furthermore, the classification model can be updated by adding new training data according to the method described above.

[0115] The structures and methods described in this embodiment mode can be used in appropriate combination with structures and methods described in other embodiment modes. [Explanation of symbols]

[0116] CL1: client computer, CL1n: client computer, CL11: client computer, CLn: client computer, DB1: database, DB2: database, LAN1: communication network, LAN2: communication network, Vlabel1: feature vector, Vlabel2: feature vector, 31: number, 32: content, 33: meta information, 34: classification information, 35: learning label, 50: GUI, 50A: GUI, 51: number, 53: feature, 54: classification information, 56: classification label, 57: score, 58a: icon, 58b: text box, 59: icon, 100: classification system, 110: GUI, 111: input unit, 112: output unit, 120: calculation unit, 121: data processing unit, 122: learning processing unit, 123: judgment processing unit, 130: memory unit

Claims

1. A system comprising a graphical user interface, a calculation unit, and a storage unit, the storage unit has a function of storing study content and content; a first feature amount and a learning label are assigned to the learning content; a second feature is assigned to the content; The calculation unit a function of generating a plurality of first classification models by machine learning using a plurality of the learning contents; a function of generating a plurality of first feature vectors from each of the plurality of training contents using the plurality of first classification models; a function of calculating an average value using the plurality of first feature vectors; generating a second classification model using a plurality of the mean values; a function of assigning determination information to the plurality of pieces of content using the second classification model; and the graphical user interface has a function of displaying a plurality of the contents and the determination information assigned to each of the plurality of the contents as a list; A content classification system.

2. A system including a graphical user interface, a calculation unit, and a storage unit, the storage unit has a function of storing study content and content; a first feature amount and a learning label are assigned to the learning content; a second feature is assigned to the content; The calculation unit a function of generating a plurality of first classification models by machine learning using a plurality of the learning contents; a function of generating a plurality of first feature vectors from each of the plurality of training contents using the plurality of first classification models; a function of performing an evaluation based on a first evaluation criterion using each of the plurality of first feature vectors; a function of performing an evaluation based on a second evaluation criterion using each of the plurality of first feature vectors; a function of generating a second classification model from a plurality of evaluation results according to the first evaluation criteria and a plurality of evaluation results according to the second evaluation criteria; a function of assigning determination information to the plurality of pieces of content using the second classification model; and the graphical user interface has a function of displaying a plurality of the contents and the determination information assigned to each of the plurality of the contents as a list; A content classification system.

3. In claim 2, the first evaluation criterion is accuracy; The second evaluation criterion is sensitivity of the content classification system.

4. In claim 1 or claim 2, A content classification system in which the learning content or the feature values ​​assigned to the content are management parameters.

5. In claim 1 or claim 2, A content classification system in which the determination information includes a classification label or a score.

6. In claim 5, The graphical user interface has a function of specifying a specific numerical range of the scores and displaying corresponding content as a list.

Images