Classification system of content

Abstract

To provide a classification model for classifying a content.SOLUTION: A learning content and a content are included. The learning content is given a first feature quantity and a learning label while the content is given a second feature quantity. A classification method of a content includes the steps of: generating a plurality of first classification models using plural learning contents by machine learning; generating a second classification model using the plurality of first classification models; and giving determination information to the plural contents using the second classification model to display the determination information on GUI. The determination information includes a classification label or a score. The GUI can specify a specific numerical range from the score, lists up the corresponding content to be displayed. The feature quantity given to the content is a management parameter (meta information).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] One aspect of the present invention relates to a content classification method, a content classification system, a classification model generation method, and a graphical user interface using a computer device.

[0002] One aspect of the present invention relates to a computer device. One aspect of the present invention relates to a method for classifying digitized content (text data, image data, audio data, or video data) using a computer device. In particular, one aspect of the present invention relates to a content classification system that efficiently classifies a collection of content using machine learning. One aspect of the present invention relates to a content classification method, a content classification system, and a classification model generation method using a graphical user interface managed by a computer device through a program. [Background technology]

[0003] Users want to easily classify and extract information on topics that they specify from a collection of content. However, when classifying content that meets a desired condition from a large amount of content, the content classification results vary depending on the knowledge or experience of each individual.

[0004] In recent years, a proposal has been made to provide a computer with the results of categorizing content based on an individual's knowledge and experience as training data, and to machine-learn how to categorize content. For example, Patent Literature 1 discloses a machine learning approach for determining documents that are highly relevant to topics specified by a user. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2009-104630 A Summary of the Invention [Problem to be solved by the invention]

[0006] A collection of content may be classified according to a purpose. In one aspect of the present invention, a case will be described in which the content is a patent. Each patent is given a unique patent number. Therefore, hereinafter, the content may be described as a patent number. Note that the content classification method used in one aspect of the present invention focuses on a plurality of management parameters given to the patent number. However, the content is not limited to patent documents. The content may include information such as text data, image data, audio data, or video data.

[0007] Content is managed using various meta-information. Meta-information does not refer to the content itself, but rather to data describing the attributes to which the content belongs, or related information. As an example, a patent number is associated with the claims, abstract, drawings, and specification as the contents of the content. Furthermore, meta-information (evaluation information, number of days since issue, family information, etc.) is given to the patent number, and it is managed using the meta-information. The patent numbers are classified according to importance using the meta-information. The accuracy and efficiency of classification depends on the contents of the target documents, but is easily affected by the experience and skill of the user, and there are challenges in improving efficiency, as a large number of documents must be classified.

[0008] The generation of classification models using machine learning has the problem that a large amount of training data is required, which places an excessive burden on users, and the problem that the variation in the number of classified contents included in the training data affects the accuracy of the classification model.

[0009] In view of the above problems, an object of one embodiment of the present invention is to provide a method for efficiently generating a classification model and classifying information by using the same. Another object of one embodiment of the present invention is to provide a graphical user interface for interactively generating a classification model. Another object of one embodiment of the present invention is to provide a program for classifying information with a high probability.

[0010] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract problems other than these from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0011] A program is stored in a storage device of the computer device. The program can display various information on a display device of the computer device via a graphical user interface (hereinafter, GUI). Note that a user can operate the program, add information, respond to a database, and give machine learning instructions to the computer device via the GUI. The program can also display the results of computation by machine learning, the contents of learning content or unclassified content downloaded from a database, and the like on the display device via the GUI. Note that hereinafter, when the term "content" is simply used, it includes learning content, unclassified content, and classified content.

[0012] The proposed content classification system uses machine learning to generate a content classification model, and classifies unclassified content using the generated content classification model. For example, content having multiple pieces of meta information is used as learning content. A feature vector is generated from the learning content by further assigning a learning label to the learning content. When generating a feature vector, the meta information or the learning label can be treated as a feature of the learning content.

[0013] The learning content is treated as training data. The classification model can be obtained by performing machine learning based on the learning content. The classification model obtained here classifies content having multiple pieces of meta information. The types of classification may be two or three or more depending on the user's purpose. By using the classification model, the user can classify all documents in a shorter time than if they were to judge all documents manually or visually.

[0014] The learning content can be downloaded from the learning content stored in the database. Alternatively, the learning content stored in the storage device of the computer device can be used. The learning content may include learning labels and be managed. Furthermore, the classification model stored in the database may be downloaded. Alternatively, the classification model stored in the storage device of the computer device may be used.

[0015] One aspect of the present invention is a content classification method including: a learning content and a content, the learning content being assigned a first feature amount and a learning label, and the content being assigned a second feature amount, the method including: generating a plurality of first classification models by machine learning using the plurality of learning contents, generating a second classification model using the plurality of first classification models, and assigning judgment information to the plurality of contents using the second classification model and displaying the contents in a graphical user interface.

[0016] One aspect of the present invention is a content classification method including: a learning content and a content, the learning content being assigned a first feature amount and a learning label, and the content being assigned a second feature amount, the content classification method including: generating a plurality of first classification models by machine learning using the plurality of learning contents, calculating an average value from the outputs of the plurality of first classification models, generating a second classification model using the plurality of average values, and assigning judgment information to the plurality of contents using the second classification model and displaying the contents in a graphical user interface.

[0017] One aspect of the present invention is a content classification method including: a learning content and a content, the learning content being assigned a first feature amount and a learning label, and the content being assigned a second feature amount. The method includes: generating a plurality of first classification models by machine learning using a plurality of learning contents; evaluating the plurality of first classification models according to a first evaluation criterion; evaluating the plurality of first classification models according to a second evaluation criterion; generating a second classification model from the evaluation results according to the plurality of first evaluation criterion and the evaluation results according to the second evaluation criterion; and assigning judgment information to the plurality of contents using the second classification model and displaying the contents in a graphical user interface.

[0018] In the above configuration, the content classification method is preferably such that the first evaluation criterion is precision and the second evaluation criterion is sensitivity.

[0019] In each of the above configurations, the content classification method preferably includes a step of generating a first classification model using any learning content.

[0020] In each of the above configurations, it is preferable that the content classification method further includes a step of providing classification information to the learning content, selecting content having the same judgment information as the classification information from the multiple contents to which classification labels have been assigned using the output of the second classification model, and displaying the content within the graphical user interface.

[0021] In each of the above configurations, the feature amount given to the learning content or the content is preferably a content classification method that is a management parameter.

[0022] In each of the above configurations, the determination information is preferably a content classification method including a classification label or a score.

[0023] In the above configuration, the graphical user interface preferably has a content classification method including a step of specifying a specific numerical range of the scores and displaying the corresponding content as a list. Effect of the Invention

[0024] One aspect of the present invention can provide a method for classifying information with high accuracy. Alternatively, one aspect of the present invention can provide a user interface for classifying information with high accuracy. Alternatively, one aspect of the present invention can provide a program for classifying information with high accuracy.

[0025] In addition, one aspect of the present invention can provide a user with an interactive interface for generating a classification model using machine learning, thereby reducing the burden on the user, such as preparing training data and evaluating learning results.

[0026] The effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. The other effects are described below and are not mentioned in this section. The effects not mentioned in this section can be derived by a person skilled in the art from the description of the specification or drawings, etc., and can be appropriately extracted from these descriptions. One embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a flow chart illustrating the classification method. [Diagram 2] FIG. 2 is a flow chart illustrating the classification method. [Diagram 3] FIG. 3 is a diagram illustrating the connection between classification system 100 and a network. [Figure 4] FIG. 4 is a block diagram illustrating the classification system. [Diagram 5] 5A and 5B are diagrams illustrating a graphical user interface. [Figure 6] FIG. 6 is a diagram for explaining a method for generating a classification model. [Figure 7] FIG. 7 is a diagram for explaining a method for generating a classification model. [Figure 8] FIG. 8 is a diagram for explaining a method for generating a classification model. [Figure 9] FIG. 9 is a diagram illustrating a graphical user interface. [Figure 10] FIG. 10 is a diagram illustrating a graphical user interface. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] In this embodiment, a content classification method will be described with reference to FIGS.

[0029] The content classification method described in this embodiment is controlled by a program that runs on a computer device. The program is stored in a memory or storage device of the computer device. Alternatively, the program is stored in a computer connected via a network (such as a local area network (LAN), a wide area network (WAN), or the Internet) or in a server computer that has a database.

[0030] The display device of the computer device can display data provided by the user to the program and the results of calculations performed on the data by the arithmetic device of the computer device. The configuration of the device will be described in detail with reference to FIG. 4.

[0031] The data displayed on the display device can be easily recognized by the user and the operability can be improved by following a display format such as a list. Therefore, the interface that allows the user to easily communicate with the programs of the computer device via the display device will be described as a GUI.

[0032] A user can use the content classification method of the program via the GUI. A user can easily classify content using the GUI. Furthermore, a user can easily visually determine the content classification results via the GUI. Furthermore, a user can easily operate a program via the GUI. Note that the content refers to information such as text data, image data, audio data, or video data.

[0033] Next, a method for classifying contents using the GUI will be described according to the operation procedure of the GUI. First, the data processing unit will be described. The data processing unit has a data collection unit and a data generation unit. For example, the data collection unit acquires a file consisting of a plurality of contents from a database via the GUI. In addition, the data generation unit can generate learning contents by having a user assign a learning label to the content via the GUI. Alternatively, learning contents to which a learning label has been assigned may be acquired from a database. Note that the plurality of contents refers to files stored in a memory or storage device of a computer device, or data stored in a database, computer, or data server connected to a network.

[0034] Therefore, it is preferable that a plurality of learning contents or a plurality of unclassified contents are stored in a list in the database. The learning contents and the unclassified contents are assigned with a plurality of features and learning labels. The learning labels can be modified by a user via a GUI. When the learning labels are assigned to the learning contents, the learning contents to which the learning labels are assigned can be stored in the database.

[0035] The training content may include validation content that is not labeled for training purposes. The validation content may be used to validate a classification model generated using the training content.

[0036] As an example, a case where the content is a patent number will be described. A patent number is assigned with a plurality of pieces of meta information as the feature quantities of the patent number. As an example, the meta information includes evaluation information, number of days elapsed, number of families, state of the family, application type, life, number of pending applications in the family, number of abandoned applications in the family, cost, number of inventors, field, or number of claims. In other words, the meta information is a management parameter of the content. Note that family includes patent family, patent family, etc.

[0037] Next, the learning processing unit will be described. The learning processing unit has a step of generating a classification model using the learning content. The learning processing unit has a classification model generating unit or a classification model evaluating unit.

[0038] The classification model generation unit can generate a classification model. The classification model generation unit includes a step of generating a plurality of first classification models by machine learning using a plurality of learning contents, and a step of generating a second classification model using the plurality of first classification models. An output value of the first classification model or the second classification model can be displayed on a GUI. A user can assign (including correction) a learning label of the first classification model to the output value. Alternatively, a user can add new learning content to the output value.

[0039] The classification model evaluation unit evaluates the classification model generated by the classification model generation unit using the verification content. When the verification content is inferred using the classification model, the classification model outputs the inference result as judgment information. The GUI can display each evaluation content by adding the judgment information.

[0040] The user can evaluate the output result of the classification model evaluation unit, modify the learning labels as necessary, and update the classification model in the classification model generation unit. Alternatively, the user can update the classification model in the classification model generation unit by adding learning contents.

[0041] Next, the judgment processing unit will be described. The judgment processing unit has a classification inference unit and a list generation unit. For example, the classification inference unit infers and classifies a plurality of unclassified contents using a first learning model and a second learning model generated by the classification model generation unit. The classification model gives the inference result to each content as judgment information.

[0042] The list generation unit can generate a list in a format desired by the user from the content given the determination information and display it on the GUI. For example, if each content is managed by application country, the application country can be used as the classification information. When the application country is used as the classification information, it is preferable that a classification model that is generated is different for each application country. Note that the classification information is not limited to the application country. For example, one of the meta information possessed by the content can be used as the classification information.

[0043] A case where meta-information is used as classification information will be described. For example, the state of a patent family may be used as classification information. In some cases, the patent number of a parent application, the patent number of a divisional application, and the like may be given as meta-information for a patent number. Different classification models can be generated depending on the state in which a divisional application is possible from the patent number of a parent application, the state in which a divisional application is not possible from the patent number of a parent application, the state in which the patent number of a parent application is in a pending state, the state in which the patent number of a parent application is in a lost state, the state in which a further divisional application is possible from the patent number of a divisional application, the state in which a further divisional application is not possible from the patent number of a divisional application, the state in which the patent number of a divisional application is in a pending state, and the state in which the patent number of a divisional application is in a lost state, and each of these classification models can be used to make inferences.

[0044] That is, the determination processing unit can infer a plurality of unclassified contents using the classification model. The inference results include a step of assigning the results to each piece of content as determination information and displaying the results on the GUI. The determination information includes at least a classification label and a score (probability). The GUI also includes a step of specifying a specific numerical range of the scores and displaying the corresponding contents as a list.

[0045] An example different from the above-mentioned classification model generation unit is shown. The classification model generation unit includes a step of generating a plurality of first classification models by machine learning using a plurality of learning contents, a step of calculating an average value from the outputs of the plurality of first classification models, and a step of generating a second classification model using the plurality of average values. The output value of the first classification model or the second classification model can be displayed on a GUI. The user can correct the learning label of the first classification model for the output value. Alternatively, the user can add learning content to the output value. The average value means that it is calculated using any one of arithmetic mean calculation, geometric mean calculation, and harmonic mean calculation.

[0046] The second classification model is generated using the average values. In the second classification model, the outputs of the first classification model are averaged, so that the influence of noise components such as outliers in the training content can be reduced.

[0047] Next, an example different from the above-mentioned classification model generating unit will be shown. The classification model generating unit includes a step of generating a plurality of first classification models by machine learning using a plurality of learning contents, a step of evaluating the plurality of first classification models by a first evaluation criterion, a step of evaluating the plurality of first classification models by a second evaluation criterion, and a step of generating a second classification model from the evaluation results by the plurality of first evaluation criterion and the evaluation results by the second evaluation criterion. The output value of the first classification model or the second classification model can be displayed on a GUI. The user can correct the learning label of the first classification model for the output value. Alternatively, the user can add learning content to the output value. The first evaluation criterion is the accuracy of the confusion matrix, and the second evaluation criterion is the sensitivity of the confusion matrix.

[0048] A second classification model is generated using the results of evaluating the outputs of the multiple first classification models using a first evaluation criterion and a second evaluation criterion. In the first evaluation criterion, the accuracy of the confusion matrix can be rephrased as a precision rate for the learning labels. In the second evaluation criterion, the sensitivity of the confusion matrix can be rephrased as a recall rate for the learning labels. The classification model thus generated can include the precision rate and the recall rate of the multiple first classification models. The second classification model generated using the multiple first classification models has improved classification accuracy.

[0049] In the above-mentioned classification model generation unit, for example, a second classification model may be generated using m (m is a natural number) of generated first classification models.

[0050] In addition, when the above-mentioned classification model generation unit has k learning contents (k represents a natural number), the first classification model can generate any learning contents up to k. In addition, when k learning contents are selected, any learning contents can include two different learning models. In addition, the first classification model can be generated using q learning contents (q represents a natural number) each in the order sorted by the k different numbers.

[0051] The program can display the content read from the database on the GUI. The content preferably has listed meta information. The GUI displays the content according to the display format of the GUI. The listed meta information assigned to the content is preferably managed in units called records. For example, each record is composed of an ID (Identification) linked to a number, content (image data, audio data, or video data), meta information, etc.

[0052] In this specification, machine learning is performed with a focus on meta information, and a classification model is generated by the machine learning. The classification model analyzes the meta information and classifies the content that has been converted into a feature vector.

[0053] In addition, in the above-mentioned content classification method, classification can be performed by machine learning without using learning labels as training data. For example, an algorithm such as K-means or DBSCAN (density-based spatial clustering of applications with noise) can be used as the classification model.

[0054] The program can also generate a classification model using machine learning with the learning content to which the multiple meta information and learning labels have been added. The classification model can use algorithms such as decision trees, naive Bayes, KNN (k Nearest Neighbor), SVM (Support Vector Machines), perceptron, logistic regression, or neural networks.

[0055] Furthermore, the program can switch the classification model according to the number of learning contents. For example, when the number of learning contents is small, a decision tree, naive Bayes, or logistic regression may be used, and when the number of learning contents is a certain amount or more, SVM, random forest, or neural network may be used. The classification model used in this embodiment uses random forest, which is one of the decision tree algorithms. Furthermore, random sampling or cross variation can be used as a method for selecting meta information, a method for selecting learning contents, or a method for selecting the first classification model. Alternatively, q items can be selected in the order in which the assigned numbers are sorted.

[0056] Next, a content classification method will be described with reference to the drawings. FIG. 1 is a flow chart for explaining a content classification method according to one embodiment of the present invention. The content classification method is controlled by a program running on a computer device. Therefore, the program can classify the content by having a data processing unit, a learning processing unit, or a judgment processing unit. The program can classify the content desired by the user via a GUI. In other words, the contents processed by each of the above-mentioned processing units correspond to the steps of the program.

[0057] In step S11, the user can instruct the loading of a file containing content via the GUI. The file is stored in a database held by the data processing unit. The file includes learning content, unclassified content, and the like.

[0058] Therefore, it is preferable that the database stores a list of multiple learning contents or multiple unclassified contents. The user can assign or modify learning labels to the learning contents displayed in the GUI. The file can include verification content that has not been assigned a learning label.

[0059] Step S12 is a learning processing unit that generates a classification model using the loaded file. The generated classification model can evaluate the verification content and display the evaluation result in the GUI. The user can instruct the user to modify the learning label, add learning content, etc. based on the evaluation result.

[0060] In addition, the user can predict the change over time of the meta information of the learning content and update the meta information. When the user updates the meta information, the classification model can include the change over time of the classification model. Therefore, the user can obtain the change over time of the classification of the content. The classification model can classify a group of content whose value is expected to increase or a group of content whose value is expected to decrease.

[0061] Step S13 is a judgment processing unit. Unclassified content is inferred using the classification model generated in step S12. The classification model can assign judgment information to the unclassified content based on the inference result. The judgment processing unit can display the content given the judgment information in a GUI in a format desired by the user. The judgment information includes at least a classification label and a score. In addition, the GUI can specify a specific numerical range of the score and display the corresponding content.

[0062] 1 will be described in more detail with reference to Fig. 2. First, step S11 will be described in detail. The data processing section in step S11 has a data collection section in step S21 and a data generation section in step S22.

[0063] The data collection unit in step S21 will be described. The data collection unit in step S21 can load a file from a database. Note that meta information or content can be managed by different databases. Meta information may differ for a company, organization, or user that handles content. Therefore, the data collection unit has a function of collecting meta information related to content from different databases. Note that each database can be installed in a different building, a different region, or a different country.

[0064] Next, the data generating unit in step S22 will be described. The data generating unit can manage content and meta information in units called records. For example, each record is composed of an ID linked to a number, content (image data, audio data, or video data), meta information, etc. Also, the user can generate learning content by adding learning labels to the content displayed in the GUI.

[0065] Next, details of step S12 will be described. The learning processing unit in step S12 has a classification model generating unit in step S23, a classification model evaluating unit in step S24, and an output result determining process in step S25.

[0066] The classification model generation unit in step S23 will be described. The classification model generation unit can generate a classification model of the content. The classification model generation unit can generate a plurality of first classification models by machine learning using a plurality of learning contents. The classification model generation unit can generate a second classification model using the plurality of first classification models. The GUI can display an output value of the first classification model or the second classification model.

[0067] After the output result determination process in step S25 described later, the user can assign (including correct) a learning label of the first classification model to the output value. Alternatively, the user can add new learning content to the output value. It is possible to predict changes over time in meta information held by the learning content and update the meta information. The effect of updating the meta information by the user can be seen in the explanation of step S12.

[0068] Next, the classification model evaluation unit in step S24 will be described. The classification model evaluation unit can evaluate the classification model generated by the classification model generation unit by using the verification content. The classification model outputs the result of inferring the verification content as judgment information. The GUI can display each evaluation content by adding the judgment information.

[0069] Next, the output result determination process of step S25 will be described. For example, the user can determine that the classification model of the content has been sufficiently trained by judging the output result of the classification model evaluation unit in step S24. The user gives an instruction to the GUI that the generation of the classification model is complete (OK). For example, the user can determine that the training of the classification model of the content is insufficient (NG). The user returns to step S23 and updates the classification model by changing the learning labels, adding learning contents, or updating meta information.

[0070] Next, step S13 will be described in detail. The determination processing section in step S13 has a classification inference section in step S26 and a list creation section in step S27.

[0071] The classification inference unit in step S26 will be described. The classification inference unit infers and classifies a plurality of unclassified contents using the first learning model and the second learning model generated by the classification model generation unit. The unclassified contents generated by the data generation unit in step S22 are provided to the classification inference unit. The classification model provides each content with an inference result as judgment information.

[0072] The list creation unit of step S27 will be described. The list creation unit can list the contents to which the judgment information is given in a format required by the user and display it on a GUI. Each content may be given classification information different from the meta information. For example, when classification information is given to the learning content, a classification model that is generated can generate a different classification model for each classification information. Alternatively, one of the meta information possessed by the content can be used as the classification information.

[0073] The judgment information includes at least a classification label and a score. The GUI can specify a specific numerical range of the score and display the corresponding content in a list on the GUI.

[0074] FIG. 3 is a diagram illustrating the connection between a classification system 100 having the above-mentioned content classification method and a network.

[0075] The classification system 100 is connected to a communication network LAN1. A database DB1 or client computers CL1 to CLn (n is a natural number) are connected to the communication network LAN1. The communication network LAN1 can be connected to a communication network LAN2 via a network. The network can be the Internet, a communication network WAN, or satellite communication. The communication network LAN2 is connected to a database DB2 or client computers CL11 to CL1n.

[0076] The classification system 100 can generate content, classify content, generate models, and classify unclassified content using files containing content stored in database DB1, database DB2, client computers CL1 to CLn, or client computers CL11 to CL1n.

[0077] Also, a user can issue instructions to the GUI from a program running on the classification system 100. For example, a user can generate the above-mentioned classification model using information from databases installed in different countries via the Internet to classify unclassified content. That is, content or meta information may be stored in different databases or client computers.

[0078] The GUI can display the classification results classified by the classification system 100 stored in the storage device of the database DB1, the database DB2, the client computers CL1 to CLn, or the computer device of the client computers CL11 to CL1n.

[0079] FIG. 4 is a block diagram for explaining the classification system 100 described in FIG. 3. The classification system 100 has a GUI (Graphical User Interface) 110, a calculation unit 120, and a storage unit 130. The GUI 110 has an input unit 111 and an output unit 112. The input unit 111 has a function of selecting a content load source and a function of inputting a learning label. The output unit 112 has a function of displaying a content list loaded from a database or the like, and a function of displaying judgment information output by the classification model. Note that the user can modify meta information included in the displayed content via the GUI.

[0080] The calculation unit 120 has a data processing unit 121, a learning processing unit 122, and a judgment processing unit 123. The data processing unit 121 has a data collection unit and a data generation unit. The learning processing unit 122 has a classification model generation unit that creates a classification model, and a classification model evaluation unit that classifies the classification model. The output result of the classification model evaluation unit has a function of evaluation result judgment processing in which a user makes a judgment. The judgment processing unit 123 has a classification inference unit and an output list creation unit that lists the results classified by the classification inference unit. The calculation unit 120 performs calculation processing using a microprocessor with a program stored in a storage unit of the computer device. However, the program can be processed using a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit).

[0081] The storage unit 130 temporarily stores a list of content and meta information generated by loading from a database or the like.

[0082] The storage unit 130 may be, for example, a DRAM (dynamic random access memory) having 1T (transistor) 1C (capacitor) type memory cells. Note that the transistors used in the DRAM memory cells may be OS transistors. An OS transistor is a transistor having a metal oxide in its semiconductor layer. A memory device in which an OS transistor is used in a memory cell is called an "OS memory." Here, as an example of an OS memory, a RAM having 1T1C type memory cells is called a "DOSRAM (Dynamic Oxide Semiconductor RAM)."

[0083] The off-current of an OS transistor is very small. Therefore, DOSRAM can reduce the frequency of refresh operations, thereby reducing the power required for refresh operations. The off-current here refers to the current that flows between the source and drain when the transistor is in the off state. When the transistor is an n-channel type, for example, if the threshold voltage is about 0 V to 2 V, the current that flows between the source and drain when the voltage between the gate and source is a negative voltage can be called the off-current.

[0084] FIG. 5A is a diagram for explaining the configuration of GUI 30. GUI 30 shows, as an example, a management screen that displays a list of p learning contents. Learning contents are managed on a record-by-record basis. The record includes a number (No) 31, a content (ID) 32, meta information (Feature) 33 (meta information (F1) 33a to meta information (Fm) 33m) indicating features, classification information (Case) 34 (classification information (C1) 34a to classification information (Cq) 34q), learning label (J-Label) 35, and the like. As an example, in FIG. 5A, learning label 35 is given one of two values, "Yes" and "No", but learning label 35 is not limited to two values ​​and may be three or more values.

[0085] 5B is a diagram for explaining the configuration of GUI 30A. GUI 30A shows a management screen in which n pieces of unclassified content are inferred by the evaluation inference unit, and the inferred judgment information is displayed in a list. Unclassified content, like learning content, has a number 31, content 32, meta information 33, and classification information 34. Furthermore, a classification label (A-Label) 36 and a score (Score) 37 are assigned to each record as judgment information.

[0086] Note that GUI 30 and GUI 30A can be managed on the same display screen. Fig. 9 or 10, which will be described later, shows an example of a GUI display that can display study content and assessment information on the same management screen.

[0087] FIG. 6 is a diagram for explaining a method for generating a classification model using a plurality of features Feature associated with the above-mentioned learning content Sample by machine learning. Each feature Feature indicates one of the meta information and corresponds to a management parameter for managing the content. In one embodiment of the present invention, a method for generating a classification model is explained using a calculation unit F, a calculation unit S, a calculation unit V, a first classification model, and a second classification model.

[0088] Each of the learning contents Sample(1) to Sample(k) is assigned j features Feature and a learning label Label. As an example, the calculation unit F1 can generate a feature vector Vlabel1(1) in a format that can be processed by a computer from the learning content Sample(1). Furthermore, the calculation unit Fk can generate a feature vector Vlabel1(k) in a format that can be processed by a computer from the learning content Sample(k). The feature vector Vlabel1(1) can be generated by the calculation unit F1 assigning different weighting coefficients to each feature. Furthermore, the feature vector Vlabel1(1) can be generated using j or less features Feature that are randomly selected.

[0089] Next, a plurality of first classification models are generated. The calculation units S1 to Sm correspond to different first classification models. As an example, the calculation unit S1 can generate the first classification model using the feature vectors Vlabel1(1) to Vlabel1(k). Note that the number of feature vectors Vlabel1 provided to the calculation unit S1 may be k or less. As a different example, the calculation unit Sm can generate the first classification model using different feature vectors Vlabel1(1) to Vlabel1(k). Thus, two different first classification models each include k or less feature vectors Vlabel1, and any one of the feature vectors Vlabel1 may include the same feature vector.

[0090] The k study content samples selected to generate the first classification model may be selected randomly or in a sorted order based on the numbers assigned to the study content. When the study content samples are selected randomly, the first classification model may include the variability of the study content. When the study content samples are selected in a sorted order based on the numbers assigned to the study content, the first classification model may include a trend along the assigned numbers based on one of the characteristics of the time series or meta information.

[0091] Therefore, the first classification model can generate the feature vector Vlabel2 using the feature vector Vlabel1 generated from the learning content Sample(1) through Sample(k).

[0092] The second classification model is generated by the calculation unit V1. As an example, the calculation unit V1 has a step of generating the second classification model using m feature vectors Vlabel2. Note that the second classification model can generate classification models having different features by using the feature vectors Vlabel2(1) to Vlabel2(m).

[0093] Therefore, the second classification model can output an output value POUT using a feature vector Vlabel1 generated from the learning content Sample(1) to Sample(k). The GUI can display the output value POUT. The output value POUT includes a classification label, which is judgment information, and a score. Therefore, the second classification model can classify contents. Furthermore, the second classification model can assign judgment information to each content.

[0094] To make an inference using the classification model described in Fig. 6, a judgment result is obtained by providing unclassified content to the training content Sample of the classification model. However, unlike the training content, the unclassified content does not have a training label attached to it.

[0095] Fig. 7 is a diagram for explaining a method for generating a classification model different from that in Fig. 6. In Fig. 7, differences from Fig. 6 are explained, and the same reference numerals are used in common between different drawings for the same parts or parts having similar functions in the configuration of the invention (or the configuration of the embodiment), and repeated explanations are omitted.

[0096] In FIG. 7, the average value Av of m feature vectors Vlabel2 is calculated to generate a feature vector Vlabel_a. The second classification model can be generated using p feature vectors Vlabel_a. The second classification model can generate a classification model having different features by calculating the average value Av of m feature vectors Vlabel2. The generated classification model can accurately classify the content.

[0097] Fig. 8 is a diagram for explaining a method for generating a classification model different from that in Fig. 7. In Fig. 8, differences from Fig. 7 are explained, and the same reference numerals are used in common between different drawings for the same parts or parts having similar functions in the configuration of the invention (or the configuration of the embodiment), and repeated explanations thereof will be omitted.

[0098] In FIG. 8, evaluation criteria for evaluating m feature vectors Vlabel2 are given to the evaluation judgment unit JG. For example, the evaluation judgment unit JG1 is given precision as a first evaluation criterion and can evaluate each feature vector Vlabel2(1). Next, the evaluation judgment unit JG1 is given sensitivity as a second evaluation criterion and can evaluate each feature vector Vlabel2(1). The evaluation judgment unit JG1 outputs the evaluation result Vlabel_b(1).

[0099] The second classification model is generated using the evaluation results Vlabel_b(1) to Vlabel_b(p). For example, the multiple feature vectors Vlabel2 may be evaluated by different first and second evaluation criteria, or may be evaluated by the same evaluation criteria. Although not shown in FIG. 8, the average value of the evaluation results Vlabel_b according to the first and second evaluation criteria can be calculated in the same way as in FIG. 7.

[0100] The second classification model can generate a classification model having different features by using the evaluation results of the m feature vectors Vlabel2. The generated classification model can accurately classify the content.

[0101] 9 is a diagram illustrating the GUI 50. The GUI 50 has a display area for contents (learning contents, uncategorized contents, and categorized contents), an icon 58a for selecting a download source of a file including the contents, a text box 58b for displaying address information where the selected file is saved, and an icon (Learning Start) 59 for executing machine learning.

[0102] In the display area, eight records are loaded as an example. Each record has components of a number (No) 51, an ID (Index) 52, a feature (Feature) 53, classification information (Case) 54, a learning label (JL) 55, a classification label (AL) 56, and a score (Prob) 57. The feature 53 can display feature F(1) 53a to feature F(j) 53j as detailed information. j is a natural number. The classification information 54 can display classification information C(1) 54a to classification information C(4) 54d as detailed information. The classification information can have a type that can be expressed by a natural number.

[0103] FIG. 9 shows an example in which the results of classification of learning content and unclassified content by the classification model are displayed on the GUI 50.

[0104] As an example, record numbers No1 to No3 correspond to study contents. Study contents are given study labels, and record numbers No1 to No3 are given classification information.

[0105] Record numbers No. 4 to No. 8 correspond to classified content. Classification labels 56 and scores 57 are given to classified content. Note that FIG. 9 shows the results of classifying record numbers No. 4 to No. 7 using a classification model obtained by learning record numbers No. 1 and No. 3. As an example, the results of classifying record number No. 8 using a classification model obtained by learning record number No. 2 are shown. Due to space limitations, FIG. 9 shows only up to 8 records, but multiple types of records can be handled.

[0106] However, when dealing with a large number of records, display issues arise. Therefore, it is preferable that classification label 56 or score 57 has a sorting function. As an example of a sorting condition, the GUI can select and display a judgment result in which classification label 56 is "Yes". In addition, the GUI can specify a numerical range for score 57 and display it. When the above-mentioned sorting conditions are given to the GUI, the GUI can classify and display content having similar characteristics to the learning content given as the teacher data.

[0107] For example, let us consider the case where the content is a patent number. A patent number is assigned multiple pieces of meta information. For patent numbers for which the patent rights are maintained, the learning label is given "Yes." For patent numbers for which the patent rights have been abandoned, the learning label is given "No." Next, machine learning is performed to generate a classification model.

[0108] The above-mentioned classification model can assign judgment information to unclassified content. The judgment information displays a classification label 56 and a score 57. As an example, a user assigns "No" to the classification label 56 using the sort function. Furthermore, the score 57 is set to "0.8" to "1.0". By providing the above-mentioned sorting conditions to the GUI, the GUI can select and display records that have similar characteristics to the study content whose patent number has been abandoned.

[0109] Fig. 10 is a diagram for explaining a GUI 50A different from Fig. 9. Fig. 10 is an example of an efficient GUI display when handling a large number of records. Note that Fig. 10 explains the differences from Fig. 9, and in the configuration of the invention (or the configuration of the embodiment), the same reference numerals are used in common between different drawings for the same parts or parts having similar functions, and repeated explanations are omitted.

[0110] Fig. 10 differs from Fig. 9 in that records relating to any selected classification information can be categorized and displayed. In Fig. 10, the display can be switched according to the type of classification information C(1) to C(4).

[0111] The user looks at the multiple feature quantities 53 assigned to the records and the judgment information of the classification model, and if the classification accuracy is sufficient, ends the update of the classification model. If the user looks at the judgment information assigned to the records and judges that the classification accuracy is not sufficient, the user can update the classification model by assigning a learning label to a record to which a user-specified label has not been assigned and pressing the icon 59. Note that the feature quantities 53 may be updated by predicting a change over time in the meta information of the learning content. When the user updates the feature quantities 53, the classification model can include a change over time in the classification model. Thus, the user can obtain a change in the classification of the content over time. The classification model becomes capable of classifying a content group whose value is expected to increase or a content group whose value is expected to decrease.

[0112] Although not shown, the order in which the numerical values ​​and label information included in the feature amount 53, the classification information 54a to 54d, the learning label 55, the classification label 56, or the score 57 are displayed can be changed, or the selected numerical values ​​and label information can be sorted and displayed in a required order using a filter function. This allows the user to efficiently evaluate the judgment results of the classification model.

[0113] The content classification method described with reference to Figs. 1 to 10 can provide a method for classifying information with a high probability. For example, a GUI is suitable for classifying information with a high probability. A program can update a classification model by providing new training data (learning labels) to the classification model. A program can classify information with a high probability by updating the classification model.

[0114] Furthermore, the generated classification model can be stored in the electronic device itself or in an external memory, and can be recalled and used when classifying a new file. Furthermore, the classification model can be updated according to the method described above while adding new training data.

[0115] The structures and methods described in this embodiment mode can be used in appropriate combination with structures and methods described in other embodiment modes. [Explanation of symbols]

[0116] CL1: client computer, CL1n: client computer, CL11: client computer, CLn: client computer, DB1: database, DB2: database, LAN1: communication network, LAN2: communication network, Vlabel1: feature vector, Vlabel2: feature vector, 31: number, 32: content, 33: meta information, 34: classification information, 35: learning label, 50: GUI, 50A: GUI, 51: number, 53: feature amount, 54: classification information, 56: classification label, 57: score, 58a: icon, 58b: text box, 59: icon, 100: classification system, 110: GUI, 111: input unit, 112: output unit, 120: calculation unit, 121: data processing unit, 122: learning processing unit, 123: judgment processing unit, 130: memory unit

Claims

[Claim 1] A learning content and a content, a first feature amount and a learning label are assigned to the learning content; A second feature is assigned to the content, generating a plurality of first classification models by machine learning using the plurality of training contents; generating a second classification model using the plurality of first classification models; assigning judgment information to the plurality of pieces of content using the second classification model and displaying the judgment information in a graphical user interface; How content is classified, including:

Images