Algorithm determining device, learning equipment production device, patent classifier, classification information determining device, algorithm determination method, learning equipment production method, classification information determination method, and program

Abstract

To solve the problem of absence of means for classifying patents with high accuracy.SOLUTION: An algorithm determining device comprises: a program storage unit in which are stored one or more learning equipment generation programs for generating a learning equipment using learning source information including two or more patent classification information including patent information and classification information, and a prediction program for acquiring predictive classification information using the learning equipment; a learning source information acceptance unit; a learning unit for executing each of the one or more learning equipment generation programs using patent classification information in at least some of the learning source information and acquiring one or more learning equipment; a prediction unit for applying a prediction program using each of the one or more learning equipment to each patent classification information in at least some of the learning source information and acquiring two or more predictive classification information for each patent classification information; an evaluation unit for acquiring the accuracy of each predictive classification information using two or more of each predictive classification information and the classification information of the learning source information and determining predictive classification information of the highest accuracy; and an algorithm information acquisition unit for acquiring algorithm information that corresponds to the determination.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to a patent classification device for classifying patent information. [Background technology]

[0002] Conventionally, there has been a patent classification device that reduces the burden on users in classifying documents such as patents (see Patent Document 1). This patent classification device is characterized by including a classification registration unit that stores in a storage unit training data that associates a document classification with the document, and a classification assignment unit that determines a classification to be assigned to a target document based on a classification of similar documents obtained based on the similarity between the training data stored in the storage unit and the target document to be assigned the classification. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-87258 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, the same algorithm was used to classify patents, and therefore the patents could not be classified with high accuracy.

[0005] In addition, in the conventional technology, the classification of each document to be annotated was determined based on the classification of similar documents obtained based on the similarity between the training data and the document to be annotated, so patents could not be classified with high accuracy.

[0006] In addition, in the prior art, patents were classified using the entire document of the patent specification, which contains a lot of information other than the key points of the invention, such as well-known technical configurations and prior art, making it impossible to classify patents with high accuracy. [Means for solving the problem]

[0007] The algorithm determination device of the first invention includes a program storage unit storing one or more learning device generation programs for generating a learning device for classifying patent information relating to a patent using learning source information having two or more patent classification information having patent information relating to a patent and classification information indicating a classification of the patent information, and a prediction program for predicting a classification of the patent classification information using the learning device and obtaining predicted classification information, a learning source information receiving unit receiving the learning source information, a learning unit executing each of the one or more learning device generation programs using at least a part of the patent classification information of the learning source information received by the learning source information receiving unit and obtaining one or more learning devices, and the algorithm determination device includes: a prediction unit that applies a prediction program to each of the one or more patent classification information using one or more learning devices generated by the learning unit for at least a portion of the patent classification information of the learning source information obtained, and obtains two or more predicted classification information for each of the one or more patent classification information; an evaluation unit that uses each of the two or more predicted classification information obtained by the prediction unit and classification information contained in the learning source information to obtain accuracy information regarding the accuracy of each of the two or more predicted classification information, and determines the predicted classification information with the highest accuracy specified by the accuracy information; an algorithm information acquisition unit that acquires algorithm information regarding an algorithm corresponding to the predicted classification information determined by the evaluation unit; and an output unit that outputs the algorithm information.

[0008] With this configuration, it is possible to determine an appropriate process for performing the prediction process.

[0009] In addition, the algorithm determination device of the second invention, compared to the first invention, further includes a learning source information acquisition unit that acquires two or more learning source information each having two or more patent classification information having different information from the patent information contained in each of the two or more patent classification information contained in the learning source information accepted by the learning source information receiving unit, the learning unit executes one or more learning device generation programs for each of the two or more learning source information acquired by the learning source information acquisition unit to acquire two or more learning devices, the prediction unit applies a prediction program using each of the two or more learning devices generated by the learning unit to at least some of the patent classification information of the learning source information accepted by the learning source information receiving unit to acquire two or more predicted classification information for each of the one or more patent classification information, and the algorithm information acquisition unit acquires algorithm information including information identifying information identifying information contained in the learning source information corresponding to the predicted classification information determined by the evaluation unit.

[0010] With this configuration, by changing the data used when performing the prediction process, it is possible to determine an appropriate process for performing the prediction process.

[0011] In addition, the algorithm determination device of the third invention is an algorithm determination device according to the first or second invention, in which the program storage unit stores two or more learning device generation programs and two or more prediction programs, the learning unit executes each of the two or more learning device generation programs using at least a portion of the patent classification information of the learning source information accepted by the learning source information accepting unit to obtain two or more learning devices, the prediction unit applies a prediction program to each of at least a portion of the patent classification information of the learning source information accepted by the learning source information accepting unit using each of the two or more learning devices generated by the learning unit to obtain two or more predicted classification information for each of the one or more patent classification information, and the algorithm information acquisition unit acquires algorithm information including program identification information that identifies the learning device generation program corresponding to the predicted classification information determined by the evaluation unit.

[0012] With this configuration, by using multiple learning device generation programs, it is possible to determine an appropriate process for performing prediction processing.

[0013] In addition, the algorithm determination device of the fourth invention is an algorithm determination device that, with respect to any one of the first to third inventions, further includes a statistical processing unit that performs one or more types of statistical processing using two or more prediction classification information acquired by the prediction unit and acquires one or more prediction classification information that is the result of the statistical processing, and the evaluation unit acquires accuracy information regarding the accuracy of each of the two or more prediction classification information using two or more prediction classification information including the one or more prediction classification information acquired by the statistical processing unit and classification information contained in the learning source information, instead of or in addition to the two or more prediction classification information acquired by the prediction unit, and determines the prediction classification information with the highest accuracy specified by the accuracy information, and the algorithm information acquisition unit acquires algorithm information including statistical processing identification information that identifies a method of statistical processing corresponding to the prediction classification information determined by the evaluation unit.

[0014] With this configuration, it is possible to determine an appropriate process for carrying out the prediction process by using a prediction result obtained by performing statistical processing on prediction results obtained using two or more learning devices.

[0015] Furthermore, the algorithm determination device of the fifth invention is an algorithm determination device according to the fourth invention, in which the statistical processing unit performs two or more types of statistical processing to obtain two or more pieces of predictive classification information as the results of the statistical processing, and the two or more types of statistical processing are two or more types of statistical processing selected from the logical sum, logical product, or majority vote of the predictive classification information.

[0016] With this configuration, by using two or more prediction results obtained by performing two or more statistical processes on prediction results obtained using two or more learning devices, it is possible to determine processing including appropriate statistical processing for performing prediction processing.

[0017] In addition, the algorithm determination device of the sixth invention, with respect to any one of the first to fifth inventions, further includes an accuracy specification information storage unit in which accuracy specification information is stored specifying which of two or more of the four accuracies, i.e., recall rate, precision rate, F-value, or accuracy rate, accuracy information for which is to be obtained, and the evaluation unit uses each of the two or more prediction classification information acquired by the prediction unit and the classification information contained in the learning source information to obtain accuracy information specified by the accuracy specification information for each of the two or more prediction classification information, and determines the prediction classification information with the highest accuracy specified by the accuracy information.

[0018] With this configuration, it is possible to determine a process that has a high score, as desired by the user, from among the recall rate, precision rate, F-measure, or accuracy rate.

[0019] In addition, the learning device production device of the seventh invention is a learning device production device that includes a learning source information receiving unit that receives learning source information, a learning unit that performs a learning process on the learning source information received by the learning source information receiving unit using an algorithm according to the algorithm information output by the algorithm determination device of any of the first to sixth inventions, to obtain a learning device, and a storage unit that stores the learning device in association with the algorithm information.

[0020] Such a configuration allows the production of suitable learning machines.

[0021] In addition, the patent classification device of the eighth invention is a patent classification device comprising: a learning device storage unit in which the learning device accumulated by the production device of the seventh invention is stored; a reception unit for receiving classification target information including a patent identifier that identifies the patent to be classified; a prediction unit for performing a prediction process on the classification target information using the learning device by an algorithm in accordance with the algorithm information corresponding to the learning device, and acquiring classification result information; and an output unit for outputting the classification result information acquired by the prediction unit in association with the patent identifier included in the classification target information received by the reception unit.

[0022] With this configuration, highly accurate prediction processing can be performed using an appropriate learning device.

[0023] In addition, the patent classification device of the ninth invention, compared to the eighth invention, further includes an important information acquisition unit that acquires important information that satisfies predetermined conditions using a patent specification corresponding to the patent identifier included in the classification target information received by the receiving unit, and a classification target processed information acquisition unit that acquires classification target processed information, which is information corresponding to the patent identifier included in the classification target information received by the receiving unit and includes the important information acquired by the important information acquisition unit, and the prediction unit applies a learning device to the classification target processed information acquired by the classification target processed information acquisition unit, predicts the classification of the classification target processed information using a machine learning algorithm, and acquires classification result information.

[0024] This arrangement allows for accurate classification of patents by using key information about patent specifications.

[0025] In addition, the patent classification device of the tenth invention is a patent classification device in which, compared to the ninth invention, the patent information also includes one or more types of information of the claims or abstract corresponding to the patent identifier, and the processed information to be classified includes the important information acquired by the important information acquisition unit and one or more types of information of the claims or abstract corresponding to the patent identifier.

[0026] This configuration allows for accurate classification of patents by using the claims and / or abstracts.

[0027] In addition, the patent classification device of the eleventh invention is a patent classification device according to the ninth or tenth invention, in which the important information acquisition unit determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the receiving unit is an effect-solution-means sentence including both an effect expression and a solution expression, and acquires one or more effect-solution-means sentences determined to be effect-solution-means sentences, and the important information includes one or more effect-solution-means sentences acquired by the effect-solution-means sentence acquisition means, or effect terms and solution terms which are terms acquired from the one or more effect-solution-means sentences.

[0028] With this configuration, patents can be classified with high accuracy by using effect / solution sentences or effect terms or solution terms.

[0029] In addition, the patent classification device of the twelfth invention is a patent classification device in which, for any one of the ninth to eleventh inventions, the important information acquisition unit is equipped with an effect sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is an effect sentence including an expression of an effect, and acquiring one or more effect sentences determined to be effect sentences, and the important information includes one or more effect sentences acquired by the effect sentence acquisition means, or effect terms which are terms acquired from the one or more effect sentences.

[0030] With this configuration, patents can be classified with high accuracy by using effect sentences or effect terms.

[0031] In addition, the patent classification device of the thirteenth invention is a patent classification device for any one of the ninth to twelfth inventions, wherein the important information acquisition unit determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the receiving unit is a solution means sentence including both a solution means expression and a solution means expression, and acquires one or more solution means sentences determined to be solution means sentences, and the important information includes one or more solution means sentences acquired by the solution means sentence acquisition means, or solution means terms which are terms acquired from the one or more solution means sentences.

[0032] With this configuration, patents can be classified with high accuracy by using solution sentences or solution terms.

[0033] In addition, the patent classification device of the fourteenth invention is a patent classification device in which, relative to any one of the ninth to thirteenth inventions, the important information acquisition unit is equipped with an object sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the receiving unit is an object sentence including a target expression, and acquiring the one or more object sentences determined to be object sentences, and the important information includes the one or more object sentences acquired by the object sentence acquisition means, or object terms which are terms acquired from the one or more object sentences.

[0034] With this configuration, patents can be classified with high accuracy by using object sentences or object terms.

[0035] In addition, the patent classification device of the fifteenth invention is a patent classification device in which, with respect to any one of the ninth to fourteenth inventions, the important information acquisition unit acquires the frequency of occurrence of one or more terms included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit, calculates a score for each of the one or more terms using the frequency of occurrence, and acquires one or more characteristic words whose score is large enough to satisfy a predetermined condition, and the important information includes the one or more characteristic words acquired by the characteristic word acquisition means.

[0036] With this configuration, patents can be classified with high accuracy by using characteristic words.

[0037] In addition, the patent classification device of the sixteenth invention is a patent classification device according to any one of the ninth to fourteenth inventions, wherein the important information acquisition unit acquires the frequency of occurrence of one or more terms included in a patent specification corresponding to the patent identifier included in the classification target information received by the receiving unit, calculates a score for each of the one or more terms using the frequency of occurrence, and acquires one or more characteristic words whose score is large enough to satisfy a predetermined condition, and related word acquisition means acquires one or more related words which are hypernyms, hyponyms, or synonyms corresponding to each of the one or more characteristic words acquired by the characteristic word acquisition means from a related word dictionary in which one or more related words among hypernyms, hyponyms, or synonyms for each of the one or more terms are stored, and the important information includes the one or more related words acquired by the related word acquisition means.

[0038] With this configuration, patents can be classified with high accuracy by using words related to characteristic words.

[0039] Furthermore, the patent classification device of the seventeenth invention is a patent classification device in which, relative to any one of the ninth to sixteenth inventions, the important information acquisition unit is equipped with an important tag corresponding sentence acquisition means for acquiring one or more important tag corresponding sentences which are sentences included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit and correspond to predetermined tags, and the important information includes one or more important tag corresponding sentences acquired by the important tag corresponding sentence acquisition means, or includes important tag corresponding terms which are terms acquired from the one or more important tag corresponding sentences.

[0040] With this configuration, patents can be classified with high accuracy by using important tag corresponding sentences or important tag corresponding terms.

[0041] Furthermore, the patent classification device of the 18th invention is a patent classification device according to any one of the 9th to 17th inventions, wherein the important information acquisition unit is equipped with a numerical range string acquisition means for acquiring one or more numerical range strings which are strings included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit and indicate a numerical range, and the important information includes the one or more numerical range strings acquired by the numerical range string acquisition means.

[0042] With this configuration, patents can be classified with high accuracy by using the numeric range strings.

[0043] In addition, the patent classification device of the 19th invention is a patent classification device according to any one of the 9th to 18th inventions, wherein the important information acquisition unit is equipped with a numeric range string acquisition means for acquiring one or more numeric range strings, which are strings included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit and indicate a numerical range, and a number acquisition means for acquiring three or more numbers included in the numerical range indicated by the numerical range string acquired by the numeric range string acquisition means, and the important information includes the numbers three or more acquired by the number acquisition means.

[0044] With this configuration, patents can be classified with high accuracy by using numbers equal to or greater than 3 that are included in the numerical range indicated by the numerical range character string.

[0045] In addition, the patent classification device of the twentieth invention, for any one of the ninth to nineteenth inventions, further includes a patent specification acquisition unit that acquires from an external server device a patent specification corresponding to the patent identifier included in the classification target information accepted by the acceptance unit, and the important information acquisition unit is a patent classification device that acquires important information that satisfies predetermined conditions using the patent specification acquired by the patent specification acquisition unit.

[0046] With this configuration, patents can be classified with high accuracy by using important information obtained from patent specifications obtained from an external server device.

[0047] The classification information determination device of the twenty-first invention performs a machine learning learning process using learning source information having two or more patent classification information including patent information and classification information of the patent information, and is equipped with a learning device storage unit in which the acquired learning device is stored, a condition information storage unit in which one or more condition information, which is a pair of a condition related to a patent and classification information of the patent information, a patent information receiving unit for receiving one or more patent information, a prediction unit for acquiring predicted classification information that identifies classification information of one or more pieces of patent information using the one or more condition information, and an output unit for outputting the predicted classification information, wherein the prediction unit is equipped with a judgment means for judging whether each of the one or more pieces of patent information matches any of the conditions held by the one or more pieces of condition information, a first judgment means for determining classification information that pairs with the condition that the judgment means judges to match, and a second judgment means for using the learning device to acquire classification information of patent information that the judgment means judges not to match any of the conditions, by machine learning prediction process.

[0048] This configuration allows patents to be classified with high accuracy.

[0049] In addition, the classification information determination device of the 22nd invention is a classification information determination device compared to the 21st invention, in which one condition is that the patent information constitutes any one of two or more patent classification information contained in the learning source information.

[0050] With this configuration, it is possible to classify with high accuracy the patents that are the training data used when obtaining the machine learning learning device, and it is also possible to classify other patents with high accuracy through predictive processing using the learning device.

[0051] A classification information decision device according to a twenty-third aspect of the present invention differs from the twenty-first aspect of the present invention in that one condition is a search formula.

[0052] With this configuration, patents that match the search query can be classified with high accuracy, and patents that do not match the search query can also be classified with high accuracy through prediction processing using a machine learning learner.

[0053] The patent classification device of the present invention is a patent classification device that includes: a learning device storage unit that uses a machine learning algorithm to learn learning source information having two or more patent records, which are patent records that do not have the full text of the patent specification and have important information obtained using a patent specification corresponding to a patent identifier that identifies a patent, and classification result information that identifies a manual classification result of the patent, and stores the acquired learning device; a reception unit that receives classification target information including a patent identifier that identifies the patent to be classified; an important information acquisition unit that acquires important information that satisfies a predetermined condition using a patent specification that corresponds to the patent identifier included in the classification target information accepted by the reception unit; a classification target processed information acquisition unit that acquires classification target processed information, which is information corresponding to the patent identifier included in the classification target information accepted by the reception unit and includes the important information acquired by the important information acquisition unit; a prediction unit that applies a learning device to the classification target processed information acquired by the classification target processed information acquisition unit, predicts the classification of the classification target processed information using a machine learning algorithm, and acquires classification result information; and an output unit that outputs the classification result information acquired by the prediction unit in association with the patent identifier included in the classification target information accepted by the reception unit.

[0054] With this configuration (first configuration), patents can be classified with high accuracy by using important information about patent specifications.

[0055] In addition, the patent classification device of the present invention is a patent classification device in which, compared to the first configuration, the patent record also includes one or more types of information of the claims or abstract corresponding to the patent identifier, and the processed information to be classified includes the important information acquired by the important information acquisition unit and one or more types of information of the claims or abstract corresponding to the patent identifier.

[0056] With this configuration (second configuration), patents can be classified with high accuracy by using the claims and / or abstracts.

[0057] In addition, in the patent classification device of the present invention, in the first or second configuration, the important information acquisition unit is equipped with an effect-solution-means sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to the patent identifier included in the classification target information received by the receiving unit is an effect-solution-means sentence including both an effect expression and a solution expression, and acquiring one or more effect-solution-means sentences determined to be effect-solution-means sentences, and the important information includes one or more effect-solution-means sentences acquired by the effect-solution-means sentence acquisition means, or effect terms and solution terms which are terms acquired from the one or more effect-solution-means sentences.

[0058] With this configuration (third configuration), patents can be classified with high accuracy by using effect / solution sentences or effect terms or solution terms.

[0059] In addition, the patent classification device of the present invention, in any one of the first to third configurations, is a patent classification device in which the important information acquisition unit is equipped with an effect sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the receiving unit is an effect sentence including an expression of an effect, and acquiring one or more effect sentences determined to be effect sentences, and the important information includes one or more effect sentences acquired by the effect sentence acquisition means, or effect terms which are terms acquired from the one or more effect sentences.

[0060] With this configuration (fourth configuration), patents can be classified with high accuracy by using effect sentences or effect terms.

[0061] In addition, the patent classification device of the present invention, in any one of the first to fourth configurations, is a patent classification device in which the important information acquisition unit is equipped with a solution means sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is a solution means sentence including both a solution means expression and a solution means expression, and acquiring one or more solution means sentences determined to be solution means sentences, and the important information includes one or more solution means sentences acquired by the solution means sentence acquisition means, or solution means terms which are terms acquired from the one or more solution means sentences.

[0062] With this configuration (the fifth configuration), patents can be classified with high accuracy by using solution sentences or solution terms.

[0063] In addition, the patent classification device of the present invention, in any one of the first to fifth configurations, is a patent classification device in which the important information acquisition unit is equipped with an object sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the receiving unit is an object sentence including a target expression, and acquiring one or more object sentences determined to be object sentences, and the important information includes one or more object sentences acquired by the object sentence acquisition means, or object terms which are terms acquired from the one or more object sentences.

[0064] With this configuration (sixth configuration), patents can be classified with high accuracy by using object sentences or object terms.

[0065] In addition, the patent classification device of the present invention, in any one of the first to sixth configurations, is a patent classification device in which the important information acquisition unit acquires the frequency of occurrence of one or more terms included in a patent specification corresponding to the patent identifier included in the classification target information accepted by the accepting unit, calculates a score for each of the one or more terms using the frequency of occurrence, and is equipped with a characteristic word acquisition means for acquiring one or more characteristic words whose score is large enough to satisfy a predetermined condition, and the important information includes the one or more characteristic words acquired by the characteristic word acquisition means.

[0066] With this configuration (seventh configuration), patents can be classified with high accuracy by using characteristic words.

[0067] In addition, the patent classification device of the present invention, in any one of the first to sixth configurations, is a patent classification device comprising: an important information acquisition unit that acquires the frequency of occurrence of one or more terms included in a patent specification corresponding to the patent identifier included in the classification target information received by the receiving unit, calculates a score for each of the one or more terms using the frequency of occurrence, and acquires one or more characteristic words whose score is large enough to satisfy a predetermined condition; and a related word acquisition means that acquires one or more related words which are hypernyms, hyponyms, or synonyms corresponding to each of the one or more characteristic words acquired by the characteristic word acquisition means from a related word dictionary in which one or more related words among hypernyms, hyponyms, or synonyms for each of the one or more terms are stored, and the important information includes the one or more related words acquired by the related word acquisition means.

[0068] With this configuration (the eighth configuration), patents can be classified with high accuracy by using related words of characteristic words.

[0069] In addition, the patent classification device of the present invention is a patent classification device in which, for any one of the first to eighth configurations, the important information acquisition unit is equipped with an important tag corresponding sentence acquisition means for acquiring one or more important tag corresponding sentences which are sentences included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the accepting unit and correspond to a predetermined tag, and the important information includes one or more important tag corresponding sentences acquired by the important tag corresponding sentence acquisition means, or includes important tag corresponding terms which are terms acquired from the one or more important tag corresponding sentences.

[0070] With this configuration (ninth configuration), patents can be classified with high accuracy by using important tag corresponding sentences or important tag corresponding terms.

[0071] In addition, the patent classification device of the present invention is a patent classification device in which, for any one of the first to ninth configurations, the important information acquisition unit is equipped with a numerical range string acquisition means for acquiring one or more numerical range strings, which are strings included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit and indicate a numerical range, and the important information includes the one or more numerical range strings acquired by the numerical range string acquisition means.

[0072] With this configuration (tenth configuration), patents can be classified with high accuracy by using numerical range character strings.

[0073] In addition, the patent classification device of the present invention, in any one of the first to ninth configurations, is a patent classification device in which the important information acquisition unit is equipped with a numeric range string acquisition means for acquiring one or more numeric range strings, which are strings included in a patent specification corresponding to the patent identifier included in the classification target information accepted by the acceptance unit and indicate a numerical range, and a number acquisition means for acquiring three or more numbers included in the numerical range indicated by the numerical range string acquired by the numeric range string acquisition means, and the important information includes the numbers three or more acquired by the number acquisition means.

[0074] With this configuration (eleventh configuration), patents can be classified with high accuracy by using numbers of 3 or more included in the numerical range indicated by the numerical range string.

[0075] In addition, the patent classification device of the present invention, in any one of the first to eleventh configurations, further includes a patent specification acquisition unit that acquires from an external server device a patent specification corresponding to the patent identifier included in the classification target information accepted by the accepting unit, and the important information acquisition unit acquires important information that satisfies predetermined conditions using the patent specification acquired by the patent specification acquisition unit.

[0076] With this configuration (twelfth configuration), patents can be classified with high accuracy by using important information acquired from patent specifications obtained from an external server device.

[0077] In addition, the learning device production device of the present invention is a learning device production device that includes: a manual evaluation information storage unit in which manual evaluation information having two or more manual evaluation records each having a patent identifier that identifies a patent and classification result information that specifies a manual classification result of the patent corresponding to the patent identifier is stored; an important information acquisition unit that acquires important information that satisfies a predetermined condition using a patent specification corresponding to the patent identifier of each of the two or more manual evaluation records in the manual evaluation information stored in the manual evaluation information storage unit; a learning source information acquisition unit that acquires, for each of the two or more manual evaluation records, a patent record having important information acquired by the important information acquisition unit and classification result information that pairs with a patent identifier corresponding to the patent specification that is the source of the important information, and acquires learning source information having two or more patent records; a learning unit that learns the learning source information acquired by the learning source information acquisition unit using a machine learning algorithm to acquire a learning device; and a storage unit that stores the learning devices acquired by the learning unit.

[0078] With this configuration (thirteenth configuration), it is possible to produce a learning machine for classifying patents with high accuracy. Effect of the Invention

[0079] According to the present invention, by performing a predictive process using an appropriate algorithm, patents can be classified with high accuracy.

[0080] In addition, according to the present invention, by obtaining predicted classification information using different decision methods for patent information that meets predetermined conditions and patent information that does not meet the conditions, patents can be classified with high accuracy.

[0081] The present invention also provides for accurate classification of patents by using key information about patent specifications. [Brief description of the drawings]

[0082] [Figure 1] Conceptual diagram of patent classification system A in embodiment 1 [Diagram 2] Block diagram of the patent classification system A [Diagram 3]Block diagram of the learning device production device 1 that constitutes the patent classification system A [Figure 4] A flowchart illustrating an example of the operation of the learning device production device 1. [Diagram 5] A flowchart for explaining an example of the important information acquisition process [Figure 6] A flowchart for explaining an example of the effect solution means statement acquisition process. [Figure 7] A flowchart for explaining an example of the effect term acquisition process. [Figure 8] Flowchart for explaining an example of the numerical value range character string acquisition process [Figure 9] Flowchart for an example of the number acquisition process [Figure 10] A flowchart explaining an example of the operation of the patent classification device 2 [Figure 11] Block diagram of patent classification system B in embodiment 2 [Figure 12] Block diagram of the patent classification device 8 that constitutes the patent classification system B [Figure 13] A flowchart for explaining an example of the operation of the algorithm determination device 5 constituting the patent classification system B [Figure 14] A flowchart for explaining the evaluation process example [Figure 15] Flowchart for explaining the K-fold cross-validation example [Figure 16] A flowchart for explaining the statistical processing evaluation example [Figure 17] A flowchart explaining an example of the operation of the learning device production device constituting the patent classification system B [Figure 18] A flowchart explaining an example of the operation of the patent classification device [Figure 19] Data structure diagram of algorithm information candidates [Figure 20] Data structure of the accuracy information [Figure 21] Block diagram of patent classification system C in embodiment 3 [Figure 22]Block diagram of patent classification system D in embodiment 4 [Figure 23] A flowchart explaining the operation of the classification information determination device 7 constituting the patent classification system D. [Figure 24] A flowchart explaining the k-th predicted classification information acquisition process. [Diagram 25] Data structure of the same condition information [Figure 26] Block diagram of patent classification system E in embodiment 5 [Figure 27] Data structure of the same condition information [Figure 28] Overview of a computer system in each embodiment [Figure 29] Block diagram of the computer system DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0083] Hereinafter, an embodiment of an algorithm determination device and the like will be described with reference to the drawings. Note that, in the embodiments, components with the same reference numerals perform similar operations, so repeated description may be omitted.

[0084] (Embodiment 1) In this embodiment, we will describe a patent classification system that includes a patent classification device that acquires important information from patent specifications, does not include the full text of the patent specifications, and classifies patent information using a learning device obtained by a machine learning algorithm to acquire learning source information having one or more patent records that have important information and manual classification.

[0085] In addition, in this embodiment, we will explain a patent classification system that includes a patent classification device that classifies patent information using a learning device obtained by a machine learning algorithm from learning source information having patent records including claims and / or abstracts.

[0086] In this embodiment, the important information includes effect / solution means sentences, effect sentences, solution means sentences, objective sentences, effect terms, solution means terms, objective terms, characteristic words, related words of characteristic words (superordinate words, subordinate words, synonyms, sentences corresponding to predetermined tags, terms included in the sentences, character strings indicating a numerical range, numbers included in the range, etc.

[0087] In addition, in this embodiment, a patent classification system including a patent classification device that acquires patent specifications from an external server device and acquires important information from the patent specifications will be described.

[0088] Furthermore, in this embodiment, a patent classification system including a learning device production device will be described.

[0089] FIG. 1 is a conceptual diagram of a patent classification system A in this embodiment. The patent classification system A includes a learning device production device 1, a patent classification device 2, a server device 3, and one or more user terminals 4. The learning device production device 1 is a device that produces learning devices. The patent classification device 2 is a device that predicts patent classifications. The server device 3 is a device that stores patent documents including patent specifications and the like, and is a well-known device. The server device 3 is, for example, a commercial patent database server (see, for example, HYPAT-i2 (URL: https: / / portal.hypatweb.jp / )). The user terminal 4 is a terminal used by a user. The user terminal 4 may be, for example, a personal computer, a smartphone, a tablet terminal, or the like.

[0090] Fig. 2 is a block diagram of a patent classification system A in this embodiment. Fig. 3 is a block diagram of a learning module production device 1 constituting the patent classification system A.

[0091] The learning device production device 1 constituting the patent classification system A includes a learning storage unit 11, a learning reception unit 12, a learning processing unit 13, and a learning output unit 14. The learning storage unit 11 includes a manual evaluation information storage unit 111. The learning processing unit 13 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a learning source information acquisition unit 133, and a learning unit 134. The important information acquisition unit 132 includes an effect solution means sentence acquisition means 1321, an effect sentence acquisition means 1322, a solution means sentence acquisition means 1323, an objective sentence acquisition means 1324, a feature word acquisition means 1325, a related word acquisition means 1326, an important tag corresponding sentence acquisition means 1327, a numerical range character string acquisition means 1328, and a numerical value acquisition means 1329. The learning output unit 14 includes an accumulation unit 141.

[0092] The patent classification device 2 includes a storage unit 21, a reception unit 22, a processing unit 23, and an output unit 24. The storage unit 21 includes a learning device storage unit 211. The processing unit 23 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a classification target processing information acquisition unit 233, and a prediction unit 234.

[0093] Various types of information are stored in the learning storage unit 11 constituting the learning module production device 1. The various types of information are, for example, manual evaluation information described later, patent specifications associated with patent identifiers, a related word dictionary, various patterns such as effect-solution-means sentence patterns described later, and various important information learning modules.

[0094] A related word dictionary is a dictionary of related words. For example, a related word dictionary has many pairs of a term and related words. A related word is a superordinate word, a hyponym, a synonym, or a similar word. A superordinate word is a term of a higher concept. A hyponym is a term of a lower concept. The structure of a related word dictionary does not matter, for example, a related word dictionary has two or more related word information. The related word information has, for example, a term and a related word of the term. The related word information is, for example, a term, a related word of the term, and a flag. The flag is information that specifies the relationship between a term and a related word of the term (for example, one of a superordinate word, a hyponym, a synonym, or a similar word).

[0095] The important information learner is a learner that judges whether information is important or not. Examples of the important information learners include an effect / solution means sentence learner, an effect term learner, a solution term learner, an effect sentence learner, a solution means sentence learner, an objective sentence learner, and an objective term learner.

[0096] The manual evaluation information storage unit 111 stores manual evaluation information having two or more manual evaluation records. The manual evaluation record is information having classification result information that specifies a manual classification result for one patent. The manual evaluation record has a patent identifier and classification result information. The classification result information of the manual evaluation record is information that specifies the result of classification of a patent identified by the patent identifier of the manual evaluation record. The patent identifier is information that identifies a patent. The patent identifier is, for example, an ID, an application number, a publication number, a registration number, a serial number, etc. The classification result information is usually information that specifies the manual classification result for a patent. The classification result information is, for example, information that specifies the technical field of a patent, information that specifies the relevance of a patent, information that specifies the importance of a patent, etc. The classification result information is, for example, "○", "×", "A", "B", "C", "D", "1", "2", "3", ... "9", a classification unique to the company used in the company, etc., and the type of information does not matter. The manual evaluation information may be, for example, a CSV file, an Excel file, or the like, and the data structure thereof is not important.

[0097] In addition to the patent identifier and classification result information, the manual review record preferably includes, for example, claims, abstracts, and patent classification codes (e.g., IPC, FI, F-terms, CPC, etc.). The manual review record may also include a patent specification.

[0098] The learning reception unit 12 receives instructions, information, and the like. The instructions, information, and the like are, for example, a learning operation start instruction. Here, the learning operation start instruction is an instruction to start the production of a learning device. The learning operation start instruction is usually an instruction from a user. The learning operation start instruction may have, for example, type specification information that specifies the type of important information acquired by the important information acquisition unit 132. The type specification information is information that specifies the type of important information used to create a learning device. The type specification information may be, for example, information that specifies the means to be operated from among the effect solution means sentence acquisition means 1321, the effect sentence acquisition means 1322, the solution means sentence acquisition means 1323, the objective sentence acquisition means 1324, the feature word acquisition means 1325, the related word acquisition means 1326, the important tag corresponding sentence acquisition means 1327, the numerical range character string acquisition means 1328, and the number acquisition means 1329. In addition, the type-specific information may be information that identifies the important information to be obtained from, for example, an effect-solution means sentence, an effect sentence, a solution means sentence, an objective sentence, an effect term, a solution means term, an objective term, a characteristic word, a related word of a characteristic word (superordinate word, subordinate word, synonym), a tag-corresponding sentence, a tag-corresponding term, a numeric range string, or important information of a number in the range indicated by the numeric range string.

[0099] The type specification information is, for example, a vector, and has a structure of, for example, (whether to acquire an effect solution means sentence, whether to acquire an effect sentence, whether to acquire an effect sentence, whether to acquire a solution means sentence, whether to acquire an object sentence, whether to acquire an effect term, whether to acquire a solution means term, whether to acquire an object term, whether to acquire a feature word, whether to acquire a related word of the feature word, whether to acquire a tag corresponding sentence, whether to acquire a tag corresponding term, whether to acquire a numeric range string, whether to acquire a number in the range indicated by the numeric range string). In this case, when the type specification information is (1,0,···0,1), the type specification information indicates that only the effect solution means sentence and the number in the range indicated by the numeric range string are acquired. In the above vector (1,0,···0,1), all elements corresponding to "···" are 0. In addition, it indicates that important information corresponding to an element "1" is acquired, and important information corresponding to an element "0" is not acquired.

[0100] An effect / solution sentence is a sentence that includes both an expression of the effect and an expression of the solution. An effect / solution sentence is a sentence that matches a pattern (which may be called an effect / solution sentence pattern) such as "<Solution> allows <effect>," "By providing <Solution>, <effect> becomes possible," "<Effect> can be achieved based on <Solution>," or "By using <Solution>, <effect> is achieved." Note that information indicated by "<" and ">" (e.g., <Solution>, <Effect>) is a variable that matches any character string. An effect / solution sentence is, for example, "By providing a protrusion, it becomes possible to adjust the airflow," or "By using key terms, the accuracy of patent classification is improved."

[0101] Moreover, an effect sentence is a sentence that includes an expression of an effect. An effect sentence is, for example, a sentence that matches a pattern (which may be called an effect sentence pattern) such as "<effect> can be achieved," "<effect> becomes possible," or "<effect> becomes." An effect sentence is, for example, "According to this embodiment, it becomes possible to adjust the air volume," or "According to this embodiment, the accuracy of patent classification is improved." An effect sentence may be considered to be the same as a problem sentence, which is a sentence that identifies a problem.

[0102] A solution sentence is a sentence that includes an expression of a solution. For example, a solution sentence is a sentence that matches a pattern (which may be called a solution sentence pattern) such as "The key point is <solution>," "Based on <solution>," or "The problem can be solved by <solution>." For example, a solution sentence is "The problem can be solved by the protrusion," or "The above problem can be solved by the key term acquisition unit."

[0103] An objective sentence is a sentence that includes an expression of an objective. An objective sentence is, for example, a sentence of a pattern (which may be called an objective sentence pattern) such as "In this invention, it is an objective to <objective>." or "The objective of the present invention is <objective>." An objective sentence is, for example, "In this invention, it is an objective to adjust the air volume." or "The objective of the present invention is to improve the accuracy of patent classification." Note that an objective sentence may be considered to be the same as an effect sentence or problem sentence. This is because objectives, effects, and problems are usually corresponding information.

[0104] An effect term is a term that indicates an effect, such as "improved accuracy," "energy saving," "air volume adjustment possible," "resource saving," "improved safety," and "improved processing speed." Note that an effect term may be considered to be the same as a problem term that specifies a problem.

[0105] The solution term is a term indicating a solution, such as "protrusion," "important term," "important information acquisition unit," "processing unit," etc.

[0106] The target term is a term that indicates a purpose, such as "improved accuracy," "energy saving," "adjustable airflow," "resource saving," "improved safety," and "improved processing speed." Note that the target term may be considered to be the same as the effect term or task term.

[0107] A characteristic word is a term that indicates the characteristics of a patent specification. A characteristic word is a term that appears frequently in a patent specification of interest. It is preferable that a characteristic word is a term that appears frequently in a patent specification of interest and that appears infrequently in other patent specifications. A characteristic word is a term in a patent specification, for example, a term whose TF / IDF value is large enough to satisfy a predetermined condition. Note that a method for calculating the TF / IDF value is a publicly known technology, so a detailed explanation is omitted.

[0108] A related word of a feature word is a superordinate word of the feature word, a subordinate word of the feature word, or a synonym of the feature word. A related word of a feature word is, for example, a term obtained by searching a related word dictionary using the feature word as a key.

[0109] The important tag corresponding sentence is a sentence corresponding to a predetermined tag. The predetermined tag is a tag that is predetermined in the patent specification. The predetermined tag is, for example, the tag of [Effects of the invention], [Problems to be solved by the invention], or [Industrial applicability].

[0110] A tag-enabled term is a term that is included in an important tag-enabled sentence.

[0111] The numerical range string is a string included in the patent specification, and is a string indicating a numerical range. The numerical range string has, for example, a unit. The numerical range string is a string that matches, for example, the patterns of "<number><unit>~<number><unit>", "<number>~<number><unit>", "<number>~<number>", "<number><unit> to <number><unit>", "<number> to <number>", "<number><unit>-<number><unit>", "<number>-<number><unit>", "<number>-<number>" and "<number>-<number>". In addition, the positive and negative numbers, the number of significant digits, and the like do not matter. In addition, the unit is, for example, a unit of length, a unit of weight, a unit of ratio, and the like, and the type does not matter. The numerical range string is, for example, "1~10mm", "2.5mg-5.2mg", "0.8% to 1.9%", and the like.

[0112] When the numeric range string is "2.5mg-5.2mg", the numbers included in the numeric range indicated by the numeric range string are, for example, "2.5mg", "2.6mg", "2.7mg", . . . "5.1mg", and "5.2mg".

[0113] Here, acceptance usually means receipt from a user terminal 4, but it may also be considered as a concept including acceptance of information input from an input device such as a keyboard, mouse, or touch panel, and acceptance of information read from recording media such as an optical disk, magnetic disk, or semiconductor memory.

[0114] The learning processing unit 13 performs various processes. The various processes are, for example, processes performed by the patent specification acquisition unit 131, the important information acquisition unit 132, the learning source information acquisition unit 133, the learning unit 134, the effect / solution means sentence acquisition unit 1321, the effect sentence acquisition unit 1322, the solution means sentence acquisition unit 1323, the objective sentence acquisition unit 1324, the feature word acquisition unit 1325, the related word acquisition unit 1326, the important tag corresponding sentence acquisition unit 1327, the numeric range character string acquisition unit 1328, and the numeric value acquisition unit 1329.

[0115] The learning processing unit 13, for example, learns important information by a machine learning algorithm, acquires a learning device that judges whether the information is important or not, and accumulates it in the learning storage unit 11. For machine learning, for example, deep learning, SVM, random forest, decision tree, etc. can be used. In machine learning, to configure a learning device, for example, a learning device is obtained by giving a group of input information (for example, a vector composed of sentences, a distributed representation of terms, etc.) and information to be output (for example, judgment information on whether the information is important or not) as arguments to a machine learning function (for example, fastText, TinySVM, various RandomForest functions, TensorFlow library, etc.). In machine learning, when making a prediction, a learning device and a group of input information (for example, a vector composed of sentences, a distributed representation of terms, etc.) are given as arguments to a machine learning function, and predicted information (for example, judgment information on whether the information is important or not) is obtained.

[0116] The learning processing unit 13 acquires an effect solution means sentence learner, which is a learning device for predicting whether or not a sentence to be judged is an effect solution means sentence, by using, for example, one or more effect solution means sentences and one or more non-effect solution means sentences, through a machine learning algorithm, and accumulates the effect solution means sentence learner in the learning storage unit 11. Note that a non-effect solution means sentence is a sentence that is not an effect solution means sentence.

[0117] More specifically, the learning processing unit 13, for example, acquires one or more effective solution means sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more effective solution means sentences, constructs a vector from the acquired one or more morphemes, and uses the vector as a positive example, and acquires one or more non-effective solution means sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more non-effective solution means sentences, constructs a vector from the acquired one or more morphemes, and uses the vector as a negative example to provide the machine learning function, and performs learning processing by machine learning, thereby acquiring an effective solution means sentence learner and storing it in the learning storage unit 11. Note that there is no restriction on the method of constructing a vector from a sentence. In other words, there is no restriction on what information is used as a feature when constructing a vector from a sentence. It goes without saying that not only one or more acquired morphemes but also n-grams may be used as features.

[0118] The learning processing unit 13 acquires an effective term learner, which is a learner for predicting whether a term to be judged is an effective term or not, by a machine learning algorithm using, for example, one or more effective terms and one or more non-effective terms, and stores the effective term learner in the learning storage unit 11. Note that a non-effective term is a term that is not an effective term.

[0119] More specifically, the learning processing unit 13, for example, acquires one or more effect terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more effect terms (for example, using word2vec), uses the distributed representation as a positive example, acquires one or more non-effect terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more non-effect terms, and provides the distributed representation as a negative example to a machine learning function, performs learning processing by machine learning, thereby acquiring an effect term learner and storing it in the learning storage unit 11.

[0120] The learning processing unit 13 acquires a solution means term learner, which is a learner for predicting whether a term to be judged is a solution means term or not, by using, for example, one or more solution means terms and one or more non-solution means terms, through a machine learning algorithm, and accumulates the solution means term learner in the learning storage unit 11. Note that a non-solution means term is a term that is not a solution means term.

[0121] More specifically, the learning processing unit 13, for example, acquires one or more solution means terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more solution means terms, and uses the distributed representation as a positive example; it also acquires one or more non-solution means terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more non-solution means terms, and uses the distributed representation as a negative example to provide the machine learning function; and performs learning processing by machine learning, thereby acquiring a solution means term learner and storing it in the learning storage unit 11.

[0122] The learning processing unit 13 acquires a solution-means sentence learner, which is a learner for predicting whether a sentence to be judged is a solution-means sentence or not, by using, for example, one or more solution-means sentences and one or more non-solution-means sentences through a machine learning algorithm, and accumulates the solution-means sentence learner in the learning storage unit 11. Note that a non-solution-means sentence is a sentence that is not a solution-means sentence.

[0123] More specifically, the learning processing unit 13, for example, acquires one or more solution means sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more solution means sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a positive example, and acquires one or more non-solution means sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more non-solution means sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a negative example, and provides the vector to a machine learning function to perform learning processing by machine learning, thereby acquiring a solution means sentence learner and storing it in the learning storage unit 11.

[0124] The learning processing unit 13 acquires an effect sentence learner, which is a learner for predicting whether or not a sentence to be judged is an effect sentence, by a machine learning algorithm using, for example, one or more effect sentences and one or more non-effect sentences, and stores the effect sentence learner in the learning storage unit 11. Note that a non-effect sentence is a sentence that is not an effect sentence.

[0125] More specifically, the learning processing unit 13, for example, acquires one or more effective sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more effective sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a positive example, and acquires one or more non-effective sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more non-effective sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a negative example, and provides the vector to a machine learning function to perform learning processing by machine learning, thereby acquiring an effective sentence learner and storing it in the learning storage unit 11.

[0126] The learning processing unit 13 acquires a target sentence learner, which is a learner for predicting whether a sentence to be judged is a target sentence or not, by using, for example, one or more target sentences and one or more non-target sentences, through a machine learning algorithm, and accumulates the target sentence learner in the learning storage unit 11. Note that a non-target sentence is a sentence that is not a target sentence.

[0127] More specifically, the learning processing unit 13, for example, acquires one or more target sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more target sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a positive example, and acquires one or more non-target sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more non-target sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a negative example, and provides the vector to a machine learning function to perform learning processing by machine learning, thereby acquiring a target sentence learner and storing it in the learning storage unit 11.

[0128] The learning processing unit 13 acquires a target term learner, which is a learner for predicting whether a term to be judged is a target term or not, by a machine learning algorithm using, for example, one or more target terms and one or more non-target terms, and stores the target term learner in the learning storage unit 11. Note that a non-target term is a term that is not a target term.

[0129] More specifically, the learning processing unit 13, for example, acquires one or more target terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more target terms, uses the distributed representation as a positive example, and acquires one or more non-target terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more non-target terms, uses the distributed representation as a negative example, provides it to a machine learning function, and performs learning processing by machine learning to acquire a target term learner and store it in the learning storage unit 11.

[0130] It goes without saying that the method by which the learning processing unit 13 acquires vectors from sentences and the method by which the learning processing unit 13 acquires vectors from terms are not important. The learning processing unit 13 can use various methods for acquiring vectors from sentences and various methods for acquiring vectors from terms.

[0131] The patent specification acquisition unit 131 acquires a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit 22. For example, the patent specification acquisition unit 131 acquires a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit 22 from an external server device 3. For example, the patent specification acquisition unit 131 acquires a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit 22 from the learning storage unit 11. For example, the patent specification acquisition unit 131 may acquire a patent specification included in a manual evaluation record.

[0132] The important information acquisition unit 132 acquires important information that satisfies a predetermined condition by using a patent specification that corresponds to a patent identifier included in the classification target information received by the reception unit 22 described below.

[0133] The important information acquisition unit 132 uses the patent specification acquired by the patent specification acquisition unit 131 to acquire important information that satisfies a predetermined condition.

[0134] The important information acquisition unit 132 acquires important information that satisfies a predetermined condition by using a patent specification corresponding to a patent identifier of each of two or more manual evaluation records of the manual evaluation information stored in the manual evaluation information storage unit 111. The important information is a sentence that satisfies the predetermined condition, or a term that satisfies the predetermined condition.

[0135] The predetermined conditions include, for example, matching an effect solution sentence, matching an effect term, matching a solution term, matching an effect sentence, matching a solution sentence, matching an objective sentence, matching an objective term, matching a feature word, matching a related word, matching an important tag corresponding sentence, matching an important tag corresponding term, matching a numeric range string, matching a numeric value that can be obtained from a numeric range string, etc.

[0136] The important information includes, for example, one or more effect solution means sentences acquired by the effect solution means sentence acquisition means 1321, or effect terms and solution means terms that are terms acquired from one or more effect solution means sentences.

[0137] The important information includes, for example, one or more effect sentences acquired by the effect sentence acquisition means 1322, or effect terms that are terms acquired from one or more effect sentences.

[0138] The important information includes, for example, one or more solution sentences acquired by the solution sentence acquisition unit 1323, or solution terms that are terms acquired from one or more solution sentences.

[0139] The important information includes, for example, one or more object sentences acquired by the object sentence acquiring unit 1324, or object terms which are terms acquired from one or more object sentences.

[0140] The important information includes, for example, one or more characteristic words acquired by the characteristic word acquisition unit 1325 .

[0141] The important information includes, for example, one or more related terms acquired by the related term acquisition unit 1326 .

[0142] The important information includes, for example, one or more important tag corresponding sentences acquired by the important tag corresponding sentence acquisition means 1327, or includes important tag corresponding terms which are terms acquired from one or more important tag corresponding sentences.

[0143] The important information includes, for example, one or more numeric range strings obtained by the numeric range string obtaining means 1328 .

[0144] The important information includes, for example, a number equal to or greater than 3 acquired by the number acquiring unit 1329. When the numeric range character string is "2.5mg-5.2mg", the numbers equal to or greater than 3 included in the numeric range indicated by the numeric range character string are, for example, "2.5mg", "2.6mg", "2.7mg", "5.1mg", and "5.2mg".

[0145] The effect / solution means sentence acquisition means 1321, effect sentence acquisition means 1322, solution means sentence acquisition means 1323, and objective sentence acquisition means 1324 constituting the important information acquisition unit 132 acquire important information using, for example, the pattern matching method (1) or the machine learning method (2), as described below. Note that it is preferable that the feature word acquisition means 1325, related word acquisition means 1326, important tag corresponding sentence acquisition means 1327, numeric range character string acquisition means 1328, and number acquisition means 1329 acquire important information by adopting a method other than the above (1) and (2). Note that the following method of acquiring important information is one example, and it goes without saying that important information may be acquired by other methods.

[0146] The important information acquisition unit 132 acquires sentences including specific expressions from the patent specification. The sentences including specific expressions are, for example, effect-solution means sentences, effect sentences, solution means sentences, or objective sentences.

[0147] The important information acquisition unit 132 acquires terms contained in sentences including specific expressions by using the patent specification. The terms contained in sentences including specific expressions are, for example, effect terms, solution terms, or object terms.

[0148] The important information acquisition unit 132 acquires characteristic words by using the patent specification. The important information acquisition unit 132 acquires related words by using the characteristic words.

[0149] The important information acquisition unit 132 uses the patent specification to acquire important tag corresponding sentences or important tag corresponding terms that correspond to a specific tag.

[0150] Below, examples of how important information can be obtained will be explained for each means. (A) Effect Solution Means Sentence Acquisition Means 1321

[0151] The effect solution means sentence acquisition means 1321 determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the receiving unit 22 is an effect solution means sentence, and acquires one or more effect solution means sentences that are determined to be effect solution means sentences. (1) Pattern matching method

[0152] The effect solution means sentence acquisition means 1321 acquires one or more sentences from the patent specification to be inspected, and judges whether each sentence matches a predetermined effect solution means sentence pattern. The effect solution means sentence acquisition means 1321 acquires a sentence that matches one or more effect solution means sentence patterns as an effect solution means sentence.

[0153] The effect solution means sentence pattern is information that specifies the pattern of the effect solution means sentence. One or more effect solution means sentence patterns are stored, for example, in the learning storage unit 11. Also, the effect solution means sentence pattern is, for example, "<effect> can be achieved by <solution means>," "<effect> can be achieved by providing <solution means>," and "<effect> can be achieved based on <solution means>." Note that the pattern matching technology is a well-known technology, so a detailed explanation will be omitted.

[0154] In addition, it is preferable to increase patterns such as effect solution sentence patterns by using a bootstrap algorithm. (2) Machine learning method

[0155] The effect solution means sentence acquisition means 1321 acquires one or more sentences from the patent specification to be inspected, applies each sentence to an effect solution means sentence learning device that judges whether it is an effect solution means sentence or not, predicts by a machine learning algorithm, and acquires judgment information that is the result of the prediction. The judgment information here is information indicating whether a sentence is an effect solution means sentence or not.

[0156] As in the learning process, machine learning can be used for prediction processing, such as deep learning, SVM, random forest, decision tree, etc. In machine learning, when making predictions, a learning device and an input information group (e.g., vectors composed of sentences, distributed representations of terms, etc.) are given as arguments to a machine learning function (e.g., fastText, TinySVM, various RandomForest functions, TensorFlow libraries, etc.), and predicted information (e.g., information on whether the information is important or not) is obtained.

[0157] Then, when the judgment information acquired by the machine learning function is judgment information that the sentence is an "effect solution means sentence", the effect solution means sentence acquisition means 1321 acquires the sentence as an effect solution means sentence. (3) Acquisition of effect terms and solution terms

[0158] The effect solution means sentence acquisition means 1321 acquires effect terms (terms corresponding to <effect>) and solution means terms (terms corresponding to <solution>) from the acquired effect solution means sentence, for example, by using an effect solution means sentence pattern corresponding to the acquired effect solution means sentence (for example, "<effect> can be achieved by <solution means>.").

[0159] Moreover, the effect / solution means sentence acquisition means 1321 acquires effect terms and solution means terms, for example, by a machine learning algorithm.

[0160] For example, the effect solution means sentence acquisition means 1321 performs morphological analysis on the acquired effect solution means sentence, applies an effect term learner to each of the acquired one or more terms, and judges whether or not it is an effect term by a machine learning algorithm. Also, for example, the effect solution means sentence acquisition means 1321 performs morphological analysis on the acquired effect solution means sentence, applies a solution term learner to each of the acquired one or more terms, and judges whether or not it is a solution term by a machine learning algorithm.

[0161] The effect / solution means sentence acquisition means 1321 may perform morphological analysis on each sentence of the patent specification, apply an effect term learner to each of the one or more acquired terms, and determine whether or not it is an effect term by a machine learning algorithm. The effect / solution means sentence acquisition means 1321 may perform morphological analysis on each sentence of the patent specification, apply a solution term learner to each of the one or more acquired terms, and determine whether or not it is a solution term by a machine learning algorithm. (B) Means of obtaining effect sentence 1322

[0162] The effect sentence acquisition means 1322 determines whether or not each of one or more sentences included in the patent specification corresponding to the patent identifier included in the classification target information accepted by the accepting unit 22 is an effect sentence, and acquires one or more effect sentences that are determined to be effect sentences. (1) Pattern matching method

[0163] The effect sentence acquisition means 1322 acquires one or more sentences from the patent specification to be inspected, and judges whether each sentence matches a predetermined effect sentence pattern. The effect sentence acquisition means 1322 acquires a sentence that matches one or more effect sentence patterns as an effect sentence.

[0164] The effect sentence pattern is information that specifies a pattern of an effect sentence. One or more effect sentence patterns are stored, for example, in the learning storage unit 11. The effect sentence pattern is, for example, "can achieve <effect>" or "becomes <effect>."

[0165] In addition, it is preferable to increase the number of patterns such as effect sentence patterns by using a bootstrap algorithm. (2) Machine learning method

[0166] The effect sentence acquisition means 1322 acquires one or more sentences from the patent specification to be inspected, applies to each sentence an effect sentence learning device that judges whether or not it is an effect sentence, makes a prediction by a machine learning algorithm, and acquires judgment information that is the result of the prediction. The judgment information here is information indicating whether or not a sentence is an effect sentence.

[0167] Then, when the judgment information acquired by the machine learning function is judgment information that the sentence is an "effect sentence", the effect sentence acquisition means 1322 acquires the sentence as an effect sentence. (3) Acquisition of effect terms

[0168] The effect sentence acquisition means 1322 acquires an effect term (a term corresponding to <effect>) from the acquired effect sentence, for example, by using an effect sentence pattern corresponding to the acquired effect sentence (for example, "<effect> can be done.").

[0169] For example, the effect sentence acquisition means 1322 performs morphological analysis on the acquired effect sentence, applies an effect term learner to each of the one or more acquired terms, and determines whether or not each term is an effect term using a machine learning algorithm.

[0170] In addition, the effect sentence acquisition means 1322 may perform morphological analysis of each sentence in the patent specification, apply an effect term learning device to each of one or more acquired terms, and determine whether or not the term is an effect term using a machine learning algorithm. (C) Solution sentence acquisition means 1323

[0171] The solution sentence acquisition means 1323 determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the receiving unit 22 is a solution sentence, and acquires one or more solution sentences determined to be solution sentences. (1) Pattern matching method

[0172] The solution sentence acquisition means 1323 acquires one or more sentences from the patent specification to be inspected, and judges whether each sentence matches a predetermined solution sentence pattern. The solution sentence acquisition means 1323 acquires a sentence that matches one or more solution sentence patterns as a solution sentence.

[0173] The solution sentence pattern is information that specifies a pattern of a solution sentence. One or more solution sentence patterns are stored, for example, in the learning storage unit 11. The solution sentence pattern is, for example, "The point is <solution>" or "Based on <solution>."

[0174] It is also preferable to increase the number of patterns such as solution sentence patterns using a bootstrap algorithm. (2) Machine learning method

[0175] The solution sentence acquisition means 1323 acquires one or more sentences from the patent specification to be inspected, applies each sentence to a solution sentence learning device that judges whether it is a solution sentence or not, predicts using a machine learning algorithm, and acquires judgment information that is the result of the prediction. The judgment information here is information indicating whether a sentence is a solution sentence or not.

[0176] Then, if the judgment information acquired by the machine learning function is "a solution sentence", the solution sentence acquisition means 1323 acquires the sentence as a solution sentence. (3) Acquisition of solution terms

[0177] The solution sentence acquisition means 1323 acquires a solution term (a term corresponding to <solution>) from the acquired solution sentence, for example, by using a solution sentence pattern corresponding to the acquired solution sentence (for example, "The point is <solution>.").

[0178] Moreover, the solution sentence acquisition unit 1323 acquires solution terms by, for example, a machine learning algorithm.

[0179] For example, the solution means sentence acquisition means 1323 performs morphological analysis on the acquired solution means sentence, applies a solution means term learner to each of the one or more acquired terms, and judges whether or not it is a solution means term by a machine learning algorithm. Also, for example, the effect solution means sentence acquisition means 1321 performs morphological analysis on the acquired effect solution means sentence, applies a solution means term learner to each of the one or more acquired terms, and judges whether or not it is a solution means term by a machine learning algorithm.

[0180] In addition, the solution means sentence acquisition means 1323 may perform morphological analysis of each sentence in the patent specification, apply a solution means term learning device to each of one or more acquired terms, and determine whether or not the term is a solution means term using a machine learning algorithm. (D) Objective sentence acquisition means 1324

[0181] The object sentence acquisition means 1324 determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the accepting unit 22 is an object sentence, and acquires one or more object sentences that are determined to be object sentences. (1) Pattern matching method

[0182] The object sentence acquisition means 1324 acquires one or more sentences from the patent specification to be inspected, and judges whether each sentence matches a predetermined object sentence pattern. The object sentence acquisition means 1324 acquires a sentence that matches one or more object sentence patterns as an object sentence.

[0183] The objective sentence pattern is information that specifies the pattern of the objective sentence. One or more objective sentence patterns are stored, for example, in the learning storage unit 11. The objective sentence pattern is, for example, "The objective of the present invention is to achieve <objective>." or "The objective of the present invention is <objective>."

[0184] In addition, it is preferable to increase the number of patterns such as objective sentence patterns by using a bootstrap algorithm. (2) Machine learning method

[0185] The object sentence acquisition means 1324 acquires one or more sentences from the patent specification to be inspected, applies an object sentence learning device to each sentence to determine whether it is an object sentence or not, predicts using a machine learning algorithm, and acquires judgment information that is the result of the prediction. The judgment information here is information indicating whether a sentence is an object sentence or not.

[0186] Then, if the judgment information acquired by the machine learning function is judgment information that the sentence is a "object sentence", the object sentence acquisition means 1324 acquires the sentence as an object sentence. (3) Acquisition of target terms

[0187] The objective sentence acquisition means 1324 acquires an objective term (a term corresponding to <objective>) from the acquired objective sentence, for example, by using an objective sentence pattern corresponding to the acquired objective sentence (for example, "In this invention, it is the objective to make <objective>.").

[0188] Furthermore, the object sentence acquiring unit 1324 acquires object terms by, for example, a machine learning algorithm.

[0189] For example, the object sentence acquisition means 1324 performs morphological analysis on the acquired object sentence, applies an object term learner to each of the one or more acquired terms, and determines whether or not each is an object term by a machine learning algorithm. Also, for example, the object sentence acquisition means 1324 performs morphological analysis on the acquired object sentence, applies an object term learner to each of the one or more acquired terms, and determines whether or not each is an object term by a machine learning algorithm.

[0190] In addition, the target sentence acquisition means 1324 may perform morphological analysis of each sentence in the patent specification, apply a target term learner to each of one or more acquired terms, and determine whether or not the term is a target term using a machine learning algorithm. (E) characteristic word acquisition means 1325

[0191] The characteristic word acquisition means 1325 acquires the frequency of occurrence of one or more terms included in the patent specification corresponding to the patent identifier included in the classification target information accepted by the accepting unit 22, calculates a score for each of the one or more terms using the frequency of occurrence, and acquires one or more characteristic words whose score is large enough to satisfy a predetermined condition.

[0192] The characteristic word acquisition means 1325 acquires, for example, the number of occurrences of one or more terms included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the accepting unit 22. The characteristic word acquisition means 1325 also acquires, for example, the number of occurrences of one or more terms in a set of target patent specifications. Then, the characteristic word acquisition means 1325 calculates a TF / IDF value (score) of one or more terms included in a patent specification corresponding to a patent identifier included in the classification target information. Next, the characteristic word acquisition means 1325 acquires, as characteristic words of the patent specification, one or more terms whose scores are high enough to satisfy a predetermined condition. Note that any technology may be used to extract characteristic words from a document. The characteristic word acquisition means 1325 can use various technologies. (F) Related word acquisition means 1326

[0193] The related word acquisition means 1326 acquires, from the related word dictionary, one or more related words corresponding to each of the one or more characteristic words acquired by the characteristic word acquisition means 1325. Such related words are an example of important information. The related word dictionary used by the related word acquisition means 1326 may be an internal dictionary of the learning module production device 1 or the patent classification device 2, or may be a dictionary of a server (not shown). (F) Means for acquiring important tag corresponding sentences 1327

[0194] The important tag corresponding sentence acquisition means 1327 acquires one or more important tag corresponding sentences, which are sentences included in the patent specification corresponding to the patent identifier included in the classification target information accepted by the acceptance unit 22 .

[0195] The important tag corresponding sentence acquisition means 1327, for example, reads important tag identification information that identifies a predetermined tag from the learning storage unit 11, and acquires one or more sentences in the tag identified by the important tag identification information from the patent specification to be processed. Note that the important tag identification information is, for example, [Effects of the invention], [Problems to be solved by the invention], or [Industrial applicability]. For example, the technology of acquiring one or more sentences corresponding to [Effects of the invention] from a patent specification is a publicly known technology.

[0196] In addition, the important tag corresponding sentence acquisition unit 1327 may acquire important tag terms from one or more sentences corresponding to the important tag identification information. For example, the important tag corresponding sentence acquisition unit 1327 performs morphological analysis on one or more sentences and acquires one or more independent words as important tag terms. (G) Numeric range character string acquisition method 1328

[0197] The numerical range string acquisition means 1328 acquires one or more numerical range strings, which are strings included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit 22 and indicate a numerical range.

[0198] The numeric range string acquisition means 1328 acquires, for example, a string that matches a numeric range string pattern from a patent specification. The numeric range string pattern is stored in the learning storage unit 11. The numeric range string patterns are, for example, "<number><unit>~<number><unit>", "<number>~<number><unit>", "<number>~<number>", "<number><unit> to <number><unit>", "<number> to <number><unit>", "<number> to <number>", "<number><unit>-<number><unit>", "<number>-<number><unit>", and "<number>-<number>".

[0199] It is preferable that the numerical value range character string acquiring means 1328 acquires one or more numerical value range character strings from one or more sentences in an area corresponding to a specific tag in a patent specification corresponding to a patent identifier included in the classification target information accepted by the accepting unit 22. The area corresponding to the specific tag is, for example, [Mode for carrying out the invention], [Example], [Experiment], etc. (H) Number acquisition means 1329

[0200] The number acquiring means 1329 acquires a number equal to or greater than 3 included in the numerical range indicated by the numerical range string acquired by the numerical range string acquiring means 1328. The number acquiring means 1329 acquires at least both ends of the numerical range indicated by the numerical range string acquired by the numerical range string acquiring means 1328 and numerical values ​​equal to or greater than 1. The number acquiring means 1329 usually acquires the upper and lower limit values ​​of the numerical range, but it is not necessary to acquire the upper and lower limit values. The number acquiring means 1329 is a numerical value within the numerical range indicated by the numerical range string, and it is preferable to acquire the numerical value of a string that does not appear in the numerical range string.

[0201] When the acquired numeric range character string is "2.5mg-5.2mg", the number acquiring means 1329 preferably acquires a number included in the numeric range indicated by the numeric range character string, for example, "2.5mg", "2.6mg", "2.7mg", "5.1mg", "5.2mg". That is, the number acquiring means 1329 generates a numeric value in the range "2.5-5.2" indicated by the acquired numeric range character string, which is a numeric value of the number of digits indicated by the number of significant digits (here, for example, one decimal place) of the numeric range character string, and acquires a numeric value of 3 or more. Note that the numeric value acquired by the number acquiring means 1329 may or may not have a unit.

[0202] The learning source information acquisition unit 133 acquires a patent record having the important information acquired by the important information acquisition unit 132 and the classification result information for each of the two or more manual evaluation records, and acquires learning source information having two or more patent records. The classification result information is information paired with a patent identifier corresponding to the patent specification that is the source of the important information, and is usually information indicating the result of manual classification of the patent.

[0203] The patent record preferably also includes one or more of the following types of information: claims or abstract corresponding to the patent identifier, and preferably includes one or more patent classification codes.

[0204] The learning unit 134 learns the learning source information acquired by the learning source information acquisition unit 133 using a machine learning algorithm, and acquires a learning device.

[0205] The learning unit 134 uses information other than the classification result information among the learning source information acquired by the learning source information acquisition unit 133 as input, and configures a learning device that outputs the classification result information using a machine learning algorithm. For machine learning, for example, deep learning, SVM, random forest, decision tree, etc. can be used. In machine learning, to configure a learning device, for example, a learning device is obtained by giving an input information group (for example, information other than the classification result information among the learning source information) and information to be output (for example, classification result information) as arguments to a machine learning function (for example, fastText, TinySVM, various RandomForest functions, TensorFlow library, etc.). The input information group and the information to be output configure a patent record.

[0206] The learning output unit 14 outputs various information, such as information about a learning device and information indicating that the learning process has been completed.

[0207] Here, output is a concept that includes display on a display, projection using a projector, printing on a printer, sound output, transmission to an external device, storage on a recording medium, and handing over the processing results to other processing devices or other programs, etc.

[0208] The accumulation unit 141 accumulates the learning device acquired by the learning unit 134. The accumulation unit 141 accumulates the learning device, for example, in the learning storage unit 11. It is preferable that the accumulation unit 141 accumulates the learning device acquired by the learning unit 134 by associating the type identification information included in the learning operation start instruction with the learning device.

[0209] Various types of information are stored in the storage unit 21 constituting the patent classification device 2. The various types of information are, for example, a learning device.

[0210] One or more learning devices are stored in the learning device storage unit 211. The learning device in the learning device storage unit 211 is, for example, a learning device acquired by the learning device production device 1. The learning device in the learning device storage unit 211 is associated with, for example, type identification information.

[0211] The learning device is a learning device obtained by learning learning source information having two or more patent records using a machine learning algorithm. The patent record does not have the full text of the patent specification. The patent record has important information obtained using the patent specification corresponding to the patent identifier that identifies the patent, and classification result information that identifies the manual classification result of the patent.

[0212] The reception unit 22 receives, for example, a prediction start instruction. The prediction start instruction is an instruction to start prediction. The prediction start instruction includes, for example, classification target information of the prediction target, information for identifying a learning device to be used, and the like.

[0213] The receiving unit 22 receives, for example, one or more pieces of classification target information. The classification target information includes a patent identifier that identifies the patent to be classified. The classification target information includes, for example, one or more pieces of information among the scope of claims and the abstract. The classification target information includes, for example, one or more patent classification codes. The classification target information includes, for example, a patent specification.

[0214] Here, acceptance usually means receipt from a user terminal 4, but it may also be considered as a concept including acceptance of information input from an input device such as a keyboard, mouse, or touch panel, and acceptance of information read from recording media such as an optical disk, magnetic disk, or semiconductor memory.

[0215] The processing unit 23 performs various types of processing. The various types of processing are, for example, processing performed by the patent specification acquisition unit 131, the important information acquisition unit 132, the classification target processing information acquisition unit 233, the prediction unit 234, and the like.

[0216] The classification target processed information acquisition unit 233 acquires classification target processed information, which is information corresponding to the patent identifier contained in the classification target information accepted by the acceptance unit 22 and which includes the important information acquired by the important information acquisition unit 132.

[0217] The processed information to be classified includes, for example, the important information acquired by the important information acquisition unit 132 and one or more types of information of the claims or abstract corresponding to the patent identifier.

[0218] When the classification target information accepted by the reception unit 22 does not have a patent specification, the classification target processed information acquisition unit 233 acquires classification target processed information consisting of, for example, some or all of the classification target information accepted by the reception unit 22 and important information acquired by the important information acquisition unit 132.

[0219] When the classification target information accepted by the reception unit 22 includes a patent specification, the classification target processed information acquisition unit 233, for example, removes some information including the patent specification from the classification target information accepted by the reception unit 22, and acquires classification target processed information consisting of the information after removing the patent specification, etc. and the important information acquired by the important information acquisition unit 132.

[0220] Moreover, it is preferable that the important information acquisition unit 132 here uses type-specific information associated with the learning device to be used.

[0221] The prediction unit 234 applies the learning device to the classification target processing information acquired by the classification target processing information acquisition unit 233, predicts the classification of the classification target processing information by a machine learning algorithm, and acquires classification result information. For machine learning, for example, deep learning, SVM, random forest, decision tree, etc. can be used. The prediction unit 234 provides the learning device and the classification target processing information as arguments to a machine learning function, and acquires classification result information. The learning device is a learning device in the learning device storage unit 211.

[0222] The output unit 24 outputs the classification result information acquired by the prediction unit 234 in association with the patent identifier included in the classification target information accepted by the acceptance unit 22.

[0223] Here, output means, for example, transmission to user terminal 4, but is a concept that also includes display on a display, projection using a projector, printing on a printer, sound output, storage on a recording medium, and handing over the processing results to other processing devices or other programs.

[0224] The server device 3 is a server that stores a large number of patent specifications corresponding to patent identifiers. When the server device 3 receives a patent acquisition instruction having a patent identifier from the learning module production device 1 or the patent classification device 2, the server device 3 acquires the patent specification corresponding to the patent identifier from the stored patent database and transmits the patent specification to the learning module production device 1 or the patent classification device 2.

[0225] The user terminal 4 receives a learning operation start instruction from the user, and transmits the learning operation start instruction to the learning device production device 1. The user terminal 4 also receives a prediction operation start instruction from the user, and transmits the prediction operation start instruction to the patent classification device 2. It is preferable that the user terminal 4 receives and outputs classification result information, which is the result of prediction, in response to the transmission of the prediction operation start instruction.

[0226] The learning storage unit 11, the manual evaluation information storage unit 111, the storage unit 21, and the learning unit storage unit 211 are preferably non-volatile recording media, but may also be realized as volatile recording media.

[0227] There is no restriction on the process by which information is stored in the learning storage unit 11, etc. For example, information may be stored in the learning storage unit 11, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the learning storage unit 11, etc., or information inputted via an input device may be stored in the learning storage unit 11, etc.

[0228] The learning acceptance unit 12, the patent specification acquisition unit 131, the learning output unit 14, and the output unit 24 can usually be realized by wireless or wired communication means or the like.

[0229] The learning processing unit 13, the important information acquisition unit 132, the learning source information acquisition unit 133, the learning unit 134, the effect solution means sentence acquisition unit 1321, the effect sentence acquisition unit 1322, the solution means sentence acquisition unit 1323, the objective sentence acquisition unit 1324, the feature word acquisition unit 1325, the related word acquisition unit 1326, the important tag corresponding sentence acquisition unit 1327, the numerical range character string acquisition unit 1328, the number acquisition unit 1329, the storage unit 141, the processing unit 23, the classification target processing information acquisition unit 233, and the prediction unit 234 can usually be realized by an MPU, a memory, etc. The processing procedure of the learning processing unit 13, etc. is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

[0230] Next, a description will be given of an example of the operation of the patent classification system A. First, a description will be given of an example of the operation of the learning module production device 1 with reference to the flowchart of FIG.

[0231] (Step S401) The learning reception unit 12 judges whether or not a learning operation start instruction has been received. If the learning operation start instruction has been received, the process proceeds to step S402, and if the learning operation start instruction has not been received, the process returns to step S401.

[0232] (Step S402) The learning processing unit 13 assigns 1 to a counter i.

[0233] (Step S403) The learning processing unit 13 judges whether or not the i-th manual evaluation record exists in the manual evaluation information of the learning target stored in the manual evaluation information storage unit 111. If the i-th manual evaluation record exists, the process proceeds to step S404; if not, the process proceeds to step S408.

[0234] (Step S404) If the patent specification exists in the i-th manual evaluation record, the patent specification acquisition unit 131 acquires the patent specification from the i-th manual evaluation record. If the patent specification does not exist in the i-th manual evaluation record, the patent specification acquisition unit 131 acquires the patent identifier of the i-th manual evaluation record, and receives the patent specification from the server device 3 using the patent identifier as a key.

[0235] (Step S405) The important information acquisition unit 132 acquires one or more types of important information from the patent specification acquired in step S404. An example of such important information acquisition processing will be described with reference to the flowchart of FIG.

[0236] (Step S406) The learning source information acquisition unit 133 uses the i-th manual evaluation record and the important information acquired in step S405 to acquire the i-th patent record that has at least the classification result information and important information in the i-th manual evaluation record, and temporarily stores it in a buffer not shown.

[0237] (Step S407) The learning processing unit 13 increments the counter i by 1. The process returns to step S403.

[0238] (Step S408) The learning source information acquisition unit 133 configures learning source information having all the patent records temporarily accumulated in step S406.

[0239] (Step S409) The learning unit 134 performs a learning process using the learning source information acquired in step S408, and acquires a learning device.

[0240] (Step S410) The accumulation unit 141 associates the type specifying information included in the learning operation start instruction with the learning device, and accumulates them in the learning storage unit 11. The process returns to step S401.

[0241] In the flowchart of FIG. 4, the process ends when the power is turned off or an interrupt is issued to end the process.

[0242] Next, an example of the important information acquisition process in step S405 will be described with reference to the flowchart in FIG.

[0243] (Step S501) The important information acquisition unit 132 acquires type-specific information included in the learning operation start instruction. In the prediction process, the important information acquisition unit 132 acquires type-specific information associated with a learning device used in the prediction process.

[0244] (Step S502) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and judges whether to acquire an effect solution means sentence as important information. If an effect solution means sentence is acquired, the process proceeds to step S503. If an effect solution means sentence is not acquired, the process proceeds to step S504.

[0245] (Step S503) The important information acquisition unit 132 performs an effect solution means sentence acquisition process. An example of the effect solution means sentence acquisition process will be described with reference to the flowchart of FIG.

[0246] (Step S504) The important information acquisition unit 132 checks the type specification information acquired in step S501 and judges whether or not to acquire an effect sentence as important information. If an effect sentence is to be acquired, the process proceeds to step S505. If an effect sentence is not to be acquired, the process proceeds to step S506.

[0247] (Step S505) The important information acquisition unit 132 performs an effect acquisition process. An example of the effect statement acquisition process will be described with reference to the flowchart of FIG.

[0248] (Step S506) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and judges whether or not to acquire a solution sentence as important information. If a solution sentence is to be acquired, the process proceeds to step S507. If a solution sentence is not to be acquired, the process proceeds to step S508.

[0249] (Step S507) The important information acquisition unit 132 performs a solution statement acquisition process. An example of the solution statement acquisition process will be described with reference to the flowchart of FIG.

[0250] (Step S508) The important information acquisition unit 132 checks the type specification information acquired in step S501 and judges whether or not to acquire the object sentence as important information. If the object sentence is to be acquired, the process proceeds to step S509. If the object sentence is not to be acquired, the process proceeds to step S510.

[0251] (Step S509) The important information acquisition unit 132 performs a purpose sentence acquisition process. An example of the purpose sentence acquisition process will be described with reference to the flowchart of FIG.

[0252] (Step S510) The important information acquisition unit 132 checks the type specification information acquired in step S501 and judges whether to acquire effect terms as important information. If effect terms are to be acquired, the process proceeds to step S511. If effect terms are not to be acquired, the process proceeds to step S512.

[0253] (Step S511) The important information acquisition unit 132 performs an effect term acquisition process. An example of the effect term acquisition process will be described with reference to the flowchart of FIG.

[0254] (Step S512) The important information acquisition unit 132 checks the type specification information acquired in step S501 and judges whether or not to acquire a solution means term as important information. If a solution means term is to be acquired, the process proceeds to step S513. If a solution means term is not to be acquired, the process proceeds to step S514.

[0255] (Step S513) The important information acquisition unit 132 performs a solution means term acquisition process. An example of the solution means term acquisition process will be described with reference to the flowchart of FIG.

[0256] (Step S514) The important information acquisition unit 132 checks the type specification information acquired in step S501 and judges whether or not to acquire the target term as important information. If the target term is acquired, the process proceeds to step S515, and if the target term is not acquired, the process proceeds to step S516.

[0257] (Step S515) The important information acquisition unit 132 performs a target term acquisition process. An example of the target term acquisition process will be described with reference to the flowchart of FIG.

[0258] (Step S516) The important information acquisition unit 132 checks the type specifying information acquired in step S501 and judges whether or not to acquire a characteristic word as important information. If a characteristic word is to be acquired, the process proceeds to step S517, and if a characteristic word is not to be acquired, the process proceeds to step S518.

[0259] (Step S517) The important information acquisition unit 132 performs a feature word acquisition process. Note that, as an example of the feature word acquisition process, for example, the above-mentioned TF / IDF algorithm is used.

[0260] (Step S518) The important information acquisition unit 132 checks the type specifying information acquired in step S501 and judges whether or not to acquire related words as important information. If related words are to be acquired, the process proceeds to step S519. If related words are not to be acquired, the process proceeds to step S520.

[0261] (Step S519) The important information acquisition unit 132 performs a related word acquisition process. As an example of the related word acquisition process, for example, as described above, the related word dictionary is searched using each of the one or more characteristic words acquired in step S517 as a key, and one or more related words that are paired with each characteristic word are acquired. It is assumed that the characteristic word acquisition process is performed as a preprocessing for acquiring the related words.

[0262] (Step S520) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether to acquire an important tag-corresponding sentence as important information. If an important tag-corresponding sentence is to be acquired, the process proceeds to step S521. If an important tag-corresponding sentence is not to be acquired, the process proceeds to step S523.

[0263] (Step S521) The important information acquisition unit 132 performs an important tag corresponding sentence acquisition process. Note that an example of the important tag corresponding sentence acquisition process has been described above, so a description thereof will be omitted here.

[0264] (Step S522) The important information acquisition unit 132 checks the type specification information acquired in step S501 and determines whether to acquire important tag terms as important information. If important tag terms are acquired, the process proceeds to step S523. If important tag terms are not acquired, the process proceeds to step S524.

[0265] (Step S523) The important information acquisition unit 132 performs an important tag term acquisition process. Note that an example of the important tag term acquisition process has been described above, so a description thereof will be omitted here.

[0266] (Step S524) The important information acquisition unit 132 checks the type identification information acquired in step S501 and determines whether to acquire a numeric range character string as important information. If a numeric range character string is acquired, the process proceeds to step S525. If a numeric range character string is not acquired, the process returns to the upper process.

[0267] (Step S525) The important information acquisition unit 132 performs a numerical value range character string acquisition process. An example of the numerical value range character string acquisition process will be described with reference to the flowchart of FIG.

[0268] (Step S526) The important information acquisition unit 132 performs a number acquisition process using one or more of the numerical value range character strings acquired in step S523. An example of the number acquisition process will be described with reference to the flowchart of FIG.

[0269] Next, examples of sentence acquisition processing such as an example of the effect / solution means sentence acquisition processing in step S503, an example of the effect acquisition processing in step S505, an example of the solution means sentence acquisition processing in step S507, and an example of the object sentence acquisition processing in step S509 will be described with reference to the flowchart in FIG.

[0270] (Step S601) The important information acquisition unit 132 acquires one or more sentences within a predetermined range of sentence extraction targets from the patent specification from which sentences are to be extracted. The predetermined range of sentence extraction targets may differ depending on the type of important information. The predetermined range of sentence extraction targets is, for example, the entire patent specification, "Effects of the Invention," and "Form for Carrying Out the Invention."

[0271] (Step S602) The important information acquiring unit 132 assigns 1 to a counter i.

[0272] (Step S603) The important information acquisition unit 132 judges whether or not the i-th sentence is present among the one or more sentences acquired in step S601. If the i-th sentence is present, the process proceeds to step S604, and if the i-th sentence is not present, the process returns to the upper process.

[0273] (Step S604) The important information acquisition unit 132 judges whether the i-th sentence acquired in step S603 corresponds to a sentence of important information of the corresponding type. An example of such a judgment process has been described above. Such a judgment process may be, for example, a method based on pattern matching or a method based on machine learning.

[0274] (Step S605) If the determination result in step S604 is that the sentence is important information of the corresponding type, the important information acquiring unit 132 proceeds to step S606, and if the sentence is not important information of the corresponding type, the important information acquiring unit 132 proceeds to step S607.

[0275] (Step S606) The important information acquiring unit 132 acquires the i-th sentence as important information, and temporarily stores it in a buffer (not shown).

[0276] (Step S607) The important information acquiring unit 132 increments the counter i by 1. The process returns to step S603.

[0277] Next, examples of term acquisition processing such as an example of effect term acquisition processing in step S511, an example of solution term acquisition processing in step S513, and an example of target term acquisition processing in step S515 will be described with reference to the flowchart in FIG.

[0278] (Step S701) The important information acquisition unit 132 performs a process of acquiring a sentence corresponding to the term. The sentence acquisition process is a process explained using the flowchart in FIG. 6. If the term is an effect term, the sentence to be acquired is one or more types of sentences among effect solution means sentences and effect sentences. If the term is a solution means term, the sentence to be acquired is one or more types of sentences among effect solution means sentences and solution means sentences. If the term is an object term, the sentence to be acquired is an object sentence.

[0279] (Step S702) The important information acquiring unit 132 assigns 1 to a counter i.

[0280] (Step S703) The important information acquisition unit 132 judges whether or not the i-th sentence is present among the one or more sentences acquired in step S701. If the i-th sentence is present, the process proceeds to step S704, and if the i-th sentence is not present, the process returns to the upper process.

[0281] (Step S704) The important information acquisition unit 132 performs morphological analysis on the i-th sentence to acquire one or more terms. The terms acquired here may be only independent words. The terms acquired here are terms acquired using the pattern of each sentence, and are terms corresponding to variables in the pattern. If the term corresponds to a variable in the pattern, then in the case where the term is an effect term, the variable in the pattern is <effect>.

[0282] (Step S705) The important information acquiring unit 132 acquires one or more terms acquired in step S704 as important information, and temporarily stores them in a buffer (not shown).

[0283] (Step S706) The important information acquisition unit 132 increments the counter i by 1. The process returns to step S703.

[0284] Next, an example of the numerical value range character string acquisition process in step S523 will be described with reference to the flowchart in Fig. 8. In the flowchart in Fig. 8, the description of the same steps as in the flowchart in Fig. 6 will be omitted.

[0285] (Step S801) The important information acquiring unit 132 assigns 1 to a counter j.

[0286] (Step S802) The important information acquisition unit 132 judges whether the j-th numeric range character string acquisition pattern is stored. If the j-th numeric range character string acquisition pattern is stored, the process proceeds to step S803. If not, the process proceeds to step S607.

[0287] (Step S803) The important information acquisition unit 132 checks whether or not a character string that matches the jth numeric range character string acquisition pattern exists in the i-th sentence. If a matching character string exists, the process proceeds to step S804. If a matching character string does not exist, the process proceeds to step S805.

[0288] (Step S804) The important information acquiring unit 132 acquires one or more character strings that match the j-th numeric value range character string acquisition pattern in the i-th sentence, and accumulates them in a buffer (not shown).

[0289] (Step S805) The important information acquiring unit 132 increments the counter j by 1. The process returns to step S802.

[0290] Next, an example of the number acquisition process in step S526 will be described with reference to the flowchart in FIG.

[0291] (Step S901) The important information acquiring unit 132 assigns 1 to a counter i.

[0292] (Step S902) The important information acquisition unit 132 judges whether or not the i-th numeric range character string acquired in step S804 exists. If it exists, the process proceeds to step S903, and if it does not exist, the process returns to the upper process.

[0293] (Step S903) The important information acquiring unit 132 acquires the lower limit numerical value of the i-th numerical value range character string.

[0294] (Step S904) The important information acquiring unit 132 acquires the upper limit numerical value of the i-th numerical value range character string.

[0295] (Step S905) The important information acquiring part 132 acquires the unit from the i-th numeric value range character string.

[0296] (Step S906) The important information obtaining unit 132 obtains the number of significant digits from the numerical value of the i-th numerical value range character string.

[0297] (Step S907) The important information acquiring unit 132 acquires a numeric value that is greater than the lower limit and less than the upper limit, and has as many digits as the number of significant digits, and is equal to or greater than 1.

[0298] (Step S908) The important information acquiring unit 132 adds the unit acquired in step S905 after each of the numerical values ​​equal to or greater than 1 acquired in step S907.

[0299] (Step S909) The important information acquiring unit 132 increments the counter i by 1. The process returns to step S902.

[0300] Next, an example of the operation of the patent classification device 2 will be described with reference to the flowchart of FIG.

[0301] (Step S1001) The reception unit 22 determines whether or not a prediction start instruction including one or more pieces of classification target information has been received. If a prediction start instruction has been received, the process proceeds to step S1002, and if a prediction start instruction has not been received, the process returns to step S1001. Note that the prediction start instruction including one or more pieces of classification target information may be a prediction start instruction including information for acquiring one or more pieces of classification target information (e.g., a file name, a URL, etc.).

[0302] (Step S1002) The processing unit 23 assigns 1 to a counter i.

[0303] (Step S1003) The processing unit 23 judges whether or not there is the i-th classification target information corresponding to the prediction start instruction received in step S1001. If there is the i-th classification target information, the processing unit 23 proceeds to step S1004, and if there is no i-th classification target information, the processing unit 23 returns to step S1001.

[0304] (Step S1004) The patent specification acquisition unit 131 judges whether or not a patent specification exists in the i-th classification target information. If a patent specification exists, the process proceeds to step S1005. If a patent specification does not exist, the process proceeds to step S1006.

[0305] (Step S1005) The patent specification acquisition unit 131 acquires a patent specification from the i-th classification target information. Then, the process proceeds to step S1008.

[0306] (Step S1006) The patent specification acquisition section 131 acquires a patent identifier from the i-th classification target information.

[0307] (Step S1007) The patent specification acquisition unit 131 receives the patent specification from the server device 3 by using the patent identifier acquired in step S1006 as a key.

[0308] (Step S1008) The important information acquisition unit 132 acquires the i-th important information using the acquired patent specification. An example of the important information acquisition process has been described with reference to the flowchart in Fig. 5. Note that it is preferable that the important information acquisition unit 132 acquires the important information using type-specific information associated with the learning device used in the prediction process.

[0309] (Step S1009) The classification target processed information acquisition unit 233 acquires the i-th classification target processed information that has the i-th important information acquired in step S1008 and does not have the full text of the patent specification.

[0310] (Step S1010) The prediction unit 234 acquires a learning device to be used in the prediction process from the learning device storage unit 211.

[0311] (Step S1011) The prediction unit 234 performs prediction processing by a machine learning algorithm using the i-th classification target processed information and the learning device acquired in step S1010, and acquires classification result information.

[0312] (Step S1012) The output unit 24 accumulates the classification result information acquired in step S1011 in association with the i-th classification target information.

[0313] (Step S1013) The processing unit 23 increments the counter i by 1. The process returns to step S1003.

[0314] In the flowchart of FIG. 10, the process ends when the power is turned off or an interrupt is issued to end the process.

[0315] As described above, according to this embodiment, patents can be classified with high accuracy by using important information about patent specifications.

[0316] Furthermore, according to this embodiment, patents can be classified with high accuracy by using the claims and / or abstracts.

[0317] Furthermore, according to this embodiment, by using effect / solution sentences or effect terms or solution terms, patents can be classified with high accuracy.

[0318] Furthermore, according to this embodiment, patents can be classified with high accuracy by using effect sentences or effect terms.

[0319] Furthermore, according to this embodiment, patents can be classified with high accuracy by using solution sentences or solution terms.

[0320] Furthermore, according to this embodiment, by using object sentences or object terms, patents can be classified with high accuracy.

[0321] Furthermore, according to this embodiment, patents can be classified with high accuracy by using characteristic words.

[0322] Furthermore, according to this embodiment, patents can be classified with high accuracy by using related words of characteristic words.

[0323] Furthermore, according to this embodiment, by using important tag corresponding sentences or important tag corresponding terms, patents can be classified with high accuracy.

[0324] Furthermore, according to the present embodiment, the use of numeric range character strings allows for accurate classification of patents.

[0325] Furthermore, according to this embodiment, by using numbers equal to or greater than 3 that are included in the numerical range indicated by the numerical range character string, patents can be classified with high accuracy.

[0326] Furthermore, according to this embodiment, patents can be classified with high accuracy by using important information acquired from patent specifications acquired from an external server device.

[0327] In addition, according to the present embodiment, by using the type-specific information to obtain only the appropriate important information from multiple types of important information, patents can be classified with higher accuracy. For example, an effect / solution statement is considered to be usable in the classification of patents in all technical fields, while a numeric range character string and numbers included in the numeric range indicated by the numeric range character string are considered to be usable in the classification of patents in some technical fields, such as the chemical field.

[0328] Furthermore, according to this embodiment, a learning machine for classifying patents with high accuracy can be produced.

[0329] The processing in this embodiment may be realized by software. This software may be distributed by software download or the like. This software may also be recorded on a recording medium such as a CD-ROM and distributed. This also applies to the other embodiments in this specification. The software that realizes the patent classification device 2 in this embodiment is a program such as the following. In other words, this program is a program for causing a computer that can access a learning source information having two or more patent records, which is a patent record that does not have the full text of a patent specification and has important information obtained using a patent specification corresponding to a patent identifier that identifies a patent, and classification result information that identifies a manual classification result of a patent, to function as a reception unit that receives classification target information including a patent identifier that identifies a patent to be classified, an important information acquisition unit that acquires important information that satisfies a predetermined condition using a patent specification corresponding to the patent identifier included in the classification target information accepted by the reception unit, a classification target processed information acquisition unit that acquires classification target processed information, which is information corresponding to the patent identifier included in the classification target information accepted by the reception unit and includes the important information acquired by the important information acquisition unit, a prediction unit that applies the learning unit to the classification target processed information acquired by the classification target processed information acquisition unit, predicts the classification of the classification target processed information using a machine learning algorithm, and acquires classification result information, and an output unit that outputs the classification result information acquired by the prediction unit in association with the patent identifier included in the classification target information accepted by the reception unit.

[0330] In addition, the software for realizing the learning device production device 1 is a program for causing a computer that can access a manual evaluation information storage unit in which manual evaluation information having two or more manual evaluation records each having a patent identifier that identifies a patent and classification result information that identifies the manual classification result of the patent corresponding to the patent identifier to function as an important information acquisition unit that acquires important information that satisfies predetermined conditions using a patent specification corresponding to the patent identifier of each of the two or more manual evaluation records in the manual evaluation information stored in the manual evaluation information storage unit, a learning source information acquisition unit that acquires, for each of the two or more manual evaluation records, a patent record having important information acquired by the important information acquisition unit and classification result information that pairs with the patent identifier corresponding to the patent specification that is the source of the important information, and acquires learning source information having the two or more patent records, a learning unit that learns the learning source information acquired by the learning source information acquisition unit using a machine learning algorithm and acquires a learning device, and a storage unit that accumulates the learning devices acquired by the learning unit.

[0331] (Embodiment 2) In this embodiment, a patent classification system B that uses a different algorithm to classify patents with high accuracy will be described.

[0332] In addition, in this embodiment, a patent classification system B that uses different data will be described.

[0333] In addition, in this embodiment, a patent classification system B in which a learning module generation program is different will be described.

[0334] In addition, in this embodiment, a patent classification system B in which statistical processing is different will be described.

[0335] In addition, in this embodiment, a patent classification system B is described in which the two or more statistical processing methods include two or more methods of OR, AND, or a method of taking a majority vote of the prediction results of each learning module.

[0336] In addition, in this embodiment, a patent classification system B will be described in which the accuracy is the recall rate, the precision rate, the F-measure, or the accuracy rate.

[0337] In this embodiment, an algorithm determination device 5 capable of determining an appropriate process for performing the prediction process will be described.

[0338] In this embodiment, a learning module production device that produces an optimal learning module using the algorithm determined by algorithm determination device 5 will be described.

[0339] In addition, in this embodiment, a patent classification device that performs highly accurate patent classification using an optimal learning device produced by a learning device production device will be described.

[0340] 11 is a block diagram of a patent classification system B in this embodiment. The patent classification system B includes an algorithm determination device 5, a learning module production device 6, a patent classification device 8, a server device 3, and one or more user terminals 4.

[0341] The algorithm determination device 5 includes a storage unit 51, a reception unit 52, a processing unit 53, and an output unit 54. The storage unit 51 includes a program storage unit 511 and an accuracy specification information storage unit 512. The reception unit 52 includes a learning source information reception unit 521. The processing unit 53 includes a learning unit 531, a prediction unit 532, an evaluation unit 533, an algorithm information acquisition unit 534, a learning source information acquisition unit 535, and a statistical processing unit 536.

[0342] The learning element production device 6 includes a learning storage unit 61, a learning reception unit 62, a learning processing unit 13, and a learning output unit 64. The learning reception unit 62 includes a learning source information reception unit 621. The learning processing unit 13 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a learning source information acquisition unit 133, and a learning unit 134. The important information acquisition unit 132 includes an effect solution means sentence acquisition means 1321, an effect sentence acquisition means 1322, a solution means sentence acquisition means 1323, an objective sentence acquisition means 1324, a feature word acquisition means 1325, a related word acquisition means 1326, an important tag corresponding sentence acquisition means 1327, a numerical range character string acquisition means 1328, and a numerical value acquisition means 1329. The learning output unit 64 includes an accumulation unit 641.

[0343] 12 is a block diagram of the patent classification device 8. The patent classification device 8 includes a storage unit 21, a reception unit 81, a processing unit 82, and an output unit 24. The processing unit 82 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a classification target processed information acquisition unit 233, a statistical processing unit 536, and a prediction unit 821.

[0344] The server device 3 and the user terminal 4 are the same elements as those in the first embodiment.

[0345] Various types of information are stored in the storage unit 51 constituting the algorithm determination device 5. The various types of information include, for example, a program described later and accuracy specification information described later. In addition, the storage unit 51 may also store, for example, learning source information described later.

[0346] The program storage unit 511 stores one or more learning device generation programs and a prediction program. Here, the prediction program corresponds to each of the one or more learning device generation programs. In other words, the learning device generation program and the prediction program usually correspond one-to-one, but may correspond many-to-one.

[0347] The learning module generation program is a program that uses learning source information to generate a learning module for classifying patent information. The learning source information is source information for generating a learning module. The learning source information has, for example, two or more pieces of patent classification information. The patent classification information is information regarding the classification of patents. The patent classification information has, for example, patent information and classification information.

[0348] Patent information is information about patents. For example, it is preferable that the patent information is a part of a patent specification, but it may be the entire text of the patent specification. The patent specification may be, for example, a patent gazette, a published patent gazette, or the like, and the type does not matter. Patent information usually includes a patent identifier. As mentioned above, the patent identifier is, for example, an ID, an application number, or the like. In addition, patent information includes, for example, one or more pieces of information such as an abstract, claims, specification, and patent classification code. It is preferable that the patent information includes the above-mentioned important information. However, the type of information included in the patent information does not matter.

[0349] Classification information is information that indicates the classification of patent information. Classification information is usually information that indicates the manual classification of patents. Classification information can be anything, such as "○,×", "A,B,C", or a technical field ID.

[0350] The patent classification information may be, for example, the patent record in the first embodiment. As described above, the patent record has important information and manual classification result information. The classification information may be the classification result information in the first embodiment. However, the classification information is not limited to manual information, and may be information with a high score among predicted classification information (for example, classification information in which the primitive score acquired by the prediction unit 532 described later is higher than or equal to a predetermined threshold value).

[0351] However, the structure of the learning source information does not matter. Also, the learning source information may be accepted by a learning source information accepting unit 521 described later, or may be acquired by a learning source information acquiring unit 535 described later.

[0352] The learning machine generation program is a program that performs learning processing using a machine learning algorithm. For example, deep learning, SVM, random forest, decision tree, etc. can be used for machine learning. In machine learning, to configure a learning machine, for example, a learning machine can be obtained by providing learning source information to a machine learning function (for example, fastText, TinySVM, various RandomForest functions, TensorFlow library, etc.).

[0353] The prediction program is a program that predicts the classification of the patent classification information using a learning device and obtains the predicted classification information. The learning device is obtained by the learning unit 531, which will be described later.

[0354] The program storage unit 511 may store, for example, two or more learning device generation programs and two or more prediction programs.

[0355] The prediction program is a program that performs prediction processing using a machine learning algorithm. The prediction program has, for example, the machine learning function described above. When the prediction program is executed by providing a learning device and patent classification information of the classification target, predicted classification information is obtained as a prediction result.

[0356] Here, the two or more prediction programs correspond to the two or more learning device generation programs, respectively. That is, the learning device generation programs and the prediction programs usually correspond one-to-one.

[0357] The accuracy specification information storage unit 512 stores accuracy specification information. The accuracy specification information is information that specifies which of two or more predetermined types of accuracy is to be acquired. The two or more predetermined types of accuracy are preferably two or more of four types of accuracy, for example, recall rate, precision rate, F value, or accuracy rate.

[0358] The recall rate, the precision rate, and the F-measure are well-known techniques. The accuracy rate is, for example, the rate of all predicted classification information acquired by the prediction unit 532 that matches the classification information included in the learning source information.

[0359] The accuracy information is information about accuracy. The accuracy information is, for example, information about recall, information about precision, information about F value, information about accuracy rate, etc., but may be any information about accuracy. The information about recall is, for example, recall, but may be information acquired based on recall. The information about precision is, for example, precision, but may be information acquired based on precision. The information about F value is, for example, F value, but may be information acquired based on F value. The information about accuracy rate is, for example, precision, but may be information acquired based on accuracy rate.

[0360] The accuracy specification information is, for example, received by the receiving unit 52 and accumulated in the accuracy specification information storage unit 512, but may be stored in the accuracy specification information storage unit 512 in advance.

[0361] The accepting unit 52 accepts various types of information, such as learning source information.

[0362] Furthermore, the receiving unit 52 receives, for example, the accuracy specification information described above.

[0363] As mentioned above, acceptance usually means receipt from the user terminal 4, but it may also be considered as a concept including acceptance of information input from an input device such as a keyboard, and acceptance of information read from a recording medium such as a disk or semiconductor memory.

[0364] The learning source information receiving unit 521 receives the learning source information described above.

[0365] The learning source information receiving unit 521 receives the learning source information from, for example, the user terminal 4. However, for example, the learning source information may be stored in a recording medium such as a disk or a semiconductor memory, and the learning source information receiving unit 521 may read the learning source information from such a recording medium. In addition, the patent information included in the learning source information received by the learning source information receiving unit 521 may be a patent identifier. In that case, the learning source information receiving unit 521 may receive the patent information corresponding to the patent identifier included in the learning source information received from the user terminal 4 from, for example, the server device 3. Note that the above items also apply to the learning source information learning receiving unit 621 constituting the learning device production device 6.

[0366] The processing unit 53 performs various processes, such as processes performed by a learning unit 531, a prediction unit 532, an evaluation unit 533, an algorithm information acquisition unit 534, a learning source information acquisition unit 535, and a statistical processing unit 536. The processing unit 53 also performs various judgments and controls of various counters, which will be described in the flowcharts.

[0367] The processing unit 53 may perform the processing performed by the important information acquisition unit 132. That is, the processing unit 53 may include the important information acquisition unit 132.

[0368] The learning unit 531 constituting the processing unit 53 executes one or more learning device generation programs using, for example, at least a portion of the patent classification information of the learning source information accepted by the learning source information accepting unit 521, and obtains one or more learning devices.

[0369] Here, the "at least a part of the patent classification information of the learning source information accepted by the learning source information accepting unit 521" used by the learning device 531 is, for example, N-percent of the patent classification information of the learning source information. N-percent is, for example, 90%, but it may also be 80% or 95%, and the numerical value is not important.

[0370] The learning unit 531 may, for example, execute one or more learning device generation programs stored in the program storage unit 511 for each of two or more pieces of learning source information acquired by the learning source information acquisition unit 535, and acquire two or more learning devices.

[0371] The learning unit 531 may, for example, execute two or more learning device generation programs using at least a portion of the patent classification information of the learning source information accepted by the learning source information accepting unit 521, and obtain two or more learning devices.

[0372] The prediction unit 532 may, for example, apply a prediction program to each of at least a portion of the patent classification information among the learning source information accepted by the learning source information accepting unit 521 using one or more learning devices generated by the learning unit 531, and obtain two or more predicted classification information for each of the one or more patent classification information.

[0373] Here, "at least some of the patent classification information among the learning source information accepted by the learning source information accepting unit 521" that is the prediction target of the prediction unit 532 is usually one or more pieces of patent classification information that are not used by the learning unit 134. For example, when the learning device 531 uses N percent (e.g., 90%) of the learning source information, it is preferable that the prediction target of the prediction unit 532 is "10-N" percent (e.g., 10%) of the learning source information. However, the learning source information may include, for example, patent classification information that is not used for learning and is not the target of prediction, or may include patent classification information that is used for learning and is also the target of prediction.

[0374] The prediction unit 532 may, for example, apply a prediction program to each of at least a portion of the patent classification information among the learning source information accepted by the learning source information accepting unit 521, using two or more learning devices generated by the learning unit 531, and obtain two or more predicted classification information for each of the one or more patent classification information.

[0375] For example, the prediction unit 532 may apply a prediction program using two or more learning devices obtained for each combination of algorithm type (e.g., deep learning 1, random forest, deep learning 2, etc.) and information type to be used (e.g., abstract and classification code, abstract, claims and classification code, etc.), obtain two or more primitive scores corresponding to two or more pieces of classification information such as "A, B, C" or "○, ×", and obtain the classification information with the highest primitive score as the predicted classification information. Note that the primitive score is the score output by the prediction program.

[0376] As a result, for example, a primitive score (=S1) for classification information "A", a primitive score (=S2) for classification information "B", and a primitive score (=S3) for classification information "C" are obtained. Then, for example, if S1>S2>S3, the classification information "A" corresponding to the highest score S1 may be obtained as the predicted classification information.

[0377] The evaluation unit 533 uses each of the two or more pieces of predicted classification information acquired by the prediction unit 532 and the classification information included in the learning source information to acquire accuracy information regarding the accuracy of each of the two or more pieces of predicted classification information, and determines the predicted classification information with the highest accuracy specified by the accuracy information. The accuracy is, for example, a recall rate, a precision rate, an F-measure, or a correct answer rate.

[0378] The evaluation unit 533 may, for example, use two or more pieces of prediction classification information including one or more pieces of prediction classification information acquired by the statistical processing unit 536 instead of or in addition to the two or more pieces of prediction classification information acquired by the prediction unit 532, and classification information contained in the learning source information, to acquire accuracy information regarding the accuracy of each of the two or more pieces of prediction classification information, and determine the prediction classification information with the highest accuracy identified by the accuracy information.

[0379] The evaluation unit 533 may, for example, use each of the two or more pieces of predicted classification information acquired by the prediction unit 532 and the classification information contained in the learning source information to acquire accuracy information for each of the two or more pieces of predicted classification information that is identified by the accuracy identification information, and determine the predicted classification information with the highest accuracy identified by the accuracy information.

[0380] The evaluation unit 533 acquires, for example, accuracy information corresponding to the accuracy specifying information stored in the accuracy specifying information storage unit 512 .

[0381] The evaluation unit 533 obtains accuracy information by, for example, using K-fold cross-validation on the learning source information.

[0382] The algorithm information acquisition unit 534 acquires algorithm information. The algorithm information is information on an algorithm corresponding to the predicted classification information determined by the evaluation unit 533. As described above, the predicted classification information determined by the evaluation unit 533 is the predicted classification information with the highest accuracy.

[0383] The algorithm information includes one or more types of information including information that specifies the type of information to be used (information specification information described later), information that specifies the learning device generation program to be used (program specification information described later), and information that specifies the statistical processing (statistical processing specification information described later).

[0384] The algorithm information is, for example, information capable of identifying the learning source information corresponding to the most accurate predicted classification information among two or more pieces of learning source information acquired by the learning source information acquisition unit 535 described later. The information capable of identifying the learning source information may be, for example, information identifying information that identifies information contained in the learning source information. The information identifying information is usually information that identifies the type of information (for example, "abstract", "claims", "abstract + claims", etc.) included in the patent information constituting the learning source information.

[0385] That is, the algorithm information acquiring unit 534 may acquire algorithm information including information specifying information that specifies information contained in the learning source information corresponding to the predicted classification information determined by the evaluating unit 533, for example.

[0386] Alternatively, the algorithm information may be, for example, information that identifies a learning device generation program that corresponds to the most accurate predicted classification information among two or more learning device generation programs stored in the program storage unit 511.

[0387] That is, the algorithm information acquisition unit 534 may acquire algorithm information including program identification information that identifies a learning device generation program corresponding to the predicted classification information determined by the evaluation unit 533, for example.

[0388] Alternatively, the algorithm information may be information that specifies, from two or more statistical processes performed by the statistical processing unit 536 described later, a statistical process that corresponds to the predicted classification information with the highest accuracy.

[0389] That is, the algorithm information acquiring unit 534 may acquire algorithm information including statistical processing specification information that specifies a statistical processing method corresponding to the predicted classification information determined by the evaluating unit 533, for example.

[0390] Alternatively, the algorithm information may be information that specifies the accuracy desired by the user, from among the recall rate, precision rate, F-measure, or accuracy rate, for example.

[0391] That is, the algorithm information acquiring unit 534 may acquire algorithm information including the accuracy specifying information accepted by the accepting unit 52, for example.

[0392] The learning source information acquisition unit 535 acquires two or more pieces of learning source information, each of which has two or more patent classification information having different information, from the patent information contained in each of the two or more patent classification information contained in the learning source information accepted by the learning source information accepting unit 521.

[0393] Here, each of the two or more learning source information having two or more patent classification information having different information is, for example, two or more learning source information in which the information contained in the patent information constituting each of the two or more patent classification information possessed by a single learning source information is different.

[0394] The learning source information acquisition unit 535 may acquire one or more types of important information from the specification of the patent information contained in each of two or more patent classification information contained in the learning source information accepted by the learning source information acceptance unit 521. In such a case, the learning source information acquisition unit 535 has the important information acquisition unit 132.

[0395] Specifically, for example, the patent information contained in each of two or more patent classification information contained in the learning source information accepted by the learning source information accepting unit 521 includes an abstract, claims, and a description, and the learning source information acquisition unit 535 may use the patent information contained in each of the two or more patent classification information contained in the accepted learning source information to acquire, for example, a first learning source information having two or more patent classification information each of which includes a summary in its patent information, a second learning source information having two or more patent classification information each of which includes a summary and claims in its patent information, a third learning source information having two or more patent classification information each of which includes a summary in its patent information, a third learning source information having two or more patent classification information each of which includes a summary and an effect solution statement, and a fourth learning source information having two or more patent classification information each of which includes an abstract, claims, and an effect solution statement in its patent information.

[0396] Furthermore, each of the two or more learning source information mentioned above is, for example, "learning source information having two or more patent classification information having patent information and classification information consisting of an abstract and claims," ​​"learning source information having two or more patent classification information having patent information and classification information consisting of an abstract," "learning source information having two or more patent classification information having patent information and classification information consisting of claims," ​​"learning source information having two or more patent classification information having patent information and classification information consisting of an abstract and important information," and "learning source information having two or more patent classification information having patent information and classification information consisting of claims and important information."

[0397] Alternatively, the two or more pieces of learning source information each having two or more pieces of patent classification information with different information may be two or more pieces of patent classification information in which, for example, patent classification information of at least a portion of the learning source information (hereinafter, the learning portion) is used for the learning process of the learning unit 531, and each patent classification information of at least a portion of the learning source information (hereinafter, the prediction portion) is the subject of prediction by the prediction unit 532, the combinations of one or more pieces of patent classification information belonging to the learning portion and one or more pieces of patent classification information belonging to the prediction portion are different.

[0398] Specifically, for example, when the learning source information receiving unit 521 receives learning source information having patent classification information for 100 items, the learning source information acquisition unit 535 may divide the 100 items equally into 10 groups, numbered first to ten (generally, into K groups), and assign 9 of the 10 groups ((K-1) groups) to the learning part and the remaining group to the prediction part, performing this process 10 times while changing the group combinations (K-fold cross-validation).

[0399] This allows 10 pieces of learning source information to be obtained, for example: first learning source information in which 90 items belonging to the first to ninth nine groups are the learning portion and 10 items belonging to the tenth group are the prediction portion; second learning source information in which 90 items belonging to the first to eighth and tenth nine groups are the learning portion and 10 items belonging to the ninth group are the prediction portion; third learning source information in which 90 items belonging to the first to seventh, ninth and tenth nine groups are the learning portion and 10 items belonging to the eighth group are the prediction portion; and tenth learning source information in which 90 items belonging to the second to tenth nine groups are the learning portion and 10 items belonging to the first group are the prediction portion.

[0400] The statistical processing unit 536 performs one or more types of statistical processing using two or more pieces of predicted classification information acquired by the prediction unit 532, and acquires one or more pieces of predicted classification information as the results of the statistical processing. The statistical processing is processing related to statistics. The statistical processing is, for example, a logical sum (OR) or logical product (AND) of the predicted classification information, or a majority vote, but any processing related to statistics is acceptable.

[0401] The statistical processing unit 536 may perform, for example, two or more types of statistical processing and obtain two or more pieces of predicted classification information as the statistical processing results. The two or more types of statistical processing may be, for example, two or more types of statistical processing among the logical sum, logical product, or majority decision of the predicted classification information.

[0402] Here, the score when taking the logical sum, logical product, or majority vote is obtained, for example, using two or more operation results of a primitive algorithm (i.e., primitive scores obtained for each of two or more pieces of classification information before the prediction unit 532 obtains the predicted classification information, which is the final result).

[0403] It is preferable that the statistical processing unit 536 compares each of the operation results of the primitive algorithm with the final result, and the greater the number or percentage of matches, the higher the score obtained.

[0404] When the scores contained in the multiple operation results of the primitive algorithm are all the same, it is preferable that the statistical processing unit 536 compares the multiple operation results of the primitive algorithm with the final result, and obtains a higher score the greater the number or percentage of matches.

[0405] For example, the statistical processing unit 536 calculates the score in the case of taking a logical sum, a logical product, or a majority vote, for example, as follows. (1) If all of the multiple operation results of the primitive algorithm match the final result (i.e., the predicted classification information obtained by the prediction unit 532), the statistical processing unit 536 assigns the highest score among the two or more operation results of the primitive algorithm. (2) If more than 50% of the results of multiple primitive algorithm operations differ from the final result, the lowest score (0.5 in the case of binary classification) is given. (3) In cases other than (1) and (2) above (i.e., when the percentage is less than 100% and more than 50% is the same as the final result), the statistical processing unit 536 assigns the lowest score among the multiple scores of the primitive algorithms.

[0406] The output unit 54 outputs various information. The various information is, for example, algorithm information. As described above, output is, for example, transmission to the user terminal 4, but it may also be considered as a concept including display on a display, printing on a printer, sound output from a speaker, storage on a recording medium, delivery of processing results to other processing devices, other programs, etc.

[0407] The output unit 54 accumulates the algorithm information acquired by the algorithm information acquisition unit 534, for example, in association with the learning source information.

[0408] The output unit 54 transmits or delivers the algorithm information acquired by the algorithm information acquisition unit 534, for example, to the learning module production device 6. However, the algorithm information may be, for example, transmitted to the user terminal 4, displayed on a display, or stored in a recording medium, and the output form and destination are not important.

[0409] The learning storage unit 61 constituting the learning device production device 6 may store various information. The various information includes, for example, a learning device, learning source information, and algorithm information. In addition, the learning storage unit 61 may store, for example, the above-mentioned effective solution means sentence, the above-mentioned non-effective solution means sentence, the above-mentioned solution means term, the above-mentioned non-solution means term, the above-mentioned effect term, the above-mentioned solution means sentence, the above-mentioned non-solution means sentence, the above-mentioned effect sentence, the above-mentioned non-effect sentence, the above-mentioned object sentence, the above-mentioned non-object sentence, the above-mentioned object term, and the above-mentioned non-object term. Note that other information may be explained at appropriate times.

[0410] The learning reception unit 62 receives various information. The various information is, for example, the learning source information described above. As described above, reception usually means reception from the user terminal 4, but it may be considered as a concept including reception of information input from an input device such as a keyboard, reception of information read from a recording medium such as a disk or semiconductor memory, etc.

[0411] The learning source information receiving unit 621 receives the learning source information. The learning source information receiving unit 621 usually receives the learning source information from the user terminal 4. However, the learning source information may be received via an input device such as a keyboard, or may be read from a recording medium such as a disk, and the manner of reception is not important.

[0412] The learning processing unit 13 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a learning source information acquisition unit 133, and a learning unit 134. The important information acquisition unit 132 includes an effect / solution means sentence acquisition means 1321, an effect sentence acquisition means 1322, a solution means sentence acquisition means 1323, an objective sentence acquisition means 1324, a feature word acquisition means 1325, a related word acquisition means 1326, an important tag corresponding sentence acquisition means 1327, a numerical value range character string acquisition means 1328, and a numerical value acquisition means 1329, which basically perform the same processing as in the first embodiment.

[0413] However, the learning unit 134 performs a learning process on the learning source information accepted by the learning source information accepting unit 621 or the learning source information acquired by the learning source information acquiring unit 133 using an algorithm according to the algorithm information output by the algorithm determination device 5, and acquires a learning device.

[0414] The learning unit 134, for example, acquires information specified by information specifying information contained in the algorithm information from the learning source information accepted by the learning source information accepting unit 621, and acquires learning source information having two or more patent classification information including the information and the classification information contained in the learning source information. In addition, the learning unit 134 provides the acquired learning source information to a program specified by program specifying information contained in the algorithm information, performs a learning process, and configures a learning device. Note that, if the information specifying information includes information specifying important information, the learning unit 134 performs important information acquisition as performed by the important information acquiring unit 132.

[0415] The learning unit 134 may, for example, acquire information specified by information specifying information contained in the algorithm information from the learning source information accepted by the learning source information accepting unit 621, and acquire two or more pieces of learning source information having two or more pieces of patent classification information containing the information and the classification information contained in the learning source information. The learning unit 134 may also provide the acquired two or more pieces of learning source information to one or more programs specified by program specifying information contained in the algorithm information, perform a learning process, and configure two or more learners. Note that when the information specifying information includes information specifying important information, the learning unit 134 performs important information acquisition as performed by the important information acquiring unit 132.

[0416] For example, when the algorithm information output by the algorithm determination device 5 is (a) information capable of identifying the learning source information corresponding to the most accurate predicted classification information among two or more pieces of learning source information, the learning unit 134 acquires a learning device using the learning source information identified by the algorithm information.

[0417] Alternatively, if the algorithm information is (b) information that identifies a learning device generation program corresponding to the most accurate predictive classification information among two or more learning device generation programs stored in the program storage unit 511, the learning unit 134 obtains a learning device using the learning device generation program identified by the algorithm information.

[0418] Alternatively, if the algorithm information is (c) information identifying a statistical processing among two or more statistical processings that corresponds to the predictive classification information with the highest accuracy, the learning unit 134 obtains two or more learning devices using two or more pieces of learning source information corresponding to the information identifying the statistical processing, two or more learning device generation programs corresponding to the information identifying the statistical processing, two or more pieces of learning source information and one or more learning device generation programs corresponding to the information identifying the statistical processing, or one or more pieces of learning source information and two or more learning device generation programs corresponding to the information identifying the statistical processing.

[0419] Furthermore, for example, when the algorithm information corresponds to information that specifies the type of accuracy desired by the user among the recall rate, the precision rate, the F-measure, or the accuracy rate (i.e., the accuracy specification information received by the reception unit 52), the learning unit 134 acquires a learning device for increasing the accuracy specified by the algorithm information. Note that the above-mentioned effect can be achieved if the learning unit 134 generates a learning device according to the algorithm information.

[0420] The storage unit 641 stores the learning devices acquired by the learning unit 134. The storage unit 641 usually stores the learning devices acquired by the learning unit 134 in association with the algorithm information output by the algorithm determination device 5. The learning devices are usually stored in the learning device storage unit 211 constituting the patent classification device 2. However, the learning devices may be stored in, for example, the learning storage unit 61, and the storage destination is not important.

[0421] The learning module storage unit 211 constituting the patent classification device 8 stores one or more learning modules acquired by the learning module production device 6. A learning module is usually associated with algorithm information.

[0422] The receiving unit 81 receives classification target information including a patent identifier that identifies a patent to be classified.

[0423] The prediction unit 821 performs prediction processing using a learning device according to an algorithm in accordance with the algorithm information associated with the learning device in the learning device storage unit 211 for the classification target information received by the receiving unit 81, and obtains classification result information.

[0424] The prediction unit 821 acquires information to be used in the prediction process, which is, for example, information corresponding to the classification target information accepted by the acceptance unit 81 and information specified by information specifying information included in the algorithm information. Next, the prediction unit 821 provides the information to be used in the prediction process to, for example, one or more prediction programs specified by one or more program specifying information included in the algorithm information, executes the one or more prediction programs, and acquires classification result information for each prediction program.

[0425] Furthermore, when the algorithm information includes statistical processing specific information, the prediction unit 821 statistically processes the classification result information for each of two or more prediction programs, and acquires final classification result information.

[0426] The prediction unit 72 may apply a learning device to the classification target processed information acquired by the classification target processed information acquisition unit 233, predict the classification of the classification target processed information using a machine learning algorithm, and acquire classification result information.

[0427] The output unit 73 outputs the classification result information acquired by the prediction unit 72 in association with the patent identifier included in the classification target information accepted by the acceptance unit 71.

[0428] The storage unit 51, the program storage unit 511, the accuracy specification information storage unit 512, the learning storage unit 61, etc. are preferably non-volatile recording media, but may also be realized as volatile recording media.

[0429] There is no restriction on the process by which information is stored in the storage unit 51 etc. For example, information may be stored in the storage unit 51 etc. via a recording medium, information transmitted via a communication line etc. may be stored in the storage unit 51 etc., or information inputted via an input device may be stored in the storage unit 51 etc.

[0430] The reception unit 52, the learning source information reception unit 521, the output unit 54, the learning reception unit 62, the learning source information reception unit 621, the learning output unit 64, the storage unit 641, and the reception unit 81 can typically be realized by wireless or wired communication means, etc.

[0431] The processing unit 53, the learning unit 531, the prediction unit 532, the evaluation unit 533, the algorithm information acquisition unit 534, the learning source information acquisition unit 535, the statistical processing unit 536, the processing unit 82, the prediction unit 821, and the like can usually be realized by a processor, a memory, and the like. The processing procedure of the processing unit 53, etc. is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, they may also be realized by hardware (dedicated circuit). The processor may be, for example, a CPU, an MPU, a GPU, or the like, and does not matter.

[0432] Next, an example of the operation of the patent classification system B will be described. First, an example of the operation of the algorithm determination device 5 will be described with reference to the flowchart of FIG.

[0433] (Step S1301) The processing unit 53 acquires all candidates for algorithm information.

[0434] For example, when one or more pieces of information specifying information (for example, three pieces of "abstract", "abstract, patent classification code", and "abstract, effect solution statement"), one or more pieces of program specifying information (for example, two pieces of "deep learning" and "random forest"), and one or more pieces of statistical processing specifying information (for example, "AND" and "OR") are stored in the storage unit 51, the processing unit 53 acquires six algorithm information candidates that are combinations of any one of the three pieces of information specifying information and any one of the two pieces of program specifying information. In addition, the processing unit 53 acquires six algorithm information candidates that are combinations of any one of the three pieces of information specifying information and two types of statistical processing specifying information of the predicted results of the programs specified by each of the two pieces of program specifying information. That is, in this example, the processing unit 53 acquires 12 algorithm information candidates.

[0435] (Step S1302) The processing unit 53 assigns 1 to a counter i.

[0436] (Step S1303) Processing unit 53 judges whether or not a candidate for the i-th algorithm information exists among the candidates for the algorithm information acquired in step S1301. If a candidate for the i-th algorithm information exists, the process proceeds to step S1304, and if a candidate for the i-th algorithm information does not exist, the process proceeds to step S1306.

[0437] (Step S1304) The evaluation unit 533 evaluates the accuracy (e.g., recall, F-value, etc.) specified by the accuracy specifying information stored in the storage unit 51 when the information and algorithm corresponding to the i-th algorithm information candidate are used, acquires the accuracy information, and temporarily stores the accuracy information in association with the i-th algorithm information candidate. This evaluation process will be described with reference to the flowchart in FIG.

[0438] (Step S1305) The processing unit 53 increments the counter i by 1. The process returns to step S1303.

[0439] (Step S1306) The algorithm information acquisition unit 534 acquires the precision information of the best value.

[0440] (Step S1307) The algorithm information acquisition unit 534 acquires, as algorithm information, candidates for algorithm information paired with the accuracy information of the best value acquired in step S1306.

[0441] (Step S1308) The algorithm information acquisition unit 534 accumulates the algorithm information.

[0442] In the flowchart of FIG. 13, the process ends when the power is turned off or an interrupt occurs to end the process.

[0443] Next, the evaluation process in step S1304 will be described with reference to the flowchart in FIG.

[0444] (Step S1401) The learning source information acquiring unit 535 acquires information identifying information corresponding to a candidate for algorithm information.

[0445] (Step S1402) The learning source information acquiring unit 535 assigns 1 to a counter i.

[0446] (Step S1403) The learning source information acquisition unit 535 judges whether the i-th patent information exists in the learning source information to be processed. If the i-th patent information exists, the process proceeds to step S1404, and if the i-th patent information does not exist, the process proceeds to step S1406. Note that when the process proceeds to step S1406, the learning source information to be evaluated has been completed. In addition, the learning source information to be processed is stored, for example, in the storage unit 51.

[0447] (Step S1404) The learning source information acquisition unit 535 acquires information corresponding to the i-th patent information and identified by the information identification information acquired in step S1401. The evaluation unit 533 acquires patent classification information consisting of classification information possessed by the learning source information to be processed and the acquired information, and temporarily stores it in a buffer not shown.

[0448] (Step S1405) The learning source information acquiring unit 535 increments the counter i by 1. The process returns to step S1403.

[0449] (Step S1406) The evaluation unit 533 judges whether or not there is statistical processing specification information corresponding to the candidate for algorithm information. If there is statistical processing specification information, the process proceeds to step S1409. If there is no statistical processing specification information, the process proceeds to step S1407.

[0450] (Step S1407) The evaluation unit 533 acquires the program specific information corresponding to the candidate for the algorithm information.

[0451] (Step S1408) The evaluation unit 533 performs K-fold cross validation using the learning program and the prediction program identified by the program identification information acquired in step S1407. A processing example of the K-fold cross validation will be described with reference to the flowchart in FIG.

[0452] (Step S1409) The evaluation unit 533 performs evaluation when there is statistical processing specification information corresponding to the candidate for algorithm information. An example of such a statistical processing evaluation process will be described with reference to the flowchart of FIG.

[0453] (Step S1410) The evaluation unit 533 temporarily stores the acquired accuracy information in association with the candidate algorithm information, and returns to the upper level process.

[0454] Next, an example of the K-fold cross-validation process in step S1408 will be described with reference to the flowchart in FIG.

[0455] (Step S1501) The evaluation unit 533 divides the learning source information into k (k is a natural number equal to or greater than 2).

[0456] (Step S1502) The evaluation unit 533 assigns 1 to a counter i.

[0457] (Step S1503) The evaluation unit 533 judges whether or not the i-th division pattern exists in the learning source information divided into k. If the i-th division pattern exists, the process proceeds to step S1504, and if the i-th division pattern does not exist, the process proceeds to step S1513. Note that when division is performed into k, k division patterns usually exist. This is a well-known technique in K-fold cross-validation.

[0458] (Step S1504) The learning unit 531 acquires information to be learned (patent classification information to be used for learning processing in the k division pattern), which is a part of the learning source information.

[0459] (Step S1505) The learning unit 531 provides the learning program identified by the program identification information with the information on the learning subject acquired in step S1504, executes the learning program, and acquires a learning device.

[0460] (Step S1506) The prediction unit 532 assigns 1 to a counter j.

[0461] (Step S1507) Prediction unit 532 judges whether or not the j-th information to be predicted exists, which is a part of the learning source information. If the j-th information to be predicted exists, the process proceeds to step S1508, and if the j-th information to be predicted does not exist, the process proceeds to step S1510.

[0462] (Step S1508) The prediction unit 532 gives the j-th information of the prediction target to the prediction program specified by the program specification information, executes the prediction program, and obtains the prediction result.

[0463] (Step S1509) The prediction unit 532 increments the counter j by 1. The process returns to step S1507.

[0464] (Step S1510) The evaluation unit 533 acquires accuracy specifying information associated with the candidate for algorithm information.

[0465] (Step S1511) The evaluation unit 533 compares the set of prediction results acquired in step S1508 with the set of classification information contained in the patent classification information of the prediction target in the learning source information, and acquires accuracy information of the accuracy (e.g., recall, F value) specified by the accuracy specification information acquired in step S1510. Note that such accuracy information is primary accuracy information.

[0466] (Step S1512) The evaluation unit 533 increments the counter i by 1. The process returns to step S1503.

[0467] (Step S1513) The evaluation unit 533 acquires accuracy information that is a representative value (for example, an average value, a median value) of the k pieces of primary accuracy information acquired in step S1511, and stores the accuracy information in association with the candidate of the algorithm information. Return to the upper process.

[0468] Next, an example of the statistical processing evaluation process in step S1409 will be described with reference to the flowchart in Fig. 16. Note that in the flowchart in Fig. 16, explanations of steps that are the same as those in the flowchart in Fig. 15 will be omitted.

[0469] (Step S1601) The evaluation unit 533 assigns 1 to a counter j.

[0470] (Step S1602) Evaluation unit 533 judges whether or not the j-th program specific information exists. If the j-th program specific information exists, the process proceeds to step S1603, and if the j-th program specific information does not exist, the process proceeds to step S1609.

[0471] (Step S1603) The learning unit 531 provides the learning program identified by the j-th program identification information with the information on the learning target acquired in step S1504, executes the learning program, and acquires a learning device.

[0472] (Step S1604) The prediction unit 532 assigns 1 to a counter l.

[0473] (Step S1605) The prediction unit 532 judges whether or not the l-th information to be predicted exists, which is a part of the learning source information. If the l-th information to be predicted exists, the process proceeds to step S1606, and if the l-th information to be predicted does not exist, the process proceeds to step S1608.

[0474] (Step S1606) The prediction unit 532 gives the l-th information of the prediction target to the prediction program specified by the program specification information, executes the prediction program, and obtains the prediction result.

[0475] (Step S1607) The prediction unit 532 increments the counter l by 1. The process returns to step S1605.

[0476] (Step S1608) The evaluation unit 533 increments the counter j by 1. The process returns to step S1602.

[0477] (Step S1609) The statistical processing unit 536 assigns 1 to counter l.

[0478] (Step S1610) The statistical processing unit 536 judges whether or not information on the l-th prediction target exists. If information on the l-th prediction target exists, the process proceeds to step S1611. If information on the l-th prediction target does not exist, the process proceeds to step S1613.

[0479] (Step S1611) The statistical processing unit 536 performs statistical processing (e.g., “AND”, “OR”, or “majority vote”) specified by the statistical processing specification information possessed by the candidate algorithm information on two or more prediction results for the information of the lth prediction target, and obtains the statistical processing result.

[0480] (Step S1612) The statistical processing unit 536 increments the counter l by 1. The process returns to step S1610.

[0481] (Step S1613) The evaluation unit 533 acquires the accuracy specifying information corresponding to the candidate for the algorithm information.

[0482] (Step S1614) The evaluation unit 533 compares the set of statistical processing results acquired in step S1611 with the set of classification information contained in the patent classification information of the prediction target in the learning source information, and acquires accuracy information of the accuracy (e.g., recall, F value) specified by the accuracy specification information acquired in step S1613. Note that such accuracy information is primary accuracy information.

[0483] (Step S1615) The evaluation unit 533 increments the counter i by 1. The process returns to step S1503.

[0484] (Step S1616) The evaluation unit 533 acquires accuracy information that is a representative value (for example, an average value, a median value) of the k pieces of primary accuracy information acquired in step S1614, and stores the accuracy information in association with the candidate of the algorithm information. Return to the upper process.

[0485] Next, an example of the operation of the learning module production device 6 will be described with reference to the flowchart of FIG.

[0486] (Step S1701) The learning receiving unit 62 acquires learning source information.

[0487] (Step S1702) The learning processing unit 13 acquires algorithm information corresponding to the learning source information acquired in step S1701.

[0488] (Step S1703) The learning processing unit 13 acquires information specifying information contained in the algorithm information.

[0489] (Step S1704) The learning processing unit 13 acquires information corresponding to each patent classification information contained in the learning source information, which is information identified by the information identification information acquired in step S1703, and acquires information consisting of the information and the classification information contained in each patent classification information. Then, the learning processing unit 13 acquires learning source information to be used in the learning process, which is learning source information having the acquired information.

[0490] (Step S1705) The learning processing unit 13 assigns 1 to the counter i.

[0491] (Step S1706) The learning processing unit 13 judges whether or not the i-th program-specific information included in the algorithm information acquired in step S1702 exists. If the i-th program-specific information exists, the process proceeds to step S1707, and if the i-th program-specific information does not exist, the process ends.

[0492] (Step S1707) The learning processing unit 13 acquires from the learning storage unit 61 the learning program identified by the program identification information acquired in step S1707.

[0493] (Step S1708) The learning processing unit 13 provides the learning source information acquired in step S1704 to the learning program acquired in step S1707, executes the learning program, and acquires a learning device. Then, the learning processing unit 13 stores the learning device in a pair with the algorithm information acquired in step S1702.

[0494] (Step S1709) The learning processing unit 13 increments the counter i by 1. The process returns to step S1706.

[0495] Next, an example of the operation of the patent classification device 8 will be described with reference to the flowchart of FIG.

[0496] (Step S1801) The reception unit 81 judges whether or not the classification target information has been received. If the classification target information has been received, the process proceeds to step S1802, and if the classification target information has not been received, the process returns to step S1801.

[0497] (Step S1802) The processing unit 82 acquires the algorithm information stored in the storage unit 21.

[0498] (Step S1803) The processing unit 82 acquires information specifying information contained in the algorithm information acquired in step S1802.

[0499] (Step S1804) The processing unit 82 acquires information that is identified by the information identifying information acquired in step S1803 and corresponds to the classification target information.

[0500] (Step S1805) The processing unit 82 acquires the program-specific information included in the algorithm information acquired in step S1802.

[0501] (Step S1806) The processing unit 82 assigns 1 to the counter i.

[0502] (Step S1807) The processing unit 82 judges whether or not the i-th program-specific information exists in the algorithm information acquired in step S1802. If the i-th program-specific information exists, the process proceeds to step S1808, and if the i-th program-specific information does not exist, the process proceeds to step S1811.

[0503] (Step S1808) The prediction unit 821 obtains from the storage unit 21 the prediction program identified by the program identification information obtained in step S1805.

[0504] (Step S1809) The prediction unit 821 provides the prediction program acquired in step S1808 with the learning device paired with the algorithm information acquired in step S1802 and the information acquired in step S1804, executes the prediction program, acquires a prediction result, and accumulates the prediction result in association with the classification target information. Note that it is preferable for the prediction unit 821 to also acquire a score.

[0505] (Step S1810) The processing unit 82 increments the counter i by 1. The process returns to step S1807.

[0506] (Step S1811) The prediction unit 821 judges whether or not statistical processing specification information is present in the algorithm information acquired in step S1802. If statistical processing specification information is present, the process proceeds to step S1812, and if statistical processing specification information is not present, the process proceeds to step S1814.

[0507] (Step S1812) The statistical processing unit 536 performs statistical processing specified by the statistical processing specification information on the prediction results of the two or more prediction processes in step S1809, and obtains the statistical processing result. Note that the statistical processing result is the final prediction result.

[0508] (Step S1813) The statistical processing unit 536 uses the score acquired by the prediction unit 821 to acquire a score corresponding to the statistical processing result.

[0509] (Step S1814) The output unit outputs the final prediction result obtained in step S1812 and the score obtained in step S1813.

[0510] Below, we will explain a specific example of the operation of the patent classification system B. Note that the following explanation can be modified in various ways and does not limit the scope of the present invention in any way.

[0511] In this example, the information specific information is either "abstract, IPC" or "abstract, effect solution statement, IPC". The program specific information is either "deep learning.exe" or "random forest.exe". The statistical information specific information is either "AND" or "OR".

[0512] The storage unit 51 of the algorithm determination device 5 stores eight algorithm information candidates 1 to 8, for example, as shown in FIG. 19, which will be described later. The algorithm information candidates are candidates for algorithm information. The algorithm information candidates 1 to 4 are four algorithm information candidates that are combinations of one of the two types of information specifying information and one of the two types of program specifying information. The algorithm information candidates 5 to 8 are four algorithm information candidates that are combinations of one of the two types of information specifying information and two types of statistical processing specifying information of the predicted results of the program specified by each of the two types of program specifying information. In this example, the eight algorithm information candidates 1 to 8 are stored in the storage unit 51 in advance, but may be acquired by the processing unit 53 in response to a user's instruction input through a setting screen, which will be described later, and stored in the storage unit 51.

[0513] 20 described later, eight pieces of accuracy information 1 to 8 corresponding to the eight algorithm information candidates 1 to 8 may be stored in the storage unit 51. The eight pieces of accuracy information 1 to 8 are acquired by the algorithm information acquisition unit 534 and stored in the storage unit 51 in association with the eight algorithm information candidates 1 to 8.

[0514] Fig. 19 is a data structure diagram of an algorithm information candidate. The algorithm information candidate has information specific information, program specific information, and statistical information specific information. Algorithm information candidate 1 associated with ID "1" has information specific information "abstract, IPC", program specific information "deep learning.exe", and statistical information specific information "-". Note that "-" is information indicating that no statistical processing will be performed.

[0515] Additionally, algorithm information candidate 2 associated with ID "2" has information specific information "abstract, effect solution statement, IPC", program specific information "deep learning.exe", and statistical information specific information "-". Similarly, algorithm information candidate 3 has information specific information "abstract, IPC", program specific information "random forest.exe", and statistical information specific information "-". Algorithm information candidate 4 has information specific information "abstract, effect solution statement, IPC", program specific information "random forest.exe", and statistical information specific information "-".

[0516] Algorithm information candidate 5 has information specific information "abstract, IPC", program specific information "deep learning.exe, random forest.exe", and statistical information specific information "OR". Algorithm information candidate 6 has information specific information "abstract, effect solution means statement, IPC", program specific information "deep learning.exe, random forest.exe", and statistical information specific information "OR". Algorithm information candidate 7 has information specific information "abstract, IPC", program specific information "deep learning.exe, random forest.exe", and statistical information specific information "AND". Algorithm information candidate 8 has information specific information "abstract, effect solution means statement, IPC", program specific information "deep learning.exe, random forest.exe", and statistical information specific information "AND".

[0517] 20 is a data structure diagram of accuracy information. Accuracy information has recall, precision, F-value, and accuracy. Accuracy information 1 associated with ID "1" has recall "a1", precision "b1", F-value "(a1+b1) / 2", and accuracy "c1". Accuracy information 2 associated with ID "2" has recall "a2", precision "b2", F-value "(a2+b2) / 2", and accuracy "c2".

[0518] Similarly, accuracy information 3 has recall "a3", precision "b3", F-value "(a3+b3) / 2", and accuracy "c3". Accuracy information 4 has recall "a4", precision "b4", F-value "(a4+b4) / 2", and accuracy "c4". Accuracy information 5 has recall "a5", precision "b5", F-value "(a5+b5) / 2", and accuracy "c5". Accuracy information 6 has recall "a6", precision "b6", F-value "(a6+b6) / 2", and accuracy "c6". The accuracy information 7 has a recall rate “a7”, a precision rate “b7”, an F-score “(a7+b7 / 2”, and an accuracy rate “c7”. The accuracy information 8 has a recall rate “a8”, a precision rate “b8”, an F-score “(a8+b8) / 2”, and an accuracy rate “c8”.

[0519] The user inputs learning source information having 100 pieces of patent classification information, each of which is composed of a patent identifier and classification information, into the user terminal 4. The user terminal 4 accepts the input of the learning source information and transmits the accepted learning source information to the algorithm determination device 5.

[0520] The program storage unit 511 stores two learning device generation programs corresponding to the above two types of program identification information, and two prediction programs corresponding to the two learning device generation programs.

[0521] The accuracy specification information storage unit 512 stores one of four pieces of accuracy specification information, namely, "recall rate," "precision rate," "F-value," and "accuracy rate" (here, "accuracy rate"). Note that the stored accuracy specification information is, for example, acquired by the processing unit 53 in response to a user's instruction inputted through a setting screen described later, and accumulated in the storage unit 51, but may be default information or automatically acquired information.

[0522] In the algorithm determination device 5, the learning source information receiving unit 521 receives the above-mentioned learning source information, and the processing unit 53 performs the following processing using the received learning source information and the eight algorithm information candidates 1 to 8 stored in the storage unit 51.

[0523] In detail, the learning source information acquisition unit 535 constituting the processing unit 53 acquires, for each of the two or more (here, 100) patent classification information contained in the received learning source information, two or more patent classification information each having two or more patent classification information having information specified by the two types of information specifying information constituting the algorithm information candidate from among the patent information contained in the patent classification information. As a result, two or more patent classification information having different information (here, two patent classification information: patent classification information having "abstract, IPC" and patent classification information having "abstract, solution statement, IPC") are acquired for one accepted patent classification information.

[0524] Next, for each of the two or more pieces of acquired learning source information, the learning unit 531 executes a learning device generation program, of the two learning device generation programs stored in the program storage unit 511, that is identified by the program identification information of the algorithm information candidate, using at least a portion (e.g., 90 pieces) of patent classification information of the learning source information, to acquire two learning devices. The prediction unit 532 applies a prediction program corresponding to the learning device generation program used to acquire the learning device, using each of the two acquired learning devices, to at least a portion (e.g., 10 pieces not used to generate a learning device) of the patent classification information of the learning source information, to acquire two or more pieces (here, four pieces) of predicted classification information for each of the two or more pieces (here, the above 10 pieces) of patent classification information.

[0525] The statistical processing unit 535 uses the two or more pieces of predicted classification information acquired to perform one of the two statistical processes (AND, OR) specified by the statistical information identification information, and acquires two or more pieces of predicted classification information (here, four pieces) as the results of the statistical processing.

[0526] The evaluation unit 533 uses the four or more pieces of prediction classification information thus acquired (here, four pieces corresponding to combinations of any of the two types of information identifying information and any of the two types of program identifying information, and four pieces corresponding to combinations of any of the two types of information identifying information and any of the two types of statistical processing identifying information of the predicted results of the program identified by each of the two types of program identifying information), as well as the classification information contained in the learning source information, to acquire accuracy information of each piece of prediction classification information, and stores it in the storage unit 51 in association with the algorithm information candidates.

[0527] Of the above eight pieces of predicted classification information, for the first four pieces of predicted classification information, accuracy information corresponding to the predicted classification information may be obtained, for example, by the K-fold cross-validation described in the flowchart of FIG. 15, and for the latter four pieces of predicted classification information, accuracy information corresponding to the predicted classification information may be obtained, for example, by the statistical processing evaluation described in the flowchart of FIG. 16.

[0528] 19. As a result, the eight pieces of accuracy information 1 to 8 shown in FIG. 20 are stored in the storage unit 51 in association with the eight algorithm information candidates 1 to 8 shown in FIG.

[0529] The algorithm information acquisition unit 534 acquires the best value (e.g., the maximum value) of the accuracy information 1 to 8 stored in the storage unit 51 for the accuracy (here, the accuracy rate) specified by the accuracy specification information stored in the accuracy specification information storage unit 512. For example, when c3 is the best value among the accuracy rates c1 to c8 corresponding to the accuracy information 1 to 8, the accuracy information 3 corresponding to the best value is acquired.

[0530] For example, if the stored accuracy specification information is information indicating that all four types of accuracy are used (for example, information indicating that for each of the four types of accuracy, x points (for example, 1 point) are added to the accuracy information corresponding to the best value to obtain the accuracy information corresponding to the highest score), the best value among the recall rates a1 to a8 is a2, the best value among the precision rates b1 to b8 is b3, the best value among the F-values ​​"(a1+b1) / 2" to "(a8+b8) / 2" is "(a2+b2) / 2", and the best value of c4 among the accuracy rates c1 to c8 is the best value, then "1 point + 1 point = 2 points" is obtained for the accuracy information 2, "1 point" is obtained for the accuracy information 3, "1 point" is obtained for the accuracy information 4, and "0 point" is obtained for each of the other accuracy information 1, 5 to 8, and the accuracy information 2 corresponding to the highest point of 2 points among the 8 points obtained in this manner may be obtained.

[0531] Also, for example, the stored accuracy specification information is information indicating that one of the four types of accuracy (for example, precision) is emphasized (for example, for each of the four types of accuracy, x points (for example, 1 point) are added to the accuracy information corresponding to the best value, and if the accuracy information to which x points have been added includes the accuracy information to be emphasized, y points (for example, 2 points) are further added to the accuracy information to be emphasized), the best value of the recall rates a1 to a8 is a2, and the best value of the precision rates b1 to b8 is y (for example, 2 points). If the good value is b3, the best value among the F-scores "(a1+b1) / 2" to "(a8+b8) / 2" is "(a2+b2) / 2", and c3 is the best value among the accuracy rates c1 to c8, then "1 point + 1 point = 2 points" is obtained for accuracy information 2, "(1 point + 2 points) + 1 point = 4 points" is obtained for accuracy information 3, and "0 points" is obtained for each of the other accuracy information 1, 4 to 8, and accuracy information 3 corresponding to 4 points, which is the highest score among the 8 scores obtained in this way, may be obtained.

[0532] The algorithm information acquisition unit 534 acquires algorithm information paired with the accuracy information of the best value (for example, when accuracy information 3 is acquired, algorithm information candidate 3 paired with the accuracy information 3) as algorithm information from the storage unit 51. The algorithm information acquisition unit 534 accumulates the acquired algorithm information in the storage unit 51 in association with the received learning source information.

[0533] In this way, in response to the learning source information being received by the learning source information receiving unit 521, the output unit 54 outputs the algorithm information acquired by the algorithm information acquiring unit 534. The output destination of the algorithm information is usually the learning machine production device 6.

[0534] In the learning machine production device 6, the learning processing unit 13 accumulates the algorithm information output from the algorithm determination device 5 in the learning storage unit 61, and the learning unit 134 acquires a learning machine using the algorithm information stored in the learning storage unit 61, for example, according to the procedure shown in the flowchart of Fig. 17. The acquired learning machine is delivered to the patent classification device 8.

[0535] The algorithm information acquired by the learning machine production device 6 is also output to the patent classification device 8. In the patent classification device 8, the processing unit 82 accumulates the learning device delivered from the learning machine production device 6 and the algorithm information output from the learning machine production device 6 in the storage unit 21. Thereafter, the reception unit 81 receives classification target information from the user terminal 4, and the processing unit 82 executes, for example, the process shown in the flowchart of FIG. 18 for the received classification target information using the learning device and algorithm information stored in the storage unit 21, and acquires predicted classification information and a score corresponding thereto. The output unit 24 transmits the acquired patent classification information and score to the user terminal 4, which is the sender of the classification target information. As a result, the patent classification information and score are displayed on the display of the user terminal 4.

[0536] The algorithm information acquired by the learning machine production device 6 may also be transmitted to the user terminal 4, which is the transmission source of the learning source information. The user terminal 4 outputs the algorithm information transmitted from the algorithm determination device 5 via an output device such as a display. As a result, the algorithm information is displayed on the display of the user terminal 4.

[0537] As described above, according to this embodiment, the algorithm determination device 5 can determine an appropriate process for performing the prediction process.

[0538] Furthermore, the algorithm determination device 5 can determine an appropriate process for performing the prediction process by changing the data used when performing the prediction process.

[0539] Furthermore, the algorithm determination device 5 can determine an appropriate process for performing the prediction process by using a plurality of learning module generation programs.

[0540] Furthermore, the algorithm determination device 5 can determine an appropriate process for performing the prediction process by using a prediction result obtained by performing statistical processing on prediction results obtained using two or more learning devices.

[0541] In addition, the algorithm determination device 5 can determine processing including an appropriate statistical processing for performing the prediction processing by using two or more prediction results obtained by performing two or more statistical processing on the prediction results obtained using two or more learning devices.

[0542] Furthermore, the algorithm determination device 5 can determine a process that has a higher score desired by the user (accuracy information specified by the accuracy specification information accepted by the accepting unit 52) ​​from among the recall rate, precision rate, F-measure, or accuracy rate.

[0543] Furthermore, the learning module production device 6 performs a learning process using an algorithm according to the algorithm information output by the algorithm determination device 5, thereby producing an appropriate learning module.

[0544] In addition, the patent classification device 2 can classify patents with high accuracy by performing prediction processing using an appropriate learning device produced by the learning device production device 6.

[0545] In addition, the patent classification device 2 can classify patents with higher accuracy by using an appropriate learning device produced by the learning device production device 6 and important information related to the patent specifications.

[0546] The software for realizing the algorithm determination device 5 in this embodiment is a program as follows. That is, this program includes a computer that can access a program storage unit 511 that stores one or more learning device generation programs for generating learning devices for classifying patent information relating to patents using learning source information having two or more patent classification information having patent information relating to patents and classification information indicating a classification of the patent information, and a prediction program for predicting the classification of the patent classification information using a learning device and acquiring the predicted classification information, the computer having a learning source information receiving unit 521 that receives the learning source information, a learning unit 531 that executes each of the one or more learning device generation programs and acquires one or more learning devices using at least a part of the patent classification information of the learning source information received by the learning source information receiving unit 521, and the program storage unit 511 that stores a prediction program for predicting the classification of the patent classification information using a learning device and acquiring the predicted classification information. a prediction unit 532 that applies a prediction program to at least a portion of the patent classification information of the attached learning source information using one or more learning devices generated by the learning unit 531, and acquires two or more predicted classification information for each of the one or more patent classification information; an evaluation unit that acquires accuracy information regarding the accuracy of each of the two or more predicted classification information using each of the two or more predicted classification information acquired by the prediction unit 532 and the classification information contained in the learning source information, and determines the predicted classification information with the highest accuracy specified by the accuracy information; an algorithm information acquisition unit 534 that acquires algorithm information regarding the algorithm corresponding to the predicted classification information determined by the evaluation unit; and a program for functioning as an output unit that outputs the algorithm information.

[0547] Moreover, the software for realizing the learning device production device 6 in this embodiment is the following program. That is, this program causes a computer to function as a learning device 134 that performs a learning process on the learning device information accepted by the learning device information acceptance unit 621 using an algorithm according to the algorithm information output by the algorithm determination device 5 to acquire a learning device, and a storage unit 641 that stores the learning device in association with the algorithm information.

[0548] Furthermore, the software for realizing the patent classification device 2 in this embodiment is the following program. That is, this program causes a computer that can access a learning device storage unit in which learning devices accumulated by the learning device production device 6 are stored to function as a reception unit that receives classification target information including a patent identifier that identifies a patent to be classified, a prediction unit 234 that performs a prediction process on the classification target information using the learning device according to an algorithm in accordance with the algorithm information associated with the learning device, and obtains classification result information, and an output unit that outputs the classification result information obtained by the prediction unit 234 in association with the patent identifier included in the classification target information received by the reception unit.

[0549] (Embodiment 3) FIG. 21 is a block diagram of a patent classification system C in this embodiment.

[0550] The patent classification system C includes an algorithm determination device 5, a learning module production device 1, a patent classification device 2, a server device 3, and one or more user terminals 4.

[0551] The algorithm determination device 5 is the same element as in the embodiment 2. The learning device production device 1, the patent classification device 2, the server device 3, and the user terminal 4 are the same elements as in the embodiment 1.

[0552] However, the algorithm used by the learning processing unit 13 constituting the learning module production device 1 is the algorithm determined by the algorithm determination device 5.

[0553] In addition, the important information acquisition unit 132 that constitutes the patent classification device 2 includes an effect / solution means sentence acquisition means 1321, an effect sentence acquisition means 1322, a solution means sentence acquisition means 1323, an objective sentence acquisition means 1324, a feature word acquisition means 1325, a related word acquisition means 1326, an important tag corresponding sentence acquisition means 1327, a numerical range string acquisition means 1328, and a number acquisition means 1329.

[0554] Next, the operation of the patent classification system C will be described. The flowchart of the algorithm determination device 5 constituting the patent classification system C is shown in Fig. 13 to Fig. 16. The flowchart of the learning module production device 1 is shown in Fig. 4 to Fig. 9. The flowchart of the patent classification device 2 is shown in Fig. 10.

[0555] As described above, according to the present embodiment, in addition to the effects described in the first embodiment, the effects described in the second embodiment can be further obtained.

[0556] (Embodiment 4) In this embodiment, we will explain a patent classification system D that is equipped with a classification information determination device, in which condition information that is a pair of a condition and a class (classification information) is stored, and for a predicted target (patent information) that matches the one condition, the class that pairs with the one condition is determined, and if the predicted target does not match the one condition, the class is determined using a machine learning algorithm.

[0557] In addition, in this embodiment, the machine learning algorithm is, for example, the same as that described in the first to third embodiments, but the type is not important. A patent classification system D including a classification information determination device will be described.

[0558] In addition, in this embodiment, we will explain a patent classification system D that is equipped with a classification information determination device, where one condition is that the patent is the training data used when creating a machine learning learning device.

[0559] 22 is a block diagram of a patent classification system D in this embodiment. The patent classification system D includes a learning module production device 1, a classification information determination device 7, a server device 3, and a user terminal 4. In the patent classification system D, a learning module production device 6 may be used instead of the learning module production device 1.

[0560] The learning device production device 1 has a configuration and operates in a similar manner to that in the embodiment 1. That is, the learning device production device 1 performs a machine learning learning process using the learning source information to obtain a learning device.

[0561] The classification information determination device 7 includes a storage unit 71, a reception unit 72, a processing unit 73, and an output unit 74. The storage unit 71 includes a learning device storage unit 711, and a condition information storage unit 712. The reception unit 72 includes a patent information reception unit 721. The processing unit 73 includes a prediction unit 731. The prediction unit 731 includes a judgment means 7311, a first determination means 7312, and a second determination means 7313.

[0562] Various types of information are stored in the storage unit 71 constituting the classification information determination device 7. The various types of information are, for example, the learning device described above and condition information described below. In addition, the storage unit 71 may store, for example, one or more pieces of patent information, or may store patent classification information.

[0563] The learning device storage unit 711 stores a learning device. In this embodiment, the learning device production device 1 performs a machine learning learning process using the learning source information to acquire a learning device. As described above, the learning source information has two or more pieces of patent classification information, and the patent classification information has patent information and classification information of the patent information. The acquired learning device is delivered to the classification information determination device 7 and is accumulated in the learning device storage unit 711 by, for example, a processing unit 73 described later.

[0564] However, the learning device may be stored in advance in the learning device storage unit 711, or may be acquired within the classification information determination device 7. In the latter case, for example, learning source information is stored in the storage unit 71, and the classification information determination device 7 may include a learning unit (not shown) that performs machine learning learning processing using the learning source information and acquires a learning device.

[0565] The condition information storage unit 712 stores one or more pieces of condition information. The condition information is a set of one condition related to a patent and classification information of the patent information. In this embodiment, the one condition is, for example, that the patent information to be predicted is patent information that constitutes any one of two or more pieces of patent classification information contained in the learning source information. The one condition is, for example, a search formula for a patent. The search formula is, for example, "Name of invention includes 'machine learning'" and "Applicant=ABC & IPC=G06F*".

[0566] As mentioned above, patent information is information related to patents, such as a part or all of a patent specification in a patent gazette or published patent gazette, etc. As mentioned above, classification information is information indicating the classification of patent information, such as "○,×" or "A,B,C", etc.

[0567] The condition information includes, for example, a patent identifier (e.g., publication number, patent number, etc.) corresponding to one of two or more pieces of patent classification information contained in the learning source information, and classification information corresponding to the one piece of patent classification information. For example, if the learning device is acquired using a part of two or more pieces of patent classification information contained in the learning source information, the condition information includes a patent identifier corresponding to one of the two or more pieces of patent classification information used when acquiring the learning device, and classification information corresponding to the one piece of patent classification information.

[0568] However, the condition information may be, for example, one of two or more pieces of patent classification information contained in the learning source information. For example, if the learning device is obtained using a part of two or more pieces of patent classification information contained in the learning source information, the condition information may be one of the two or more pieces of patent classification information used when obtaining the learning device.

[0569] The learning device production device 1 may also deliver, for example, the learning device to the classification information determination device 7, together with the learning device, learning source information used when obtaining the learning device.

[0570] In the classification information determination device 7, for example, the processing unit 73 receives the learning source information, and uses the learning source information to acquire two or more pieces of condition information having a patent identifier and classification information, and stores the two or more pieces of condition information in the learning device storage unit 711. However, the acquisition of the two or more pieces of condition information may be performed on the learning device production device 1 side, and the acquired two or more pieces of condition information may be delivered to the classification information determination device 7.

[0571] As a result, the condition information storage unit 712 stores two or more condition information (for example, 90 pieces of condition information 1 to 90 shown in Figure 25 described later) corresponding to two or more pieces of patent classification information (for example, 90 pieces of patent classification information 11 to 100 out of the 100 patent classifications 1 to 100 that constitute the patent classification information described above) used by the learning device production device 1 when acquiring the learning device.

[0572] The reception unit 72 receives various types of information. The various types of information are, for example, patent information. As described above, reception usually means reception from the user terminal 4, but it may be considered to be a concept including, for example, reception of information input from an input device such as a keyboard, reception of information read from a recording medium such as a disk or semiconductor memory, and the like.

[0573] The patent information receiving unit 721 receives one or more pieces of patent information. The patent information receiving unit 721 usually receives one or more pieces of patent information from the user terminal 4. However, for example, one or more pieces of patent information may be stored in a recording medium such as a disk, and the patent information receiving unit 721 may read out one or more pieces of patent information from such a recording medium.

[0574] The processing unit 73 performs various processes. The various processes are, for example, processes of the prediction unit 731, the judgment means 7311, the first decision means 7312, and the second decision means 7313. The processing unit 73 also performs, for example, various judgments described in the flowcharts. Other processes performed by the processing unit 73 may be described as appropriate.

[0575] The prediction unit 731 obtains predicted classification information that specifies classification information of each of one or more pieces of patent information, using one or more pieces of condition information.

[0576] The prediction unit 731 executes a predicted classification information acquisition process for each of one or more pieces of patent information accepted by the patent information acceptance unit 721. The predicted classification information acquisition process is a process for acquiring predicted classification information. The predicted classification information is information that specifies the classification information of the patent information.

[0577] The judgment means 7311 judges whether or not each of the one or more pieces of patent information matches any of the conditions contained in each of the one or more pieces of condition information.

[0578] The judgment means 7311 judges, for example, for each of one or more patent information accepted by the patent information accepting unit 721, whether or not the patent information matches any of one or more conditions corresponding to one or more condition information stored in the condition information storage unit 712.

[0579] In this embodiment, for example, the condition information is a pair of a patent identifier and classification information, and the condition information storage unit 712 stores one or more patent identifiers in correspondence with the classification information, and the judgment means 7311 may judge, for each of the one or more accepted patent information, whether or not the patent identifier held by the patent information matches any of the one or more patent identifiers stored in the condition information storage unit 712.

[0580] The first determining means 7312 obtains classification information paired with the condition that the determining means 7311 has determined to be met.

[0581] The first determining means 7312, for example, for each of one or more pieces of patent information, obtains classification information paired with a patent identifier that is determined to match the patent identifier possessed by the patent information.

[0582] The second determination means 7313 obtains classification information of the patent information by machine learning prediction processing using the learning device stored in the learning device storage unit 711. For the patent information that the judgment means 7311 judges not to match any of the conditions, the second determination means 7313 obtains classification information by machine learning prediction processing using the learning device stored in the learning device storage unit 711.

[0583] In this embodiment, the second determination means 7313, for example, obtains classification information of patent information corresponding to a patent identifier that is determined not to match any of one or more patent identifiers stored in the condition information storage unit 712, by a machine learning prediction process.

[0584] The output unit 74 outputs various information. The various information is, for example, predicted classification information. For example, for each of one or more accepted patent information, the output unit 74 outputs the classification information determined by the first determination means 7312 or the classification information acquired by the second determination means 7313 as predicted classification information in a pair with a patent identifier corresponding to the patent information.

[0585] As mentioned above, output means, for example, transmission to user terminal 4, but it may also be considered as a concept including display on a display, printing on a printer, sound output from a speaker, storage on a recording medium, and delivery of processing results to other processing devices or other programs.

[0586] The output unit 74 outputs the predicted classification information acquired by the prediction unit 731. The output unit 74 normally transmits the acquired predicted classification information to the user terminal 4. However, the predicted classification information may be, for example, displayed on a display, stored in a recording medium, or delivered to another processing device, another program, or the like, and the output form and output destination are not important.

[0587] The server device 3 performs the same operations as those in the first embodiment.

[0588] The user terminal 4 performs the same operations as in the first embodiment.

[0589] The storage unit 71, the learning unit storage unit 711, and the condition information storage unit 712 are preferably non-volatile recording media, but may also be realized as volatile recording media.

[0590] There is no restriction on the process by which information is stored in the storage unit 71, etc. For example, information may be stored in the storage unit 71, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the storage unit 1, etc., or information inputted via an input device may be stored in the storage unit 1, etc.

[0591] The reception unit 72, the patent information reception unit 721, and the output unit 74 can usually be realized by wireless or wired communication means, etc.

[0592] The processing unit 73, the prediction unit 731, the judgment means 7311, the first decision means 7312, and the second decision means 7313 can usually be realized by an MPU, a memory, etc. The processing procedure of the processing unit 73, etc. is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may also be realized by hardware (dedicated circuit).

[0593] Next, the operation of the patent classification system D will be described with reference to the flowcharts of Figures 23 and 24. Figure 23 is a flowchart illustrating the operation of the classification information determination device 7.

[0594] (Step S2301) The processing unit 73 judges whether or not the patent information receiving unit 721 has received one or more pieces of patent information. If it is judged that the patent information receiving unit 721 has received one or more pieces of patent information, the process proceeds to step S2302, and if it is judged that the patent information receiving unit 721 has not received any pieces of patent information, the process returns to step S2501.

[0595] (Step S2302) The processing unit 73 assigns 1 to a counter k.

[0596] (Step S2303) The processing unit 73 judges whether the k-th patent information exists or not. If it is judged that the k-th patent information exists, the process proceeds to step S2303, and if it is judged that the k-th patent information does not exist, the process returns to step S2301.

[0597] (Step S2304) The prediction unit 731 executes a process for acquiring predicted classification information k that identifies classification information of the k-th patent information (hereinafter, "k-th predicted classification information acquisition process"). The k-th predicted classification information acquisition process will be described with reference to the flowchart of FIG. 24.

[0598] (Step S2305) The processing unit 73 increments the counter k by 1. The process returns to step S2303.

[0599] In the flowchart of FIG. 23, the process ends when the classification information determination device 7 is powered off or when an interrupt occurs to end the process.

[0600] FIG. 24 is a flowchart illustrating the "k-th predicted classification information acquisition process" in step S2304.

[0601] (Step S2401) The prediction unit 731 assigns 1 to a counter n.

[0602] (Step S2402) Prediction unit 731 judges whether or not the n-th condition exists. If it is judged that the n-th condition exists, the process proceeds to step S2403, and if it is judged that the n-th condition does not exist, the process proceeds to step S2407.

[0603] (Step S2403) The judgment means 7311 constituting the prediction unit 731 judges whether or not the k-th patent information matches the n-th condition. If it is judged that the k-th patent information matches the n-th condition, the process proceeds to step S2605, and if it is judged that the k-th patent information does not match the n-th condition, the process proceeds to step S2404.

[0604] (Step S2404) The prediction unit 731 increments the counter n by 1. The process returns to step S2402.

[0605] (Step S2405) The first determining means 7312 determines classification information paired with the n-th condition among the one or more conditions corresponding to the one or more pieces of condition information stored in the condition information storage unit 712.

[0606] (Step S2406) The prediction unit 731 sets the category information determined in step S2405 to "predicted category information k." Then, the process returns to the upper level process.

[0607] (Step S2407) The second determination means 7313 uses the learning device stored in the learning device storage unit 711 to obtain classification information of the k-th patent information through machine learning prediction processing.

[0608] (Step S2408) The prediction unit 731 sets the classification information acquired in step S2407 to "predicted classification information k." Then, the process returns to the upper level process.

[0609] A specific example of the operation of the patent classification system D in this embodiment will be described below.

[0610] As described in the first embodiment, the learning device production device 1 performs a machine learning learning process using the learning source information to acquire a learning device. The learning device production device 1 delivers the acquired learning device and the learning source information used when acquiring the learning device to the classification information determination device 7. In this example, it is assumed that a learning device is acquired using learning source information having 100 pieces of patent classification information 1 to 100, and the learning device is delivered together with the learning source information.

[0611] In the classification information determination device 7, the processing unit 73 receives the learning device and stores it in the learning device storage unit 711. The processing unit 73 also receives the learning source information and obtains two or more pieces of condition information corresponding to the learning source information (in this example, 90 pieces of condition information corresponding to the 90 pieces of patent classification information 11 to 100 used when obtaining the learning device, out of the 100 pieces of patent classification information 1 to 100 contained in the learning source information), and stores them in the condition information storage unit 712.

[0612] As a result, the learning device storage unit 711 stores the learning device acquired by the learning device production device 1 using the patent classification information 11 to 100 out of the 100 pieces of patent classification information contained in the received learning source information. Also, the condition information storage unit 712 stores, for example, two or more pieces of condition information (90 pieces in this example corresponding to the 90 pieces of patent classification information 11 to 100 used when acquiring the learning device) as shown in Fig. 25.

[0613] FIG. 25 is a data structure diagram of condition information in this embodiment. The condition information has a patent identifier and classification information. Two or more pieces of condition information stored in the condition information storage unit 712 are associated with an ID (e.g., "1", "2", etc.). For example, the condition information associated with ID "1" (hereinafter, condition information 1) has a patent identifier "JP Patent Publication 2017-dddd" and classification information "B". Moreover, the condition information 2 associated with ID "2" has a patent identifier "JP Patent Publication 2017-eeee" and classification information "A". Similarly, the condition information 100 has a patent identifier "JP Patent Publication 2018-ffff" and classification information "C".

[0614] In this example, it is assumed that 100 pieces of patent information corresponding to the 100 pieces of patent classification information 1 to 100 contained in the received learning source information are read out from an existing medium such as a disk.

[0615] In the classification information determination device 7, the patent information receiving unit 721 receives the 100 pieces of patent information read out as described above, and the prediction unit 731 executes a predicted classification information acquisition process for acquiring predicted classification information that identifies the classification information of each of the received 100 pieces of patent information.

[0616] That is, the judgment means 7311 constituting the prediction unit 731 first judges whether or not the first patent information corresponding to the patent classification information 1 matches any of the ten conditions corresponding to the ten pieces of condition information 1 to 10 stored in the condition information storage unit 712. It is judged that the patent information does not match any of the conditions.

[0617] In response to the above-mentioned determination result that none of the conditions are met, the second determination means 7313 uses the learning device stored in the learning device storage unit 711 to obtain classification information of the first patent information by machine learning prediction processing. In this example, it is assumed that classification information "A" is obtained. As a result, classification information "A" by the learning device is obtained as predicted classification information of the first patent information.

[0618] Similarly, for patent classification information 2 to 10, it is determined that none of the conditions are met, and predicted classification information by the learning device is obtained (for example, predicted classification information "A" for patent classification information 2, predicted classification information "B" for patent classification information 10, etc.). As a result, predicted classification information "A"..."B" by the learning device is obtained as predicted classification information for each of the second to tenth patent information.

[0619] Next, the judgment means 7311 judges whether or not the 11th patent information corresponding to the patent classification information 11 matches any one of the 90 conditions corresponding to the 90 condition information 1 to 90 stored in the condition information storage unit 712. It is judged that the condition information matches the first condition of the 90 conditions.

[0620] In response to the above-mentioned determination result that the first condition is met, the first determination means 7312 determines classification information "B" that pairs with the patent identifier "JP Patent Publication 2017-dddd", which is the first condition among the 90 conditions corresponding to the 90 pieces of condition information stored in the condition information storage unit 712. As a result, classification information "B" that constitutes patent classification information 11 is obtained as predicted classification information of the 11th patent information.

[0621] Similarly, patent classification information 12-100 is determined to match the 12th to 90th conditions of the 90 conditions, and classification information "B"..."C" that pairs with the patent identifiers "JP Patent Publication 2017-eeee"..."JP Patent Publication 2018-ffff", which are the 12th to 100th conditions, is determined. As a result, classification information "A"..."C" that constitutes each of patent classification information 12-100 is obtained as predicted classification information for each of the 12th to 100th patent information.

[0622] The output unit 74 stores the classification information thus obtained for each of the 100 pieces of patent information in a recording medium such as a disk in association with the patent information.

[0623] As a result, for example, 100 pairs of patent information and classification information of the patent information are stored on the recording medium.

[0624] As described above, according to this embodiment, the learning device production device 1 performs a machine learning learning process using learning source information having two or more patent classification information having patent information and classification information, acquires a learning device, and the acquired learning device is stored in the learning device storage unit 711. The condition information storage unit 712 stores one or more condition information that is a pair of a condition related to a patent and classification information. The classification information determination device 7 accepts one or more patent information, acquires predicted classification information that specifies classification information of each of the one or more patent information using the one or more condition information, and outputs the predicted classification information.

[0625] When obtaining predicted classification information, the classification information determination device 7 determines whether or not one or more pieces of patent information match any of the conditions contained in one or more pieces of condition information, and determines classification information that pairs with the condition that is determined to match, while using a learning device to obtain classification information for patent information that is determined not to match any of the conditions through machine learning prediction processing, thereby enabling patents to be classified with high accuracy.

[0626] Furthermore, in the above configuration, one condition is that the patent information constitutes any one of two or more patent classification information contained in the learning source information, thereby making it possible to classify with high accuracy the patents in the training data used when obtaining the machine learning learning device, and also to classify other patents with high accuracy by predictive processing using the learning device.

[0627] The software for realizing the classification information determination device 7 in this embodiment is, for example, a program as follows. That is, this program performs machine learning learning processing using learning source information having two or more patent classification information having patent information and classification information of the patent information, and includes a computer that can access a learning device storage unit 711 in which the acquired learning device is stored and a condition information storage unit 712 in which one or more condition information, which is a pair of one condition related to a patent and classification information of the patent information, a patent information receiving unit 721 that receives one or more patent information, and a prediction unit 73 that acquires predicted classification information that identifies classification information of each of the one or more patent information using the one or more condition information. and an output unit 74 that outputs the predicted classification information, wherein the prediction unit 731 includes a determination means 7311 that determines whether or not each of the one or more patent information matches any of the conditions contained in the one or more condition information, a first determination means 7312 that determines classification information that pairs with the condition that the determination means 7311 determines to match, and a second determination means 7313 that uses the learning device to obtain classification information of patent information that the determination means 7311 determines to not match any of the conditions by machine learning prediction processing.

[0628] (Embodiment 5) In this embodiment, a patent classification system E is described that is equipped with a classification information determination device in which condition information, which is a pair of a condition and a class (classification information), is stored, and for a predicted target (patent information) that matches the one condition, the class that pairs with the one condition is determined, and when the one condition is not matched, the class is determined using a machine learning algorithm.

[0629] In addition, in this embodiment, the machine learning algorithm is, for example, the same as that described in the first to third embodiments, but the type is not important. A patent classification system E including a classification information determination device will be described.

[0630] In addition, in this embodiment, a patent classification system E including a classification information determination device will be described, in which one condition is a patent search formula (for example, "the title of the invention contains 'machine learning'").

[0631] 26 is a block diagram of a patent classification system E in this embodiment. The patent classification system E includes an algorithm determination device 5, a learning device production device 6, a classification information determination device 7a, a server device 3, and a user terminal 4.

[0632] The classification information determination device 7a includes a storage unit 71, a reception unit 72a, a processing unit 73a, and an output unit 74. The reception unit 72a includes a patent information reception unit 721, and a search formula reception unit 722. The processing unit 73a includes a patent specification acquisition unit 131, an important information acquisition unit 132, a classification target processing information acquisition unit 233, a prediction unit 731, and a search unit 732. The prediction unit 731 includes a judgment means 7311, a first determination means 7312, and a second determination means 7313.

[0633] The user terminal 4 includes a user accepting unit 41 , a user processing unit 42 , a user transmitting unit 43 , a user receiving unit 44 , and a user output unit 45 .

[0634] The algorithm determination device 5 has the same configuration as that in the second embodiment, and performs the same operations.

[0635] The learning module production device 6 has the same configuration as that in the second embodiment, and performs the same operation.

[0636] The classification information determining device 7a has the same configuration and operates in the same manner as the classification information determining device 7 in the fourth embodiment, except for the points described below.

[0637] That is, one condition constituting one or more pieces of condition information stored in the condition information storage unit 712 is a search formula. A search formula is information indicating conditions when searching patent information (hereinafter, may be referred to as search conditions). Search conditions are, for example, keywords such as "machine learning", patent classification codes such as IPC, F-term, and FI, and may be period information specifying a period to which the application date or publication date belongs, and the type is not limited.

[0638] In addition, the search criteria may include a combination of item information and a keyword, such as "Name of invention includes machine learning." Item information is information that identifies an item in patent information. Item information is, for example, "Name of invention," "Abstract," "Effect of invention," etc., but may also be "Full text," and the type is not important.

[0639] The search formula may be, for example, two or more conditions such as keywords or patent classification codes linked with a logical operator (for example, AND or OR).The search formula may also be a combination of multiple search formulas (linked with logical operators), and the format is not important.

[0640] The receiving unit 72a receives, for example, a search query in addition to patent information.

[0641] The search formula receiving unit 722 receives a search formula. The search formula receiving unit 722 usually receives the search formula from the user terminal 4. However, the search formula may be received via an input device such as a keyboard in the classification information determination device 7a, and the manner of reception is not important.

[0642] The processing unit 73a performs the processes of the prediction unit 731 and the like, as well as the processes of the patent specification acquisition unit 131, the important information acquisition unit 132, the classification target processed information acquisition unit 233, and the search unit 732. The patent specification acquisition unit 131, the important information acquisition unit 132, and the classification target processed information acquisition unit 233 perform the same operations as those in the first embodiment.

[0643] The search unit 732 performs a search using the search formula received by the search formula receiving unit 722, and obtains search results. The search target is, for example, one or more pieces of patent information in the storage unit 71, but may also be one or more pieces of patent information in the server device 3. The search result is one or more pieces of patent information that satisfy the search conditions indicated by the search formula, among the one or more pieces of patent information that are the search target. The search result may be one or more patent identifiers corresponding to the one or more pieces of patent information.

[0644] The user reception unit 41 constituting the user terminal 4 receives various types of information. The various types of information are, for example, the above-mentioned search formula. The reception by the user reception unit 41 is usually received via an input device such as a keyboard in the user terminal 4. However, the reception may be, for example, reading from a recording medium or receiving from an external device, and the manner is not important. The same matters also apply to the reception of a search formula by the search formula reception unit 411.

[0645] The search query receiving unit 411 receives a search query.

[0646] The user processing unit 42 performs various types of processing, such as judging whether the user reception unit 41 has received information such as a search query, converting the received information into information to be transmitted, and the like, but the type of processing is not important.

[0647] The user transmitting unit 43 transmits various types of information. The various types of information are, for example, a search formula. The user transmitting unit 43 transmits, for example, the search formula accepted by the user accepting unit 41 to the classified information determining device 7a.

[0648] The user receiving unit 44 receives various information. The various information is, for example, a search formula. For example, the user receiving unit 44 receives the search result from the classified information determination device 7a in response to the transmission of the search formula by the user transmitting unit 43.

[0649] The user output unit 45 outputs various information. The various information is, for example, search results. The user output unit 45 displays the information such as the search results on a display or stores it in a recording medium. However, the information such as the search results may be handed over to another program or transmitted to another device, and the output form is not important.

[0650] The search query reception unit 411 and the user reception unit 41 may or may not include an input device such as a keyboard or a touch panel. The search query reception unit 411 and the like may be realized by driver software for an input device, or by an input device and its driver software.

[0651] The user transmitting unit 43 and the user receiving unit 44 can usually be realized by wireless or wired communication means or the like.

[0652] The search unit 732, the user processing unit 42, etc. can usually be realized by an MPU, a memory, etc. The processing procedure of the processing unit 73a, etc. is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may also be realized by hardware (dedicated circuit).

[0653] The operation of the patent classification system E is similar to that explained in the fourth embodiment using the flowcharts of Figures 23 and 24.

[0654] A specific example of the operation of the patent classification system E in this embodiment will be described below.

[0655] The algorithm determination device 5 determines an appropriate process for performing the prediction process, as described in the second embodiment. The learning device production device 6 uses the learning source information to perform a learning process, which is a machine learning learning process and is determined by the algorithm determination device 5, as described in the second embodiment, to acquire a learning device. The learning device production device 6 delivers the acquired learning device and the learning source information used when acquiring the learning device to the classification information determination device 7a. In this example, it is assumed that a learning device is acquired using learning source information having 100 pieces of patent classification information, and the learning device is delivered together with the learning source information.

[0656] In the classification information determination device 7a, the processing unit 73a receives the learning device and stores it in the learning device storage unit 711. The processing unit 73a also receives the learning source information, acquires two or more pieces of condition information corresponding to the learning source information (in this example, 90 pieces of condition information corresponding to the 90 pieces of patent classification information 11 to 100 used when acquiring the learning device) and stores them in the condition information storage unit 712.

[0657] As a result, the learning device storage unit 711 stores the learning devices acquired by the learning device production device 6 using the 90 pieces of patent classification information 11 to 100 from among the above learning source information. Also, the condition information storage unit 712 stores, for example, two or more pieces of condition information (condition information 1 to 3, etc.) as shown in FIG.

[0658] FIG. 27 is a data structure diagram of condition information in this embodiment. Condition information has a search formula and classification information. Two or more pieces of condition information stored in the condition information storage unit 712 are associated with an ID (for example, "1", "2", etc.). For example, condition information 1 associated with ID "1" has a search formula "'Name of invention' contains 'machine learning'" and classification information "A". Condition information 2 associated with ID "2" has a search formula "'machine learning' and 'natural language processing'" and classification information "B". Similarly, condition information 3 has a search formula "'H06 aa / aa' or 'G06 bb / bb' or..." and classification information "C".

[0659] A user inputs one or more pieces of patent information via an input device of the user terminal 4. The patent information input here is a patent identifier such as a publication number. For example, assume that 10 patent identifiers corresponding to 10 pieces of patent classification information 1 to 10 out of 100 pieces of patent classification information 1 to 100 contained in the learning source information described above are input.

[0660] In the user terminal 4, the user receiving unit 41 receives the ten input patent identifiers, and the user transmitting unit 43 transmits the ten received patent identifiers to the classification information determining device 7a.

[0661] In the classification information determination device 7a, the patent information receiving unit 721 receives the 10 patent identifiers transmitted above, the output unit 74 transmits a request including the received 10 patent information identifiers to the server device 3, and the patent information receiving unit 721 receives 10 patent information corresponding to the transmitted 10 patent information identifiers from the server device 3. The prediction unit 731 executes a predicted classification information acquisition process for acquiring predicted classification information that specifies classification information of the patent information for each of the received 10 patent information.

[0662] That is, the judgment means 7311 constituting the prediction unit 731 first judges whether or not the first patent information corresponding to the patent classification information 1 matches any one of two or more conditions corresponding to two or more pieces of condition information 1 to 3 stored in the condition information storage unit 712. Here, it is assumed that the patent information is judged not to match any of the conditions.

[0663] In response to the above-mentioned determination result that none of the conditions are met, the second determination means 7313 uses the learning device stored in the learning device storage unit 711 to obtain classification information of the first patent information by machine learning prediction processing. In this example, it is assumed that classification information "A" is obtained. As a result, classification information "A" by the learning device is obtained as predicted classification information of the first patent information.

[0664] Similarly, for patent classification information 2 to 9, it is determined that none of the conditions are met, and predicted classification information by the learning device (for example, predicted classification information "A" for patent classification information 2) is obtained. As a result, predicted classification information "A", "A", ... by the learning device is obtained as predicted classification information for each of the second to ninth patent information.

[0665] Next, the judgment means 7311 judges whether or not the tenth patent information corresponding to the patent classification information 10 satisfies any of the above two or more conditions. It is assumed that the condition information is judged to satisfy the third condition corresponding to the condition information 3.

[0666] In response to the above judgment result that the third condition is met, the first determination means 7312 acquires classification information "C" that pairs with the search formula "H06 aa / aa" or "G06 bb / bb" or...", which is the third condition out of the three or more conditions corresponding to the ten condition information stored in the condition information storage unit 712. As a result, classification information "C", which constitutes patent classification information 10, is acquired as the predicted classification information of the tenth patent information.

[0667] The output unit 74 transmits the classification information thus acquired for each of the 10 pieces of patent information to the user terminal 4 in association with the patent information. As a result, 10 pieces of group information, which are pairs of patent information and classification information, are transmitted.

[0668] In the user terminal 4, the user receiving unit 44 receives the 10 pieces of group information transmitted above, and the user output unit 45 outputs the received 10 pieces of group information via a display. The output group information does not have to be the same as the received group information. For example, the patent information constituting the output group information may be a part of the patent information constituting the received group information (e.g., a patent identifier and an abstract, etc.).

[0669] As a result, ten pairs of patent information and classification information are displayed on the display of the user terminal 4.

[0670] As described above, according to this embodiment, the learning device production device 6 performs a machine learning learning process using learning source information having two or more pieces of patent classification information each having patent information and classification information, acquires a learning device, and the acquired learning device is stored in the learning device storage unit 711. The condition information storage unit 712 stores one or more pieces of condition information which are a pair of a condition related to a patent and classification information. The classification information determination device 7a receives one or more pieces of patent information, acquires predicted classification information that specifies classification information of each of the one or more pieces of patent information using the one or more pieces of condition information, and outputs the predicted classification information.

[0671] When obtaining predicted classification information, the classification information determination device 7a judges whether or not one or more pieces of patent information match any of the conditions contained in one or more pieces of condition information, and determines classification information that pairs with the condition that is determined to match, while using a learning device to obtain classification information for patent information that is determined not to match any of the conditions through machine learning prediction processing, thereby enabling patents to be classified with high accuracy.

[0672] In addition, in the above configuration, since one condition is a search formula, patents that match the search formula can be classified with high accuracy, and patents that do not match the search formula can also be classified with high accuracy through prediction processing using a machine learning learner.

[0673] In addition, the algorithm determination device 5 determines an appropriate process for performing predictive processing, and the learning module production device 6 performs learning processing using an algorithm according to that determination, thereby producing an appropriate learning module, which enables patents to be classified with high accuracy.

[0674] The software for realizing the classification information determination device 7a in this embodiment is, for example, a program as follows. That is, this program performs machine learning learning processing using learning source information having two or more patent classification information having patent information and classification information of the patent information, and includes a computer that can access a learning device storage unit 711 in which the acquired learning device is stored and a condition information storage unit 712 in which one or more condition information, which is a pair of one condition related to a patent and classification information of the patent information, a patent information reception unit 721 that receives one or more patent information, and a prediction unit 722 that acquires predicted classification information that identifies classification information of each of the one or more patent information using the one or more condition information. 31 and functioning as an output unit that outputs the predicted classification information, and the prediction unit 731 includes a determination means 7311 that determines whether or not each of the one or more patent information matches any of the conditions contained in the one or more condition information, a first determination means 7312 that determines classification information that pairs with the condition that the determination means 7311 determines to match, and a second determination means 7313 that uses the learning device to obtain classification information of patent information that the determination means 7311 determines does not match any of the conditions, by machine learning prediction processing.

[0675] FIG. 28 shows the appearance of a computer that executes the program described in this specification to realize the patent classification device 2 and the like of the various embodiments described above. The above-mentioned embodiments can be realized by computer hardware and a computer program executed thereon. FIG. 28 is an overview of this computer system 300, and FIG. 29 is a block diagram of the system 300. FIG. 28 and FIG. 29 are diagrams showing the appearance of a computer that realizes an engagement system.

[0676] In FIG. 28, computer system 300 includes a computer 301 which includes a CD-ROM drive 3012 , a keyboard 302 , a mouse 303 , and a monitor 304 .

[0677] 29, computer 301 includes, in addition to CD-ROM drive 3012, MPU 3013, bus 3014 connected to CD-ROM drive 3012 etc., ROM 3015 for storing programs such as a boot-up program, RAM 3016 connected to MPU 3013 for temporarily storing instructions of application programs and providing temporary storage space, and hard disk 3017 for storing application programs, system programs, and data. Although not shown here, computer 301 may further include a network card for providing connection to a LAN.

[0678] A program that causes the computer system 300 to execute the functions of the learning device production device 1, the patent classification device 2, the algorithm determination device 5, the classification information determination device 7, and the like in each of the above-mentioned embodiments may be stored on a CD-ROM 3101, inserted into the CD-ROM drive 3012, and further transferred to the hard disk 3017. Alternatively, the program may be transmitted to the computer 301 via a network (not shown) and stored on the hard disk 3017. The program is loaded into the RAM 3016 when executed. The program may be loaded directly from the CD-ROM 3101 or the network.

[0679] The program does not necessarily include an operating system (OS) or a third party program that causes the computer 301 to execute the functions of the learning device production device 1, the patent classification device 2, the algorithm determination device 5, etc., of the above-mentioned embodiments. The program only needs to include an instruction portion that calls appropriate functions (modules) in a controlled manner to obtain a desired result. How the computer system 300 operates is well known, and a detailed description will be omitted.

[0680] In addition, in the above program, the steps of transmitting information and receiving information do not include processing performed by hardware, such as processing performed by a modem or interface card in the transmitting step (processing that is performed only by hardware).

[0681] Furthermore, the computer that executes the above program may be a single computer or a plurality of computers. That is, the program may be executed in a centralized manner or in a distributed manner.

[0682] Furthermore, in each of the above embodiments, it goes without saying that two or more communication means present in one device may be physically realized by one medium.

[0683] In addition, in each of the above embodiments, each process may be realized by centralized processing by a single device, or may be realized by distributed processing by multiple devices. In other words, the learning device production device 1, the patent classification device 2, and the algorithm determination device 5 may operate as standalone devices. When the learning device production device 1, the patent classification device 2, and the algorithm determination device 5 operate as standalone devices, the reception unit receives instructions, information, etc. from a user, etc. Furthermore, the output unit displays information, outputs sound, or transmits information, etc. to a display device.

[0684] The present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these modifications are also included within the scope of the present invention. [Industrial Applicability]

[0685] As described above, the patent classification system of the present invention has the effect of being able to classify patents with high accuracy by performing predictive processing using an appropriate algorithm, and is useful as a patent classification system, etc.

[0686] Furthermore, the patent classification system according to the present invention has the effect of being able to classify patents with high accuracy by using important information related to patent specifications, and is useful as a patent classification system, etc.

[0687] In addition, the classification information determination device of the present invention has the effect of being able to classify patents with high accuracy by obtaining predicted classification information using different determination methods for patent information that meets predetermined conditions and patent information that does not meet the conditions, and is useful as a classification information determination device, etc. [Explanation of symbols]

[0688] 1, 6 Learning device production device 2, 8 Patent classification device 3. Server equipment 4. User terminal 5. Algorithm decision device 7, 7a Classification information determination device 11, 61 Learning storage section 12, 62 Learning Reception 13 Learning processing section 14, 64 Learning output section 21, 51, 71 Storage area 22, 52, 72, 81 Reception 23, 53, 73, 73a, 82 Processing section 24, 54, 74 Output section 41 User Reception Department 42 User Processing Section 43 User Transmission Unit 44 User Receiving Unit 45 User output section 111 Human evaluation information storage unit 131 Patent Specification Acquisition Department 132 Important information acquisition department 133, 535 Learning source information acquisition unit 134, 531 Learning Department 141, 641 Storage unit 211, 711 Learning unit storage unit 233 Classification object processing information acquisition unit 234, 532, 731, 821 Prediction Department 421, 732 Search Department 511 Program Storage Unit 512 Accuracy specific information storage unit 521, 621 Learning Source Information Reception Department 531 Learning Machine 533 Evaluation Department 534 Algorithm Information Acquisition Unit 536 Statistical Processing Unit 621 Learning Source Information Learning Reception Department 712 Condition information storage section 721 Patent Information Reception Department 1321 Effect resolution means text acquisition means 1322 Means of obtaining effect text 1323 Solution means sentence acquisition means 1324 Objective sentence acquisition means 1325 Means of acquiring characteristic words 1326 Related Word Acquisition Method 1327 Means for obtaining important tag-corresponding sentences 1328 Number range string acquisition method 1329 Number acquisition means 7311 Judgment means 7312 First determining means 7313 Second determining means

Claims

1. a program storage unit storing one or more learning device generation programs for generating a learning device for classifying patent information relating to a patent using learning source information having two or more patent classification information having patent information relating to a patent and classification information indicating a classification for the patent information, and a prediction program for predicting a classification of the patent classification information using the learning device and acquiring predicted classification information; A learning source information receiving unit that receives learning source information; a learning unit that executes the one or more learning device generation programs by using at least a part of patent classification information of the learning device information received by the learning device information receiving unit, and obtains one or more learning devices; a prediction unit that applies a prediction program to each of at least a portion of the patent classification information among the learning source information received by the learning source information receiving unit, using one or more learning devices generated by the learning unit, to obtain two or more predicted classification information for each of the one or more patent classification information; an evaluation unit that uses each of the two or more pieces of predicted classification information acquired by the prediction unit and classification information included in the learning source information to acquire accuracy information regarding the accuracy of each of the two or more pieces of predicted classification information, and determines the predicted classification information having the highest accuracy specified by the accuracy information; an algorithm information acquisition unit that acquires algorithm information related to an algorithm corresponding to the predicted classification information determined by the evaluation unit; an output unit that outputs the algorithm information, The method further includes a statistical processing unit that performs one or more types of statistical processing using the two or more pieces of predicted classification information acquired by the prediction unit, and acquires one or more pieces of predicted classification information as a result of the statistical processing, The evaluation unit is acquiring accuracy information regarding accuracy of each of the two or more pieces of predicted classification information using, instead of or in addition to the two or more pieces of predicted classification information acquired by the prediction unit, each of the two or more pieces of predicted classification information including the one or more pieces of predicted classification information acquired by the statistical processing unit and classification information included in the learning source information, and determining the predicted classification information having the highest accuracy specified by the accuracy information; The algorithm information acquisition unit An algorithm determination device that acquires algorithm information including statistical processing specification information that specifies a statistical processing method corresponding to the predicted classification information determined by the evaluation unit.

2. The statistical processing unit Performing two or more types of statistical processing to obtain two or more pieces of predicted classification information that are the results of the statistical processing; 2. The algorithm determination device according to claim 1, wherein the two or more types of statistical processing are two or more types of statistical processing selected from among a logical sum, a logical product, and a majority decision of predicted classification information.

3. The learning source information acquisition unit further includes an acquisition unit for acquiring two or more pieces of learning source information each having two or more pieces of patent classification information having different information from the patent information contained in the two or more pieces of patent classification information contained in the learning source information received by the learning source information receiving unit, The learning unit is Executing the one or more learning module generation programs for each of the two or more pieces of learning module information acquired by the learning module information acquisition unit to acquire two or more learning modules; The prediction unit is applying a prediction program to each of at least a portion of the patent classification information among the learning source information received by the learning source information receiving unit, using two or more learning devices generated by the learning unit, to obtain two or more predicted classification information for each of the one or more patent classification information; The algorithm information acquisition unit 3. The algorithm determination device according to claim 1, further comprising: acquiring algorithm information including information specifying information that specifies information contained in the learning source information corresponding to the predicted classification information determined by the evaluation unit.

4. The program storage unit stores two or more learning device generation programs and two or more prediction programs, The learning unit is Executing each of the two or more learning module generation programs using at least a part of patent classification information of the learning module information received by the learning module information receiving unit, thereby obtaining two or more learning modules; The prediction unit is applying a prediction program to each of at least a portion of the patent classification information among the learning source information received by the learning source information receiving unit, using two or more learning devices generated by the learning unit, to obtain two or more predicted classification information for each of the one or more patent classification information; The algorithm information acquisition unit The algorithm determination device according to claim 1 , wherein the evaluation unit acquires algorithm information including program identification information that identifies a learning device generation program corresponding to the determined predictive classification information.

5. The method further includes a reception unit that receives accuracy specification information that specifies which of two or more of the four accuracies, namely, recall, precision, F-value, or accuracy rate, accuracy information is to be acquired; The evaluation unit is 5. The algorithm determination device according to claim 1, further comprising: a prediction unit that uses each of two or more pieces of predicted classification information acquired by the prediction unit and classification information contained in the learning source information to obtain accuracy information for each of the two or more pieces of predicted classification information that is specified by accuracy specifying information received by the reception unit; and a prediction unit that determines the predicted classification information with the highest accuracy specified by the accuracy information.

6. A learning source information receiving unit that receives learning source information; a learning unit that performs a learning process on the learning source information received by the learning source information receiving unit to obtain a learning device using an algorithm according to algorithm information output by the algorithm determination device of any one of claims 1 to 5; and a storage unit that stores the learning device in association with the algorithm information.

7. a learning module storage unit in which learning modules accumulated by the production device according to claim 6 are stored; A receiving unit that receives classification target information including a patent identifier that identifies a patent to be classified; a prediction unit that performs a prediction process on the classification target information using a learning device according to an algorithm in accordance with algorithm information associated with the learning device, and acquires classification result information; A patent classification device comprising: an output unit that outputs the classification result information acquired by the prediction unit in correspondence with the patent identifier contained in the classification target information accepted by the acceptance unit.

8. An algorithm determination method implemented by the algorithm determination device according to any one of claims 1 to 5, comprising: An algorithm determination method comprising the steps performed by the algorithm determination device.

9. A method for producing a learning device, which is realized by the production device according to claim 6, comprising the steps of: A method for producing a learning device comprising the steps performed by the production device.

10. A patent classification method implemented by the patent classification device according to claim 7, comprising: A patent classification method comprising the steps performed by the patent classification device.

11. On the computer, A program for executing all the processes performed by the algorithm determination device according to any one of claims 1 to 5, the learning device production device according to claim 6, or the patent classification device according to claim 7.

Images